4. Add a Submit button to the ``modal-footer`` element of the Set Boundary modal in :file:`templates/earth_engine/viewer.html`:
.. code-block:: html+django
:emphasize-lines: 3
5. Navigate to ``` with id ``boundary-file-form-group`` with this updated version in :file:`templates/earth_engine/viewer.html`:
.. code-block:: html+django
:emphasize-lines: 3-6
3. The modal is not open by default when the page loads, which is normally the desired behaviour. However, when the page refreshes after a form submission that yields errors, the errors will be obscured from the user until they open the dialog again. Automatically open the Set Boundary modal if there is an error to display. Replace the **INITIALIZATION / CONSTRUCTOR** section of :file:`public/js/gee_datasets.js` with the following:
.. code-block:: javascript
:emphasize-lines: 21-26
/************************************************************************
* INITIALIZATION / CONSTRUCTOR
*************************************************************************/
$(function() {
// Initialize Global Variables
bind_controls();
// EE Products
EE_PRODUCTS = $('#ee-products').data('ee-products');
// Initialize values
m_platform = $('#platform').val();
m_sensor = $('#sensor').val();
m_product = $('#product').val();
INITIAL_START_DATE = m_start_date = $('#start_date').val();
INITIAL_END_DATE = m_end_date = $('#end_date').val();
m_reducer = $('#reducer').val();
m_map = TETHYS_MAP_VIEW.getMap();
// Open boundary file modal if it has an error
if ($('#boundary-file').hasClass('is-invalid')) {
let boundary_modal_elem = document.getElementById('set-boundary-modal');
let boundary_modal_inst = bootstrap.Modal.getOrCreateInstance(boundary_modal_elem);
boundary_modal_inst.show();
}
});
4. Navigate to `
`_ and upload a non-zip file. Verify that the error message is displayed in the modal and that it opens automatically. Upload a zip file and verify that the modal does not open automatically and no error is displayed when you open it.
6. Validate File is a Shapefile Containing Polygons
===================================================
In this step you will add the logic to validate that the file contained in the ZIP archive is a shapefile. You will use the ``pyshp`` library to do this, which will introduce a new dependency for the app.
1. Install ``pyshp`` library into your Tethys conda environment using conda or pip. Run the following command in the terminal with your Tethys environment activated:
.. code-block:: bash
# conda: conda-forge channel highly recommended
conda install -c conda-forge pyshp
# pip
pip install pyshp
2. Add ``pyshp`` as a new dependency in the ``install.yml``:
.. code-block:: yaml
:emphasize-lines: 16
# This file should be committed to your app code.
version: 1.0
# This should be greater or equal to your tethys-platform in your environment
tethys_version: ">=4.0.0"
# This should match the app - package name in your setup.py
name: earth_engine
requirements:
# Putting in a skip true param will skip the entire section. Ignoring the option will assume it be set to False
skip: false
conda:
channels:
- conda-forge
packages:
- earthengine-api
- oauth2client
- geojson
- pyshp
pip:
post:
3. Add the following imports and create a new helper function ``find_shapefile`` in :file:`helpers.py`:
.. code-block:: python
def find_shapefile(directory):
"""
Recursively find the path to the first file with an extension ".shp" in the given directory.
Args:
directory (str): Path of directory to search for shapefile.
Returns:
str: Path to first shapefile found in given directory.
"""
shapefile_path = ''
# Scan the temp directory using walk, searching for a shapefile (.shp extension)
for root, dirs, files in os.walk(directory):
for f in files:
f_path = os.path.join(root, f)
f_ext = os.path.splitext(f_path)[1]
if f_ext == '.shp':
shapefile_path = f_path
break
return shapefile_path
4. Use the new ``find_shapefile`` helper function and ``pyshp`` in ``handle_shapefile_upload`` to validate that the unzipped directory contains a shapefile. Update ``handle_shapefile_upload`` in :file:`helpers.py`:
.. code-block:: python
import shapefile
.. code-block:: python
:emphasize-lines: 31-45
def handle_shapefile_upload(request):
"""
Uploads shapefile to Google Earth Engine as an Asset.
Args:
request (django.Request): the request object.
Returns:
str: Error string if errors occurred.
"""
# Write file to temp for processing
uploaded_file = request.FILES['boundary-file']
with tempfile.TemporaryDirectory() as temp_dir:
temp_zip_path = os.path.join(temp_dir, 'boundary.zip')
# Use with statements to ensure opened files are closed when done
with open(temp_zip_path, 'wb') as temp_zip:
for chunk in uploaded_file.chunks():
temp_zip.write(chunk)
try:
# Extract the archive to the temporary directory
with zipfile.ZipFile(temp_zip_path) as temp_zip:
temp_zip.extractall(temp_dir)
except zipfile.BadZipFile:
# Return error message
return 'You must provide a zip archive containing a shapefile.'
# Verify that it contains a shapefile
try:
# Find a shapefile in directory where we extracted the archive
shapefile_path = find_shapefile(temp_dir)
if not shapefile_path:
return 'No Shapefile found in the archive provided.'
with shapefile.Reader(shapefile_path) as shp_file:
# Check type (only Polygon supported)
if shp_file.shapeType != shapefile.POLYGON:
return 'Only shapefiles containing Polygons are supported.'
except TypeError:
return 'Incomplete or corrupted shapefile provided.'
5. Download :download:`USA_simplified.zip <./resources/USA_simplified.zip>`, a zip archive containing a simplified shapefile of the boundary of the United States. Also download :download:`points.zip <./resources/points.zip>`, an archive containing a shapefile with only points.
6. Navigate to `
`_ and verify the following:
* Upload the :file:`USA_simplified.zip` and verify that no errors are shown.
* Upload the :file:`points.zip` and verify that an error *is* shown.
* Create a zip archive that does not contain a shapefile and upload it. Verify an error *is* shown.
7. Save Shapefile to the User's Workspace Directory
===================================================
At this point you have confirmed that the user uploaded a ZIP archive containing a shapefile of polygons but the file is still stored as a temporary file and will be deleted as soon as the code finishes executing. In this step you will add the logic to write the file to the user's workspace directory. This will involve creating a few new helper functions and using the :ref:`tethys_workspaces_api`.
1. The shapefile and its sidecars will be stored in a directory called :file:`boundary` within the user's workspace. Only one boundary shapefile will be stored for each user, so if the :file:`boundary` directory already exists, it will need to be cleared out. The ``prep_boundary_dir`` helper function will be responsible for initializing the :file:`boundary` directory in the user's workspace and clearing it out if needed. Add the following imports and create the ``prep_boundary_dir`` function in :file:`helpers.py`:
.. code-block:: python
import glob
.. code-block:: python
def prep_boundary_dir(root_path):
"""
Setup the workspace directory that will store the uploaded boundary shapefile.
Args:
root_path (str): path to the root directory where the boundary directory will be located.
Returns:
str: path to boundary directory for storing boundary shapefile.
"""
# Copy into new shapefile in user directory
boundary_dir = os.path.join(root_path, 'boundary')
# Make the directory if it doesn't exist
if not os.path.isdir(boundary_dir):
os.mkdir(boundary_dir)
# Clear the directory if it exists
else:
# Find all files in the directory using glob
files = glob.glob(os.path.join(boundary_dir, '*'))
# Remove all the files
for f in files:
os.remove(f)
return boundary_dir
2. The ``write_boundary_shapefile`` helper function takes the ``shapefile.Reader`` object that was used to validate the shapefile and uses it to write a copy of the shapefile to the given directory. Create the ``write_boundary_shapefile`` function in :file:`helpers.py`:
.. code-block:: python
def write_boundary_shapefile(shp_file, directory):
"""
Write the shapefile to the given directory. The shapefile will be called "boundary.shp".
Args:
shp_file (shapefile.Reader): A shapefile reader object.
directory (str): Path to directory to which to write shapefile.
Returns:
str: path to shapefile that was written.
"""
# Name the shapefiles boundary.* (boundary.shp, boundary.dbf, boundary.shx)
shapefile_path = os.path.join(directory, 'boundary')
# Write contents of shapefile to new shapfile
with shapefile.Writer(shapefile_path) as out_shp:
# Based on https://pypi.org/project/pyshp/#examples
out_shp.fields = shp_file.fields[1:] # skip the deletion field
# Add the existing shape objects
for shaperec in shp_file.iterShapeRecords():
out_shp.record(*shaperec.record)
out_shp.shape(shaperec.shape)
return shapefile_path
3. The ``controller`` decorator provides arguments for adding the user and app workspaces. Add the ``user_workspace`` argument and set it to ``True`` in the ``controller`` decorator for the ``viewer`` function. The decorator passes the user workspace object as an additional argument to the controller, so you will need to add an additional argument to accept the user workspace in :file:`controllers.py`:
.. code-block:: python
:emphasize-lines: 1-2
@controller(user_workspace=True)
def viewer(request, user_workspace):
"""
Controller for the app viewer page.
"""
.. tip:
For more information about Tethys Workspaces, see :ref:`tethys_workspaces_api`.
4. The ``viewer`` controller will need to be able to pass the ``user_workspace`` to the ``handle_shapefile_upload`` function. Modify the ``handle_shapefile_upload`` helper function to accept the ``user_workspace`` as an additional argument in :file:`helpers.py`:
.. code-block:: python
:emphasize-lines: 1
def handle_shapefile_upload(request, user_workspace):
"""
Uploads shapefile to Google Earth Engine as an Asset.
Args:
request (django.Request): the request object.
user_workspace (tethys_sdk.workspaces.Workspace): the User workspace object.
Returns:
str: Error string if errors occurred.
"""
5. Add logic to write the uploaded shapefile to the user workspace in ``handle_shapefile_upload`` in :file:`helpers.py`:
.. code-block:: python
:emphasize-lines: 45-49
def handle_shapefile_upload(request, user_workspace):
"""
Uploads shapefile to Google Earth Engine as an Asset.
Args:
request (django.Request): the request object.
user_workspace (tethys_sdk.workspaces.Workspace): the User workspace object.
Returns:
str: Error string if errors occurred.
"""
# Write file to temp for processing
uploaded_file = request.FILES['boundary-file']
with tempfile.TemporaryDirectory() as temp_dir:
temp_zip_path = os.path.join(temp_dir, 'boundary.zip')
# Use with statements to ensure opened files are closed when done
with open(temp_zip_path, 'wb') as temp_zip:
for chunk in uploaded_file.chunks():
temp_zip.write(chunk)
try:
# Extract the archive to the temporary directory
with zipfile.ZipFile(temp_zip_path) as temp_zip:
temp_zip.extractall(temp_dir)
except zipfile.BadZipFile:
# Return error message
return 'You must provide a zip archive containing a shapefile.'
# Verify that it contains a shapefile
try:
# Find a shapefile in directory where we extracted the archive
shapefile_path = find_shapefile(temp_dir)
if not shapefile_path:
return 'No Shapefile found in the archive provided.'
with shapefile.Reader(shapefile_path) as shp_file:
# Check type (only Polygon supported)
if shp_file.shapeType != shapefile.POLYGON:
return 'Only shapefiles containing Polygons are supported.'
# Setup workspace directory for storing shapefile
workspace_dir = prep_boundary_dir(user_workspace.path)
# Write the shapefile to the workspace directory
write_boundary_shapefile(shp_file, workspace_dir)
except TypeError:
return 'Incomplete or corrupted shapefile provided.'
6. Modify the ``handle_shapefile_upload`` call in the ``viewer`` controller in :file:`controllers.py` to pass the user workspace path:
.. code-block:: python
:emphasize-lines: 4
# Handle Set Boundary Form
set_boundary_error = ''
if request.POST and request.FILES:
set_boundary_error = handle_shapefile_upload(request, user_workspace)
7. Navigate to `
`_ and upload the :file:`USA_simplified.zip`. Verify that the shapefile is saved to the active user's workspace directory with its sidecar files (e.g. :file:`workspaces/user_workspaces/admin/boundary/boundary.shp`).
8. Redirect Upon Successful File Upload
=======================================
As a final user experience improvement, issue a redirect response instead of the normal response when there are now errors. This will clear the form and reset the state of the page.
1. Add the logic to the ``viewer`` controller in :file:`controllers.py`:
.. code-block:: python
from django.http import HttpResponseRedirect
.. code-block:: python
:emphasize-lines: 6-8
# Handle Set Boundary Form
set_boundary_error = ''
if request.POST and request.FILES:
set_boundary_error = handle_shapefile_upload(request, user_workspace)
if not set_boundary_error:
# Redirect back to this page to clear form
return HttpResponseRedirect(request.path)
2. Navigate to `
`_ and upload the :file:`USA_simplified.zip`. Navigate to a different page of the app and verify that no warning messages are displayed indicating that changes to the form may be lost.
9. Test and Verify
==================
Browse to `
`_ in a web browser and login if necessary. Verify the following:
1. Verify that Set Boundary button opens the Set Boundary Modal.
2. Upload a non-zip file and verify that the appropriate error is displayed.
3. Upload a zip archive that does not contain a shapefile and verify that the appropriate error is displayed.
4. Upload the :file:`points.zip` and verify that the appropriate error is displayed.
5. Upload the :file:`USA_simplified.zip` and verify that **no** errors are displayed.
6. Verify that the :file:`boundary.shp` is written to the user workspace of the active user (e.g. :file:`workspaces/user_workspace/admin/boundary/boundary.shp`).
7. Press the *Home* button in the header to navigate to the home page. Verify that no warnings are displayed after a successful upload when navigating away.
10. Solution
============
This concludes this portion of the GEE Tutorial. You can view the solution on GitHub at `
`_ or clone it as follows:
.. parsed-literal::
git clone https://github.com/tethysplatform/tethysapp-earth_engine.git
cd tethysapp-earth_engine
git checkout -b file-upload-solution file-upload-solution-|version|