Tethys Platform
Table Of Contents
Table Of Contents

Visualize Google Earth Engine Datasets

Last Updated: November 2019

In this tutorial you will load the GEE dataset the user has selected into the map view. The following topics will be reviewed in this tutorial:

  • Tethys MapView Gizmo JavaScript API

  • JQuery AJAX Calls

  • Authenticating with GEE in Tethys Apps

  • Retrieving GEE XYZ Tile Layer Endpoints

  • Logging in Tethys

../../_images/vis_gee_layers_solution.png

0. Start From Previous Solution (Optional)

If you wish to use the previous solution as a starting point:

git clone https://github.com/tethysplatform/tethysapp-earth_engine.git
cd tethysapp-earth_engine
git checkout -b map-view-solution map-view-solution-3.0

1. Handle GEE Authentication

To use GEE services, your app will need to authenticate using a GEE Account. This step will illustrate one way that this can be handled.

  1. Create a new Python module in the gee package called params.py with the following contents:

service_account = ''  # your google service account
private_key = ''  # path to the json private key for the service account
  1. Create a new Python module in the gee package called methods.py with the following contents:

import logging
import ee
from ee.ee_exception import EEException
from . import params as gee_account
from .products import EE_PRODUCTS
from . import cloud_mask as cm

log = logging.getLogger(f'tethys.apps.{__name__}')

if gee_account.service_account:
    try:
        credentials = ee.ServiceAccountCredentials(gee_account.service_account, gee_account.private_key)
        ee.Initialize(credentials)
    except EEException as e:
        print(str(e))
else:
    try:
        ee.Initialize()
    except EEException as e:
        from oauth2client.service_account import ServiceAccountCredentials
        credentials = ServiceAccountCredentials.from_p12_keyfile(
            service_account_email='',
            filename='',
            private_key_password='notasecret',
            scopes=ee.oauth.SCOPE + ' https://www.googleapis.com/auth/drive '
        )
        ee.Initialize(credentials)


def image_to_map_id(image_name, vis_params={}):
    """
    Get map_id parameters
    """
    pass


def get_image_collection_asset(platform, sensor, product, date_from=None, date_to=None, reducer='median'):
    """
    Get tile url for image collection asset.
    """
    pass

Important

The code at the top of this module handles authenticating with Google Earth Engine automatically when it is imported. By default it will check the params.py module for service account credentials and then fall back to checking the credentials file you generated earlier (see: 6. Authenticate with Google Earth Engine of Google Earth Engine Primer for Tethys Developers). Authenticating using the credential file works well for development but it will not work when you deploy the app. For production you will need to obtain and use a Google Earth Engine Service Account. Then add the credentials to the gee.param.py module. DO NOT COMMIT THESE CREDENTIALS IN A PUBLIC REPOSITORY.

2. Implement GEE Methods

Google Earth Engine provides XYZ tile services for each of their datasets. In this step, you'll write the necessary GEE logic to retrieve a tile service endpoint for a given dataset product.

  1. Some of the datasets require functions for filtering out the clouds in the images, so you'll create a module with functions for removing the clouds. Create a new Python module in the gee package called cloud_mask.py with the following contents:

import ee


def mask_l8_sr(image):
    """
    Cloud Mask for Landsat 8 surface reflectance. Derived From: https://developers.google.com/earth-engine/datasets/catalog/LANDSAT_LC08_C01_T1_SR
    """
    # Bits 3 and 5 are cloud shadow and cloud, respectively.
    cloudShadowBitMask = (1 << 3)
    cloudsBitMask = (1 << 5)

    # Get the pixel QA band.
    qa = image.select('pixel_qa')

    # Both flags should be set to zero, indicating clear conditions.
    mask = qa.bitwiseAnd(cloudShadowBitMask).eq(0).And(qa.bitwiseAnd(cloudsBitMask).eq(0))
    return image.updateMask(mask)


def cloud_mask_l457(image):
    """
    Cloud Mask for Landsat 7 surface reflectance. Derived From: https://developers.google.com/earth-engine/datasets/catalog/LANDSAT_LE07_C01_T1_SR
    """
    qa = image.select('pixel_qa')

    # If the cloud bit (5) is set and the cloud confidence (7) is high
    # or the cloud shadow bit is set (3), then it's a bad pixel.
    cloud = qa.bitwiseAnd(1 << 5).And(qa.bitwiseAnd(1 << 7)).Or(qa.bitwiseAnd(1 << 3))

    # Remove edge pixels that don't occur in all bands
    mask2 = image.mask().reduce(ee.Reducer.min())

    return image.updateMask(cloud.Not()).updateMask(mask2)


def mask_s2_clouds(image):
    """
    Cloud Mask for Sentinel 2 surface reflectance. Derived from: https://developers.google.com/earth-engine/datasets/catalog/COPERNICUS_S2
    """
    qa = image.select('QA60')

    # Bits 10 and 11 are clouds and cirrus, respectively.
    cloudBitMask = 1 << 10
    cirrusBitMask = 1 << 11

    # Both flags should be set to zero, indicating clear conditions.
    mask = qa.bitwiseAnd(cloudBitMask).eq(0).And(qa.bitwiseAnd(cirrusBitMask).eq(0))

    return image.updateMask(mask).divide(10000)
  1. The get_image_collection_asset function builds the map tile service URL for the given platform, sensor, and product and filters it by the dates and reducer method. Implement the get_image_collection_asset function as follows in methods.py:

def get_image_collection_asset(platform, sensor, product, date_from=None, date_to=None, reducer='median'):
    """
    Get tile url for image collection asset.
    """
    ee_product = EE_PRODUCTS[platform][sensor][product]

    collection = ee_product['collection']
    index = ee_product.get('index', None)
    vis_params = ee_product.get('vis_params', {})
    cloud_mask = ee_product.get('cloud_mask', None)

    log.debug(f'Image Collection Name: {collection}')
    log.debug(f'Band Selector: {index}')
    log.debug(f'Vis Params: {vis_params}')

    tile_url_template = "https://earthengine.googleapis.com/map/{mapid}/{{z}}/{{x}}/{{y}}?token={token}"

    try:
        ee_collection = ee.ImageCollection(collection)

        if date_from and date_to:
            ee_filter_date = ee.Filter.date(date_from, date_to)
            ee_collection = ee_collection.filter(ee_filter_date)

        if index:
            ee_collection = ee_collection.select(index)

        if cloud_mask:
            cloud_mask_func = getattr(cm, cloud_mask, None)
            if cloud_mask_func:
                ee_collection = ee_collection.map(cloud_mask_func)

        if reducer:
            ee_collection = getattr(ee_collection, reducer)()

        map_id_params = image_to_map_id(ee_collection, vis_params)

        return tile_url_template.format(**map_id_params)

    except EEException:
        log.exception('An error occurred while attempting to retrieve the image collection asset.')
  1. Implement the image_to_map_id function as follows in methods.py:

def image_to_map_id(image_name, vis_params={}):
    """
    Get map_id parameters
    """
    try:
        ee_image = ee.Image(image_name)
        map_id = ee_image.getMapId(vis_params)
        map_id_params = {
            'mapid': map_id['mapid'],
            'token': map_id['token']
        }
        return map_id_params

    except EEException:
        log.exception('An error occurred while attempting to retrieve the map id.')

3. Create Endpoint for Getting Map Images

In this step you'll create a new endpoint that can be used to call the get_image_collection_asset function from the client-side of the application.

  1. Add a new controller called get_image_collection to controllers.py:

import logging
from django.http import JsonResponse, HttpResponseNotAllowed
from .gee.methods import get_image_collection_asset

log = logging.getLogger(f'tethys.apps.{__name__}')
@login_required()
def get_image_collection(request):
    """
    Controller to handle image collection requests.
    """
    response_data = {'success': False}

    if request.method != 'POST':
        return HttpResponseNotAllowed(['POST'])

    try:
        log.debug(f'POST: {request.POST}')

        platform = request.POST.get('platform', None)
        sensor = request.POST.get('sensor', None)
        product = request.POST.get('product', None)
        start_date = request.POST.get('start_date', None)
        end_date = request.POST.get('end_date', None)
        reducer = request.POST.get('reducer', None)

        url = get_image_collection_asset(
            platform=platform,
            sensor=sensor,
            product=product,
            date_from=start_date,
            date_to=end_date,
            reducer=reducer
        )

        log.debug(f'Image Collection URL: {url}')

        response_data.update({
            'success': True,
            'url': url
        })

    except Exception as e:
        response_data['error'] = f'Error Processing Request: {e}'

    return JsonResponse(response_data)

Tip

In this step you added logging to the new endpoint. Tethys and Django leverage Python's built-in logging capabilities. Use logging statements in your code to provide useful debugging information, system status, or error capture in your production logs. The logging for a portal can be configured in the Tethys Portal Configuration. To learn more about logging in Tethys/Django see: Django Logging

  1. Add a new UrlMap to the url_maps method of the app class in app.py:

UrlMap(
    name='get_image_collection',
    url='earth-engine/get-image-collection',
    controller='earth_engine.controllers.get_image_collection'
),

4. Stub Out the Map JavaScript Methods

In this step you'll stub out the methods and variables you'll need to add the GEE layers to the map.

  1. Create new variables to store a reference to the Tethys MapView object and the GEE layer in public/js/gee_datasets.js:

// Map Variables
    var m_map,
        m_gee_layer;
  1. Add the following module function declarations in public/js/gee_datasets.js below the dataset select function declarations:

// Map Methods
    var update_map, update_data_layer, create_data_layer, clear_map;
  1. Add the following module function stubs in public/js/gee_datasets.js, just below the collect_data implementation:

// Map Methods
update_map = function() {};

update_data_layer = function(url) {};

create_data_layer = function(url) {};

clear_map = function() {};
  1. Use the Tethys MapView JavaScript API to retrieve the underlying OpenLayers Map object when the module initializes. Having a handle on this object gives us full control over the map using the OpenLayers JavaScript API.

/************************************************************************
*                  INITIALIZATION / CONSTRUCTOR
*************************************************************************/
$(function() {
    // Initialize Global Variables
    bind_controls();

    // EE Products
    EE_PRODUCTS = $('#ee-products').data('ee-products');

    // Initialize values
    m_platform = $('#platform').val();
    m_sensor = $('#sensor').val();
    m_product = $('#product').val();
    INITIAL_START_DATE = m_start_date = $('#start_date').val();
    INITIAL_END_DATE = m_end_date = $('#end_date').val();
    m_reducer = $('#reducer').val();

    m_map = TETHYS_MAP_VIEW.getMap();
});

5. Implement Adding Layers to the Map

In this step you'll implement the new methods with logic to (1) retrieve the XYZ map service URL by calling the new get-image-collection endpoint using AJAX and then (2) create a new OpenLayers Layer with an XYZ Source and add it to the map.

  1. Call the get-image-collection endpoint using jQuery.ajax() passing it the parameters from the controls in the``update_map`` method in public/js/gee_datasets.js:

update_map = function() {
    let data = collect_data();

    let xhr = $.ajax({
        type: 'POST',
        url: 'get-image-collection/',
        dataType: 'json',
        data: data
    });

    xhr.done(function(response) {
        if (response.success) {
            console.log(response.url);
            update_data_layer(response.url);
        } else {
            alert('Oops, there was a problem loading the map you requested. Please try again.');
        }
    });
};
  1. Bind the update_map method to the Load button click event in bind_controls the method:

$('#load_map').on('click', function() {
    update_map();
});

Note

If you test the Load button at this point, the AJAX call to the get-image-collection endpoint will fail because it is missing the CSRF token. The token is used to verify that the call came from our client-side code and not from a site posing to be our site. As a security precaution, the server will reject any POST requests that don't include this token. you'll add the CSRF token in the next step. For more information about CSRF see: Cross Site Request Forgery protection.

  1. Add the following code to the public/js/main.js file to automatically attach the CSRF Token to every AJAX request that needs it:

// Get a cookie
function getCookie(name) {
    var cookieValue = null;
    if (document.cookie && document.cookie != '') {
        var cookies = document.cookie.split(';');
        for (var i = 0; i < cookies.length; i++) {
            var cookie = jQuery.trim(cookies[i]);
            // Does this cookie string begin with the name we want?
            if (cookie.substring(0, name.length + 1) == (name + '=')) {
                cookieValue = decodeURIComponent(cookie.substring(name.length + 1));
                break;
            }
        }
    }
    return cookieValue;
}

// find if method is csrf safe
function csrfSafeMethod(method) {
    // these HTTP methods do not require CSRF protection
    return (/^(GET|HEAD|OPTIONS|TRACE)$/.test(method));
}

// add csrf token to appropriate ajax requests
$(function() {
    $.ajaxSetup({
        beforeSend: function(xhr, settings) {
            if (!csrfSafeMethod(settings.type) && !this.crossDomain) {
                xhr.setRequestHeader("X-CSRFToken", getCookie("csrftoken"));
            }
        }
    });
}); //document ready;
  1. The update_data_layer method lazily creates the m_gee_layer if it doesn't exist or reuses it if it does exist. Implement the update_data_layer method in public/js/gee_datasets.js:

update_data_layer = function(url) {
    if (!m_gee_layer) {
        create_data_layer(url);
    } else {
        m_gee_layer.getSource().setUrl(url);
    }
};
  1. The create_data_layer method creates a new ol.layer.Tile layer with an ol.source.XYZ source using the URL provided. The new layer is assigned to m_gee_layer so it can be reused in subsquent calls and then added to the map below the drawing layer so that drawn features will show up on top. Implement the create_data_layer method in public/js/gee_datasets.js.

create_data_layer = function(url) {
    let source = new ol.source.XYZ({
        url: url,
        attributions: '<a href="https://earthengine.google.com" target="_">Google Earth Engine</a>'
    });

    m_gee_layer = new ol.layer.Tile({
        source: source,
        opacity: 0.7
    });
    // Insert below the draw layer (so drawn polygons and points render on top of data layer).
    m_map.getLayers().insertAt(1, m_gee_layer);
};
  1. Verify that the layers are being loaded on the map at this point. Browse to http://localhost:8000/apps/earth-engine in a web browser and login if necessary. Use the dataset controls to select a dataset product and press the Load button. Changing to a new dataset and pressing Load should replace the current layer with the new one.

6. Implement Clearing Layers on the Map

Users can now visualize GEE layers on the map, but there is no way to clear the data from the map. In this step, you'll add a button that will remove layers and clear the map.

  1. Add Clear button to home controller in controllers.py:

clear_button = Button(
    name='clear_map',
    display_text='Clear',
    style='default',
    attributes={'id': 'clear_map'}
)

...

context = {
    'platform_select': platform_select,
    'sensor_select': sensor_select,
    'product_select': product_select,
    'start_date': start_date,
    'end_date': end_date,
    'reducer_select': reducer_select,
    'load_button': load_button,
    'clear_button': clear_button,
    'ee_products': EE_PRODUCTS,
    'map_view': map_view
}
  1. Add Clear button to the app_navigation_items block of the templates/earth_engine/home.html template:

{% block app_navigation_items %}
  <li class="title">Select Dataset</li>
  {% gizmo platform_select %}
  {% gizmo sensor_select %}
  {% gizmo product_select %}
  {% gizmo start_date %}
  {% gizmo end_date %}
  {% gizmo reducer_select %}
  <p class="help">Change variables to select a data product, then press "Load" to add that product to the map.</p>
  {% gizmo load_button %}
  {% gizmo clear_button %}
{% endblock %}
  1. The clear_map method removes the layer from the map and removes all references to it. Implement clear_map method in public/js/gee_datasets.js:

clear_map = function() {
    if (m_gee_layer) {
        m_map.removeLayer(m_gee_layer);
        m_gee_layer = null;
    }
};
  1. Bind the clear_map method to the click event of the Clear button (in the bind_controls method):

$('#clear_map').on('click', function() {
    clear_map();
});
  1. Verify that the Clear button works. Browse to http://localhost:8000/apps/earth-engine in a web browser and login if necessary. Load a dataset as before and then press the Clear button. The currently displayed layer should be removed from the map. Repeat this process a few times, loading several datasets before clearing at least one of the times, to ensure it is working properly.

7. Implement Map Loading Indicator

You may have noticed while testing the app, that it can take some time for a layer to load. In this step you will add a loading image to indicate to the user that the map is loading, so they don't keep pressing the Load button impatiently.

  1. Download this animated map loading image or find one that you like and save it to the public/images directory.

  2. Create a new stylesheet called public/css/loader.css with styles for the loader image:

#loader {
    display: none;
    position: absolute;
    top: calc(50vh - 185px);
    left: calc(50vw - 186px);
}

#loader img {
    border-radius: 10%;
    box-shadow: 0 0 10px rgba(0, 0, 0, 0.2);
}

#loader.show {
    display: block;
}
  1. Add the image to the after_app_content block of the templates/earth_engine/home.html template and include the new public/css/loader.css:

{% block content_dependent_styles %}
    {{ block.super }}
    <link rel="stylesheet" href="{% static 'earth_engine/css/map.css' %}" />
    <link rel="stylesheet" href="{% static 'earth_engine/css/loader.css' %}" />
{% endblock %}
{% block after_app_content %}
  <div id="ee-products" data-ee-products="{{ ee_products|jsonify }}"></div>
  <div id="loader">
    <img src="{% static 'earth_engine/images/map-loader.gif' %}">
  </div>
{% endblock %}
  1. Show the loader image when the map starts loading tiles by binding to tile load events on the layer Source. Update the create_data_layer method in public/js/gee_datasets.js:

create_data_layer = function(url) {
    let source = new ol.source.XYZ({
        url: url,
        attributions: '<a href="https://earthengine.google.com" target="_">Google Earth Engine</a>'
    });

    source.on('tileloadstart', function() {
        $('#loader').addClass('show');
    });

    source.on('tileloadend', function() {
        $('#loader').removeClass('show');
    });

    source.on('tileloaderror', function() {
        $('#loader').removeClass('show');
    });

    m_gee_layer = new ol.layer.Tile({
        source: source,
        opacity: 0.7
    });

    // Insert below the draw layer (so drawn polygons and points render on top of data layer).
    m_map.getLayers().insertAt(1, m_gee_layer);
};

8. Test and Verify

Browse to http://localhost:8000/apps/earth-engine in a web browser and login if necessary. Verify the following:

  1. Use the dataset controls to select a dataset and press the Load button to add it to the map.

  2. Subsequent dataset loads should replace the previous dataset.

  3. Use the Clear button to clear the map.

  4. When a layer is loading tiles, a loading image should display to indicate to the user that the app is working.

9. Solution

This concludes this portion of the GEE Tutorial. You can view the solution on GitHub at https://github.com/tethysplatform/tethysapp-earth_engine/tree/vis-gee-layers-solution-3.0 or clone it as follows:

git clone https://github.com/tethysplatform/tethysapp-earth_engine.git
cd tethysapp-earth_engine
git checkout -b vis-gee-layers-solution vis-gee-layers-solution-3.0