THREDDS Primer

Last Updated: May 2023

In this tutorial you will be introduced to THREDDS using the Docker container that is included with Tethys Platform. This primer is adapted from the THREDDS Data Server Documentation. Topics covered include:

  • Thematic Real-Time Environmental Distributed Data Services (THREDDS)

  • THREDDS Data Server (TDS)

  • Open-source Project for a Network Data Access Protocol (OpeNDAP)

  • NetCDF Subset Serivce (NCSS)

  • Web Mapping Service (WMS)

  • Web Coverage Service (WCS)

  • TDS Content Directory

  • Catalog

  • Datasets

  • Services

  • Tethys Docker Containers

  • THREDDS Docker Image

1. Install TDS

The THREDDS Data Server (TDS) can be installed either manually (see: Getting Started with the TDS) or using the Docker image developed by Unidata (see THREDDS Docker image). For this tutorial, we recommend using the Docker image to install THREDDS. We have made it easier to install the THREDDS Docker image by adding it as an option to the Tethys Docker command.

Warning

Docker needs to be installed to use the tethys docker command (see Install Docker). You will also need to add your user to the docker group and logout and log back in (see Linux Post Install).

  1. Initialize the THREDDS Docker container, making sure to bind the data directory when prompted:

bash
tethys docker init -c thredds

Note

The command tethys docker init only needs to be run the first time you are creating a container. If it already exists, you can skip to the next step.

  1. Start the THREDDS Docker container:

bash
tethys docker start -c thredds

Tip

To stop the docker container:

bash
tethys docker stop -c thredds

For more information about the Docker interface in Tethys Platform see the docker command reference.

  1. Obtain the endpoint for the THREDDS Docker container:

bash
tethys docker ip

Tip

Alternatively, you may install TDS manually using the Getting Started with the TDS guide. You may also use an existing TDS server, provided you have access to it's content directory.

2. Explore the Content Directory

Navigate to the TDS Content Directory and explore the contents. This directory stores all of the TDS configuration information and data. Review the TDS Content Directory tutorial for an explanation of each of the files and folders in the content directory.

Tip

When using the THREDDS docker container, the location of this directory is the folder that you specified during the bind step. The default bind location is ~/.tethys/thredds.

3. Customize TDS

The TDS Configuration File (threddsConfig.xml) is used to control the behavior of the TDS including metadata about the server and which services are enabled.

  1. Open threddsConfig.xml in a text editor.

  2. Customize the following parameters in the serverInformation section:

  • name: Name of the TDS Server

  • logoUrl: Logo for the TDS Server

  • logoAltText: Text description of logo for screen readers

  • abstract: Description of the data hosted on the TDS Server

  • keywords: Keywords for the type of data hosted on the TDS Server

  • contact
    • name: Name of TDS Server Maintainer

    • organization: Organization of TDS Server Maintainer

    • email: Email of TDS Server Maintainer

  • hostInstitution
    • name: Institution/organization hosting the TDS Server

    • webSite: Website of institution/organization hosting the TDS Server

    • logoUrl: Logo of institution/organization hosting the TDS Server

    • logoAltText: Text description of logo for screen readers

  1. Save your changes and restart the TDS:

bash
tethys docker restart -c thredds

Tip

You can view the logs TDS server in the Docker container using the Docker CLI:

bash
docker logs tethys_thredds

This is useful for debugging issues when the container won't start, such as malformed XML in the threddsConfig.xml.

  1. Navigate to the following locations to see how this metadata is incorporated into the TDS:

Tip

See Basic TDS Configuration for more information about configuring TDS.

  1. The TDS Configuration File is also used to control which data services are enabled. Services that are supported by TDS include Web Coverage Service (WCS), Web Map Service (WMS), ncISO, and NetCDF Subset Service (NCSS). Most of these services are disabled by default in the docker version of TDS and need to be enabled. Review the following documentation about how to enable the various data services in TDS:

  1. You can further customize the look and feel of your TDS using CSS. Navigate to the htmlSetup section of the TDS Configuration File (threddsConfig.xml) and note the names of the css files for different pages (e.g. tds.css, tdsCat.css, tdsDap.css). If you create these files in the public directory, you'll be able to customize the CSS of the associated pages. See the Customizing the TDS Look and Feel documentation for more details.

4. Client/Configuration Catalogs

  1. Review the following documentation on catalog configuration (ignore exercises):

Tip

It is easy to confuse the TDS Root Configuration Catalog (catalog.xml) and the TDS Configuration File (threddsConfig.xml). Remember:

  • TDS Root Configuration Catalog (catalog.xml): Used to define and configure datasets hosted by the TDS server.

  • TDS Configuration File (threddsConfig.xml): Used to customize TDS server information and behaviour.

5. NetCDF Data Exercise

  1. Download this National Water Model Short Range Forecast data and extract it. The archive should contain the first 3 of 18 NetCDF files each representing a 1-hour increment in an 18 hour forecast produced by the National Water Model. Each file contains over 2.7 million forecast points where each point is associated a different stream reach on the National Hydrogrophy Dataset.

  2. Create a new nwm directory in the public directory of the TDS Content Directory (i.e.: <TDS_CONTENT_DIRECTORY>/public/nwm).

  3. Copy the NetCDF files from step 1 into <TDS_CONTENT_DIRECTORY>/public/nwm.

  4. Create a new catalog coniguration file at the root of the TDS Content Directory called nwmCatalog.xml with the following contents:

xml
<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink"
   name="Unidata THREDDS-IDD NetCDF-OpenDAP Server" version="1.0.1">

  <service name="latest" serviceType="Resolver" base="" />
  <service name="all" serviceType="Compound" base="">
    <service name="ncdods" serviceType="OPENDAP" base="/thredds/dodsC/" />
    <service name="HTTPServer" serviceType="HTTPServer" base="/thredds/fileServer/" />
  </service>

  <dataset name="National Water Model Data" collectionType="TimeSeries">
    <metadata inherited="true">
      <serviceName>all</serviceName>
      <authority>edu.ucar.unidata</authority>
      <dataType>Points</dataType>
      <dataFormat>NetCDF</dataFormat>
      <documentation type="rights">Freely available</documentation>
      <documentation xlink:href="https://water.noaa.gov/about/nwm" xlink:title="National Water Model documentation"></documentation>
      <creator>
        <name vocabulary="DIF">DOC/NOAA/NWS/OWP</name>
        <contact url="https://water.noaa.gov/" email="nws.nwc.ops@noaa.gov" />
      </creator>
      <timeCoverage>
        <start>2020-01-06T00:00:00</start>
        <duration>18 hours</duration>
      </timeCoverage>
    </metadata>

    <datasetScan name="NWM Short Range Data" ID="nwm_short_range" path="nwm" location="content/nwm/" harvest="true">
      <metadata inherited="true">
        <documentation type="summary">National Water Model (NWM) - a hydrologic modelling framework that simulates observed and forecast streamflow over the entire continental United States.</documentation>
        <geospatialCoverage>
          <northsouth>
            <start>24.637987</start>
            <size>24.795109</size>
            <units>degrees_north</units>
          </northsouth>
          <eastwest>
            <start>-125.946552</start>
            <size>60.346914</size>
            <units>degrees_east</units>
          </eastwest>
          <updown>
            <start>0.0</start>
            <size>0.0</size>
            <units>km</units>
          </updown>
        </geospatialCoverage>
      </metadata>
      <sort>
        <lexigraphicByName increasing="true"/>
      </sort>
    </datasetScan>
  </dataset>
</catalog>
  1. Add a new catalog reference to the nwmCatalog.xml at the bottom of the catalog section of catalog.xml:

xml
<catalogRef xlink:title="National Water Model Catalog" xlink:href="nwmCatalog.xml" name=""/>
  1. Restart the THREDDS server:

bash
tethys docker restart -c thredds
  1. Navigate to http://localhost:8383/thredds/catalog/nwm/catalog.html to verify that the new data is available.