Last Updated: May 2020
This document provides a number of resources that can be used to troubleshoot your production installation of Tethys Portal.
Check the Logs¶
Your first stop when troubleshooting should be to check the logs to determine if there are any errors occuring. There are several logs that may contain useful information related to any issues you are having.
The supervisor logs are most helpful if you encounter errors while running one of the
The NGINX error log is useful for debugging issues related to configuration problems with NGINX or other errors related to NGINX.
The access log records all incoming web traffic and response from NGINX. This file is helpful for debugging problems connecting to Tethys Portal. For example, if NGINX is running and there are no new entries in the
access.log when you try to access Tethys Portal, then the connection is not making it to NGINX. This could be caused by network issues with your server or closed ports in your firewall.
The Tethys Portal log is useful for determining errors that are raised by Tethys Portal such as 500 errors.
If you are having trouble with the Postfix email server, then you should check the
syslog. Look for entries with
postfix in them.
A 250 status code means a send operation was successful.
Learning a few of the common Linux commands can help you become more proficient at debugging issues with production installations.
The tail command is often useful for checking logs. It allows you to view the last lines of a file, which is especially helpful for logs b/c they tend to get long:
tail -n <number-of-lines> <path-to-file>
You can also have
tail follow the logs, so you can see live print outs to the logs as you interact with the website. Just add the -f option to follow the log file:
tail -f -n <number-of-lines> <path-to-file>
The grep command is another useful utility when inspecting logs. You can pipe the output from a tail command into a grep command to filter the output to only lines containing a query string or pattern. For example:
tail -n 100 /var/log/syslog | grep "postfix"
The chown command can be used to change the ownership of files and directories. For example, change the ownership of all files in a directory to a certain user:
sudo chown -R <username> /path/to/dir
The chmod command can be used to change permission levels of owners, groups, and everyone else on files and directories. For example to add execute permissions of the owners of the file you could run:
sudo chmod +ux /path/to/file.ext
Many issues with a Tethys Portal production installation come down to a configuration issue. This is especially true if you are having issues starting NGINX or Daphne (ASGI). If the issue is not readily apparent in the logs, then a next step should be to review the configuration files.
You should verify the following:
Spelling errors in variables
There are two Tethys specific configuration files for supervisor, one for NGINX and one for Daphne (ASGI).
Also verify that these files are correctly linked to the appropriate directory in
/etc (see Supervisor & Daphne Configuration). Listing the contents of the directory with the -l option will show you if the links are valid or not:
ls -l /etc/supervisor/conf.d/
ls -l /etc/supervisord.d/
The Tethys-specific configuration file for NGINX is usually located at:
Also verify that this file is correctly linked to the appropriate directory in
/etc (see NGINX Configuration). Listing the contents of the directory with the -l option will show you if the link is valid or not:
ls -l /etc/nginx/sites-enabled/
ls -l /etc/nginx/conf.d/
All Tethys and Django settings are configured using the
Any Django setting can be added to the
settings section of the
portal_config.yml. DO NOT EDIT THE settings.py FILE DIRECTLY
If you suspect an issue with SELinux then inspecting the
tethys-selinux.te file may be worthwhile:
If you make a change to the
tethys-selinux.te file, you will need to run the
semodule_package commands again and then update the policy (see: 3. Security-Enhanced Linux File Permissions (CentOS, May not Apply)).
The production Tethys Portal server processes will be managed by the
NGINX_USER. If you encounter issues with file permissions (i.e. permission denied), you may need to grant the
NGINX_USER access to additional directories.
The minimum directories that should be owned by the
NGXINX_USER at runtime are the
TETHYS_WORKSPACES_ROOT directories (see: File Permissions). Listing the contents of these directories is a good sanity check to ensure the contents are owned by the
sudo ln -l <STATIC_ROOT> sudo ln -l <TETHYS_WORKSPACES_ROOT>
You may also need to modify what level of access the
NGINX_USER has. This can be done using the
chmod command. For example, to set the owner to have read-only access to all files in a directory, you could run:
sudo chmod -R =ur /path/to/dir
NGINX_USER access to directories and files on your server should be done judiciously. Remember anything the
NGINX_USER can access is potentially accessible to the internet at large and the internet is a hostile environment. When possible, you should also restrict access to read-only.
Your first step in addressing an issue with your production installation is to do an internet search. As each production installation is different, it is likely that the issue you are encountering is specific to your setup. It is also likely that someone else has encountered this same issue and there is a solution to it on one of the many online forums.
The following tips can help you when searching for an issue:
Include the software name somewhere in the search (i.e.: supervisor, nginx, Django)
Include the error message from any traceback, but remove anything that is specific to your machine or instance
Tethys Portal is built on Django, so adding Django to your search terms often yields helpful results.
Recognize that many issues you may encounter won't be Tethys specific. If you can figure out where the error is coming from then you can narrow your search (see: Check the Logs). For example, if the error is occurring in one of the NGINX logs, then adding "NGINX" to your search terms would be more helpful than adding "Tethys".
Tethys Platform Forum¶
The Tethys Platform Forum is an excellent place to search for Tethys-specific problems. Many members of the forum respond to questions posted on the forum, including the primary developers of Tethys Platform.
Please search for your issue before posting a new question, as someone likely has already asked the question you want to ask. If you do post a question, please provide as much information as possible. At a minimum, include:
Steps to reproduce the problem.
A clear description of the problem with traceback if applicable.
What the expected behavior should be.
Helpful metadata such as the operating system and version of Tethys.