Troubleshooting
Last Updated: September 2022
This document provides a number of resources that can be used to troubleshoot your production installation of Tethys Portal.
Check the Logs
Your first stop when troubleshooting should be to check the logs to determine if there are any errors occuring. There are several logs that may contain useful information related to any issues you are having.
Supervisor Logs
The supervisor logs are most helpful if you encounter errors while running one of the supervisorctl commands:
/var/log/supervisor/supervisor.log
NGINX Logs
The NGINX error log is useful for debugging issues related to configuration problems with NGINX or other errors related to NGINX.
/var/log/nginx/error.log
The access log records all incoming web traffic and response from NGINX. This file is helpful for debugging problems connecting to Tethys Portal. For example, if NGINX is running and there are no new entries in the access.log when you try to access Tethys Portal, then the connection is not making it to NGINX. This could be caused by network issues with your server or closed ports in your firewall.
/var/log/nginx/access.log
Tethys Logs
The Tethys Portal log is useful for determining errors that are raised by Tethys Portal such as 500 errors.
/var/log/tethys/tethys.log
Email Logs
If you are having trouble with the Postfix email server, then you should check the syslog. Look for entries with postfix in them.
/var/log/syslog
Note
A 250 status code means a send operation was successful.
Linux Commands
Learning a few of the common Linux commands can help you become more proficient at debugging issues with production installations.
Tail Command
The tail command is often useful for checking logs. It allows you to view the last lines of a file, which is especially helpful for logs b/c they tend to get long:
tail -n <number-of-lines> <path-to-file>
You can also have tail follow the logs, so you can see live print outs to the logs as you interact with the website. Just add the -f option to follow the log file:
tail -f -n <number-of-lines> <path-to-file>
Grep Command
The grep command is another useful utility when inspecting logs. You can pipe the output from a tail command into a grep command to filter the output to only lines containing a query string or pattern. For example:
tail -n 100 /var/log/syslog | grep "postfix"
Chown Command
The chown command can be used to change the ownership of files and directories. For example, change the ownership of all files in a directory to a certain user:
sudo chown -R <username> /path/to/dir
Chmod Command
The chmod command can be used to change permission levels of owners, groups, and everyone else on files and directories. For example to add execute permissions of the owners of the file you could run:
sudo chmod +ux /path/to/file.ext
Review Configuration
Many issues with a Tethys Portal production installation come down to a configuration issue. This is especially true if you are having issues starting NGINX or Daphne (ASGI). If the issue is not readily apparent in the logs, then a next step should be to review the configuration files.
You should verify the following:
Paths
Syntax errors
Spelling errors in variables
Other inconsistencies
Supervisor
There are two Tethys specific configuration files for supervisor, one for NGINX and one for Daphne (ASGI).
~/.tethys/asgi_supervisord.conf
~/.tethys/nginx_supervisord.conf
Also verify that these files are correctly linked to the appropriate directory in /etc (see Supervisor & Daphne Configuration). Listing the contents of the directory with the -l option will show you if the links are valid or not:
Ubuntu:
ls -l /etc/supervisor/conf.d/
Rocky Linux:
ls -l /etc/supervisord.d/
NGINX
The Tethys-specific configuration file for NGINX is usually located at:
~/.tethys/tethys_nginx.conf
Also verify that this file is correctly linked to the appropriate directory in /etc (see NGINX Configuration). Listing the contents of the directory with the -l option will show you if the link is valid or not:
Ubuntu:
ls -l /etc/nginx/sites-enabled/
Rocky Linux:
ls -l /etc/nginx/conf.d/
Tethys
All Tethys and Django settings are configured using the portal_config.yml.
~/.tethys/portal_config.yml
Important
Any Django setting can be added to the settings section of the portal_config.yml. DO NOT EDIT THE settings.py FILE DIRECTLY
SELinux
If you suspect an issue with SELinux then inspecting the tethys-selinux.te file may be worthwhile:
~/.tethys/tethys-selinux.te
If you make a change to the tethys-selinux.te file, you will need to run the checkmodule and semodule_package commands again and then update the policy (see: 3. Security-Enhanced Linux File Permissions (Rocky Linux, May not Apply)).
File Permissions
The production Tethys Portal server processes will be managed by the NGINX_USER. If you encounter issues with file permissions (i.e. permission denied), you may need to grant the NGINX_USER access to additional directories.
The minimum directories that should be owned by the NGXINX_USER at runtime are the STATIC_ROOT and TETHYS_WORKSPACES_ROOT directories (see: File Permissions). Listing the contents of these directories is a good sanity check to ensure the contents are owned by the NGINX_USER:
sudo ln -l <STATIC_ROOT>
sudo ln -l <TETHYS_WORKSPACES_ROOT>
You may also need to modify what level of access the NGINX_USER has. This can be done using the chmod command. For example, to set the owner to have read-only access to all files in a directory, you could run:
sudo chmod -R =ur /path/to/dir
Caution
Granting the NGINX_USER access to directories and files on your server should be done judiciously. Remember anything the NGINX_USER can access is potentially accessible to the internet at large and the internet is a hostile environment. When possible, you should also restrict access to read-only.
Internet Search
Your first step in addressing an issue with your production installation is to do an internet search. As each production installation is different, it is likely that the issue you are encountering is specific to your setup. It is also likely that someone else has encountered this same issue and there is a solution to it on one of the many online forums.
The following tips can help you when searching for an issue:
Include the software name somewhere in the search (i.e.: supervisor, nginx, Django)
Include the error message from any traceback, but remove anything that is specific to your machine or instance
Tethys Portal is built on Django, so adding Django to your search terms often yields helpful results.
Recognize that many issues you may encounter won't be Tethys specific. If you can figure out where the error is coming from then you can narrow your search (see: Check the Logs). For example, if the error is occurring in one of the NGINX logs, then adding "NGINX" to your search terms would be more helpful than adding "Tethys".
Tethys Platform GitHub Discussions
The Tethys Platform GitHub Discussions is an excellent place to search for Tethys-specific problems. Many members of the group respond to questions posted, including the primary developers of Tethys Platform.
Please search for your issue before posting a new question, as someone likely has already asked the question you want to ask. If you do post a question, please provide as much information as possible. At a minimum, include:
Steps to reproduce the problem.
A clear description of the problem with traceback if applicable.
What the expected behavior should be.
Helpful metadata such as the operating system and version of Tethys.