Compute API¶
Last Updated: February 11, 2015
Distributed computing in Tethys Platform is made possible with HTCondor. Portal wide HTCondor computing resources are managed through the Tethys Compute Admin Pages. Accessing these resources in your app and configuring app specific resources is made possible through the Compute API.
See also
For more information on HTCondor see Overview of HTCondor or the HTCondor User Manual.
Key Concepts¶
HTCondor is a job and resources management middleware. It can be used to create High-Throughput Computing (HTC) systems from diverse computing units including desktop computers or cloud-computing resources. These HTC systems are known as HTCondor pools or clusters. In Tethys the Python library TethysCluster is used to automatically provision HTCondor clusters on Amazon Web Services (AWS) or Microsoft Azure. Portal-wide clusters can be configured by the Tethys Portal admin using the Tethys Compute Admin Pages, or app-specific clusters can be configured in apps using the ClusterManager. To run jobs to a clusters, it must have a Scheduler configured. Portal-wide schedulers can also be configured by the Tethys Portal admin using the Tethys Compute Admin Pages, or app-specific schedulers can be set up through the Compute API.
See also
To see how to configure a job with a Scheduler see the Jobs API.
Working with the Cluster Manager¶
The cluster manager can be used to create new computing clusters. It is accessed through the get_cluster_manager function.
from tethys_sdk.compute import get_cluster_manager
tethyscluster_config_file = '/path/to/TethysCluster/config/file'
cluster_manager = get_cluster_manager(tethyscluster_config_file)
For more information on how to use the cluster manager see the TethysCompute documentation
Working with Schedulers¶
Portal-wide schedulers can be accessed through the list_schedulers and the get_scheduler functions.
from tethys_sdk.compute import list_schedulers, get_scheduler
scheduler = list_schedulers()[0]
# this assumes the Tethys Portal administrator has created a scheduler named 'Default'.
scheduler = get_scheduler('Default')
App-specific schedulers can be created with the create_scheduler function.
from tethys_sdk.compute import create_scheduler
# For Condor Job
scheduler = create_scheduler(name='my_app_scheduler', host='example.com', type='condor', username='root', private_key_path='/path/to/private/key')
# For Dask Job
scheduler = create_scheduler(name='my_app_scheduler', host='localhost:8786', type='dask', timeout=30, heartbeat_interval=2, dashboard='localhost:8787')
API Documentation¶
- tethys_sdk.compute.list_schedulers()¶
Gets a list of all scheduler objects registered in the Tethys Portal
- Returns
List of Schedulers
- tethys_sdk.compute.get_scheduler(name)¶
Gets the scheduler associated with the given name
- Parameters
name (str) -- The name of the scheduler to return
- Returns
The scheduler with the given name or None if no scheduler has the name given.
- tethys_sdk.compute.create_scheduler(name, host, scheduler_type='condor', **kwargs)¶
Creates a new scheduler of the type given.
- Parameters
name (str) -- The name of the scheduler
host (str) -- The hostname or IP address of the scheduler
scheduler_type (str) -- Type of scheduler to create. Either 'dask' or 'condor'. Defaults to 'condor'.
kwargs -- Keyword arguments of scheduler-specific options. See: create_dask_scheduler and create_condor_scheduler.
- Returns
The newly created scheduler
Note
The newly created scheduler object is not committed to the database.
- tethys_sdk.compute.create_condor_scheduler(name, host, username=None, password=None, private_key_path=None, private_key_pass=None)¶
Creates a new condor scheduler
- Parameters
name (str) -- The name of the scheduler
host (str) -- The hostname or IP address of the scheduler
username (str, optional) -- The username to use when connecting to the scheduler
password (str, optional) -- The password for the username
private_key_path (str, optional) -- The path to the location of the SSH private key file
private_key_pass (str, optional) -- The passphrase for the private key
- Returns
The newly created condor scheduler
Note
The newly created condor scheduler object is not committed to the database.
- tethys_sdk.compute.create_dask_scheduler(name, host, timeout=None, heartbeat_interval=None, dashboard=None)¶
Creates a new dask scheduler
- Parameters
name (str) -- The name of the scheduler
host (str) -- The hostname or IP address of the scheduler
timeout (str, optional) --
heartbeat_interval (str, optional) --
dashboard (str, optional) --
- Returns
The newly created dask scheduler
Note
The newly created dask scheduler object is not committed to the database.