Compute API

Last Updated: February 11, 2015

Distributed computing in Tethys Platform is made possible with HTCondor. Portal wide HTCondor computing resources are managed through the Tethys Compute Admin Pages. Accessing these resources in your app and configuring app specific resources is made possible through the Compute API.

See also

For more information on HTCondor see Overview of HTCondor or the HTCondor User Manual.

Key Concepts

HTCondor is a job and resources management middleware. It can be used to create High-Throughput Computing (HTC) systems from diverse computing units including desktop computers or cloud-computing resources. These HTC systems are known as HTCondor pools or clusters. In Tethys the Python library TethysCluster is used to automatically provision HTCondor clusters on Amazon Web Services (AWS) or Microsoft Azure. Portal-wide clusters can be configured by the Tethys Portal admin using the Tethys Compute Admin Pages, or app-specific clusters can be configured in apps using the ClusterManager. To run jobs to a clusters, it must have a Scheduler configured. Portal-wide schedulers can also be configured by the Tethys Portal admin using the Tethys Compute Admin Pages, or app-specific schedulers can be set up through the Compute API.

See also

To see how to configure a job with a Scheduler see the Jobs API.

Working with the Cluster Manager

The cluster manager can be used to create new computing clusters. It is accessed through the get_cluster_manager function.

from tethys_sdk.compute import get_cluster_manager

tethyscluster_config_file = '/path/to/TethysCluster/config/file'
cluster_manager = get_cluster_manager(tethyscluster_config_file)

For more information on how to use the cluster manager see the TethysCompute documentation

Working with Schedulers

Portal-wide schedulers can be accessed through the list_schedulers and the get_scheduler functions.

from tethys_sdk.compute import list_schedulers, get_scheduler

scheduler = list_schedulers()[0]

# this assumes the Tethys Portal administrator has created a scheduler named 'Default'.
scheduler = get_scheduler('Default')

App-specific schedulers can be created with the create_scheduler function.

from tethys_sdk.compute import create_scheduler

scheduler = create_scheduler(name='my_app_scheduler', host='example.com', username='root', private_key_path='/path/to/private/key')

API Documentation

tethys_sdk.compute.list_schedulers()

Gets a list of all scheduler objects registered in the Tethys Portal

Returns:List of Schedulers
tethys_sdk.compute.get_scheduler(name)

Gets the scheduler associated with the given name

Parameters:name (str) – The name of the scheduler to return
Returns:The scheduler with the given name or None if no scheduler has the name given.
tethys_sdk.compute.create_scheduler(name, host, username=None, password=None, private_key_path=None, private_key_pass=None)

Creates a new scheduler

Parameters:
  • name (str) – The name of the scheduler
  • host (str) – The hostname or IP address of the scheduler
  • username (str, optional) – The username to use when connecting to the scheduler
  • password (str, optional) – The password for the username
  • private_key_path (str, optional) – The path to the location of the SSH private key file
  • private_key_pass (str, optional) – The passphrase for the private key
Returns:

The newly created scheduler

Note

The newly created scheduler object is not committed to the database.