Compute API¶
Last Updated: February 11, 2015
Distributed computing in Tethys Platform is made possible with HTCondor. Portal wide HTCondor computing resources are managed through the Tethys Compute Admin Pages. Accessing these resources in your app and configuring app specific resources is made possible through the Compute API.
See also
For more information on HTCondor see Overview of HTCondor or the HTCondor User Manual.
Key Concepts¶
HTCondor is a job and resources management middleware. It can be used to create High-Throughput Computing (HTC) systems from diverse computing units including desktop computers or cloud-computing resources. These HTC systems are known as HTCondor pools or clusters. In Tethys the Python library TethysCluster is used to automatically provision HTCondor clusters on Amazon Web Services (AWS) or Microsoft Azure. Portal-wide clusters can be configured by the Tethys Portal admin using the Tethys Compute Admin Pages, or app-specific clusters can be configured in apps using the ClusterManager. To run jobs to a clusters, it must have a Scheduler configured. Portal-wide schedulers can also be configured by the Tethys Portal admin using the Tethys Compute Admin Pages, or app-specific schedulers can be set up through the Compute API.
See also
To see how to configure a job with a Scheduler see the Jobs API.
Working with the Cluster Manager¶
The cluster manager can be used to create new computing clusters. It is accessed through the get_cluster_manager function.
from tethys_sdk.compute import get_cluster_manager
tethyscluster_config_file = '/path/to/TethysCluster/config/file'
cluster_manager = get_cluster_manager(tethyscluster_config_file)
For more information on how to use the cluster manager see the TethysCompute documentation
Working with Schedulers¶
Portal-wide schedulers can be accessed through the list_schedulers and the get_scheduler functions.
from tethys_sdk.compute import list_schedulers, get_scheduler
scheduler = list_schedulers()[0]
# this assumes the Tethys Portal administrator has created a scheduler named 'Default'.
scheduler = get_scheduler('Default')
App-specific schedulers can be created with the create_scheduler function.
from tethys_sdk.compute import create_scheduler
scheduler = create_scheduler(name='my_app_scheduler', host='example.com', username='root', private_key_path='/path/to/private/key')
API Documentation¶
- tethys_sdk.compute.get_cluster_manager(config_file=None, cache=False)¶
Factory for ClusterManager class that attempts to load AWS credentials from the TethysCluster config file. Returns a ClusterManager object if successful
- tethys_sdk.compute.list_schedulers()¶
Gets a list of all scheduler objects registered in the Tethys Portal
Returns: List of Schedulers
- tethys_sdk.compute.get_scheduler(name)¶
Gets the scheduler associated with the given name
Parameters: name (str) – The name of the scheduler to return Returns: The scheduler with the given name or None if no scheduler has the name given.
- tethys_sdk.compute.create_scheduler(name, host, username=None, password=None, private_key_path=None, private_key_pass=None)¶
Creates a new scheduler
Parameters: - name (str) – The name of the scheduler
- host (str) – The hostname or IP address of the scheduler
- username (str, optional) – The username to use when connecting to the scheduler
- password (str, optional) – The password for the username
- private_key_path (str, optional) – The path to the location of the SSH private key file
- private_key_pass (str, optional) – The passphrase for the private key
Returns: The newly created scheduler
Note
The newly created scheduler object is not committed to the database.