Last Updated: February 11, 2015
Distributed computing in Tethys Platform is made possible with HTCondor. Portal wide HTCondor computing resources are managed through the Tethys Compute Admin Pages. Accessing these resources in your app and configuring app specific resources is made possible through the Compute API.
HTCondor is a job and resources management middleware. It can be used to create High-Throughput Computing (HTC) systems from diverse computing units including desktop computers or cloud-computing resources. These HTC systems are known as HTCondor pools or clusters. In Tethys the Python library TethysCluster is used to automatically provision HTCondor clusters on Amazon Web Services (AWS) or Microsoft Azure. Portal-wide clusters can be configured by the Tethys Portal admin using the Tethys Compute Admin Pages, or app-specific clusters can be configured in apps using the ClusterManager. To run jobs to a clusters, it must have a
Scheduler configured. Portal-wide schedulers can also be configured by the Tethys Portal admin using the Tethys Compute Admin Pages, or app-specific schedulers can be set up through the Compute API.
To see how to configure a job with a
Scheduler see the Jobs API.
Working with the Cluster Manager¶
The cluster manager can be used to create new computing clusters. It is accessed through the
from tethys_sdk.compute import get_cluster_manager tethyscluster_config_file = '/path/to/TethysCluster/config/file' cluster_manager = get_cluster_manager(tethyscluster_config_file)
For more information on how to use the cluster manager see the TethysCompute documentation
Working with Schedulers¶
Portal-wide schedulers can be accessed through the
list_schedulers and the
from tethys_sdk.compute import list_schedulers, get_scheduler scheduler = list_schedulers() # this assumes the Tethys Portal administrator has created a scheduler named 'Default'. scheduler = get_scheduler('Default')
App-specific schedulers can be created with the
from tethys_sdk.compute import create_scheduler scheduler = create_scheduler(name='my_app_scheduler', host='example.com', username='root', private_key_path='/path/to/private/key')
Gets a list of all scheduler objects registered in the Tethys Portal
Returns: List of Schedulers
Gets the scheduler associated with the given name
Parameters: name (str) – The name of the scheduler to return Returns: The scheduler with the given name or None if no scheduler has the name given.
create_scheduler(name, host, username=None, password=None, private_key_path=None, private_key_pass=None)¶
Creates a new scheduler
- name (str) – The name of the scheduler
- host (str) – The hostname or IP address of the scheduler
- username (str, optional) – The username to use when connecting to the scheduler
- password (str, optional) – The password for the username
- private_key_path (str, optional) – The path to the location of the SSH private key file
- private_key_pass (str, optional) – The passphrase for the private key
The newly created scheduler
The newly created scheduler object is not committed to the database.