Condor Job Type

Last Updated: December 27, 2018

The Condor Job Type (and Condor Workflow Job Type) enable the real power of the jobs API by combining it with the Compute API. This make it possible for jobs to be offloaded from the main web server to a scalable computing cluster, which in turn enables very large scale jobs to be processed.

See also

The Condor Job and the Condor Workflow job types use the CondorPy library to submit jobs to HTCondor compute pools. For more information on CondorPy and HTCondor see the CondorPy documentation and specifically the Overview of HTCondor.

Creating a Condor Job

To create a job call the create_job method on the job manager. The required parameters are name, user and job_type. Any other job attributes can also be passed in as kwargs.

from tethys_sdk.compute import list_schedulers
from .app import MyApp as app

def some_controller(request):

    # get the path to the app workspace to reference job files
    app_workspace = app.get_app_workspace().path

    # create a new job from the job manager
    job = job_manager.create_job(
        name='myjob_{id}',  # required
        user=request.user,  # required
        job_type='CONDOR',  # required

        # any other properties can be passed in as kwargs
        attributes=dict(
            transfer_input_files=('../input_1', '../input_2'),
            transfer_output_files=('example_output1', example_output2),
        ),
        condorpy_template_name='vanilla_transfer_files',
        remote_input_files=(
            os.path.join(app_workspace, 'my_script.py'),
            os.path.join(app_workspace, 'input_1'),
            os.path.join(app_workspace, 'input_2')
        )
    )

    # properties can also be added after the job is created
    job.extended_properties = {'one': 1, 'two': 2}

    # each job type may provided methods to further specify the job
    job.set_attribute('executable', 'my_script.py')

    # get a scheduler for the job
    my_scheduler = list_schedulers()[0]
    job.scheduler = my_scheduler

    # save or execute the job
    job.save()
    # or
    job.execute()

Before a controller returns a response the job must be saved or else all of the changes made to the job will be lost (executing the job automatically saves it). If submitting the job takes a long time (e.g. if a large amount of data has to be uploaded to a remote scheduler) then it may be best to use AJAX to execute the job.

API Documentation

class tethys_compute.models.CondorJob(*args, **kwargs)

CondorPy Job job type

class tethys_sdk.jobs.CondorJobTemplate(name, job_description, scheduler=None, **kwargs)

DEPRECATED A subclass of the JobTemplate with the type argument set to CondorJob.

Parameters

• name (str) -- Name to refer to the template.

• job_description (CondorJobDescription) -- An object containing the attributes for the condorpy job.

• scheduler (Scheduler) -- An object containing the connection information to submit the condorpy job remotely.