Condor Workflow Job Type

Last Updated: December 27, 2018

A Condor Workflow provides a way to run a group of jobs (which can have hierarchical relationships) as a single (Tethys) job. The hierarchical relationships are defined as parent-child relationships. For example, suppose a workflow is defined with three jobs: JobA, JobB, and JobC, which must be run in that order. These jobs would be defined with the following relationships: JobA is the parent of JobB, and JobB is the parent of JobC.

See also

The Condor Workflow job type uses the CondorPy library to submit jobs to HTCondor compute pools. For more information on CondorPy and HTCondor see the CondorPy documentation and specifically the Overview of HTCondor.

Creating a Condor Workflow

Creating a Condor Workflow job involves 3 steps:

  1. Create an empty Workflow job from the job manager.

  2. Create the jobs that will make up the workflow with CondorWorkflowJobNode

  3. Define the relationships among the nodes

from tethys_sdk.jobs import CondorWorkflowJobNode
from .app import MyApp as app

def some_controller(request):

    # get the path to the app workspace to reference job files
    app_workspace = app.get_app_workspace().path

    workflow = job_manager.create_job(
        name='MyWorkflowABC',
        user=request.user,
        job_type='CONDORWORKFLOW',
        scheduler=None,
    )
    workflow.save()

    job_a = CondorWorkflowJobNode(
        name='JobA',
        workflow=workflow,
        condorpy_template_name='vanilla_transfer_files',
        remote_input_files=(
            os.path.join(app_workspace, 'my_script.py'),
            os.path.join(app_workspace, 'input_1'),
            os.path.join(app_workspace, 'input_2')
        ),
        attributes=dict(
            executable='my_script.py',
            transfer_input_files=('../input_1', '../input_2'),
            transfer_output_files=('example_output1', 'example_output2'),
        )
    )
    job_a.save()

    job_b = CondorWorkflowJobNode(
        name='JobB',
        workflow=workflow,
        condorpy_template_name='vanilla_transfer_files',
        remote_input_files=(
            os.path.join(app_workspace, 'my_script.py'),
            os.path.join(app_workspace, 'input_1'),
            os.path.join(app_workspace, 'input_2')
        ),
        attributes=dict(
            executable='my_script.py',
            transfer_input_files=('../input_1', '../input_2'),
            transfer_output_files=('example_output1', 'example_output2'),
        ),
    )
    job_b.save()

    job_c = CondorWorkflowJobNode(
        name='JobC',
        workflow=workflow,
        condorpy_template_name='vanilla_transfer_files',
        remote_input_files=(
            os.path.join(app_workspace, 'my_script.py'),
            os.path.join(app_workspace, 'input_1'),
            os.path.join(app_workspace, 'input_2')
        ),
        attributes=dict(
            executable='my_script.py',
            transfer_input_files=('../input_1', '../input_2'),
            transfer_output_files=('example_output1', 'example_output2'),
        ),
    )
    job_c.save()

    job_b.add_parent(job_a)
    job_c.add_parent(job_b)

    workflow.save()
    # or
    workflow.execute()

Note

The CondorWorkflow object must be saved before the CondorWorkflowJobNode objects can be instantiated, and the CondorWorkflowJobNode objects must be saved before you can define the relationships.

Before a controller returns a response the job must be saved or else all of the changes made to the job will be lost (executing the job automatically saves it). If submitting the job takes a long time (e.g. if a large amount of data has to be uploaded to a remote scheduler) then it may be best to use AJAX to execute the job.

API Documentation

class tethys_compute.models.CondorWorkflow(*args, **kwargs)

CondorPy Workflow job type

class tethys_compute.models.CondorWorkflowNode(*args, **kwargs)

Base class for CondorWorkflow Nodes

Parameters
  • name (str) --

  • workflow (CondorWorkflow) -- instance of a CondorWorkflow that node belongs to

  • parent_nodes (list) -- list of CondorWorkflowNode objects that are prerequisites to this node

  • pre_script (str) --

  • pre_script_args (str) --

  • post_script (str) --

  • post_script_args (str) --

  • variables (dict) --

  • priority (int) --

  • category (str) --

  • retry (int) --

  • retry_unless_exit_value (int) --

  • pre_skip (int) --

  • abort_dag_on (int) --

  • dir (str) --

  • noop (bool) --

  • done (bool) --

For a description of the arguments see http://research.cs.wisc.edu/htcondor/manual/v8.6/2_10DAGMan_Applications.html

class tethys_compute.models.CondorWorkflowJobNode(*args, **kwargs)

CondorWorkflow JOB type node

class tethys_sdk.jobs.CondorWorkflowTemplate(name, parameters=None, jobs=None, max_jobs=None, config='', **kwargs)

DEPRECATED A subclass of the JobTemplate with the type argument set to CondorWorkflow.

Parameters
  • name (str) -- Name to refer to the template.

  • parameters (dict, DEPRECATED) -- A dictionary of key-value pairs. Each Job type defines the possible parameters.

  • jobs (list) -- A list of CondorWorkflowJobTemplates.

  • max_jobs (dict, optional) -- A dictionary of category-max_job pairs defining the maximum number of jobs that will run simultaneously from each category.

  • config (str, optional) -- A path to a configuration file for the condorpy DAG.

class tethys_sdk.jobs.CondorWorkflowJobTemplate(name, job_description, **kwargs)

DEPRECATED A subclass of the CondorWorkflowNodeBaseTemplate with the type argument set to CondorWorkflowJobNode.

Parameters
  • name (str) -- Name to refer to the template.

  • job_description (CondorJobDescription) -- An instance of CondorJobDescription containing of key-value pairs of job attributes.