Running Renku on HPC

Renku CLI supports various backends for executing workflows. Currently, there are two different providers are implemented, namely cwltool and toil. Implementing a workflow provider documents gives a more detailed description of how to implement your own workflow provider.

By default all workflows are executed by the cwltool provider, that exports the workflow to CWL and then uses cwltool to execute the given CWL.

The workflow backend can be changed by using the -p/--provider <PROVIDER> command line option. A backend’s default configuration can be overridden by providing the -c/--config <config.yaml> command line parameter. The following renku commands support the above mentioned workflow provider related command line options:

For example, to execute a previously created my_plan workflow with toil, one simply would run the following command:

$ renku execute -p toil my_plan

Using toil as a workflow provider has the advantage that it supports running the workflows on various high-performance computing and cloud platforms.

Renku on Slurm

Slurm is a highly configurable open-source workload manager, which is in widespread use at government laboratories, universities and companies world wide and performs workload management for over half of the top 10 systems in the TOP500.

As toil supports Slurm, one can easily execute the previously created renku workflows on Slurm. One just needs to provide a simple configuration file to the provider (--config):

batchSystem: slurm
disableCaching: true

The disableCaching is necessary to be enabled for Slurm, for more details see the related toil issue TOIL-1006.

Additional Slurm specific parameters can be provided with the TOIL_SLURM_ARGS environment variable.

Taking the example above, the following command line will execute my_plan on Slurm:

$ TOIL_SLURM_ARGS="-A my_account --export=ALL" renku execute -p toil -c provider.yaml my_plan

where

  • provider.yaml file contains the above mentioned two parameters for toil,

  • -A my_account specifies which account should be charged for the used resources,

  • --export=ALL specifies that all environment variables are propagated to the Slurm workers. It is often required in academic Slurm installations.