Renku Workflow

Renku uses PROV-O and its own Renku ontology to represent workflows.

plan

Represent run templates.

class renku.domain_model.workflow.plan.AbstractPlan(*args, **kwargs)

Bases: Persistent, ABC

Abstract base class for all plans.

assign_new_id(*, uuid=None)

Assign a new UUID.

This is required only when there is another plan which is exactly the same except the parameters’ list.

delete(when=None)

Mark a plan as deleted.

NOTE: Don’t call this function for deleting plans since it doesn’t delete the whole plan derivatives chain. Use renku.core.workflow.plan::remove_plan instead.

property deleted

True if plan is deleted.

derive(creator=None)

Create a new AbstractPlan that is derived from self.

find_parameter(parameter)

Find if a parameter exists on this plan.

find_parameter_workflow(parameter)

Return the workflow a parameter belongs to.

static generate_id(*, uuid=None, **_)

Generate an identifier for Plan.

is_derivation()

Return if an AbstractPlan has correct derived_from.

resolve_direct_reference(reference)

Resolve a direct parameter reference.

resolve_mapping_path(mapping_path)

Resolve a mapping path to its reference parameter.

static validate_name(name)

Check a name for invalid characters.

class renku.domain_model.workflow.plan.Plan(*args, **kwargs)

Bases: AbstractPlan

Represent a renku run execution template.

assign_new_id(*, uuid=None)

Assign a new UUID.

This is required only when there is another plan which is exactly the same except the parameters’ list.

copy()

Create a copy of this plan.

Required where a plan is used several times in a workflow, but we need to set different values on them.

derive(creator=None)

Create a new Plan that is derived from self.

find_parameter(parameter)

Find if a parameter exists on this plan.

find_parameter_workflow(parameter)

Return the workflow a parameter belongs to.

get_field_by_id(id)

Return an in Input/Output/Parameter by its id.

get_parameter_by_id(parameter_id)

Get a parameter on this plan by id.

get_parameter_path(parameter)

Get the path to a parameter inside this plan.

hidden_inputs = []

Includes a list of dependencies that are defined by Renku and should be hidden from users.

is_derivation()

Return if an Plan has correct derived_from.

is_equal_to(other)

Return true if plan hasn’t changed from the other plan.

property keywords_csv

Comma-separated list of keywords associated with workflow.

resolve_direct_reference(reference)

Resolve a direct parameter reference.

resolve_mapping_path(mapping_path)

Resolve a mapping path to its reference parameter.

set_parameters_from_strings(params_strings)

Set parameters by parsing parameters strings.

to_argv(with_streams=False, quote_string=True)

Convert a Plan into argv list.

class renku.domain_model.workflow.plan.PlanDetailsJson(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

Bases: Schema

Serialize a plan to a response object.

renku.domain_model.workflow.plan.get_duplicate_arguments_names(plan)

Return a list of duplicate inputs/outputs/parameters names in a plan.

renku.domain_model.workflow.plan.validate_plan_name(name, extra_valid_characters='_-')

Check a name for invalid characters.

composite_plan

Represent a group of run templates.

class renku.domain_model.workflow.composite_plan.CompositePlan(*args, **kwargs)

Bases: AbstractPlan

A plan containing child plans.

add_link(source, sinks)

Validate and add a ParameterLink.

add_mapping(mapping)

Add a mapping to this run.

derive(creator=None)

Create a new CompositePlan that is derived from self.

find_link_by_target(target)

Find a link on this or a child workflow that has target as a sink.

find_parameter(parameter)

Check if a parameter exists on this run or one of its children.

find_parameter_workflow(parameter)

Return the workflow a parameter belongs to.

get_parameter_by_id(parameter_id)

Get a parameter on this plan by id.

get_parameter_path(parameter)

Get the path to a parameter inside this plan.

is_derivation()

Return if an CompositePlan has correct derived_from.

is_equal_to(other)

Return true if plan hasn’t changed from the other plan.

map_all_inputs()

Map all unmapped inputs from child steps to the parent.

map_all_outputs()

Map all unmapped outputs from child steps to the parent.

map_all_parameters()

Map all unmapped parameters from child steps to the parent.

resolve_direct_reference(reference)

Resolve a direct parameter reference.

resolve_mapping_path(mapping_path)

Resolve a mapping path to its reference parameter.

set_links_from_strings(link_strings)

Set links between parameters of child steps.

set_mapping_defaults(default_strings)

Set default value based on a default specification string.

set_mapping_descriptions(mapping_descriptions)

Set descriptions for mappings.

set_mappings_from_strings(mapping_strings)

Set mappings by parsing mapping strings.

parameter

Classes to represent inputs/outputs/parameters in a Plan.

class renku.domain_model.workflow.parameter.CommandInput(*, default_value=None, description=None, id, mapped_to=None, name=None, name_set_by_user=False, position=None, prefix=None, encoding_format=None, derived_from=None, postfix=None)

Bases: CommandParameterBase

An input to a command.

derive(plan_id)

Create a new CommandInput that is derived from self.

static generate_id(plan_id, position=None, postfix=None, name=None)

Generate an id for CommandInput.

is_equal_to(other)

Return if attributes that cause a change in the parameter, are the same.

to_stream_representation()

Input stream representation.

update_from(other)

Update this input with values from another input, if applicable.

class renku.domain_model.workflow.parameter.CommandOutput(*, create_folder=False, default_value=None, description=None, id, mapped_to=None, name=None, name_set_by_user=False, position=None, prefix=None, encoding_format=None, derived_from=None, postfix=None)

Bases: CommandParameterBase

An output from a command.

derive(plan_id)

Create a new CommandOutput that is derived from self.

static generate_id(plan_id, position=None, postfix=None, name=None)

Generate an id for CommandOutput.

is_equal_to(other)

Return if attributes that cause a change in the parameter, are the same.

to_stream_representation()

Input stream representation.

update_from(other)

Update this output with values from another output, if applicable.

class renku.domain_model.workflow.parameter.CommandParameter(*, default_value=None, description=None, id, name=None, name_set_by_user=False, position=None, prefix=None, derived_from=None, postfix=None)

Bases: CommandParameterBase

An argument to a command that is neither input nor output.

derive(plan_id)

Create a new CommandParameter that is derived from self.

static generate_id(plan_id, position=None, postfix=None, name=None)

Generate an id for CommandParameter.

update_from(other)

Update this output with values from another output, if applicable.

class renku.domain_model.workflow.parameter.CommandParameterBase(*, default_value, description, id, name, name_set_by_user=False, position=None, prefix=None, derived_from=None, postfix=None)

Bases: object

Represents a parameter for a Plan.

property actual_value

Get the actual value to be used for execution.

property actual_value_set

Whether the actual_value on this parameter has been set at least once.

abstract derive(plan_id)

Create a new command parameter from self.

is_equal_to(other)

Return if attributes that cause a change in the parameter, are the same.

property role

Return a unique role for this parameter within its Plan.

to_argv(quote_string=True)

String representation (sames as cmd argument).

class renku.domain_model.workflow.parameter.HiddenInput(*, default_value=None, description=None, id, mapped_to=None, name=None, name_set_by_user=False, position=None, prefix=None, encoding_format=None, derived_from=None, postfix=None)

Bases: CommandInput

An input to a command that is added by Renku and should be hidden from users.

class renku.domain_model.workflow.parameter.MappedIOStream(*, id=None, stream_type)

Bases: object

Represents an IO stream (stdin, stdout, stderr).

classmethod from_str(stream_type)

Create an instance from the given stream type string.

static generate_id(stream_type)

Generate an id for parameters.

class renku.domain_model.workflow.parameter.ParameterLink(source, sinks, id)

Bases: object

A link between a source and one or more sink parameters.

apply()

Apply source value to sinks.

static generate_id(plan_id)

Generate an id for parameters.

class renku.domain_model.workflow.parameter.ParameterMapping(*, default_value, description=None, id, name=None, name_set_by_user=False, mapped_parameters, **kwargs)

Bases: CommandParameterBase

A mapping of child parameter(s) to a parent CompositePlan.

property actual_value

Get the actual value to be used for execution.

derive(plan_id)

Create a new CommandParameter that is derived from self.

static generate_id(plan_id, position=None, postfix=None, name=None)

Generate an id for ParameterMapping.

property leaf_parameters

Return leaf (non-Mapping) parameters contained by this Mapping.

to_stream_representation()

Input stream representation.

renku.domain_model.workflow.parameter.generate_parameter_name(parameter, kind)

Generate a name for an input, output, or parameter.

workflow_file

Models to represent a workflow file run templates.

class renku.domain_model.workflow.workflow_file.WorkflowFileCompositePlan(*args, **kwargs)

Bases: CompositePlan

A workflow file composite plan.

assign_new_id(*, sequence=None, **_)

Assign a new UUID or a deterministic.

static generate_id(path=None, sequence=None, uuid_only=False, **_)

Generate an identifier for Plan.

is_equal_to(other)

Return true if plan hasn’t changed from the other plan.

class renku.domain_model.workflow.workflow_file.WorkflowFilePlan(*args, **kwargs)

Bases: Plan

Represent a Plan that is converted from a workflow file.

assign_new_id(*, sequence=None, **_)

Assign a new UUID or a deterministic.

static generate_id(path=None, name=None, sequence=None, uuid_only=False, **_)

Generate an identifier for Plan.

is_equal_to(other)

Return true if plan hasn’t changed from the other plan.

set_parameters_from_strings(params_strings)

Set parameters by parsing parameters strings.

property unqualified_name

Name of the plan as appears in the workflow file definition.

static validate_name(name)

Check a name for invalid characters.

Renku Workflow Conversion

Renku allows conversion of tracked workflows to runnable workflows in supported tools (Currently CWL)

CWL

Converter for workflows to CWL.

class renku.core.workflow.converters.cwl.CWLExporter

Converts a CompositePlan or a Plan to CWL format.

workflow_convert(workflow, basedir, output, output_format, resolve_paths, nest_workflows)

Converts the specified workflow to CWL format.

Parameters:

• worflow (Union[CompositePlan, Plan]) – The plan or composite plan to be converted to cwl.

• basedir (Path) – The path of the base location used as a prefix for all workflow input and outputs.

• output (Optional[Path]) – The file where the CWL specification should be saved, if None then no file is created.

• output_format (Optional[str]) – Not used. Only YAML is generated, regardless of what is provided.

• resolve_paths (Optional[bool]) – Whether to make all paths absolute and resolve all symlinks, True by default.

• nest_workflows (Optional[bool]) – Whether nested CWL workflows should be used or each sub-workflow should be a separate file, False by default.

Returns:

The contents of the CWL workflow as string. If nested workflows are used then only the parent specification is returned.

workflow_format()

Workflow format name.