Renku Workflow

Renku uses PROV-O and its own Renku ontology to represent workflows.

Plans

Represent run templates.

class renku.core.models.workflow.plan.AbstractPlan(*args, **kwargs)

Abstract base class for all plans.

assign_new_id()

Assign a new UUID.

This is required only when there is another plan which is exactly the same except the parameters’ list.

find_parameter(parameter)

Find if a parameter exists on this plan.

find_parameter_workflow(parameter)

Return the workflow a parameter belongs to.

static generate_id(uuid=None)

Generate an identifier for Plan.

resolve_direct_reference(reference)

Resolve a direct parameter reference.

resolve_mapping_path(mapping_path)

Resolve a mapping path to its reference parameter.

static validate_name(name)

Check a name for invalid characters.

class renku.core.models.workflow.plan.Plan(*args, **kwargs)

Represent a renku run execution template.

assign_new_id()

Assign a new UUID.

This is required only when there is another plan which is exactly the same except the parameters’ list.

copy()

Create a copy of this plan.

Required where a plan is used several times in a workflow but we need to set different values on them.

derive()

Create a new Plan that is derived from self.

find_parameter(parameter)

Find if a parameter exists on this plan.

find_parameter_workflow(parameter)

Return the workflow a parameter belongs to.

get_parameter_by_id(parameter_id)

Get a parameter on this plan by id.

get_parameter_path(parameter)

Get the path to a parameter inside this plan.

is_similar_to(other)

Return true if plan has the same inputs/outputs/arguments as another plan.

property keywords_csv

Comma-separated list of keywords associated with workflow.

resolve_direct_reference(reference)

Resolve a direct parameter reference.

resolve_mapping_path(mapping_path)

Resolve a mapping path to its reference parameter.

set_parameters_from_strings(params_strings)

Set parameters by parsing parameters strings.

to_argv(with_streams=False)

Convert a Plan into argv list.

class renku.core.models.workflow.plan.PlanDetailsJson(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

Serialize a plan to a response object.

class renku.core.models.workflow.plan.PlanSchema(*args, commit=None, client=None, **kwargs)

Plan schema.

Create an instance.

class Meta

Meta class.

model

alias of renku.core.models.workflow.plan.Plan

Represent a group of run templates.

class renku.core.models.workflow.composite_plan.CompositePlan(*args, **kwargs)

A plan containing child plans.

add_link(source, sinks)

Validate and add a ParameterLink.

add_mapping(mapping)

Add a mapping to this run.

derive()

Create a new CompositePlan that is derived from self.

find_link_by_target(target)

Find a link on this or a child workflow that has target as a sink.

find_parameter(parameter)

Check if a parameter exists on this run or one of its children.

find_parameter_workflow(parameter)

Return the workflow a parameter belongs to.

get_parameter_by_id(parameter_id)

Get a parameter on this plan by id.

get_parameter_path(parameter)

Get the path to a parameter inside this plan.

map_all_inputs()

Map all unmapped inputs from child steps to the parent.

map_all_outputs()

Map all unmapped outputs from child steps to the parent.

map_all_parameters()

Map all unmapped parameters from child steps to the parent.

resolve_direct_reference(reference)

Resolve a direct parameter reference.

resolve_mapping_path(mapping_path)

Resolve a mapping path to its reference parameter.

set_links_from_strings(link_strings)

Set links between parameters of child steps.

set_mapping_defaults(default_strings)

Set default value based on a default specification string.

set_mapping_descriptions(mapping_descriptions)

Set descriptions for mappings.

set_mappings_from_strings(mapping_strings)

Set mappings by parsing mapping strings.

class renku.core.models.workflow.composite_plan.CompositePlanSchema(*args, commit=None, client=None, **kwargs)

Plan schema.

Create an instance.

class Meta

Meta class.

model

alias of renku.core.models.workflow.composite_plan.CompositePlan

Parameters

Classes to represent inputs/outputs/parameters in a Plan.

class renku.core.models.workflow.parameter.CommandInput(*, default_value=None, description=None, id, mapped_to=None, name=None, position=None, prefix=None, encoding_format=None, derived_from=None, postfix=None)

An input to a command.

derive(plan_id)

Create a new CommandInput that is derived from self.

static generate_id(plan_id, position=None, postfix=None)

Generate an id for CommandInput.

to_stream_representation()

Input stream representation.

class renku.core.models.workflow.parameter.CommandInputSchema(*args, commit=None, client=None, **kwargs)

CommandInput schema.

Create an instance.

class Meta

Meta class.

model

alias of renku.core.models.workflow.parameter.CommandInput

class renku.core.models.workflow.parameter.CommandOutput(*, create_folder=False, default_value=None, description=None, id, mapped_to=None, name=None, position=None, prefix=None, encoding_format=None, derived_from=None, postfix=None)

An output from a command.

derive(plan_id)

Create a new CommandOutput that is derived from self.

static generate_id(plan_id, position=None, postfix=None)

Generate an id for CommandOutput.

to_stream_representation()

Input stream representation.

class renku.core.models.workflow.parameter.CommandOutputSchema(*args, commit=None, client=None, **kwargs)

CommandOutput schema.

Create an instance.

class Meta

Meta class.

model

alias of renku.core.models.workflow.parameter.CommandOutput

class renku.core.models.workflow.parameter.CommandParameter(*, default_value=None, description=None, id, name=None, position=None, prefix=None, derived_from=None, postfix=None)

An argument to a command that is neither input nor output.

derive(plan_id)

Create a new CommandParameter that is derived from self.

static generate_id(plan_id, position=None, postfix=None)

Generate an id for CommandParameter.

class renku.core.models.workflow.parameter.CommandParameterBase(*, default_value, description, id, name, position=None, prefix=None, derived_from=None, postfix=None)

Represents a parameter for a Plan.

property actual_value

Get the actual value to be used for execution.

property actual_value_set

Whether the actual_value on this parameter has been set at least once.

abstract derive(plan_id)

Create a new command parameter from self.

property role

Return a unique role for this parameter within its Plan.

to_argv()

String representation (sames as cmd argument).

class renku.core.models.workflow.parameter.CommandParameterBaseSchema(*args, commit=None, client=None, **kwargs)

CommandParameterBase schema.

Create an instance.

class Meta

Meta class.

model

alias of renku.core.models.workflow.parameter.CommandParameterBase

class renku.core.models.workflow.parameter.CommandParameterSchema(*args, commit=None, client=None, **kwargs)

CommandParameter schema.

Create an instance.

class Meta

Meta class.

model

alias of renku.core.models.workflow.parameter.CommandParameter

class renku.core.models.workflow.parameter.MappedIOStream(*, id=None, stream_type)

Represents an IO stream (stdin, stdout, stderr).

static generate_id(stream_type)

Generate an id for parameters.

class renku.core.models.workflow.parameter.MappedIOStreamSchema(*args, commit=None, client=None, **kwargs)

MappedIOStream schema.

Create an instance.

class Meta

Meta class.

model

alias of renku.core.models.workflow.parameter.MappedIOStream

class renku.core.models.workflow.parameter.ParameterLink(source, sinks, id)

A link between a source and one or more sink parameters.

apply()

Apply source value to sinks.

static generate_id(plan_id)

Generate an id for parameters.

class renku.core.models.workflow.parameter.ParameterLinkSchema(*args, commit=None, client=None, **kwargs)

ParameterLink schema.

Create an instance.

class Meta

Meta class.

model

alias of renku.core.models.workflow.parameter.ParameterLink

class renku.core.models.workflow.parameter.ParameterMapping(*, default_value=None, description=None, id, name=None, mapped_parameters=None, **kwargs)

A mapping of child parameter(s) to a parent CompositePlan.

property actual_value

Get the actual value to be used for execution.

static generate_id(plan_id, position=None, postfix=None)

Generate an id for CommandOutput.

property leaf_parameters

Return leaf (non-Mapping) parameters contained by this Mapping.

to_stream_representation()

Input stream representation.

class renku.core.models.workflow.parameter.ParameterMappingSchema(*args, commit=None, client=None, **kwargs)

ParameterMapping schema.

Create an instance.

class Meta

Meta class.

model

alias of renku.core.models.workflow.parameter.ParameterMapping

Renku Workflow Logic

Execution Graph

Build an execution graph for a workflow.

class renku.core.management.workflow.concrete_execution_graph.ExecutionGraph(workflows, virtual_links=False)

Represents an execution graph for one or more workflow steps.

calculate_concrete_execution_graph(virtual_links=False)

Create an execution DAG between Plans showing dependencies between them.

Resolve ParameterLink’s involving ParameterMapping’s to the underlying actual parameters and potentially also virtual links determined by parameter values.

property cycles

Get potential cycles in execution graph.

property workflow_graph

Return a subgraph with only workflows and their dependencies.

Value Resolution

Resolution of Workflow execution values precedence.

class renku.core.management.workflow.value_resolution.CompositePlanValueResolver(plan, values)

Value resolution class for a CompositePlan.

Applies values and default_values to a nested workflow.

Order of precedence is as follows (from lowest to highest): - Default value on a parameter - Default value on a mapping to the parameter - Value passed to a mapping to the parameter - Value passed to the parameter - Value propagated to a parameter from the source of a ParameterLink

apply()

Applies values and default_values to a CompositePlan.

class renku.core.management.workflow.value_resolution.PlanValueResolver(plan, values)

Value resolution class for a Plan.

Applies values and default_values to a workflow.

apply()

Applies values and default_values to a Plan.

class renku.core.management.workflow.value_resolution.ValueResolver(plan, values)

Value resolution class for an AbstractPlan.

abstract apply()

Applies values and default_values to a potentially nested workflow.

Returns

The AbstractPlan with the user provided values set.

static get(plan, values)

Factory method to obtain the specific ValueResolver for a workflow.

Parameters

• plan – a workflow.

• values – user defined dictionary of runtime values for the provided workflow.

Returns

A ValueResolver object

renku.core.management.workflow.value_resolution.apply_parameter_links(workflow)

Apply values from parameter links.

Plan Factory

Used to create Plan objects based on command line arguments

Represent a PlanFactory for tracking workflows.

class renku.core.management.workflow.plan_factory.PlanFactory(command_line, explicit_inputs=None, explicit_outputs=None, directory=None, working_dir=None, no_input_detection=False, no_output_detection=False, success_codes=None, stdin=None, stdout=None, stderr=None)

Factory for creating a plan from a command line call.

add_command_input(default_value, prefix=None, position=None, postfix=None, encoding_format=None)

Create a CommandInput.

add_command_output(default_value, prefix=None, position=None, postfix=None, encoding_format=None)

Create a CommandOutput.

add_command_output_from_input(input)

Create a CommandOutput from an input.

add_command_output_from_parameter(parameter)

Create a CommandOutput from a parameter.

add_command_parameter(default_value, prefix=None, position=None, name=None)

Create a CommandParameter.

add_explicit_inputs()

Add explicit inputs .

add_indirect_inputs()

Read indirect inputs list and add them to explicit inputs.

add_indirect_outputs()

Read indirect outputs list and add them to explicit outputs.

add_inputs_and_parameters(*arguments)

Yield command input parameters.

add_outputs(candidates)

Yield detected output and changed command input parameter.

get_stream_mapping_for_value(value)

Return a stream mapping if value is a path mapped to a stream.

guess_type(value, ignore_filenames=None)

Return new value and CWL parameter type.

iter_input_files(basedir)

Yield tuples with input id and path.

split_command_and_args()

Return tuple with command and args from command line arguments.

to_plan(project_gateway, name=None, description=<class 'NoneType'>, keywords=None)

Return an instance of Plan based on this factory.

watch(client_dispatcher, no_output=False)

Watch a Renku repository for changes to detect outputs.

renku.core.management.workflow.plan_factory.add_indirect_parameter(working_dir, name, value)

Add a parameter to indirect parameters.

renku.core.management.workflow.plan_factory.delete_indirect_files_list(working_dir)

Remove indirect inputs, outputs, and parameters list.

renku.core.management.workflow.plan_factory.get_indirect_inputs_path(client_path)

Return path to file that contains indirect inputs list.

renku.core.management.workflow.plan_factory.get_indirect_outputs_path(client_path)

Return path to file that contains indirect outputs list.

renku.core.management.workflow.plan_factory.get_indirect_parameters_path(client_path)

Return path to file that contains indirect parameters list.

renku.core.management.workflow.plan_factory.read_indirect_parameters(working_dir)

Read and return indirect parameters.

Renku Provenance Logic

Use for building a direct acyclic graph of past executions.

Renku workflow commands.

renku.core.management.workflow.activity.create_activity_graph(activities, remove_overridden_parents=True, with_inputs_outputs=False)

Create a dependency DAG from activities.

renku.core.management.workflow.activity.get_activities_until_paths(paths, sources, activity_gateway, client_dispatcher, revision=None)

Get all current activities leading to paths, from sources.

renku.core.management.workflow.activity.sort_activities(activities, remove_overridden_parents=True)

Returns a sorted list of activities based on their dependencies and execution order.

Renku Workflow Conversion

Renku allows conversion of tracked workflows to runnable workflows in supported tools (Currently CWL)

CWL

Converter for workflows to CWL.

class renku.core.management.workflow.converters.cwl.CWLExporter

Converts a CompositePlan or a Plan to CWL format.

workflow_convert(workflow, basedir, output, output_format)

Converts the specified workflow to CWL format.

workflow_format()

Workflow format name.