Repository API¶
This API is built on top of Git and Git-LFS.
Renku repository management.
-
class
renku.core.management.
LocalClient
(path=<function default_path>, renku_home='.renku', parent=None, commit_activity_cache=NOTHING, activity_index=None, remote_cache=NOTHING, migration_type=<MigrationType.ALL: 7>, external_storage_requested=True, *, data_dir='data')[source]¶ A low-level client for communicating with a local Renku repository.
Method generated by attrs for class LocalClient.
Datasets¶
Client for handling datasets.
-
class
renku.core.management.datasets.
DatasetsApiMixin
[source]¶ Client for handling datasets.
Method generated by attrs for class DatasetsApiMixin.
-
CACHE
= 'cache'¶ Directory to cache transient data.
-
DATASETS
= 'datasets'¶ Directory for storing dataset metadata in Renku.
-
DATASETS_PROVENANCE
= 'dataset.json'¶ File for storing datasets’ provenance.
-
DATASET_IMAGES
= 'dataset_images'¶ Directory for dataset images.
-
POINTERS
= 'pointers'¶ Directory for storing external pointer files.
-
add_data_to_dataset
(dataset, urls, force=False, overwrite=False, sources=(), destination='', ref=None, external=False, extract=False, all_at_once=False, destination_names=None)[source]¶ Import the data into the data directory.
-
add_dataset_tag
(dataset, tag, description='', force=False)[source]¶ Adds a new tag to a dataset.
Validates if the tag already exists and that the tag follows the same rules as docker tags. See https://docs.docker.com/engine/reference/commandline/tag/ for a documentation of docker tag syntax.
Raises: errors.ParameterError
-
create_dataset
(name=None, title=None, description=None, creators=None, keywords=None, images=None, safe_image_paths=[])[source]¶ Create a dataset.
-
dataset_commits
(dataset, max_results=None)[source]¶ Gets the newest commit for a dataset or its files.
Commits are returned sorted from newest to oldest.
-
datasets
¶ Return mapping from path to dataset.
-
datasets_provenance
¶ Return dataset provenance if available.
-
datasets_provenance_path
¶ Path to store activity files.
-
load_dataset_from_provenance
(name, strict=False)[source]¶ Load latest dataset’s metadata from dataset provenance file.
Removes tags from a dataset.
-
static
remove_file
(filepath)[source]¶ Remove a file/symlink and its pointer file (for external files).
-
renku_dataset_images_path
¶ Return a
Path
instance of Renku dataset metadata folder.
-
renku_datasets_path
¶ Return a
Path
instance of Renku dataset metadata folder.
-
renku_pointers_path
¶ Return a
Path
instance of Renku pointer files folder.
-
update_dataset_git_files
(files, ref, delete=False)[source]¶ Update files and dataset metadata according to their remotes.
Parameters: - files – List of files to be updated
- delete – Indicates whether to delete files or not
Returns: List of files that should be deleted
-
update_datasets_provenance
(dataset, remove=False)[source]¶ Update datasets provenance for a dataset.
-
Repository¶
Client for handling a local repository.
-
class
renku.core.management.repository.
PathMixin
(path=<function default_path>)[source]¶ Define a default path attribute.
Method generated by attrs for class PathMixin.
-
class
renku.core.management.repository.
RepositoryApiMixin
(renku_home='.renku', parent=None, commit_activity_cache=NOTHING, activity_index=None, remote_cache=NOTHING, migration_type=<MigrationType.ALL: 7>, *, data_dir='data')[source]¶ Client for handling a local repository.
Method generated by attrs for class RepositoryApiMixin.
-
ACTIVITY_INDEX
= 'activity_index.yaml'¶ Caches activities that generated a path.
-
DEPENDENCY_GRAPH
= 'dependency.json'¶ File for storing dependency graph.
-
DOCKERFILE
= 'Dockerfile'¶ Name of the Dockerfile in the repo.
-
LOCK_SUFFIX
= '.lock'¶ Default suffix for Renku lock file.
-
METADATA
= 'metadata.yml'¶ Default name of Renku config file.
-
PROVENANCE_GRAPH
= 'provenance.json'¶ File for storing ProvenanceGraph.
-
WORKFLOW
= 'workflow'¶ Directory for storing workflow in Renku.
-
activities_for_paths
(paths, file_commit=None, revision='HEAD')[source]¶ Get all activities involving a path.
-
activity_index_path
¶ Path to the activity filepath cache.
-
data_dir
= None¶ Define a name of the folder for storing datasets.
-
dependency_graph
¶ Return dependency graph if available.
-
dependency_graph_path
¶ Path to the dependency graph file.
-
docker_path
¶ Path to the Dockerfile.
-
find_previous_commit
(paths, revision='HEAD', return_first=False, full=False)[source]¶ Return a previous commit for a given path starting from
revision
.Parameters: - revision – revision to start from, defaults to
HEAD
- return_first – show the first commit in the history
- full – return full history
Raises: KeyError – if path is not present in the given commit
- revision – revision to start from, defaults to
-
import_from_template
(template_path, metadata, force=False)[source]¶ Render template files from a template directory.
-
latest_agent
¶ Returns latest agent version used in the repository.
-
lock
¶ Create a Renku config lock.
-
migration_type
¶ Type of migration that is being executed on this client.
-
parent
= None¶ Store a pointer to the parent repository.
-
path_activity_cache
¶ Cache of all activities and their generated paths.
-
process_and_store_run
(command_line_tool, name, client)[source]¶ Create Plan and Activity from CommandLineTool and store them.
-
process_commit
(commit=None, path=None)[source]¶ Build an
Activity
.Parameters: - commit – Commit to process. (default:
HEAD
) - path – Process a specific CWL file.
- commit – Commit to process. (default:
-
project
¶ Return the Project instance.
-
provenance_graph_path
¶ Path to store activity files.
-
remote
¶ Return host, owner and name of the remote if it exists.
-
renku_home
= None¶ Define a name of the Renku folder (default:
.renku
).
-
renku_metadata_path
¶ Return a
Path
instance of Renku metadata file.
-
renku_path
= None¶ Store a
Path
instance of the Renku folder.
-
template_checksums
¶ Return a
Path
instance to the template checksums file.
-
update_graphs
(activity_run)[source]¶ Update Dependency and Provenance graphs from a ProcessRun/WorkflowRun.
-
workflow_path
¶ Return a
Path
instance of the workflow folder.
-
Git Internals¶
Wrap Git client.
-
class
renku.core.management.git.
GitCore
[source]¶ Wrap Git client.
Method generated by attrs for class GitCore.
-
candidate_paths
¶ Return all paths in the index and untracked files.
-
commit
(commit_only=None, commit_empty=True, raise_if_empty=False, commit_message=None, abbreviate_message=True)[source]¶ Automatic commit.
-
dirty_paths
¶ Get paths of dirty files in the repository.
-
modified_paths
¶ Return paths of modified files.
-
repo
= None¶ Store an instance of the Git repository.
-
-
renku.core.management.git.
get_mapped_std_streams
(lookup_paths, streams=('stdin', 'stdout', 'stderr'))[source]¶ Get a mapping of standard streams to given paths.
Git utilities.
-
class
renku.core.models.git.
GitURL
(href, pathname=None, protocol='ssh', hostname='localhost', username=None, password=None, port=None, owner=None, name=None, regex=None)[source]¶ Parser for common Git URLs.
Method generated by attrs for class GitURL.
-
image
¶ Return image name.
-
-
class
renku.core.models.git.
Range
(start, stop)[source]¶ Represent parsed Git revision as an interval.
Method generated by attrs for class Range.