Compi: a framework for portable and reproducible pipelines

XML for pipeline definition

Compi pipelines are defined in a single XML (eXtensible Markup Language) file that includes: (i) tasks and dependencies between them, (ii) pipeline input parameters that are forwarded to tasks, and (iii) metadata, including parameters and tasks descriptions, which are useful for automatic generation of the user interface, as well as for pipeline documentation.

We have chosen XML instead of JSON, YAML, or a custom DSL (Domain Specific Language) to reconcile the various requirements simultaneously. First, we wanted a high degree of interoperability. XML, JSON, and YAML can be easily generated and parsed in almost any programming language, although YAML can pose portability issues in some cases as YAML parsers in different languages can produce different results. In contrast, a DSL (Di Tommaso et al., 2017) is less interoperable, being difficult to produce or consume from languages other than the one on which the DSL is based. Second, XML is appropriate for dealing with long chunks of text, such as the embedded source code for pipeline tasks. Since Compi is language agnostic, these tasks can be defined in any programming language. Thanks to the XML CDATA (character data) blocks, which allow declaring a section of the XML that should not be parsed as XML, it is possible to include source code without any alteration. Embedding task code in JSON is virtually not feasible, as tabs and line breaks, which are key characters in languages like Python, must be escaped in JSON files. In YAML, however, it is easier thanks to its multiline literals. In DSLs it depends on whether the programming language the DSL is based on allows you to declare multiline strings easily, as Python does. Third, XML is easy to validate syntactically and semantically through schemas, which are also present in JSON and YAML. DSLs take advantage of the language parser the DSL is based on. Fourth, for security reasons, since Compi is designed to run pipelines defined by third-party programmers, YAML is a less secure language in this regard, as runnable code could be embedded in fields that were not intended for this purpose (see https://www.arp242.net/yaml-config.html). Fifth comes readability. YAML is the clearer winner in this regard, as readability is the key feature of this format. Although XML is less readable than YAML, the comparison with JSON and DSLs is a more subjective matter. Based on these five requirements, the choice was between XML and YAML: while XML is more secure and portable, YAML is more readable. Finally, we selected XML, prioritizing its security, portability and popularity. Also, while writing XML files could be verbose, YAML is syntactically aware of whitespace, which is a welcome feature in the Python community, but still debated outside of it.

The workflow execution engine

The workflow execution engine of Compi is implemented purely in Java and is responsible for multi-thread task scheduling, monitoring, and standard error and output logging of tasks. First, it computes the DAG (Direct Acyclic Graph) of the task dependencies, since a task may depend on a set of tasks that must be run before the given task. Right after starting, or whenever a task finishes its execution, the engine reacts and all the tasks that can be run, i.e., the tasks whose dependencies are now complete, are sent to a worker thread pool that has a parameterizable size. When there are no more tasks to run, the pipeline execution ends.

Every time a task is about to run and there is a free thread in the pool, a new subprocess is spawned by invoking the system Bash interpreter to execute the task script. Pipeline parameters are passed to the task script through environment variables, which is a robust, standard mechanism with two main benefits. On the one hand, environment variables are easily accessed via “$variable_name” or more complex expressions, so pipeline parameters are available directly within the task script. On the other hand, this allows Compi to pass parameters to scripts written in languages other than Bash, because virtually all programming languages give access to environment variables.

The way to execute languages differently from Bash is through task interpreters. Any task can have a task interpreter defined in the pipeline specification, which is an intermediate, user-defined bash script intended to take the task script as input and call an interpreter from a different programming language.

Moreover, it is possible to define task runners, which are the same concept as interpreters, but with a different purpose. They are not defined within the pipeline specification, but rather at execution time, and are intended to tailor task execution to specific computing resources, without modifying the pipeline itself. In this way, the workflow definition is decoupled from the workflow execution, making it possible to change the way tasks are executed without modifying the workflow XML file. For instance, if the tasks must be run in a cluster environment such as SGE or Slurm, computations must be initiated via a submission command (qsub in SGE or srun in Slurm). Task runners intercept task execution by changing the default Bash interpreter with queue submissions.

Compi project architecture

Compi comprises three main modules. The most important is the core module, which contains the workflow execution engine, with its main data structures. On top of it, there are two additional modules. On the one hand, the cli (command-line interface) module contains the command-line user interface for running pipelines, which generates a specific pipeline application tailored for pipeline tasks and parameters. On the other hand, the dk (Development Kit) module allows to create a portable application in the form of a Docker image and publishing the pipelines at Compi Hub.

Compi Hub

As discussed above, Compi Hub is a public repository where Compi pipelines can be published. The Compi Hub front-end was implemented using the Angular v7 web application framework, while the back-end was implemented using TypeScript and offers a RESTful API that supports all the functionality of the front-end. This REST API is also used by the compi-dk tool to allow pipeline developers to publish their pipelines from the command line. The Compi Hub back-end runs in a Node.js server and uses a MongoDB database to store the data.

Pipeline design

As explained previously, Compi pipelines are defined in an XML document that includes user parameters, tasks, and metadata. Figure 2 shows an example of a minimal pipeline. This sample pipeline defines two parameters (“name” and “output”), described in the corresponding “params” section, and two tasks (“greetings” and “bye”) described in the “tasks” section. The parameter descriptions are used to automatically generate both the user interface and the documentation for the pipeline. In the same way and for the same purpose, tasks are described in the “metadata” section, which allows the pipeline developer to describe them in a human-readable manner.

Tasks are defined as “task” elements within the “tasks” section. The main components of a task are: source code, parameters and dependencies. The source code is placed inside a “task” element, and the parameters used by a task are defined within the “params” attribute, and those tasks that the current task depends on are defined in the “after” attribute. For instance, in Fig. 2, the “bye” task, uses the “name” and “output” parameters, and depends on the “greetings” task. Parameter values are passed to the tasks as environment variables, which is a standard method that any programming language gives access to. In this way, the Compi workflow execution engine does not need to process the task code in any way (e.g., to perform variable substitution), guaranteeing the possibility of using any programming language to define the code of the task. In this sense, the task code is written in Bash by default, although other scripting languages (e.g., Python, R, AWK, etc.) can be used under a suitable interpreter through an “interpreter” parameter. For example, in Fig. 2, the “bye” task is written in Perl and the “interpreter” parameter indicates how to invoke the Perl interpreter to run the task code.

A special type of tasks are parallel iterative tasks (or loop tasks), defined via “foreach” elements, which spawn multiple parallel processes with the same code on a collection of items. This collection of items is provided by a user-specified source of items, which could be a comma-separated list of values, a range of numbers, the files in a specific directory, a parameter whose value is a list of values separated by commas, or even a custom command whose output lines are taken as items. Each item is available to the task code as an environment variable.

When all spawned tasks have finished, the foreach task ends as well and subsequent dependent tasks can be run. Nevertheless, there are several scenarios in which a pipeline developer can define multiple consecutive foreach tasks intended to iterate over the same collection of independent items. In these scenarios, when one iteration of a loop has finished, the corresponding iteration of the next loop could start without waiting for the entire previous loop to finish. Compi supports this type of interaction between foreach tasks by simply adding the ‘*’ prefix to the task name when declaring the dependency on the “after” attribute. For example, in a scenario where a pipeline must process a set of samples (e.g: “case-1”, “case-2”, “control-1”, “control-2”) by performing two consecutive operations: preprocess and analyze, the dependency of the “preprocess” task on the “analyze” task can be prefixed with ‘*’ (i.e., after =”*preprocess”) to tell Compi that iterations over the “analyze” loop could start when the corresponding iterations over the “preprocess” loop have finished.

Figure 3 shows an example of this, where two foreach tasks “preprocess” and “analyze” iterate over the same set of items (“samples”). The “analyze” task depends on the “preprocess” task, but at an iteration-level (after =”*preprocess”). Without the ‘*’ prefix, the “analyze” task will only start when the whole “preprocess” task has finished.

Pipeline execution and dependency management

One of the main features of Compi is the creation of a classic CLI for the entire pipeline based on its parameter specifications to aid users run the pipeline. This CLI is displayed when a user executes “compi run -p pipeline.xml –help”, which describes the Compi execution parameters and specification of each pipeline task. File S1 shows this CLI for the RNA-Seq Compi pipeline (https://www.sing-group.org/compihub/explore/5d09fb2a1713f3002fde86e2).

As Fig. 4 illustrates, the execution of Compi pipelines can be controlled using multiple parameters that fall in three main categories: pipeline inputs (i.e., the pipeline definition and its input parameters), logging, and execution control. In Fig. 4A, the “compi run” command receives the pipeline definition file explicitly, while the “–params” option indicates that the input parameters must be read from the “compi.params” file. On the other hand, in Fig. 4B the pipeline definition file is omitted and compi assumes that the pipeline definition must be read from a file named “pipeline.xml“ located in the current working directory. Also, pipeline parameters are passed on the command line after all the “compi run” parameters and separated by the ‘–’ delimiter.

Regarding logging options, both include the “–logs” option to specify a directory to save the standard (stdout) and error (stderr) outputs of each task along with the specific parameter values used in each execution. They are saved in three different files named with the name of the corresponding task as prefix (e.g., “task-name.out.log“,“task-name.err.log“, and “task-name.params“). To avoid unnecessary file creation, Compi does not save task outputs unless the “–logs” option is used. However, it is possible to specify which tasks should be logged using the “–log-only-task” or “–no-log-task” parameters. In addition to the task specific logs, Compi displays its own log messages during pipeline execution. These messages can be disabled by including “–quiet“ as shown in Fig. 4B. In contrast, in the example given in Fig. 4A, the “–show-std-outs” option forces Compi to forward each task log to the corresponding Compi output, which is very useful for debugging purposes during pipeline development.

The third group of options allows to control the execution of the pipeline. For instance, the “–num-tasks” parameter used in Fig. 4A sets the maximum number of tasks that can be run in parallel. It is important to note that this is not necessarily equivalent to the number of threads the pipeline will use, as some tasks may use parallel processes themselves. The “–abort-if-warnings” option, also used in the example in Fig. 4A, tells Compi to abort the pipeline execution if there are warnings in the pipeline validation. This is a useful and recommended option for pipeline testing during development, to avoid undesired effects that may arise from ignoring such warnings. A typical scenario that causes a warning is when the name of a pipeline parameter is found inside the task code, but the task does not have access to it, because it is not a global parameter nor defined in the set of task parameters.

One of the most notable features of the Compi workflow execution engine is that it allows a fine-grained control over the execution of the pipeline tasks. While many workflow engines only allow launching the entire pipeline or partially relaunching a pipeline from a point of failure, which can also be done in Compi using the “resume” command, Compi also allows launching sub-pipelines using modifiers such as “–from”, “–after“, “–until“ or “–before“. In the example shown in Fig. 4B, a combination of the “–from“ and “–until“ modifiers is used, resulting in the execution of all tasks in the path between “task-1” and “task-10”, including “task-1” and “task-10”. If the “–after“ and “–before“ are used instead, then “task-1” and “task-10” are not executed. A fifth modifier is “–single-task“, which allows only the specified task to execute and is not compatible with the other four modifiers. The Compi documentation (https://www.sing-group.org/compi/docs/running_pipelines.html#examples) includes several examples to illustrate how each of these options work.

Compi runs each task as a local command by default (Fig. 5A). This means that if a task invokes a certain tool (e.g., a ClustalOmega alignment running “clustalo -i /path/to/input.fasta -o /path/to/output.fasta”), this tool must be available either from the path environment variable or by including the absolute path to the binary executable. Since dependency management is always cumbersome, a special effort has been made to offer developers and users various alternatives to deal with this problem, which are explained below.

One way to address this issue is by means of a file with a custom XML runner definition, as done in the example shown in Fig. 4A with “–runners-config pipeline-runners.xml”. Individual runners are defined through a “runner“ element within a runners file, where the “task“ attribute is used to specify the list of tasks that the runner must execute. In this way, when a task identifier is assigned to a runner, Compi will ask the runner to run the corresponding task code (instead of running it as a local command). The usage of pipeline runners to handle dependencies allows Docker images to take responsibility for them (Fig. 5D). For instance, Fig. 6A shows a pipeline task named “align” that uses a tool (defined by the pipeline parameter “clustalomega”) that receives a file as input and produces an output. The runner defined in Fig. 6B for the same task runs the specified task code (available in the environment variable “task_code”) using a Docker image. The runner here is almost a generic Docker runner, and the key points are:

• First, the creation of a variable (“$envs”) with the list of parameters that must be passed as environment variables to the Docker container.

• Second, run the Docker image with the list of environment variables and mount the directory where the command has the input and output files (“workingDir” in this example).

Such a Docker runner would allow to follow an image-per-task execution pattern, where each task is executed using a different container image (Spjuth et al., 2018). An example of this execution pattern can be found in the GenomeFastScreen pipeline (https://sing-group.org/compihub/explore/5e2eaacce1138700316488c1), although in this case “docker run” commands are included in each task rather than provided in a runners file for the sake of simplicity. Following this image-per-task execution pattern, it is possible for a pipeline to use different versions of the same software or two tools that require different versions of some dependencies.

In addition, custom runners can also be used to submit pipeline tasks to a job scheduler such as SGE, Torque or SLURM in supercomputers or computer clusters (Fig. 5B). For instance, Fig. 7 shows a generic Slurm runner. Some srun parameters may need to be adjusted for each specific cluster and the “–export” parameter must be used to export all environment variables to the process to be executed, as the task parameters are declared as environment variables.

In the same way, when using Compi runners, it is also possible to combine containerized execution of Compi tasks with Docker in a clustered environment, using a container orchestration system such as Kubernetes (Fig. 5E).

Another way to achieve dependency management is by building a monolithic Docker image with Compi, the pipeline itself and all its dependencies (Fig. 5C). This topic is explained in “Reproducible Application Packaging with Docker”.

Finally, dependency management can be also delegated to external systems. For example, Compi allows the use of Conda/Bioconda packages seamlessly. Each task can use them simply by activating and deactivating the corresponding Conda environments before executing the specific commands. The Metatax pipeline (https://www.sing-group.org/compihub/explore/5d807e5590f1ec002fc6dd83) illustrates this. For instance, Fig. 8 shows the execution of a script (whose name is defined by the “validate_mapping_file” parameter, highlighted in bold) from the Qiime Bioconda package.

Reproducible application packaging with docker

Compi enables the creation of portable end-user CLI applications for pipelines that can be distributed as Docker images. As noted in the previous section, this is another way to deal with dependency management, as such Docker images contain all the dependencies required by the pipeline. Pipelines distributed in this way follow an image-per-pipeline execution pattern in which all tasks are executed using the same image container (Spjuth et al., 2018) (Fig. 5C), and can even be run using Docker-compatible container technologies such as Singularity.

The compi-dk command-line tool is provided to assist in the development and packaging of Compi-based applications into Docker images. Pipeline development starts with the creation of a new compi-dk project with the “compi-dk new-project” command, which creates a project directory and inizialites two template files: pipeline.xml and Dockerfile. After this, the definition of the pipeline can start by modifying the pipeline.xml template and the subsequent local testing (using the compi command). Also, it can be tested by building a Docker image (using compi-dk) and running the containerized pipeline. As for the latter case, when the compi-dk build command is executed on the project directory, a Docker image for the pipeline is created. This image contains the compi executable file and a specific pipeline.xml file, along with the pipeline dependencies as defined in the Dockerfile. Figure 9 shows the Dockerfile of the MINC Computer Vision pipeline for image classification based on Deep Learning (https://www.sing-group.org/compihub/explore/5d08a9e41713f3002fde86d5). The Dockerfile skeleton was automatically generated by compi-dk, being only necessary to add the “RUN” commands for the installation of “gluoncv” and “gnuplot” dependencies. In this regard, it is important to note that special attention was placed on the ability to derive a pipeline application image from any preexisting Docker image of interest (e.g., images with bioinformatics packages). For instance, the RNA-Seq Compi pipeline discussed above was created using the DEWE (López-Fernández et al., 2019) Docker image as a base image and the MINC Computer Vision was created using one of the Apache MXNet Docker images as a base image.

When working with a compi-dk project, it is also possible to create a Docker image for a pipeline that follows the image-per-task execution pattern. This is the case of the GenomeFastScreen pipeline (https://sing-group.org/compihub/explore/5e2eaacce1138700316488c1), in which, as explained above, a “docker run” command is included within each task instead of managing the execution of Docker using an external runner file. Regardless, most of the tasks within this pipeline run under external Docker images, thus following the image-per-task execution pattern. As the GenomeFastScreen pipeline itself is also distributed as a Docker image, it must be able to run Docker images as well (please refer to the pipeline documentation for details on how to do this).

Pipeline distribution via Compi Hub

Compi enables the creation of portable end-user CLI applications for pipelines that can be distributed as Docker images. As noted in the previous section, this is another way to deal with dependency management, as such Docker images contain all the dependencies required by the pipeline. Pipelines distributed this way follow an image-per-pipeline execution pattern in which all tasks are executed using the same image container (Spjuth et al., 2018) (Fig. 5C), and can even be run using Docker-compatible container technologies such as Singularity.

Once the pipeline development is completed, it can be released through Compi Hub to increase the visibility and benefit from the Compi Hub features. Pipelines can be registered using the Compi Hub web interface (https://www.sing-group.org/compihub) or using the “compi-dk hub-push” command. Compi Hub can store several versions of a pipeline, each of them associated to a pipeline.xml file where the specific pipeline version is defined. In addition, since Compi Hub does not store either the full source code (e.g., scripts included in the pipeline.xml as source files) or the Docker images themselves, pipeline publishers are encouraged to: (i) publish the source code (i.e., the compi-dk project) in public repositories such as GitHub or GitLab to allow users to re-build the project locally at any time; and (ii) push the corresponding Docker image to the Docker Hub registry so that users can pull the image and follow the instructions to run the pipeline application.

The Compi Hub website lists publicly available pipelines and gives access to all pipelines. When a pipeline is selected, the main pipeline information is displayed, including title and description, creation date, as well as links to external repositories in GitHub or Docker Hub. In addition, for each pipeline version, Compi Hub displays the following information:

• Overview: this section is headed by the pipeline DAG, generated in the backend using the “compi export-graph” command. Since it is an interactive graph, visitors can use it to navigate to each task description. Figure 10 shows the Metatax pipeline DAG (https://www.sing-group.org/compihub/explore/5d807e5590f1ec002fc6dd83). The DAG is followed by two tables, one containing the pipeline tasks and their associated descriptions, and a second one containing global parameters of the pipeline. Finally, this section encloses one table for each task with descriptions and specific parameters. It is important to note that all this information is automatically generated from the pipeline XML.

• Readme: this section shows the content of the README.md file when it is present in the compi-dk project. This file should be used to provide a comprehensive description of the pipeline, as well as instructions on how to use it.

• Dependencies: this section shows the content of the DEPENDENCIES.md file when it is present in the compi-dk project. We recommend that pipeline developers include this file with a human-readable description of the pipeline dependencies and the specific versions used to develop the pipeline.

• License: this section shows the content of the LICENSE file when it is present in the compi-dk project. We encourage pipeline developers to include this file in their compi-dk projects so that the terms of use of the pipeline are clear.

• Dataset: this section contains a list of datasets that can be used to test the different versions of the pipeline. It is shown when a pipeline publisher associates a test dataset with the displayed pipeline. In addition to the instructions given in the Readme section, we also recommend that pipeline publishers provide test datasets to help users test the pipelines themselves.

• Runners: this section displays a list of example runner configurations when they are present in the compi-dk project. Runner configurations must be stored as XML files within the “runners-example” directory of the project.

• Params: this section shows a list of example parameter configurations when they are present in the compi-dk project. Parameter configurations must be stored as plain-text key-value files in the “params-example” directory of the project.

As can be seen, Compi Hub was not designed to be a merely pipeline repository. We seek that developers accompany each pipeline with all the necessary information to ensure its portability and reproducibility by other researchers.