pipeline submitting engine
What is looper?
Looper is a job submitting engine. Looper deploys arbitrary shell commands for each sample in a standard PEP project. You can think of looper as providing a single user interface to running, monitoring, and managing all of your sample-intensive research projects the same way, regardless of data type or pipeline used.
What makes looper better?
Looper decouples job handling from the pipeline process. In a typical pipeline, job handling (managing how individual jobs are submitted to a cluster) is delicately intertwined with actual pipeline commands (running the actual code for a single compute job). In contrast, the looper approach is modular: looper only manages job submission. This approach leads to several advantages compared with the traditional integrated approach:
- pipelines do not need to independently re-implement job handling code, which is shared.
- every project uses a universal structure, so datasets can move from one pipeline to another.
- users must learn only a single interface that works with any project for any pipeline.
- running just one or two samples/jobs is simpler, and does not require a distributed compute environment.
Releases are posted as GitHub releases, or you can install using
pip install --user looper
pip install --user --upgrade looper
looper executable in not automatically in your
$PATH, add the following line to your
looper, follow the Hello Looper example repository to run your first looper project:
# download and unzip the hello_looper repository wget https://github.com/databio/hello_looper/archive/master.zip unzip master.zip # Run looper: cd hello_looper-master looper run project/project_config.yaml
Detailed explanation of results is in the Hello world tutorial.