Update Workflow management
parent
d8056fe707
commit
d733d8aa09
|
@ -32,17 +32,6 @@ acyclic graph*. This allows two things:
|
||||||
These two features together provide a solid step towards reproducible simulation
|
These two features together provide a solid step towards reproducible simulation
|
||||||
work.
|
work.
|
||||||
|
|
||||||
# GNU Make
|
|
||||||
Make is a program specifically designed to be a build system, i.e. a tool that
|
|
||||||
coordinates the compilation of a program's source code so that an executable or
|
|
||||||
library can be built. Each file of the build process is called a *target* and is
|
|
||||||
the output of some rule. Although it's primary purpose is creating build files,
|
|
||||||
it can easily be made to manage outputs of simulations. While it has the
|
|
||||||
advantage of being installed on virtually every Linux machine used for
|
|
||||||
scientific work, it lacks some features (most notably integration with queue
|
|
||||||
systems) which only make it practical for small cases (although I am sure some
|
|
||||||
shortcomings could be solved with a strong knowledge of Make).
|
|
||||||
|
|
||||||
# Snakemake
|
# Snakemake
|
||||||
Snakemake is a tool written in Python to managed rule-based workflows. The
|
Snakemake is a tool written in Python to managed rule-based workflows. The
|
||||||
workflow definition is a rather simple text file (usually a `Snakefile`), which
|
workflow definition is a rather simple text file (usually a `Snakefile`), which
|
||||||
|
@ -171,3 +160,30 @@ Here is a list of useful features:
|
||||||
the name.
|
the name.
|
||||||
- [Parameter space
|
- [Parameter space
|
||||||
exploration](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#parameter-space-exploration).
|
exploration](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#parameter-space-exploration).
|
||||||
|
|
||||||
|
# GNU Make
|
||||||
|
Make is a program specifically designed to be a build system, i.e. a tool that
|
||||||
|
coordinates the compilation of a program's source code so that an executable or
|
||||||
|
library can be built. Each file of the build process is called a *target* and is
|
||||||
|
the output of some rule. Although it's primary purpose is creating build files,
|
||||||
|
it can easily be made to manage outputs of simulations. While it has the
|
||||||
|
advantage of being installed on virtually every Linux machine used for
|
||||||
|
scientific work, it lacks some features (most notably integration with queue
|
||||||
|
systems) which only make it practical for small cases (although I am sure some
|
||||||
|
shortcomings could be solved with a strong knowledge of Make).
|
||||||
|
|
||||||
|
For reference, here is a `Makefile` defining the same rules as the Snakemake example above.
|
||||||
|
|
||||||
|
```makefile
|
||||||
|
# One input, one output
|
||||||
|
groups_with_users.txt: /etc/group
|
||||||
|
cat $< | awk -F ':' '$$4 != "" { print $$1,$$4; }' > $@
|
||||||
|
|
||||||
|
# Multiple outputs with grouped targets
|
||||||
|
sorted_groups.txt only_users.txt &: groups_with_users.txt
|
||||||
|
sort < $< | tee sorted_groups.txt | cut -d ' ' -f 2 > only_users.txt
|
||||||
|
|
||||||
|
# Rule with pattern
|
||||||
|
start_with_letter_%.txt: groups_with_users.txt
|
||||||
|
grep '^$*' < $< > $@
|
||||||
|
```
|
Loading…
Reference in New Issue