flo¶

design goals¶

This package is deliberately designed to help users write small, but compact workflow prototypes using whatever tools they prefer — R, pandas, scipy, hadoop. The goal here is not to provide a substitute for these tools, but rather to be the glue that sticks them together. It takes inspiration from a number of existing tools that have similar aims, particularly GNU make that has many desirable properties of working with data workflows. Specifically, the design goals for this project are to:

• Provide an easy-to-use framework. This applies for n00bs and pros alike. Use human-readable syntax.
• Prevent, as much as reasonably possible, costly mistakes. Avoid inadvertently rerunning commands that overwrite results or executing commands that take a long time.
• Encourage good development practices, but allow for flexibility. There’s a tradeoff here, but we have an opinion on how to do this in a good way.

prior art¶

creates¶

The creates key uniquely identifies the resource that is created. By default, it is interpreted as a path to a file (relative paths are interpreted as relative to the flo.yaml file) or a directory. Importantly, every task is intended to create a single file or directory. If you have a task that creates multiple files, you can either (i) split that into separate tasks or (ii) have all of those files embedded in a directory and use the directory name as the creates value like this:

---
creates: "path/to/output/directory"
depends: "path/to/some/script.py"
command:
  - "mkdir -p {{creates}}"
  - "python {{depends}} {{creates}}"

In this case, the directory path/to/output/directory is passed as the first argument to path/to/some/script.py, which can then add as many files as necessary to that directory. When this task is complete, flo checks the hash of all files in path/to/output/directory and all of its child directories to determine if it is in sync or not.

depends¶

The depends key defines the resource(s) on which this task depends. It is common for depends to specify many things, including data analysis scripts or other tasks from within the flo.yaml. Multiple dependencies can be defined in a YAML list like this:

depends:
  - "path/to/some/script.py"
  - "another/task/creates/target.txt"

These dependencies are what flo uses to determine if a task is out of sync and needs to be re-executed. Importantly, flo obeys the dependencies when it constructs the task graph but always runs in a deterministic order. If a specified depends does not exist immediately prior to flo running the task, flo throws an informative error.

command¶

The command key is mandatory and it defines the command(s) that should be executed to produce the resource specified by the creates key. Like the depends key, multiple steps can be defined in a YAML list like this:

command:
  - "mkdir -p $(dirname {{creates}})"
  - "python {{depends}} > {{creates}}"

templating variables¶

Importantly, the command is rendered as a jinja template to avoid duplication of information that is already defined in that task. Its quite common to use {{depends}} and {{creates}} in the command specification, but you can also use other variables like this:

---
creates: "path/to/some/output/file.txt"
sigma: "2.137"
depends: "path/to/some/script.py"
command: "python {{depends}} {{sigma} > {{creates}}"

In the aforementioned example, sigma is only available when rendering the jinja template for that task. If you’d like to use sigma in several other tasks, you can alternatively put it in a global namespace in a flo.yaml like this (similar example here):

---
sigma: "2.137"
tasks:
  -
    creates: "path/to/some/output/file.txt"
    depends: "path/to/some/script.py"
    command: "python {{depends}} {{sigma} > {{creates}}"
  -
    creates: "path/to/another/output/file.txt"
    depends:
      - "path/to/another/script.py"
      - "path/to/some/output/file.txt"
    command: "python {{depends[0]}} {{sigma}} < {{depends[1]}} > {{creates}}"

Another common use case for global variables is when you have several tasks that all depend on the same file. You can also use jinja templating in the creates and depends attributes of your flo.yaml like this:

---
input: "data/sp500.html"
tasks:
  -
    creates: "{{input}}"
    command:
      - "mkdir -p $(dirname {{creates}})"
      - "wget http://en.wikipedia.org/wiki/List_of_S%26P_500_companies -O {{creates}}"
  -
    creates: "data/names.dat"
    depends:
      - "src/extract_names.py"
      - "{{input}}"
    command: "python {{depends|join(' ')}} > {{creates}}"
  -
    creates: "data/symbols.dat"
    depends:
      - "src/extract_symbols.py"
      - "{{input}}"
    command: "python {{depends|join(' ')}} > {{creates}}"

There are several examples for more inspiration on how you could use the flo.yaml specification. If you have suggestions for other ideas, please add them!

deterministic execution order¶

flo is guaranteed to run in the exact same order every single time and its important that users understand how it works. When flo is executed, it makes sure to obey the dependencies specified in the YAML configuration. In the event of ties flo is executed in the same order as the tasks appear in the YAML configuration. Technically, this is very similar to a breadth first search originating from the set of tasks that have no dependencies except that we order things based on the maximum distance that each task is from any given source node and we break ties based on the order in the YAML configuration file.

The deterministic order example contains a few different YAML configuration files to demonstrate how this works in practice, the highlights of which are summarized here.

For sibling tasks, sibling tasks are executed in the order in which they appear in the YAML configuration file, but always after the their dependencies have been satisfied. In this example, the task graph looks like this and the tasks are guaranteed to run in alphabetical order.

For parallel threads, task threads are executed based on their distance from the source tasks and secondarily based on their ordering in the YAML configuration file. In this example, the task graph looks something like this and the tasks are guaranteed to run in alphabetical order.

For merging task graphs, tasks are executed based on their maximal distance from any source task. In this example, the task graph looks something like this and the tasks are guaranteed to run in alphabetical order.

running workflows¶

By default, the flo run command will execute the entire workflow, or at least the portion of it that is “out of sync” since the last time it ran. Executing flo run twice in a row without editing any files in the interim will not rerun any steps. If you edit a particular file in the workflow and re-execute flo run, this will only re-execute the parts that have been affected by the change. This makes it very easy to iterate quickly on data analysis problems without having to worry about re-running an arsenal commands — you only have to remember one, flo run.

flo run                # runs everything for the first time
flo run                # nothing changed; runs nothing
edit path/to/some/script.py
flo run                # only runs the parts that are affected by change

Importantly, if you edit a particular task in the flo.yaml itself, this will cause that particular task to be re-run as well:

flo run
edit flo.yaml          # change a particular task's command
flo run                # rerun's that command and any dependent task

The flo command is able to do this by tracking the status of all creates, depends, and task definitions by hashing the contents of these resources. If the contents in any depends or the task itself has changed since the last time that task was run, flo will run that task. For reference, the hashes of all of the creates, depends, and workflow task definitions are in .flo/state.csv.

same project, different workflows¶

Naturally, there will be times when you’ll want to have separate sets of steps to accomplish different things. One simple way to separate your workflow configuration is by putting them in two separate files, say figures.yaml and analysis.yaml. You can then specify running these separate workflows on the command line with the --config option like this:

flo run --config figures.yaml
flo run --config analysis.yaml

All other behaviors of the YAML configuration and use of the flo command remain exactly the same.

limiting flo run execution¶

Oftentimes we do not want to run the entire workflow, but only a particular component of it. Like GNU make, you can specify a particular task by its creates value on the command line like this:

flo run path/to/some/output/file.txt

This limits flo to only executing the task defined in path/to/some/output/file.txt and all of its recursive upstream dependencies. Other times we do not want to run the entire workflow, but run everything after a specific task. We can do that like this:

flo run --start-at path/to/some/file.txt

This limits flo to only executing the task defined in path/to/some/file.txt and all of its recursive downstream dependencies. This can be combined with flo run task_id to only all tasks between two specified tasks like this:

flo run --start-at path/to/some/file.txt path/to/some/output/file.txt

If you ever want to only run one task, say a task that creates path/to/some/file.txt, you can specify that task as both the starting and ending point of the workflow run with --only:

# these two things are the same
flo run --only path/to/some/file.txt
flo run --start-at path/to/some/file.txt path/to/some/file.txt

In some situations — especially with very long-running tasks that you know haven’t been affected by changes — it is convenient to be able to skip particular tasks like this:

flo run --skip path/to/some/file.txt

This eliminates the task associated with path/to/some/file.txt from the workflow but preserves the dependency chain so that other tasks are still executed in the proper order.

Sometimes it is convenient to rerun an entire workflow, regardless of the current status of the files that were generated.

flo run
# don't do anything for several months
echo "Rip Van Winkle awakens and wonders, where did I leave off again?"
echo "Screw it, lets just redo the entire analysis"
flo run --force

For long-running workflows, it is convenient to be alerted when the entire workflow completes. The --notify command line option makes it possible to have the last 100 lines of the .flo/flo.log sent to an email address specified on the command line.

flo run --notify j.doe@example.com

I’m nervous, what’s going to happen?¶

While we don’t recommend it, its not uncommon to get “in the zone” and make several edits to analysis scripts before re-running your workflow. Because we’re human, its easy to incorrectly remember the files you edited and how they may affect re-running the workflow. To help, the flo status command lets you see which commands will be run and approximately how much time it should take (!!!).

flo run
edit path/to/some/script.py
edit path/to/another/script.py
echo "a long time passes"
flo status             # don't run anything, just report what would be done

For reference, flo stores the duration of each task in .flo/duration.csv. Another way you can comfort yourself is by looking at the status visualization.

flo status --serve

which displays something like this:

Starting over¶

Sometimes you want to start with a clean slate. Perhaps the data you originally started with is dated or you want to be confident a workflow properly runs from start to finish before inviting collaborators. Whatever the case, the flo clean command can be useful for removing all creates targets that are defined in flo.yaml. With the --force command line option, you can remove all files without having to confirm that you want to remove them. If you just want to remove a particular target, you can use flo clean task_id to only remove that creates target.

flo clean              # asks user if they want to remove `creates` results
flo clean --force      # removes all `creates` targets without confirmation
flo clean a/task       # only remove the a/task target

Saving results¶

Before removing or totally redoing an analysis, I’ve often found it useful to backup my results and compare the differences later. The flo archive command makes it easy to quickly backup an entire flo (including generated creates targets, source code specified in depends, and the underlying flo.yaml) and compare it to previous versions.

flo archive            # store archive in .flo/archives/*.tar.bz2
for i in `seq 20`; do
    edit path/to/some/script.py
    flo run
done
echo 'oh crap, this sequence of changes was a mistake'
flo archive --restore  # uncompresses archive

latest¶

• enforce that depends must exist prior to running any commands (#59)
• more informative error messages (#56, #57, #58)
• several bug fixes, including:
  • properly handling backspacing output of subprocessed commands (#53)

1.0.0¶

• removed pseudotask creation (every task must have a command key)
• specifying alternative yaml configuration (#62)
• incorporated deterministic ordering in a predictable and explainable manner (#65, #70)
• introduced flo status command; removed ``flo run –dry-run`` in favor of ``flo status`` (#55)
• incorporated the --only option (#37)
• several bug fixes, including:
  • making sure that the TaskGraph is a directed acyclic graph (#61)
  • ensuring that creates exists after a task has been run (#60)
  • clarifying output on flo status and flo run (#64)

0.2.0¶

• Initial release