The goal of makepipe is to allow for the construction of make-like pipelines in R with very minimal overheads. In contrast to targets (and its predecessor drake) which offers an opinionated pipeline framework that demands highly functionalised code, makepipe is easy-going, being adaptable to a wide range of data science workflows.
A minimal example can be found here: https://github.com/kinto-b/makepipe_example
You can install the released version of makepipe from CRAN with:
And the development version from GitHub with:
To construct a pipeline, one simply needs to chain together a number of make_with_*() statements. When the pipeline is run through, each make_with_*() block is evaluated if and only if the targets are out-of-date with respect to the dependencies (and source file). But, whether or not the block is evaluated, a segment will be added to the Pipeline object behind the scenes. At the end of the script, once the entire pipeline has been run through, one can display the accumulated Pipeline object to produce a flow-chart visualisation of the pipeline. For example:
library(makepipe)
make_with_source(
dependencies = c("data/0_raw_data.csv", "lookup/concordance.csv"),
source = "1 data_prep.R",
targets = "data/1_data.Rds"
)
make_with_recipe(
dependencies = c("data/1_data.Rds", "data/0_pop.Rds"),
recipe = {
dat <- readRDS("data/raw_data.Rds")
pop <- readRDS("data/pop_data.Rds")
merged_dat <- merge(dat, pop, by = "id")
saveRDS(merged_dat, "data/2_data.Rds")
},
targets = c("data/2_data.Rds")
)
show_pipeline(labels = c(
"lookup/concordance.csv" = "Postcode concordance",
"data/0_raw_data.csv" = "Raw survey data"
))