Are you working with big data? Do you need to pass your data through various software? If you’ve ever been in this situation, you would know it can become very difficult to maintain reproducibility and accuracy; wait, have I updated this output file? The more manual steps we do, the more human errors are inevitably introduced into our analysis, hampering accuracy and reproducibility.
Be lazy, the machine does it better.
Workflow languages automate your data analysis workflow. But this isn’t all, they ensure all your analysis logs are captured in an organised fashion, explicitly outline the software used, capture the input and output files at each step and even allow you to restart the pipeline from where it errored out, avoiding wasted time and resources re-running your workflow from the start. Additionally, when your data inevitably becomes big data, workflow languages allow you to easily scale up meaning you can move your analysis to a high performance cluster (HPC) without stress.
In this hands-on workshop, ESR’s Leah Kemp and the NeSI team will guide you through an introduction to Snakemake, a workflow language with its basis in the popular programming language, Python. Attendees can expect to learn:
- The benefits of using Snakemake or other workflow languages,
- how to create a workflow to organise your computations, and
- how an HPC scheduler (such as Slurm) fits into your workflow
This workshop is being run as part of NZ Research Software Engineering Conference 2021 which has the theme of Open Research – Workflows, Data and Communities; However, you do not need to register for the conference in order to register for the workshop. The workshop is free to attend and entirely online.
This workshop is intended for anyone who has several steps in their data analysis workflow. It is a hands-on workshop, meaning you will be coding along with Leah and the NeSI team.
This is an online workshop that will use Jupyter on NeSI. You will need to have a NeSI account in order to fully participate in the workshop. Those that register will be sent instructions on how to do this approximately 10 days before the event. Ticket sales for this event end on 3 September 2021.
Workshop sections:
- 01 - Introduction
- 02 - Setup
- 03 - Create a basic workflow
- 04 - Leveling up your workflow
- 05 - We want more!
The workflows we will create: