Skip to content

Using Farm

Jump to bottom Edit New page
amob edited this page Dec 12, 2016 · 56 revisions

Farm is the College of Agriculture and Environmental Sciences computer cluster that runs Ubuntu Linux version 13.04. It's our primary high performance computing resource in the Ross-Ibarra lab.

This document serves as a reference for the most commonly used SLURM commands and workflows. Setup and other topics can be found on the pages listed below:

Getting Started

Quick Introdution to Computer Clusters

Farm has a head node, which controls the cluster and compute nodes which is where the action happens. Farm runs on a cluster workload management system called Slurm. For the most part, you interact with Farm using scripts to launch jobs on the compute nodes; you don't run processes on the head node and you don't log into the compute nodes directly. The only tasks that acceptable on the head node are:

  • Downloading files (with wget or curl)
  • Compiling/building files
  • Installing R packages
  • Submitting or checking on jobs

Connecting to Farm

The address is username@agri.cse.ucdavis.edu. username here will be your UCD kerberos ID. You will have to have generated a SSH key (see this section) and given the public key part (do not share the private key!) to CSE Help.

SSH Config

Make your life a little easier by adding the following to ~/.ssh/config:

Host farm
    HostName agri.cse.ucdavis.edu
    User username

Replace username with your username. This will allow you to ssh to farm with just ssh farm in the future.

Getting to Know Slurm

Slurm is a lot like SGE: you submit jobs via batch scripts. These batch scripts have common headers; we will see one below.

First, we can get a sense of our lovely cluster with sinfo, which is pronounced sin-fay (Etymology unknown, but we think French origin):

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
hi           up   infinite      4 drain* c8-[25,39-40],c9-35
hi           up   infinite     64    mix c8-[22-24,30-38,45,54-58,60-61,84-87]
hi           up   infinite      6   idle c8-[42-44,59],c9-[94,97]
serial       up   infinite      2  down* c10-[12,40]
serial       up   infinite     15    mix c10-[8-9,11,13-22,41-42]
serial       up   infinite     18   idle c10-[10,23-39]
bigmeml      up   infinite      1  mixed bigmem2
bigmeml      up   infinite      5    mix bigmem[1,3-6]
bigmemm      up   infinite      1  mixed bigmem2
bigmemm      up   infinite      5    mix bigmem[1,3-6]
bigmemh      up   infinite      1  mixed bigmem2
bigmemh      up   infinite      5    mix bigmem[1,3-6]

Here we see our bigmems (more on those later), and all of their inferior but still useful friends.

Note that there is a column of STATE, which indicates the state of the machine. A better way of looking at what's going on on each machine is with squeue, which is the job queue.

$ squeue
         JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
       1541913   bigmemh some_job someone  R 6-04:27:02      1 bigmem6
       1613530   bigmemm some_job someone PD       0:00      1 (Resources)
       1544863   bigmeml     some_job someone R 5-10:59:32      1 bigmem2
       1472908        hi  some_job someone  R 5-22:16:23      1 c8-22
       1477386    serial some_job  someone  R 14-03:19:29      1 c10-11

This shows each job ID (very important), partition the job is running on, name of person running the job. Also note TIME which is how long a job has been running.

This queue is very important: it can tell us who is running what where, and how long it's been running. Also, if we realize that we're accidentally doing something silly like mapping maize reads to the human genome, we can use squeue to find the job ID, allowing us to cancel a job with scancel. Let's kill vince251's job eva:

$ scancel 5370

It's that easy! Slurm is pretty boring so far; all we can do is look at the cluster and try to kill jobs. Let's see how to submit jobs.

An Example Slurm Batch Script Header

We wrap our jobs in little batch scripts, which is nice because these also help make steps reproducible. We'll see how to write batch scripts for Slurm in the next section, but suppose we had one written called steve.sh. To keep your directory organized, I usually keep a scripts/ directory (or even slurm-scripts/ if you have lots of other little scripts).

I like to organize each of my projects in their own directory in a general ~/projects/ directory. In each project directory, I make a directory called slurm-log for Slurm's logs. Tip: use these logs, as these are very helpful in debugging. I separate them from my project because they fill up directories rather quickly.

Let's look at an example batch script header for a job called steve (which is run with script steve.sh) that's in a project directory named your-cool-project (you're going to change these parts).

#!/bin/bash -l
#SBATCH -D /home/vince251/projects/your-cool-project/
#SBATCH -o /home/vince251/projects/your-cool-project/slurm-log/steve-stdout-%j.txt
#SBATCH -e /home/vince251/projects/your-cool-project/slurm-log/steve-stderr-%j.txt
#SBATCH -J steve
#SBATCH -t 24:00:00
set -e
set -u

# insert your script here
  • -D sets your project directory.
  • -o sets where standard output (of your batch script) goes.
  • -e sets where standard error (of your batch script) goes.
  • -J sets the job name.
  • -t sets the time limit for the job, 24:00:00 indicates 24 hours.

Note that the programs in your batch script can redirect their output however they like — something you will like want to do. This is the standard output and standard error of the batch script itself.

Also note that these directories must already be made — Slurm will not create them if they don't exist. If they don't exist, sbatch will not work and die silently (since there's no place to write standard error). If you keep trying something and it doesn't log the error, make sure all these directories exist.

As mentioned, the jobname is how you distinguish your jobs in squeue. If we ran this, we'd see "steve" in the JOBS column.

The time limit for the job should be greater than the estimated time to complete your job. Time-and-a-half or twice as much time as you think it will take are good rules. If your job reaches this time limit it will be killed. It's frustrating to lose a job because you underestimate the time. Alternatively, you can set this with the --time flag (instead of -t, e.g. --time=1-00:00 sets a time limit of one day)

An example script

Try running this test script:

#!/bin/bash -l
#SBATCH -D /home/USERNAME
#SBATCH -J bob
#SBATCH -o /home/USERNAME/out-%j.txt
#SBATCH -e /home/USERNAME/error-%j.txt
#SBATCH -t 24:00:00
#SBATCH --array=0-8

bob=( 1 1 1 2 2 2 3 3 3 )
sue=( 1 2 3 1 2 3 1 2 3 )

block=${bob[$SLURM_ARRAY_TASK_ID]}
min=${sue[$SLURM_ARRAY_TASK_ID]}

echo "$block is $min"

Make sure you switch your user name for USERNAME. You should see a bunch of files named "error" and "out" show up in your home directory. Try launching this using sbatch -p bigmem and sbatch -p serial to make sure you have access to both queues. More info on array jobs can be found below.

Array jobs can also process a fixed number of files. This set of files can be streamed into sed which grabs the appropriate line (given by the array job), and then passes that to the pipeline. It's wise to use sort (with -s for stable) to ensure your input order does not change. For example:

find data/aln -name "*.bam" | xargs -n1 -I{} basename {} .bam | \
   sort -s | sed -n "$SLURM_ARRAY_TASK_ID"p | \
   xargs -n1 -I{} samtools sort -@ 10 -m2G data/aln/{}.bam $ODIR/{}.sorted

Do not use this if you have spaces in your file name. We could use -print0, but that would interfere with other commands.

Know About The Queue

We share the cluster. There is a queue established with multiple users submitting jobs. This means often times SLURM will allocate resources to your jobs only when nodes in your desired partition are open and not occupied by jobs from other users. In order to submit a job (covered in the next section), you must specify which partition your job(s) will run on. Based on your memory and cpu requirements, you can choose to run with the following partition options:

bigmeml (low): This means that your job might be killed at any time. Great for soaking up unused cycles with short jobs; a particularly good fit for large array jobs with short run times.

bigmemm (medium): This means your jobs might be suspended, but will resume when a high priority job finishes. Good for long jobs that need lots of cpu, but NOT recommended for MPI jobs. Up to 100% of idle resources can be used.

bigmemh (high) or (hell yeah, I want this to run): Your job will kill/suspend lower priority jobs. High priority means your jobs will keep the allocated hardware until it's done or there's a system or power failure. Limited to the number of CPUs your group contributed. Please check with Jeff before running anything using more than 8cpu on bigmemh, as we only have 64 slots on this queue.

serial: These are the older serial nodes. Here jobs will keep the allocated hardware until it's done or there's a system or power failure. For many jobs not needing high memory, the serial queue is still the way to go. There are many hundreds of cpu there, and we have guaranteed access to 128 of them. Note that each node in this queue only has 16 cpu and 24G of RAM, so plan your scripts accordingly.

NOTE: The serial queue is unregulated and may be subject to large array jobs with long run times (from personal experience).

Submitting Jobs

We submit our batch scripts with sbatch. For example, we can submit the steve.sh job (assuming it's in a scripts/ directory):

$ sbatch -p serial scripts/steve.sh

The -p and -t are both necessary. Most people will want to submit to the serial partition, indicated by -p serial. You also must set a time limit after which your job will be automatically terminated using -t; here we have chosen two hours by indicating hours, minutes, and seconds (-t 2:00:00). Alternatively, you can use the --time flag, it works simiary. . It is good to pick a time beyond what you think you need (e.g. twice as much time as you think it needs).

It's that easy! After submitting jobs, check with squeue that it's still running (and didn't immediately fail, do to syntax error or a program not being in your $PATH or a module not loaded). If you don't see a steve job in squeue, then it's time to debug. Use these slurm-log/ directory to standard output and standard error to figure out what happened. I use ls -lrt (ls with reverse time sort) to see the most recent Slurm log, i.e.:

$ ls -lrt slurm-log/
-rw-rw-r-- 1 vince251 vince251            0 Sep 25 11:07 dwgeval-stderr-5370.txt
drwxrwxr-x 2 vince251 vince251        12288 Sep 25 11:07 .
-rw-rw-r-- 1 vince251 vince251           47 Sep 25 11:34 dwgeval-stderr-5368.txt

Hey cool, that's the name of the last Slurm job (but double check against the job ID that sbatch gives you when you run it. Then look at this in less or something.

Specifying a Partition like bigmem

You may want to use the big memory machines that the Ross-Ibarra lab has access to, which is an option because our fearless leader contributed a machine to the bigmem pool.

Let's first see these machines using sinfo:

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
hi           up   infinite      4 drain* c8-[25,39-40],c9-35
hi           up   infinite     64    mix c8-[22-24,30-38,45,54-58,60-61,84-87]
hi           up   infinite      6   idle c8-[42-44,59],c9-[94,97]
serial       up   infinite      2  down* c10-[12,40]
serial       up   infinite     15    mix c10-[8-9,11,13-22,41-42]
serial       up   infinite     18   idle c10-[10,23-39]
bigmeml      up   infinite      1  mixed bigmem2
bigmeml      up   infinite      5    mix bigmem[1,3-6]
bigmemm      up   infinite      1  mixed bigmem2
bigmemm      up   infinite      5    mix bigmem[1,3-6]
bigmemh      up   infinite      1  mixed bigmem2
bigmemh      up   infinite      5    mix bigmem[1,3-6]

Voilà! We see that we have two bigmems (3 and 4), noting these are in a special partition called bigmeml. To use these, we specify the partition in either sbatch or our batch script itself.

To do so with sbatch, do:

$ sbatch -p bigmeml steve.sh

Or, we can do so in the batch script itself by adding a line:

#SBATCH --partition=bigmeml

Allocating Resources

If you're using a process that requires more than one CPU/thread/process, you'll need to allocate more tasks. It's very important you do this, or else the cluster manager won't know how much CPU your program is running, and allocate resources thinking it has more available than it does. This really gums up the works, so don't do this. To specify the number of tasks (roughly, number of processes), use sbatch --ntasks=x where x is the number.

Farm plans for around 8GB of memory per CPU/task for all bigmem queues, and 1.5GB of memory per CPU/task for the serial queue. So if your task needs more memory but only one CPU, use --mem=16000 (memory in MB, so this is requesting 16GB remember that you can only request up to 24GB on serial). This makes sure that machines don't get overtaxed.

Monitoring Stuff

You can monitor jobs a few ways:

  • Agri Ganglia

  • squeue: see if they are still running.

  • Watching your files grow.

  • Advanced: ssh to a node and use top, but do not run anything on the nodes this way. Every time you do this, Bill, Jeff, or I will have to ruthlessly strangle a young kitten. So if you not sure what this section means, save the kittens and don't ssh to the nodes.

Managing Your Home Directory

An extension of Monitoring Stuff, you should also monitor the amount of space you data and files are taking up.

Good cluster practice involves realizing that many things on the cluster are shared, that includes storage. Currently, there is ~45 Tb of space for the entire lab. That sounds like a lot, but after data and outputs are generated from the many jobs you are bound to run, the cluster will begin to feel a little too clustered.

Use du -sh to check the amount of storage being used in any given directory. Just cd to the directory of your choice and enter that command and options. Check directory size regularly.

Use df -h to check the size of your drive, it's total storage being used, how much you have left on your drive, and where that drive is mounted.

$ df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/sda1                     854G  638G  173G  79% /
none                          4.0K     0  4.0K   0% /sys/fs/cgroup
udev                           32G  4.0K   32G   1% /dev
tmpfs                         6.3G  1.4M  6.3G   1% /run
none                          5.0M     0  5.0M   0% /run/lock
none                           32G     0   32G   0% /run/shm
none                          100M     0  100M   0% /run/user
nas-8-0:/export/1/sbhadral     11T  7.9T  2.3T  78% /home/sbhadral

Looks like it's time for me to start cleaning up my directories...

Transferring Big Data onto Farm (screen and rsync )

When transferring files to/from a server, scp is a nice command for small data, as those are very quick and do not require babysitting (enter man scp in the command-line for more details). With larger files, the transfers can become problematic as they take quite some time. No one wants to sit and watch files transfer. Solution: screen and rsync.

screen allows you to enter an overlay of your terminal. The screen terminal environment is nearly identical to your previous terminal command-line, but you can now simply "detach" the screen and have it run in the background. No risk of connection drops, you can shutdown your computer and still have your tasks running. To get back to your screen, just "reattach" and continue. (refer to man screen for more details).

Examples:

Enter screen:

$ screen

To detach: press "ctrl + a" then "d". This will return you to your regular command-line.

To reattach:

$ screen -r

This will recover your screen immediately if you only have 1 running. If you have multiple screens running, this will return a list of those screens and their respective IDs in this format [pid.tty.host].

With multiple sessions to choose from, reattach with the following.

$ screen -D -r [4137.pts-65.farm]

(NOTE: check up on your active screen sessions with screen -list )

To terminate screen, enter the respective screen and enter exit.

(NOTE: From personal experience. When you enter a screen, it is possible to enter a screen within that screen, and a screen within that screen...etc. You can get lost in screen limbo. If you have watched the film Inception(2010) you will understand. Just terminate each subscreen with exit. You will know when you're out, when you see [screen is terminating] at the top of your window.)

Now that you know how screen works, you can use it in conjunction with rsync.

rsync copies/transfers files fast and reliably with lots of options and versatility (refer to man rsync). After entering screen, rsync the files of your choosing from their location into the target destination on farm.

Example:

$ screen

$ rsync -avz <my files> farm:/home/sbhadral/Projects/Rice_project/fixed_fastqs/

Detach screen and check up on your transfer whenever.

(NOTE: You cannot transfer from server to server remotely, you need to be logged into one of the servers. e.g. rilab@169.237.206.32:/home/blah/.. to farm:/home/sbhadral/.. )

Using R with Slurm

Often, we need to work with R interactively on a server. To do this, we use srun with the following options:

$ srun -p your-partition -t 2:00:00 --pty R

This will drop you into an interactive R session on the partition specified by -p. --pty launches srun in terminal in pseudoterminal mode, which makes R behave as it would on your local machine. With srun you still have to set a time limit.

If you want to render R plots to your local computer through X11, here is a solution:

$ srunx your-partition

Make sure your X11 is open before you typing the command.

Using ssh -Y with X11 may seem like good idea, but this does not specify a partition to work interactively in, so you will end up running things on the head node. Bad idea.

Warnings

Do not run anything on the headnode except cluster management tools (squeue, sbatch, etc), compilation tasks (but usually ask CSE help for big apps), or downloading files. If you run anything on the headnode, you will disgrace your lab. How embarrassing is this? Imagine if you had to give your QE dressed up like Richard Simmons. It's that embarrassing.

Back up your stuff, as Farm does not back up your stuff. Using git is a good option.

Monitor your disk space, as it can fill up quickly.

Array jobs

Slurm supports job arrays, which make it easy to submit many jobs of a similar type at once. For our cluster, we can submit a max of 1000 jobs in a single array. To get more jobs than that, submit multiple arrays. The basic syntax is:

$ sbatch --array=<start>-<end>

You can also specify this in the script you are submitting via sbatch:

#!/bin/bash -l
#SBATCH --array=0-50

Slurm also comes with some built in variables for managing file names. SLURM_ARRAY_JOB_ID will be set to the first job ID in the array, and SLURM_ARRAY_TASK_ID will be set to the index of the current job array. You can use these in filenames by inserting %A or %a into your bash script, which will be replaced with SLURM_ARRAY_JOB_ID and SLURM_ARRAY_TASK_ID respectively. For example, slurm-%A_%a.out will be replaced with slurm-7358_1.out

More information is available here and an example script is above.

Add a custom footer

RILab Wiki

Farm Subwiki

Other Computing Information

Maize

Protocols

Clone this wiki locally