Batch Overview
Last updated
Last updated
The general principle behind batch processing is automating repetitive tasks. Single tasks are known as jobs, while a set of jobs is known as a batch. This distinction is mostly academic, since the terms job and batch job are now mostly synonymous, but here we'll use the terms separately.
There are three basic steps in a batch or job-oriented workflow:
Copy input data from archival storage to scratch space
Run computational tasks over the input data
Copy output to archival storage
On Chinook the first and last steps must occur on login nodes, and the computation step on compute nodes. This is enforced by the login nodes having finite CPU set and not being present on the compute nodes.
Depending on the scale and characteristics of a particular job, different jobs may require different combinations of computational resources. Garnering these resources is a combination of:
Choosing which to submit the job to
Choosing what resources to request from the partition
This is done by writing whose directives specify these resources.