torque-roll-userguide

Included software

The torque roll contains software collected from the following places

Links

Support

The rocks mailing list is the preferred place to get help and discuss the torque-roll as there are a lot of people on this list with hands-on experience from using the torque-roll on Rocks. Before posting questions to the list you should search the list archives for the terms pbs or torque, as the answer to your problems might be there already.

Installation

It is assumed that you know how to install a Rocks roll on a frontend, see the main Rocks documentation for an intro to installation of a Rocks cluster. You can either burn the roll iso on a CD or install from a central server, both methods are equivalent.

A serial job

It is useful to give info about expected walltime and the number of cpus the job needs. Here is how runscript.sh could look like for a single cpu job:

#!/bin/sh
#PBS -lwalltime=1:00:00
#PBS -lnodes=1

./do-my-work

This script asks for 1 hour runtime and will run on one cpu. The job will terminate when the script exits or will be terminated by the batch system if it passes the 1 hour runtime limit. The #PBS directives can also be given as commandline arguments to qsub like:

qsub -lnodes=1,walltime=1:00:00 runscript.sh

Commandline arguments takes precedence over runscript directives. Note that #PBS must be given exactly like this as the first characters on the line, no extra #s or spaces. All #PBS directives must come before any shell statements or else they will be ignored by the batch system.

When the job is finished you will get back two files with the standard output and standard error for the job in the same directory you submitted the job from. See man qsub.

A parallel job

If you have a parallel application using MPI you can run parallel jobs within the batch system. Let us take a look at the following script:

#!/bin/sh
#PBS -lwalltime=1:00:00
#PBS -lnodes=10
#PBS -lpmem=2gb
#PBS -N parallel_simulation

mpirun ./do-my-work

Note: this runscript will probably not work in its current form as different MPI-implementations need different commands to start the application, see below.

The runscript above is a parallel job that asks for 10 cpus and 2 gigabytes of memory per cpu, the scheduler will then make sure these resources are available to the job before it can start. The runscript will be run on the first node in the nodelist assigned to this job and mpirun will take care of launching the parallel programme named do-my-work on all of the cpus assigned to this jobs, possibly on several compute nodes. If you ask for more resources than is possibly available on a node the job will either be rejected at submit time or will never start.

Different kinds of MPI libraries

Since quite a few implementations of the MPI libraries exist, both free and commercial, it is not possible to cover all possible ways to start any MPI-application in this document. The focus will be on the ones that ships with Rocks: OpenMPI and MPICH2.

#!/bin/sh
#PBS -lwalltime=1:00:00
#PBS -lnodes=10
#PBS -lpmem=2gb
#PBS -N parallel_simulation

cd $PBS_O_WORKDIR

. /opt/torque/etc/openmpi-setup.sh

mpirun ./do-my-work

#!/bin/sh
#PBS -lwalltime=1:00:00
#PBS -lnodes=10
#PBS -lpmem=2gb
#PBS -N parallel_simulation

cd $PBS_O_WORKDIR

/opt/mpiexec/bin/mpiexec ./do-my-work

Pre torque-roll 5.3

As the command rocks sync config would overwrite the torque node-list the only way to make node properties persistent was to turn off automatic updates of the node list by editing /etc/torque-roll.conf. This method still works for torque-roll v5.3 and upwards.

Torque-roll 5.3 and onwards

As of torque-roll v5.3 and up node properties can be set using the rocks concept of node attributes with the rocks command line tool. This is best illustrated by an example:

# rocks set host attr compute-0-0  torque_properties fast
# rocks set host attr compute-0-1  torque_properties slow
# rocks report pbsnodes | sh

This method will make the node properties sticky and automatic node list updates will still work.

The node properties will now appear in the node info and users can now submit jobs to only run on either fast or slow nodes:

$ pbsnodes compute-0-0
$ pbsnodes compute-0-1
$ qsub -lnodes=1:fast runscript.sh
$ qsub -lnodes=1:slow runscript.sh

If no flag on the qsub command is given then scheduling will be done as if the node properties were not set.

Each node can have more than one property. Names are separated by commas, for instance:

# rocks set host attr compute-0-0  torque_properties fast,highmem

Maui vs torque

Torque is the resource manager, its task is to collect info about the state of the compute nodes and jobs. Maui is the scheduler, its task is to decide when and where to run the jobs submitted to torque.

Most things can be achieved by modifying /opt/maui/maui.cfg. Maui needs a restart after changing the config file:

service maui restart

Advice: If you can achieve the same thing by changing either torque or maui, use maui. Restarting maui is rather lightweight operation, and seldom causes problems for live systems. Restarting pbs_server can make the system oscillatory for a few minutes as pbs_server needs to contact all pbs_moms to get back in state.

Needed job info

To make the maui scheduler able to make informed decisions on how to prioritize jobs and on what nodes they should be started on it needs info about the jobs. The minimum requirement is the number of cpus and walltime. Information about memory requirements for the job is also useful. For instance:

#PBS -lwalltime=HH:MM:SS
#PBS -lnodes=10:ppn=8
#PBS -lpmem=1gb

Memory handling on linux

torque/maui supports two memory specification types, (p)mem and (p)vmem on linux.

• pmem is not enforced, it is used only as information to the scheduler.
• pvmem is enforced, procs that exceed the limit will be terminated. The pbs_mom daemon limits vmem size by setting the equivalent of ulimit -v on the processes it controls.

It is currently not possible to limit the amount of physical memory a process can allocate on a linux system. One can only limit the amount of virtual memory. Virtual memory is the physical memoroy + swap. Seeman pbs_resources_linux for details.

Tuning the batch system

Torque is installed in /opt/torque. qmgr is the torque management command

Friendly advice: backup your working config before modifying the setup:

# qmgr -c “print server” > /tmp/pbsconfig.txt

Roll back to escape from a messed up system:

# qterm; pbs_server -t create
# qmgr < /tmp/pbsconfig.txt

This will bring you back to where you started. Remark: this will wipe the whole queue setup and all currently queued and running jobs will be lost!

The default batch configuration from the torque-roll is saved in /opt/torque/pbs.default. Do this to get back the original setup that came with the torque-roll:

# qterm; pbs_server -t create
# qmgr < /opt/torque/pbs.default

Prioritizing short jobs

Often it is useful to give shorter jobs higher priority. It is recommended to use the XFACTOR feature in maui rather than torque queues with different priorites.:

XFACTORWEIGHT 1000

XFACTOR is defined as:

XFACTOR=(walltime+queuetime)/walltime

XFACTOR will increase faster for shorter walltimes thus giving higher priorities for short jobs. Depends on users giving reasonable walltime limits.

Prioritizing large jobs (maui)

In a cluster with a diverse mix of jobs it is often desirable to prioritize the large jobs and make the smaller ones fill in the gaps.:

CPUWEIGHT 1000
MEMWEIGHT 100

This should be combined with fairshare to avoid starving users falling outside this prioritization.

Fairshare (maui)

Also known as

“Keeping all users equally unhappy”

Can be done on several levels users, groups.....

Set a threshold:

USERCFG[DEFAULT] FSTARGET=10
FSWEIGHT 100

Users having used more than 10% will get reduced priority and vice versa.

Adjusting your policy

You can play with the weights to fine-tune your scheduling policies:

XFACTORWEIGHT 100
FSWEIGHT 1000
RESWEIGHT 10
CPUWEIGHT 1000
MEMWEIGHT 100

Analyze the prioritization with diagnose -p

Job node distribution

Default is MINRESOURCE Run on the nodes which gives the least unused resources.

Spread or pack?:

NODEALLOCATIONPOLICY PRIORITY

Select the most busy nodes first:

NODECFG[DEFAULT] PRIORITYF=JOBCOUNT

Select the least busy nodes first:

NODECFG[DEFAULT] PRIORITYF=-1.0*JOBCOUNT

Node access policy

Default access policy is SHARED Can choose to limit this to SINGLEJOB or SINGLEUSER, for instance:

NODEACCESSPOLICY SINGLEUSER

Single user access prevents users from stepping on each others toes while allowing good utilization for serial jobs.

Throttling policies

Sometimes one needs to limit the user from taking over the system:

MAXPROC, MAXPE, MAXPS, MAXJOB, MAXIJOB

All can be set for all or individual users and groups:

USERCFG[DEFAULT], USERCFG[UserA] etc.

Debugging and analyzing

Lot of tools:

pbsnodes      -- node status
qstat -f              -- all details of a job
diagnose -n   -- node status from maui
diagnose -p   -- job priority calculation
showres -n    -- job reservation per node
showstart     -- obvious
checkjob/checknode – also pretty obvious..