Showing posts with label 0-script. Show all posts

Wednesday, April 1, 2015

Comparison of PBS and SLURM script

PBS Torque and SLURM scripts are two frameworks for specifying the resource requirements and settings for the job you want to run. For the most part, there are equivalent settings in each script. To help translate from one system to the other, the following table lists examples of equivalent options for PBS and SLURM job scripts.

Please note: SLURM can read PBS Torque script options, but we advise that it is safer to use pure SLURM commands.

Ref: https://vlsci.org.au/documentation/comparison-pbs-and-slurm-script-commands

A PBS script is a standard Unix/Linux shell script that contains a few extra comments at the beginning that specify directives to PBS. These comments all begin with #PBS.

PBS USER GUIDE

Overview

PBS is a job resource manager. A job is defined as a computational task such as computational simulation or data analysis. PBS provides job queuing and execution services in a batch cluster environment.
In the HPC systems, PBS works with the Moab job scheduler. PBS provides job information to Moab and Moab tells PBS which jobs to run and on what compute nodes in a cluster to run the jobs on.

Job Limits

PBS is configured on each system to have a number of separate job queues. There is a default queue on each system that every user has access to. Each funding group has its own queue.
These limits are placed on funding group queues:

**Funding Group Queue Limits**
Limit	Lion-X*
Maximum Walltime (Hours)	96 (default) to 336+
Maximum Processors in Use Per User	No Limit
Maximum Job Size (Processors)	32

Note that the walltime limits placed on funding group queues are arbitrary and can be adjusted at the request of the group's PI.
If you believe you're part of a funding group of a system and you don't know what queue you should be using, please email us at support@ics.psu.edu stating what group you are part of and that you need to know your group queue.
These limits are placed on system default queues:

**System Default Queue Limits**
Limit	Lion-X*	Lion-XK	CyberStar	Clsf
Maximum Walltime (Hours)	24	24	96	No Limit
Maximum Processors in Use Per User	32	No Limit	No Limit	No Limit
Maximum Job Size (Processors)	32	512 (nodes=64:ppn=8)	256 (nodes=32:ppn=8)	128 (nodes=1:ppn=128)

Most queue limits can be checked by running the command qstat -q.

Submitting a Job

Jobs are submitted to a PBS queue so that PBS can dispatch them to be run on one or more of a cluster's compute nodes. There are two main types of PBS jobs:

Non-interactive Batch Jobs: this is the most common PBS job. A job script is created that contains PBS resource requests and the commands necessary to execute the job. The job script is then submitted to PBS to be run non-interactively.
Interactive Batch Jobs: this is a way to get an interactive terminal on one or more of the compute nodes of a cluster. Commands can then be run interactily through that terminal directly on the compute nodes for the duration of the job. Interactive jobs are helpful for such things as program debugging and running many short jobs.

Non-interactive Batch Jobs

There are two steps to running a non-interactive batch job:

Create a PBS Script
A PBS script is a standard Unix/Linux shell script that contains a few extra comments at the beginning that specify directives to PBS. These comments all begin with #PBS. The most important PBS directives are:

**Definition of Important PBS Directives**
PBS Directive	Description
#PBS -l walltime=HH:MM:SS	This directive specifies the maximum walltime (real time, not CPU time) that a job should take. If this limit is exceeded, PBS will stop the job. Keeping this limit close to the actual expected time of a job can allow a job to start more quickly than if the maximum walltime is always requested.
*#PBS -l pmem=SIZEgb*	This directive specifies the maximum amount of physical memory used by any process in the job. For example, if the job would run four processes and each would use up to 2 GB (gigabytes) of memory, then the directive would read #PBS -l pmem=2gb. The default for this directive on Lion-XF and Lion-LSP is 1 GB (gigabyte) of memory. Other Lion clusters do not currently set a default.
#PBS -l nodes=N:ppn=M	This specifies the number of nodes (nodes=N) and the number of processors per node (ppn=M) that the job should use. PBS treats a processor core as a processor, so a system with eight cores per compute node can have ppn=8 as its maximum ppn request. Note that unless a job has some inherent parallelism of its own through something like MPI or OpenMP, requesting more than a single processor on a single node is usually wasteful and can impact the job start time.
#PBS -q queuename	This specifies what PBS queue a job should be submitted to. This is only necessary if a user has access to a special queue. This option can and should be omitted for jobs being submitted to a system's default queue.
#PBS -j oe	Normally when a command runs it prints its output to the screen. This output is often normal output and error output. This directive tells PBS to put both normal output and error output into the same output file.

The following is an example PBS script.

# This is a sample PBS script. It will request 1 processor on 1 node
# for 4 hours.
#   
#   Request 1 processors on 1 node 
#   
#PBS -l nodes=1:ppn=1
#
#   Request 4 hours of walltime
#
#PBS -l walltime=4:00:00
#
#   Request 1 gigabyte of memory per process
#
#PBS -l pmem=1gb
#
#   Request that regular output and terminal output go to the same file
#
#PBS -j oe
#
#   The following is the body of the script. By default,
#   PBS scripts execute in your home directory, not the
#   directory from which they were submitted. The following
#   line places you in the directory from which the job
#   was submitted.
#
cd $PBS_O_WORKDIR
#
#   Now we want to run the program "hello".  "hello" is in
#   the directory that this script is being submitted from,
#   $PBS_O_WORKDIR.
#
echo " "
echo " "
echo "Job started on `hostname` at `date`"
./hello
echo " "
echo "Job Ended at `date`"
echo " "

Note that the above example script is for a non-MPI job. Information on how to write PBS scripts for MPI jobs can be found in the MPI software pages.

Submit the PBS Script to PBS for Execution
Once a PBS script is created, it needs to be submitted to PBS so that it becomes eligible to be run. The command to submit a script to PBS is called qsub. The syntax of qsub is:

qsub scriptfile
The following is an example of using qsub to submit a PBS script called myjob.

% qsub myjob
95.lionxj.rcc.psu.edu
The job script myjob has just been submitted to PBS and has been assigned the Job_ID 95.lionxj.rcc.psu.edu. This Job_ID can later be used to control the job.

Interactive Batch Jobs

Interactive PBS jobs are similar to non-interactive PBS jobs in that they are submitted to PBS via the command qsub. Submitting an interactive PBS job differs from a non-interactive PBS job in that a PBS script is not necessary. All PBS directives can be specified on the command line.
The syntax for qsub for submitting an interactive PBS job is:

qsub -I ... pbs directives ...

The -I flag above tells qsub that this is an interactive job. The following example shows using qsub to submit an interactive job using one processor on one node for four hours.

lionxi:~$ qsub -I -l nodes=1:ppn=1 -l walltime=4:00:00
qsub: waiting for job 1064159.lionxi.rcc.psu.edu to start
qsub: job 1064159.lionxi.rcc.psu.edu ready

lionxi25:~$

There are two things of note here. The first is that the qsub command doesn't exit when run with the interactive -I flag. Instead, it waits until the job is started and gives a prompt on the first compute node assigned to a job. The second thing of note is the prompt lionxi25:~$ - this shows that commands are now being executed on the compute node lionxi25.

Checking Job Status

The command to check job status is qstat. qstat has many options. Some common ones are:

PBS Commands for Checking Job Status

Command Name Description of Command Functionality

qstat Shows the status of all PBS jobs. The time displayed is the CPU time used by the job.

qstat -s Shows the status of all PBS jobs. The time displayed is the walltime used by the job.

qstat -u userid Shows the status all PBS jobs submitted by the user userid. The time displayed is the walltime used by the job.

qstat -n Shows the status all PBS jobs along with a list of compute nodes that the job is running on.

qstat -f jobid Shows detailed information about the job jobid.

**PBS Commands for Checking Job Status**
Command Name	Description of Command Functionality
qstat	Shows the status of all PBS jobs. The time displayed is the CPU time used by the job.
qstat -s	Shows the status of all PBS jobs. The time displayed is the walltime used by the job.
qstat -u userid	Shows the status all PBS jobs submitted by the user userid. The time displayed is the walltime used by the job.
qstat -n	Shows the status all PBS jobs along with a list of compute nodes that the job is running on.
qstat -f jobid	Shows detailed information about the job jobid.

A job can be in several different states. The most common ones are:

**PBS Job States**
State	Meaning
Q	The job is queued and is waiting to start.
R	The job is currently running.
E	The job is currently ending.
H	The job has a user or system hold on it and will not be eligible to run until the hold is removed.

Example: qstat output

lionxj:~$ qstat
Job id        Name           User     Time Use S Queue
------------- -------------- -------- -------- - -----
10.lionxj     sparse         abc123   188:20:2 R lionxj
11.lionxj     test           jwh128   00:00:18 R lionxj-admin
...

Job id: the job's unique indentifier
Name: name of the job
User: user that owns the job
Time Use: CPU time used by the job
S: state of the job
Queue: the queue the job is in

Example: qstat -s output

lionxj:~$ qstat -s

lionxj.rcc.psu.edu: 
                                                        Req'd  Req'd   Elap
Job ID        Username Queue    Jobname  SessID NDS TSK Memory Time  S Time
------------- -------- -------- -------- ------ --- --- ------ ----- - -----
10.lionxj.rcc abc123   lionxj   sparse   5793     4  --    2gb 190:0 R 189:2
    --
11.lionxj.rcc jwh128   lionxj-a test     11946    3  --    --  500:0 R 166:5
    -- 
...

Job id: the job's unique indentifier
Username: user that owns the job
Queue: the queue the job is in
Jobname: the name of the job
NDS: the number of compute nodes the job is using
Req'd Memory: the memory requested for the job
Req'd Time: the walltime requested for the job
S: the state of the job
Elap Time: the elapsed walltime for the job

Deleting a Job

The command to delete a job is qdel. Its syntax is "qdel Job_ID".

PBS Commands for Deleting Jobs

Command Name Description of Command Functionality

qdel Job_ID Deletes the job identified by Job_ID.

qdel $(qselect -u username) Deletes all jobs belonging to user username.

**PBS Commands for Deleting Jobs**
Command Name	Description of Command Functionality
qdel Job_ID	Deletes the job identified by Job_ID.
*qdel $(qselect -u username)*	Deletes all jobs belonging to user username.

Example: deleting a job with Job_ID 10

lionxj:~$ qdel 10
Example: deleting all jobs belonging to user abc123

lionxj:~$ qdel $(qselect -u abc123)

Viewing Job Output

By default PBS will write screen output from a job to the follwing files:

PBS Output Files

Output File Name Contents of Output File

Jobname.oJob_ID This file would contain the non-error output that would normally be written to the screen.

Jobname.eJob_ID This file would contain the error output that would normally be written to the screen.

**PBS Output Files**
Output File Name	Contents of Output File
Jobname.oJob_ID	This file would contain the non-error output that would normally be written to the screen.
Jobname.eJob_ID	This file would contain the error output that would normally be written to the screen.

If the PBS directive #PBS -j oe is used in a PBS script, the non-error and the error output are both written to the Jobname.oJob_ID file.

More Information

More information on PBS and PBS scripts can be found in the man pages for the commands qsub, pbs_resources, qstat, and qdel.

Ref: https://rcc.its.psu.edu/user_guides/system_utilities/pbs/

PBS script

Sample PBS script for serial job

For most applications, you should only need to change items indicated in red. Items shown in amber may also warrant your attention.

#!/bin/sh -f

###############################################################
# #
# Bourne shell script for submitting a serial job to the #
# PBS queue using the qsub command. #
# #
###############################################################

# Remarks: A line beginning with # is a comment.
# A line beginning with #PBS is a PBS directive.
# PBS directives must come first; any directives
# after the first executable statement are ignored.
#

##########################
# #
# The PBS directives #
# #
##########################

# Set the name of the job (up to 15 characters,
# no blank spaces, start with alphanumeric character)

#PBS -N JobName

# By default, the standard output and error streams are sent
# to files in the current working directory with names:
# job_name.osequence_number <- output="" p=""># job_name.esequence_number <- error="" p=""># where job_name is the name of the job and sequence_number
# is the job number assigned when the job is submitted.
# Use the directives below to change the files to which the
# standard output and error streams are sent.

# #PBS -o stdout_file
# #PBS -e stderr_file

# The directive below directs that the standard output and
# error streams are to be merged, intermixed, as standard
# output.

#PBS -j oe

# Specify the maximum cpu and wall clock time. The wall
# clock time should take possible queue waiting time into
# account. Format: hhhh:mm:ss hours:minutes:seconds
# Be sure to specify a reasonable value here.
# If the job does not finish by the time reached,
# the job is terminated.

#PBS -l cput=6:00:00
#PBS -l walltime=6:00:00

# Specify the queue. The CMU cluster currently has three queues:
# "green", "blue", and "red". Jobs submitted to these queues
# will run in cpu-dedicated mode; if all cpu's assigned to the
# queue are occupied with a job, then new jobs are queued and will
# not run until a cpu is freed up. You should take this waiting
# time into account when setting "walltime".

#PBS -q green

# Specify the maximum amount of physical memory required.
# kb for kilobytes, mb for megabytes, gb for gigabytes.
# Take some care in setting this value. Setting it too large
# can result in your job waiting in the queue for sufficient
# resources to become available.

#PBS -l mem=512mb

# PBS can send informative email messages to you about the
# status of your job. Specify a string which consists of
# either the single character "n" (no mail), or one or more
# of the characters "a" (send mail when job is aborted),
# "b" (send mail when job begins), and "e" (send mail when
# job terminates). The default is "a" if not specified.
# You should also specify the email address to which the
# message should be send via the -M option.

# #PBS -m abe
# #PBS -m ae

#PBS -M user_email_address

# Declare the time after which the job is eligible for execution.
# If you wish the job to be immediately eligible for execution,
# comment out this directive. If you wish to run at some time in
# future, the date-time argument format is
# [DD]hhmm
# If the day DD is not specified, it will default to today if the
# time hhmm is in the future, otherwise, it defaults to tomorrow.
# If the day DD is specified as in the future, it defaults to the
# current month, otherwise, it defaults to next month.

# #PBS -a 2215 commented out

# Specify the priority for the job. The priority argument must be
# an integer between -1024 and +1023 inclusive. The default is 0.

# #PBS -p 0

# Specify the number of nodes requested and the
# number of processors per node.

#PBS -l nodes=1:ppn=1

# Define the interval at which the job will be checkpointed,
# if checkpointing is desired, in terms of an integer number
# of minutes of CPU time.

# #PBS -c c=2

##########################################
# #
# Output some useful job information. #
# #
##########################################

echo ------------------------------------------------------
echo -n 'Job is running on node '; cat $PBS_NODEFILE
echo ------------------------------------------------------
echo PBS: qsub is running on $PBS_O_HOST
echo PBS: originating queue is $PBS_O_QUEUE
echo PBS: executing queue is $PBS_QUEUE
echo PBS: working directory is $PBS_O_WORKDIR
echo PBS: execution mode is $PBS_ENVIRONMENT
echo PBS: job identifier is $PBS_JOBID
echo PBS: job name is $PBS_JOBNAME
echo PBS: node file is $PBS_NODEFILE
echo PBS: current home directory is $PBS_O_HOME
echo PBS: PATH = $PBS_O_PATH
echo ------------------------------------------------------

##############################################################
# #
# The prologue script automatically makes a directory #
# on the local disks for you. The name of this directory #
# depends on the job id, but you need only refer to it #
# using ${WORKDIR}. #
# #
##############################################################

SERVER=$PBS_O_HOST
WORKDIR=/scratch/PBS_$PBS_JOBID
SCP=/usr/bin/scp
SSH=/usr/bin/ssh

######################################################################
# #
# To minimize communications traffic, it is best for your job #
# to work with files on the local disk of the compute node. #
# Hence, one needs to transfer files from your permanent home #
# directory tree to the directory ${WORKDIR} automatically #
# created by PBS on the local disk before program execution, #
# and to transfer any important output files from the local #
# disk back to the permanent home directory tree after program #
# execution is completed. #
# #
# There are essentially two ways to achieve this: (1) to use the #
# PBS stagein and stageout utilities, or (2) to manually copy the #
# files by commands in this script. The stagein and stageout #
# features of OpenPBS are somewhat awkward, especially since #
# wildcards and macros in the file lists cannot be used. This #
# method also has some timing issues. Hence, we ask you to use #
# the second method, and to use secure copy (scp) to do the file #
# transfers to avoid NSF bottlenecks. #
# #
######################################################################

#####################################################
# #
# Specify the permanent directory(ies) on the #
# server host. Note that when the job begins #
# execution, the current working directory at #
# the time the qsub command was issued becomes #
# the current working directory of the job. #
# #
#####################################################

PERMDIR=${HOME}/work

SERVPERMDIR=${PBS_O_HOST}:${PERMDIR}

echo server is $SERVER
echo workdir is $WORKDIR
echo permdir is $PERMDIR
echo servpermdir is $SERVPERMDIR
echo ------------------------------------------------------
echo -n 'Job is running on node '; cat $PBS_NODEFILE
echo ------------------------------------------------------
echo ' '
echo ' '

###############################################################
# #
# Transfer files from server to local disk. #
# #
###############################################################

stagein()
{
echo ' '
echo Transferring files from server to compute node
echo Writing files in node directory ${WORKDIR}
cd ${WORKDIR}

${SCP} ${SERVPERMDIR}/input_file .
${SCP} ${SERVPERMDIR}/program_executable .
echo Files in node work directory are as follows:
ls -l
}

############################################################
# #
# Execute the run. Do not run in the background. #
# #
############################################################

runprogram()
{
program_executable < input_file > output_file }

###########################################################
# #
# Copy necessary files back to permanent directory. #
# #
###########################################################

stageout()
{
echo ' '
echo Transferring files from compute nodes to server
echo Writing files in permanent directory ${PERMDIR}
cd ${WORKDIR}

${SCP} output_file ${SERVPERMDIR}
echo Final files in permanent data directory:
${SSH} ${SERVER} "cd ${PERMDIR}; ls -l"
}

#####################################################################
# #
# The "qdel" command is used to kill a running job. It first #
# sends a SIGTERM signal, then after a delay (specified by the #
# "kill_delay" queue attribute (set to 60 seconds), unless #
# overridden by the -W option of "qdel"), it sends a SIGKILL #
# signal which eradicates the job. During the time between the #
# SIGTERM and SIGKILL signals, the "cleanup" function below is #
# run. You should include in this function commands to copy files #
# from the local disk back to your home directory. Note: if you #
# need to transfer very large files which make take longer than #
# 60 seconds, be sure to use the -W option of qdel. #
# #
#####################################################################

early()
{
echo ' '
echo ' ############ WARNING: EARLY TERMINATION #############'
echo ' '
}

trap 'early; stageout' 2 9 15

##################################################
# #
# Staging in, running the job, and staging out #
# were specified above as functions. Now #
# call these functions to perform the actual #
# file transfers and program execution. #
# #
##################################################

stagein
runprogram
stageout

###############################################################
# #
# The epilogue script automatically deletes the directory #
# created on the local disk (including all files contained #
# therein. #
# #
###############################################################

exit

Ref:
http://qcd.phys.cmu.edu/QCDcluster/pbs/run_serial.html

"Ex-ample" of Online Services

Online Services headline animator, feedBurner