SLURM_CONF
The location of the Slurm configuration file.
Display the status information of a running job/step.
sstat [OPTIONS ...]
Status information for running jobs invoked with Slurm.
The sstat command displays job status information for your analysis. The sstat command displays information pertaining to CPU, Task, Node, Resident Set Size (RSS) and Virtual Memory (VM). You can tailor the output with the use of the --fields= option to specify the fields to be shown.
For the root user, the sstat command displays job status data for any job running on the system.
For the non-root user, the sstat output is limited to the user's jobs.
NOTE: The sstat command requires that the jobacct_gather plugin be installed and operational.
NOTE: Availability of metrics rely on the jobacct_gather plugin used. For example the jobacct_gather/cgroup in combination with cgroup/v2 does not provide Virtual Memory metrics due to limitations in the kernel cgroups interfaces and will show a 0 for the related fields.
-a, --allsteps
Print all steps for the given job(s) when no step is specified.
-o, --format, --fields
Comma separated list of fields. (use '--helpformat' for a list of available fields).
NOTE: When using the format option for listing various fields you can put a %NUMBER afterwards to specify how many characters should be printed.
i.e. format=name%30 will print 30 characters of field name right justified. A -30 will print 30 characters left justified.
-h, --help
Displays a general help message.
-e, --helpformat
Print a list of fields that can be specified with the '--format' option.
-j, --jobs
Format is <job(.step)>. Stat this job step or comma-separated list of job steps. Jobs may be specified by numeric job ID or by SLUID. This option is required. When no step is specified, only the lowest numbered normal step (excluding batch, extern, etc.) is displayed. Use the --allsteps flag to display all running steps for the given job(s). NOTE: A step id of 'batch' will display the information about the batch step. NOTE: A step id of 'extern' will display the information about the extern step. This step is only available when using PrologFlags=contain
--noconvert
Don't convert units from their original type (e.g. 2048M won't be converted to 2G).
-n, --noheader
No heading will be added to the output. The default action is to display a header.
-p, --parsable
output will be '|' delimited with a '|' at the end
-P, --parsable2
output will be '|' delimited without a '|' at the end
-i, --pidformat
Predefined format to list the pids running for each job step. (JobId,Nodes,Pids)
--usage
Display a command usage summary.
-v, --verbose
Primarily for debugging purposes, report the state of various variables during processing.
-V, --version
Print version.
Descriptions of each field option can be found below. Note that the Ave*, Max* and Min* accounting fields look at the values for all the tasks of each step in a job and return the average, maximum or minimum values of the task for that job step. For example, for MaxRSS, the returned value is the maximum memory consumption seen by one of the tasks of the step, and MaxRSSTask shows which task it is.
AllocTRESAllocated TRES of all tasks in job.
AveCPUAverage (system + user) CPU time of all tasks in job.
AveCPUFreqAverage weighted CPU frequency of all tasks in job, in kHz.
AveDiskReadAverage number of bytes read by all tasks in job.
AveDiskWriteAverage number of bytes written by all tasks in job.
AvePagesAverage number of page faults of all tasks in job.
AveRSSAverage resident set size of all tasks in job.
AveVMSizeAverage Virtual Memory size of all tasks in job.
ConsumedEnergyTotal energy consumed by all tasks in job, in joules. Note: Only in case of exclusive job allocation this value reflects the jobs' real energy consumption.
JobIDThe number of the job or job step. It is in the form: job.jobstep
MaxDiskReadMaximum number of bytes read by all tasks in job.
MaxDiskReadNodeThe node on which the maxdiskread occurred.
MaxDiskReadTaskThe task ID where the maxdiskread occurred.
MaxDiskWriteMaximum number of bytes written by all tasks in job.
MaxDiskWriteNodeThe node on which the maxdiskwrite occurred.
MaxDiskWriteTaskThe task ID where the maxdiskwrite occurred.
MaxPagesMaximum number of page faults of all tasks in job.
MaxPagesNodeThe node on which the maxpages occurred.
MaxPagesTaskThe task ID where the maxpages occurred.
MaxRSSMaximum resident set size of all tasks in job.
MaxRSSNodeThe node on which the maxrss occurred.
MaxRSSTaskThe task ID where the maxrss occurred.
MaxVMSizeMaximum Virtual Memory size of all tasks in job.
MaxVMSizeNodeThe node on which the maxvsize occurred.
MaxVMSizeTaskThe task ID where the maxvsize occurred.
MinCPUMinimum (system + user) CPU time of all tasks in job.
MinCPUNodeThe node on which the mincpu occurred.
MinCPUTaskThe task ID where the mincpu occurred.
NTasksTotal number of tasks in a job or step.
ReqCPUFreqRequested CPU frequency for the step, in kHz.
The Slurm Lexicographically-sortable Unique Identifier assigned to the job. This value remains constant during the lifetime of the job, even after a resize. Be aware that the accounting database tracks a separate SLUID per accounting record that changes on resize or requeue.
TresUsageInAveTres average usage in by all tasks in job. NOTE: If corresponding TresUsageInMaxTask is -1 the metric is node centric instead of task.
TresUsageInMaxTres maximum usage in by all tasks in job. NOTE: If corresponding TresUsageInMaxTask is -1 the metric is node centric instead of task.
TresUsageInMaxNodeNode for which each maximum TRES usage out occurred.
TresUsageInMaxTaskTask for which each maximum TRES usage out occurred.
TresUsageOutAveTres average usage out by all tasks in job. NOTE: If corresponding TresUsageOutMaxTask is -1 the metric is node centric instead of task.
TresUsageOutMaxTres maximum usage out by all tasks in job. NOTE: If corresponding TresUsageOutMaxTask is -1 the metric is node centric instead of task.
TresUsageOutMaxNodeNode for which each maximum TRES usage out occurred.
TresUsageOutMaxTaskTask for which each maximum TRES usage out occurred.
Executing sstat sends a remote procedure call to slurmctld. If enough calls from sstat or other Slurm client commands that send remote procedure calls to the slurmctld daemon come in at once, it can result in a degradation of performance of the slurmctld daemon, possibly resulting in a denial of service.
Do not run sstat or other Slurm client commands that send remote procedure calls to slurmctld from loops in shell scripts or other programs. Ensure that programs limit calls to sstat to the minimum necessary for the information you are trying to gather.
Some sstat options may be set via environment variables. These environment variables, along with their corresponding options, are listed below. (Note: Command line options will always override these settings.)
SLURM_CONF
The location of the Slurm configuration file.
SLURM_DEBUG_FLAGS
Specify debug flags for sstat to use. See DebugFlags in the slurm.conf(5) man page for a full list of flags. The environment variable takes precedence over the setting in the slurm.conf.
$ sstat --format=AveCPU,AvePages,AveRSS,AveVMSize,JobID -j 11 25:02.000 0K 1.37M 5.93M 9.0
$ sstat -p --format=AveCPU,AvePages,AveRSS,AveVMSize,JobID -j 11 25:02.000|0K|1.37M|5.93M|9.0|
Copyright (C) 2009 Lawrence Livermore National Security. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
Copyright (C) 2010-2022 SchedMD LLC.
This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.
Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
10 releases since Debian 5.0 (Lenny); the text changed 9 times.
--allsteps since Debian 6.0 (Squeeze);
--fields since Debian 7 (Wheezy);
--format since Debian 7 (Wheezy);
--helpformat since Debian 6.0 (Squeeze);
--jobs since Debian 6.0 (Squeeze);
--noconvert since Debian 9 (Stretch);
--parsable since Debian 6.0 (Squeeze);
--parsable2 since Debian 6.0 (Squeeze);
--pidformat since Debian 7 (Wheezy);
--version since Debian 6.0 (Squeeze);
-P since Debian 6.0 (Squeeze);
-V since Debian 6.0 (Squeeze);
-a since Debian 6.0 (Squeeze);
-e since Debian 6.0 (Squeeze);
-i since Debian 7 (Wheezy);
-j since Debian 6.0 (Squeeze);
-n since Debian 6.0 (Squeeze);
-o since Debian 7 (Wheezy);
-p since Debian 6.0 (Squeeze).
--help-fields until Debian 5.0 (Lenny).