---
name: "srun_cr"
section: "1"
platform: "linux"
package: "slurm-client"
version: "16.05.9-1+deb9u4"
release: "Debian 9 (Stretch)"
platforms: ["linux"]
url: "https://new.carta.tech/srun_cr"
---

# srun\_cr(1)

run parallel jobs with checkpoint/restart support

## NAME

srun\_cr - run parallel jobs with checkpoint/restart support

## SYNOPSIS

**srun\_cr** \[*OPTIONS*...\]

## DESCRIPTION

The design of **srun\_cr** is inspired by **mpiexec\_cr** from MVAPICH2 and **cr\_restart** form BLCR. It is a wrapper around the **srun** command to enable batch job checkpoint/restart support when used with Slurm's **checkpoint/blcr** plugin.

## OPTIONS

The **srun\_cr** execute line options are identical to those of the **srun** command. See "man srun" for details.

## DETAILS

After initialization, **srun\_cr** registers a thread context callback function. Then it forks a process and executes "cr\_run --omit srun" with its arguments. **cr\_run** is employed to exclude the **srun** process from being dumped upon checkpoint. All catchable signals except SIGCHLD sent to **srun\_cr** will be forwarded to the child **srun** process. SIGCHLD will be captured to mimic the exit status of **srun** when it exits. Then **srun\_cr** loops waiting for termination of tasks being launched from **srun**.

The step launch logic of Slurm is augmented to check if **srun** is running under **srun\_cr**. If true, the environment variable **SLURM\_SRUN\_CR\_SOCKET** should be present, the value of which is the address of a Unix domain socket created and listened to be **srun\_cr**. After launching the tasks, **srun** tries to connect to the socket and sends the job ID, step ID and the nodes allocated to the step to **srun\_cr**.

Upon checkpoint, srun\_cr checks to see if the tasks have been launched. If not srun\_cr first forwards the checkpoint request to the tasks by calling the Slurm API **slurm\_checkpoint\_tasks()** before dumping its process context.

Upon restart, **srun\_cr** checks to see if the tasks have been previously launched and checkpointed. If true, the environment variable SLURM\_RESTART\_DIR is set to the directory of the checkpoint image files of the tasks. Then **srun** is forked and executed again. The environment variable will be used by the **srun** command to restart execution of the tasks from the previous checkpoint.

## COPYING

Copyright (C) 2009 National University of Defense Technology, China. Produced at National University of Defense Technology, China (cf, DISCLAIMER).

This file is part of Slurm, a resource management program. For details, see \<https://slurm.schedmd.com/\>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.

## SEE ALSO

[**srun**(1)](/srun/1.md)