Pyxis being a SPANK plugin, the new command-line arguments it introduces are directly added to srun.

$ srun --help
...
      --container-image=[USER@][REGISTRY#]IMAGE[:TAG]|PATH
                              [pyxis] the image to use for the container
                              filesystem. Can be either a docker image given as
                              an enroot URI, or a path to a squashfs file on the
                              remote host filesystem.

      --container-mounts=SRC:DST[:FLAGS][,SRC:DST...]
                              [pyxis] bind mount[s] inside the container. Mount
                              flags are separated with "+", e.g. "ro+rprivate"

      --container-workdir=PATH
                              [pyxis] working directory inside the container
      --container-name=NAME   [pyxis] name to use for saving and loading the
                              container on the host. Unnamed containers are
                              removed after the slurm task is complete; named
                              containers are not. If a container with this name
                              already exists, the existing container is used and
                              the import is skipped.
      --container-save=PATH   [pyxis] Save the container state to a squashfs
                              file on the remote host filesystem.
      --container-mount-home  [pyxis] bind mount the user's home directory.
                              System-level enroot settings might cause this
                              directory to be already-mounted.

      --no-container-mount-home
                              [pyxis] do not bind mount the user's home
                              directory
      --container-remap-root  [pyxis] ask to be remapped to root inside the
                              container. Does not grant elevated system
                              permissions, despite appearances.

      --no-container-remap-root
                              [pyxis] do not remap to root inside the container
      --container-entrypoint  [pyxis] execute the entrypoint from the container
                              image

      --no-container-entrypoint
                              [pyxis] do not execute the entrypoint from the
                              container image
      --container-writable    [pyxis] make the container filesystem writable
      --container-readonly    [pyxis] make the container filesystem read-only

      --container-env=NAME[,NAME...]
                              [pyxis] names of environment variables to preserve
                              from the host environment

Each command-line argument has a corresponding PYXIS_* environment variable:

Example:

$ export PYXIS_CONTAINER_IMAGE=ubuntu:24.04
$ srun grep PRETTY /etc/os-release
PRETTY_NAME="Ubuntu 24.04.3 LTS"

Command-line arguments always take precedence over environment variables.

$ export PYXIS_CONTAINER_IMAGE=ubuntu:24.04
$ srun --container-image=almalinux:9 grep PRETTY /etc/os-release
PRETTY_NAME="AlmaLinux 9.6 (Sage Margay)"

This argument activates the Pyxis plugin and containerizes the submitted job. If no container registry is specified, the image will be pulled from Docker Hub:

$ srun --container-image=centos grep PRETTY /etc/os-release
PRETTY_NAME="CentOS Linux 8 (Core)"

You can pull the container image from any container registry, like you would do with the docker CLI:

$ srun --container-image nvcr.io/nvidia/pytorch:20.03-py3

You can use a squashfs file (from --container-save or enroot export) by passing its path as the argument:

$ srun --container-image ~/ubuntu.sqsh

If this file is on a shared filesystem, this is is useful for avoiding to pull the same image on all nodes of your cluster.

Pyxis passes the image specification to enroot for import. Enroot supports multiple image URI formats:

• Docker Hub (implicit): IMAGE:TAG or IMAGE@sha256:DIGEST

  $ srun --container-image ubuntu:24.04 hostname
  $ srun --container-image=ubuntu@sha256:d22e4fb389065efa4a61bb36416768698ef6d955fe8a7e0cdb3cd6de80fa7eec hostname

• Custom Registry (# separator): REGISTRY#IMAGE:TAG

  $ srun --container-image nvcr.io#nvidia/pytorch:25.10-py3 hostname

• Custom Registry (/ separator, like docker): REGISTRY/IMAGE:TAG

  $ srun --container-image nvcr.io/nvidia/pytorch:25.10-py3 hostname

• Explicit docker:// protocol:

  $ srun --container-image docker://ubuntu:24.04 hostname

• Local Docker daemon (dockerd://): Pull from a locally-running Docker daemon instead of a remote registry:

  $ srun --container-image dockerd://my-local-image:latest hostname

  Note: Requires Docker daemon to be running on compute nodes.

• Local Podman (podman://): Pull from a locally-running Podman repository:

  $ srun --container-image podman://my-local-image:latest hostname

  Note: Requires Podman to be installed on compute nodes.

• Absolute path:

  $ srun --container-image /lustre/containers/pytorch.sqsh hostname

• Relative path:

  $ srun --container-image ./pytorch.sqsh hostname

This argument can be used to expose folders or files from the host system to the container. It is similar to the -v (or --mount type=bind) argument of docker run.

For instance, to bind-mount the /mnt folder from the host as /data inside the container:

$ srun --container-image ubuntu --container-mounts /mnt:/data ls /data

Using the same syntax, you can also mount files:

$ srun --container-image ubuntu --container-mounts /etc/os-release:/host/os-release cat /host/os-release

If the source and destination are identical, you can use the short-form with a single path:

$ srun --container-image ubuntu --container-mounts /mnt ls /mnt

You can also use relative paths (using the job's current working directory):

$ srun --container-image ubuntu --container-mounts ./config:/root/config cat /root/config

Finally, you can use additional mount flags such as ro (read-only), to prevent the container from unintentionally modifying the content from the host:

$ srun --container-image ubuntu --container-mounts /tmp/config:/root/config:ro sh -c 'echo oops > /root/config'
/usr/bin/sh: 1: cannot create /root/config: Read-only file system

In addition to regular bind mounts, pyxis supports special mount types:

Create a temporary filesystem in memory at a specific location inside the container:

$ srun --container-image ubuntu:24.04 --container-mounts tmpfs:/tmp df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           64G     0  64G   0% /tmp

You can specify tmpfs-specific options, such as huge page support:

$ srun --container-image ubuntu:24.04 --container-mounts tmpfs:/tmp:huge=within_size df -h /tmp

Remove existing mounts from inside the container:

$ srun --container-image=ubuntu:24.04 findmnt /sys
TARGET SOURCE FSTYPE OPTIONS
/sys   sysfs  sysfs  ro,nosuid,nodev,noexec,relatime

$ srun --container-image=ubuntu:24.04 --container-mounts umount:/sys ls -l /sys
total 0

This argument is used to save the state of the container filesystem, in order to reuse it across srun commands. This is similar to docker run --name, and it is used to run or install additional tools required by the application.

# The file utility is not installed by default.
$ srun --container-image=ubuntu:20.04 which file
srun: error: luna-0173: task 0: Exited with exit code 1

# The following command creates a named container with the name "myubuntu", starting from the ubuntu 20.04 image.
$ srun --container-image=ubuntu:20.04 --container-name=myubuntu sh -c 'apt-get update && apt-get install -y file'

# Use the container filesystem created above, you don't need to specify --container-image anymore.
$ srun --container-name=myubuntu which file
/usr/bin/file

If you don't need to add anything to the container, you can also use this argument combined with a no-op command like true, to behave like docker pull and docker create, to prepare the container on all nodes before launching the application.

$ srun --container-image=ubuntu:20.04 --container-name=myubuntu true

If the container is running, --container-name will behave like docker exec. This is particularly useful on the login node of the cluster combined with --jobid, to join a running container without having to ssh to the compute node:

# From a compute node, or inside a sbatch script
$ srun --container-name=myapp --container-mounts /mnt:/data ./myapp

# From the login node
$ srun --overlap --jobid=432788 --container-name=myapp findmnt /data
TARGET SOURCE               FSTYPE OPTIONS
/data  /dev/nvme2n1p2[/mnt] ext4   rw,relatime,errors=remount-ro

As you will land in the same container, this approach can be used to debug or profile your app with gdb, perf_events, strace, etc.

The --container-name argument supports optional flags to control container creation behavior:

--container-name=NAME[:FLAGS]

Available flags:

• auto (default): Reuse the container if it exists, create it if it doesn't
• create: Always create a new container, fail if a container with this name already exists
• exec: Only attach to a running container, fail if the container doesn't exist or isn't running
• no_exec: Never attach to a running container, even if it exists

Ensure a fresh container is created (fail if it already exists):

$ srun --container-image ubuntu:24.04 --container-name=ubuntu:create true
$ srun --container-image ubuntu:24.04 --container-name=ubuntu:create true
pyxis: error: "create" flag was passed to --container-name but the container already exists

Only attach to a running container (useful for exec-ing into a running job):

$ srun --container-image ubuntu:24.04 --container-name=ubuntu true

$ srun --container-name=ubuntu:exec true
pyxis: error: "exec" flag was passed to --container-name but the container is not running

$ srun --container-name=ubuntu sleep 120 &
$ srun --overlap --container-name=ubuntu:exec hostname
ioctl

Prevent attaching to a running container (always start a new container instance):

$ srun --container-image ubuntu:24.04 --container-name=ubuntu:no_exec hostname

Exports the container filesystem to a squashfs file after the job completes. This file can then be passed to --container-image.

This option is useful to avoid storming a container registry with requests when running a large distributed job, all the nodes will pull the image simultaneously (unless the layers are cached already on some nodes).

Instead you can have a single job pull the container image, save it to a parallel filesystem, and then all the nodes can use the squashfs file from this shared filesystem.

$ srun --ntasks=1 --container-image nvcr.io#nvidia/pytorch:20.03-py3 --container-save /lustre/felix/pytorch.sqsh true
$ srun --nodes=128 --container-image /lustre/felix/pytorch.sqsh python train.py

This argument can also be useful to save the state of a container across jobs, as it is not possible with --container-name.

By default, the working directory of the job will be taken from the container image (WORKDIR in a Dockerfile). This argument is equivalent to docker run --workdir and allows to override this path.

$ srun --container-image nvcr.io#nvidia/pytorch:20.03-py3 pwd
/workspace

$ srun --container-image nvcr.io/nvidia/pytorch:20.03-py3 --container-workdir /root pwd
/root

These arguments control whether the user will see themselves as UID 0 (root) or their usual UID when inside the container. This feature relies on the user namespaces feature of the Linux kernel, and thus the container is never granted additional privileges.

$ whoami 
fabecassis

$ srun --container-image ubuntu:20.04 --container-remap-root whoami
root

$ srun --container-image ubuntu:20.04 --no-container-remap-root whoami
fabecassis

Being root inside the container is useful to install packages, or in general for any application that expects to be root (e.g. checks that the UID is 0).

Being yourself inside the container is useful for applications that refuse to run as root (like OpenMPI), or scripts that get confused by the sudden change of UID and home directory.

There is a positive and negative form for this option, as the default behavior (when none of these arguments are used) depends on the pyxis configuration. The default behavior is to remap root, hence --container-remap-root is a no-op in this situation.

These arguments control whether the user's home directory should be mounted inside the container. This can also be achieved with --container-mounts.

If you are root inside the container (see above section), the directory will be mounted to /root:

$ srun --container-image ubuntu --container-mount-home --container-remap-root sh -c 'echo $HOME ; ls $HOME'
/root
...

If your UID is unchanged inside the container, the directory will be mounted at the same location than the host:

$ srun --container-image ubuntu --container-mount-home --no-container-remap-root sh -c 'echo $HOME ; ls $HOME'
/home/fabecassis
...

Mounting your home directory inside of the container is useful if you need to use code or configuration files (such as your .bashrc) stored outside of the container image.

There is a positive and negative form for this option, as the default behavior (when none of these arguments are used) depends on the enroot setting ENROOT_MOUNT_HOME. Mounting the home directory by default can create problems (such as the user's .bashrc overriding environment variables from the container), so it is not recommended.

These arguments control whether the entrypoint defined in the container image is executed (ENTRYPOINT in the Dockerfile).

This is useful when the container image has an entrypoint that appends arguments to an existing binary. This is the pattern described as "allowing that image to be run as though it was that command (and then use CMD as the default flags)" in the docker documentation. When that's the case, pyxis will fail when starting the container, and then suggest to try with --no-container-entrypoint

$ srun --container-entrypoint --container-image myimage true
pyxis: importing docker image ...
pyxis: creating container filesystem ...
pyxis: starting container ...
slurmstepd: error: pyxis: container start failed with error code: 1
slurmstepd: error: pyxis: printing contents of log file ...
...
slurmstepd: error: pyxis: couldn't start container
slurmstepd: error: pyxis: if the image has an unusual entrypoint, try using --no-container-entrypoint

Entrypoints that follow the consistency recommendation (only wrapping the execution of a command) of Docker Hub will work fine, so --container-entrypoint can be used to have the entrypoint executed. Some entrypoints serve a useful purpose such as injecting environment variables or preparing the container filesystem.

There is a positive and negative form for this option, as the default behavior (when none of these arguments are used) depends on the pyxis configuration. The default behavior is to not execute the entrypoint, hence --no-container-entrypoint is a no-op in this situation.

This argument allows you to preserve specific environment variables from the host environment inside the container, overriding any values set by the container image. Variables are passed as a comma-separated list.

This is useful override container defaults with host values.

$ srun --container-image=nvidia/cuda:12.9.1-base-ubuntu24.04 sh -c 'echo $CUDA_VERSION'
12.9.1

$ CUDA_VERSION=42.0.0 srun --container-image=nvidia/cuda:12.9.1-base-ubuntu24.04 --container-env=CUDA_VERSION sh -c 'echo $CUDA_VERSION'
42.0.0

Note: when not using --container-writable / --contaienr-readonly, the default behavior depends on the enroot configuration setting ENROOT_ROOTFS_WRITABLE.

Makes the container filesystem writable, allowing you to modify files and install packages:

$ srun --container-image ubuntu:24.04 --container-writable sh -c 'touch /test && ls -l /test'
-rw-rw-r-- 1 root root 0 Nov 11 11:00 /test

This is useful for installing additional packages at runtime or applications that need to write logs or temporary files.

Note that changes are temporary unless you use --container-save or --container-name to persist the container state.

Note: when not using --container-writable / --contaienr-readonly, the default behavior depends on the enroot configuration setting ENROOT_ROOTFS_WRITABLE.

Makes the container filesystem read-only:

$ srun --container-image ubuntu:24.04 --container-readonly touch /test
pyxis: importing docker image: ubuntu:24.04
pyxis: imported docker image: ubuntu:24.04
/usr/bin/touch: cannot touch '/test': Read-only file system

This is useful for preventing accidental modifications to the container filesystem.

This argument causes pyxis to print the output of the container's entrypoint script to the job's stdout/stderr.

By default, when using --container-entrypoint, the entrypoint runs silently. Use --container-entrypoint-log to see what the entrypoint is doing:

$ srun --container-image nvcr.io/nvidia/pytorch:25.10-py3 --container-entrypoint --container-entrypoint-log true
pyxis: importing docker image: nvcr.io/nvidia/pytorch:25.10-py3
pyxis: imported docker image: nvcr.io/nvidia/pytorch:25.10-py3

=============
== PyTorch ==
=============

NVIDIA Release 25.10 (build 222058775)
PyTorch Version 2.9.0a0+145a3a7
[...]