Configuration file reference
Configuration file entries
securityChecks (bool, REQUIRED)
Enable/disable runtime security checks to verify that security critical files are not tamperable by non-root users. Disabling this may be convenient when rapidly iterating over test and development installations. It is strongly recommended to keep these checks enabled for production deployments. Refer to the section about security requirements for more details about these checks.
Recommended value: true
OCIBundleDir (string, REQUIRED)
Absolute path to where Sarus will generate an OCI bundle, from which a container
will be created. An OCI bundle is composed by a config.json configuration file
and a directory which will form the root filesystem of the container. The
OCIBundleDir
directory must satisfy the security requirements for critical files and directories.
Recommended value: /var/sarus/OCIBundleDir
rootfsFolder (string, REQUIRED)
The name Sarus will assign to the directory inside the OCI bundle; this
directory will become the root filesystem of the container. The value must be
a valid filename and not a path (that is, /
characters are not allowed).
Recommended value: rootfs
prefixDir (string, REQUIRED)
Absolute path to the base directory where Sarus has been installed. This path is used to find all needed Sarus-specific utilities.
Example value: /opt/sarus/<version>
hooksDir (string, OPTIONAL)
Absolute path to the directory containing the OCI hook JSON configuration files. See Configuring Sarus for OCI hooks. Sarus will process the JSON files in hooksDir and generate the configuration file of the OCI bundle accordingly. The hooks will effectively be called by the OCI-compliant runtime specified by runcPath.
Example value: /opt/sarus/<version>/etc/hooks.d
tempDir (string, REQUIRED)
Absolute path to the directory where Sarus will create temporary files and folders for various purposes, like unpacking filesystem layers when pulling and loading images.
Recommended value: /tmp
localRepositoryBaseDir (string, REQUIRED)
Absolute base path to individual user directories, where Sarus will create
(if necessary) and access local image repositories. The repositories will be
located in <localRepositoryBaseDir>/<user name>/.sarus
.
Example value: /home
centralizedRepositoryDir (string, OPTIONAL)
Absolute path to where Sarus will create (if necessary) and access the system-wide centralized image repository. This repository is intended to satisfy use cases where an image can only be broadcasted to the users, without allowing them to pull the image directly (e.g. images that cannot be redistributed due to licensing agreements).
The centralized repository is meant to be read-only for regular users, and its contents should be modifiable only by the system administrators.
Example value: /var/sarus/centralized_repository
skopeoPath (string, REQUIRED)
Absolute path to a trusted skopeo
binary, which will be used to pull images
from container registries or load them from local files.
umociPath (string, REQUIRED)
Absolute path to a trusted umoci
binary, which will be used to unpack image
contents before converting them to SquashFS format.
mksquashfsPath (string, REQUIRED)
Absolute path to trusted mksquashfs
binary.
This executable must satisfy the security requirements for critical files and directories.
mksquashfsOptions (string, OPTIONAL)
String with whitespace-separated command line options which will be passed to
the binary at mksquashfsPath when
creating SquashFS-based images.
This parameter can be used, for example, to control the compression algorithm,
the compression level, or the number of processors used by mksquashfs
.
Please refer to the help message or manpage of your mksquashfs
installation
for the exact list of options available.
The following online manpage can serve as a general reference:
mksquashfs(1).
initPath (string, REQUIRED)
Absolute path to trusted init process static binary which will launch the
user-specified applications within the container when the --init
option
to sarus run is used.
This executable must satisfy the security requirements for critical files and directories.
By default, within the container Sarus only executes the user-specified application, which is assigned PID 1. The PID 1 process has unique features in Linux: most notably, the process will ignore signals by default and zombie processes will not be reaped inside the container (see [1] , [2] for further reference).
Running the container application through an init system provides a solution for signaling container applications or reaping processes of long-running containers.
The standalone package of Sarus uses tini as its default init process.
Warning
Some HPC applications may be subject to performance losses when run with an init process. Our internal benchmarking tests with tini showed overheads of up to 2%.
runcPath (string, REQUIRED)
Absolute path to trusted OCI-compliant runtime binary, which will be used by Sarus to spawn the actual low-level container process. This executable must satisfy the security requirements for critical files and directories.
ramFilesystemType (string, REQUIRED)
The type of temporary filesystem Sarus will use for setting up the base VFS
layer for the container. Must be either tmpfs
or ramfs
.
A filesystem of this type is created inside a dedicated mount namespace unshared by Sarus for each container. The temporary filesystem thus generated will be used as the location of the OCI bundle, including the subsequent mounts (loop, overlay and, if requested, bind) that will form the container's rootfs. The in-memory and temporary nature of this filesystem helps with performance and complete cleanup of all container resources once the Sarus process exits.
Warning
When running on Cray Compute Nodes (CLE 5.2 and 6.0), tmpfs
will not work
and ramfs
has to be used instead.
Recommended value: tmpfs
siteMounts (array, OPTIONAL)
List of JSON objects defining filesystem mounts that will be automatically performed from the host system into the container bundle. This is typically meant to make network filesystems accessible within the container but could be used to allow certain other facilities.
Each object in the list must define the following fields:
type
(string): The type of the mount. Currently, onlybind
(for bind-mounts) is supported.source
(string): Absolute path to the host file/directory that will be mounted into the container.destination
(string): Absolute path to where the filesystem will be made available inside the container. If the directory does not exist, it will be created.
Bind mounts
In addition to type
, source
and destination
, bind mounts can optionally
add the following field:
flags
(object, OPTIONAL): Object defining the flags for the bind mount. Can have the following fields:readonly (string, empty value expected): Mount will be performed as read-only.
By default, bind mounts will always be of recursive private
flavor. Refer to the
Linux docs
for more details.
General remarks
siteMounts
are not subject to the limitations of user mounts requested
through the CLI. More specifically, these mounts:
Can specify any path in the host system as source
Can specify any path in the container as destination
It is not recommended to bind things under /usr
or other common critical
paths within containers.
It is OK to perform this under /var
or /opt
or a novel path that your
site maintains (e.g. /scratch
).
siteDevices (array, OPTIONAL)
List of JSON object defining device files which will be automatically mounted from the host filesystem into the container bundle. The devices will also be whitelisted in the container's device cgroup (Sarus disables access to custom device files by default).
Each object in the list supports the following fields:
source
(string, REQUIRED): Absolute path to the device file on the host.destination
(string, OPTIONAL): Absolute path to the desired path for the device in the container. In the absence of this field, the device will be bind mounted at the same path within the container.access
(string, OPTIONAL): Flags defining the the type of access the device will be whitelisted for. Must be a combination of the charactersrwm
, standing for read, write and mknod access respectively; the characters may come in any order, but must not be repeated. Permissions default torwm
if this field is not present.As highlighted in the related section of the User Guide, Sarus cannot grant more access permissions than those allowed in the host device cgroup.
environment (object, OPTIONAL)
JSON object defining operations to be performed on the environment of the container process. Can have four optional fields:
set
(object): JSON object with fields having string values. The fields represent the key-value pairs of environment variables. The variables defined here will be set in the container environment, possibly replacing any previously existing variables with the same names. This can be useful to inform users applications and scripts that they are running inside a Sarus container.prepend
(object): JSON object with fields having string values. The fields represent the key-value pairs of environment variables. The values will be prepended to the corresponding variables in the container, using a colon as separator. This can be used, for example, to prepend site-specific locations to PATH.append
(object): JSON object with fields having string values. The fields represent the key-value pairs of environment variables. The values will be appended to the corresponding variables in the container, using a colon as separator. This can be used, for example, to append site-specific locations to PATH.unset
(array): List of strings representing environment variable names. Variables with the corresponding names will be unset in the container.
userMounts (object, OPTIONAL)
Normal users have to possibility of requesting custom paths available to them
in the host environment to be mapped to another path inside the container.
This is achieved through the --mount
option of sarus run
.
The userMounts
object offers the means to set limitations for this feature
through two arrays:
notAllowedPrefixesOfPath
: list of strings representing starting paths. The user will not be able to enter these paths or any path under them as a mount destination. Default set to["/etc","/var","/opt/sarus"]
.notAllowedPaths
: list of strings representing exact paths. The user will not be able to enter these paths as a mount destination. Default set to["/opt"]
.
Both these fields and userMounts
itself are optional: remove them to lift
any restriction.
These limitations apply only to mounts requested through the command line;
Mounts entered through siteMounts
are not affected by them.
seccompProfile (string, OPTIONAL)
Absolute path to a file defining a seccomp profile in accordance with the JSON format specified by the OCI Runtime Specification. This profile will be applied to the container process by the OCI runtime.
Seccomp (short for "SECure COMPuting mode") is a Linux kernel feature allowing to filter the system calls which are performed by a given process. It is intended to minimize the kernel surface exposed to an application.
For reference, you may refer to the default seccomp profiles used by Docker, Singularity CE or Podman.
apparmorProfile (string, OPTIONAL)
Name of the AppArmor profile which will be
applied to the container process by the OCI runtime.
The profile must already be loaded in the kernel and listed under
/sys/kernel/security/apparmor/profiles
.
selinuxLabel (string, OPTIONAL)
SELinux label which will be applied to the container process by the OCI runtime.
selinuxMountLabel (string, OPTIONAL)
SELinux label which will be applied to the mounts performed by the OCI runtime into the container.
containersPolicy (object, OPTIONAL)
The containers-policy.json(5) file (formally called the "signature verification policy file") is used by Skopeo and other container tools to define a set of policy requirements (for example trusted keys) which have to be satisfied in order to qualify a container image, or individual signatures of that image, as valid and secure to download.
By default, a user-specific policy is read from ${HOME}/.config/containers/policy.json
;
if such file does not exists, the system-wide /etc/containers/policy.json
is used instead. This system-wide file is usually provided by the
containers-common package.
The containersPolicy
object defines fallback and enforcement options for the
policy file and supports the following fields:
path
(string, REQUIRED): Absolute path to a fallback containers-policy.json(5) file, which will be passed by Sarus to Skopeo in case neither the user-specific nor the system-wide default policy files exist. This allows to use a policy also on systems which don't have the default files present on all nodes. If no default file exists and thecontainersPolicy
parameter is not defined, Sarus throws an error.enforce
(bool, OPTIONAL): If true, always use the policy file atcontainersPolicy/path
, even if any default file exists. This allows to have a Sarus-specific policy different from the one(s) used by other tools on the system.
Important
Sarus installations come with a policy file at <prefixDir>/etc/policy.json
,
which is set as the starting value of containersPolicy/path
.
This policy file is very permissive and is in line with the defaults provided
by package managers for the most popular Linux distributions. It is intended
only as a starting point in case a system does not feature default policy files.
containersRegistries.dPath (string, OPTIONAL)
Absolute path to a containers-registries.d(5)
directory for registries configurations. If defined, this directory will be
used by Skopeo instead of the default ${HOME}/.config/containers/registries.d
or /etc/containers/registries.d
directories.
defaultMPIType (string, OPTIONAL)
Default value which will be set in the com.hooks.mpi.type=<value>
bundle
annotation when the --mpi
option of sarus run is used without
an accompanying --mpi-type
option providing an explicit MPI type.
In practice, this parameter identifies the default MPI-related hook when
multiple hooks are configured with a
com.hooks.mpi.type=<value>
condition, and enables users to just use --mpi
to access the default MPI support mechanism configured for the system.
This parameter also provides compatibility with workflows which do not use the
--mpi-type
option for reasons of portability or legacy.
If this parameter is not defined and the command line does not feature the
--mpi-type
option, the com.hooks.mpi.type=<value>
annotation is not set.
This could fit the case of a system where only a single MPI hook is configured.
Refer to this section for more details about the interaction between command line options, annotations, and hook selection.
repositoryMetadataLockTimings (object, OPTIONAL)
To prevent race conditions when accessing metadata files of local image repositories, Sarus implements a mechanism based on a lock file.
The repositoryMetadataLockTimings
JSON object allows to control the
behavior of Sarus while it is waiting to acquire the lock on the metadata file.
This object can have the following optional fields:
timeoutMs
(integer): Maximum amount of time (in milliseconds) for which Sarus attempts to acquire the lock. If Sarus is still waiting after this time has elapsed, the program times out and exits with an error. The value must be a positive integer. If the parameter is not defined, defaults to 60000 ms (60 seconds).warningMs
: While waiting to acquire the lock, Sarus prints regular warning messages to inform of its status (i.e. access to the metadata file is taking longer but the program is not stalled for another reason) until the timeout is reached or acquisition of the lock is successful. This parameter determines the time interval (in milliseconds) between such warning messages. The value must be a positive integer. If the parameter is not defined, defaults to 10000 ms (10 seconds).
enablePMIxv3Support (bool, OPTIONAL) (experimental)
Enable experimental support for propagating a PMIx v3 context into the container. If support is enabled and use of PMIx is detected during container creation, Sarus will perform additional bind mounts and ensure specific environment variables used by the PMIx v3 interface are set.
Default value: False
Example configuration file
{
"securityChecks": true,
"OCIBundleDir": "/var/sarus/OCIBundleDir",
"rootfsFolder": "rootfs",
"prefixDir": "/opt/sarus/1.7.0",
"hooksDir": "/opt/sarus/1.7.0/etc/hooks.d",
"tempDir": "/tmp",
"localRepositoryBaseDir": "/home",
"centralizedRepositoryDir": "/var/sarus/centralized_repository",
"skopeoPath": "/usr/bin/skopeo",
"umociPath": "/usr/bin/umoci",
"mksquashfsPath": "/usr/sbin/mksquashfs",
"mksquashfsOptions": "-comp gzip -processors 4 -Xcompression-level 6",
"runcPath": "/usr/local/sbin/runc.amd64",
"ramFilesystemType": "tmpfs",
"siteMounts": [
{
"type": "bind",
"source": "/home",
"destination": "/home",
"flags": {}
}
],
"siteDevices": [
{
"source": "/dev/fuse",
"access": "rw"
}
],
"environment": {
"set": {
"VAR_TO_SET_IN_CONTAINER": "value"
},
"prepend": {
"VAR_WITH_LIST_OF_PATHS_IN_CONTAINER": "/path/to/prepend"
},
"append": {
"VAR_WITH_LIST_OF_PATHS_IN_CONTAINER": "/path/to/append"
},
"unset": [
"VAR_TO_UNSET_IN_CONTAINER_0",
"VAR_TO_UNSET_IN_CONTAINER_1"
]
},
"userMounts": {
"notAllowedPrefixesOfPath": [
"/etc",
"/var",
"/opt/sarus"
],
"notAllowedPaths": [
"/opt"
]
},
"seccompProfile": "/opt/sarus/1.7.0/etc/seccomp/default.json",
"apparmorProfile": "sarus-default",
"selinuxLabel": "system_u:system_r:svirt_sarus_t:s0:c124,c675",
"selinuxMountLabel": "system_u:object_r:svirt_sarus_file_t:s0:c715,c811"
"containersPolicy": {
"path": "/opt/sarus/1.7.0/etc/policy.json",
"enforce": false
},
"containersRegistries.dPath": "/opt/sarus/1.7.0/etc/registries.d"
"defaultMPIType": "mpich",
"repositoryMetadataLockTimings": {
"timeoutMs": 120000,
"warningMs": 15000
}
}