Environment module tasks
The ScriptEngine HPC Task package allows interaction with environment modules, often used on HPC systems to configure the user’s environment for installed software packages. This task package supports the two most common module implementations: Lmod (https://lmod.readthedocs.io) and Environment Modules (https://modules.readthedocs.io).
The ScriptEngine tasks in this package allow modules to be loaded or unloaded in SE scripts and can thus modify the environment and available software during the execution of scripts.
Prerequisites
A fairly recent version of either Lmod (source code at https://github.com/TACC/Lmod) or Environment modules (http://modules.sourceforge.net) is needed. In particular, the module version should provide Python3 initialisation scripts.
If, however, the module version installed on an HPC system does not provide Python3 init scripts, it is possible to initialise the tasks from a user-provided initialisation script. This allows to use the ScriptEngine module tasks even on systems with an outdated module system. See The init argument below.
The hpc.module
task
Runs any module command.
Usage:
- hpc.module:
cmd: <COMMAND_NAME>
args: <LIST_OF_ARGS> # optional
Note that no checking is done as to wether the command and arguments are valid! In particular, there is no guarantee that the command will run given the particular module implementation (Environment modules or Lmod) and the version installed on the HPC system. The command name and arguments are passed to the underlying module system and runtime errors reported via ScriptEngine.
For example:
- hpc.module:
cmd: list
will run module list
and write the result to standard output.
If the module command requires arguments, they are given via the task argument
args
. The arguments have to be specified as a list (even if there is only
one):
- hpc.module:
cmd: show
args: [ gcc/10.2 ]
The hpc.module.load
task
This task is provided for convenience, as it allows for a shorter syntax to load
modules (compared to the hpc.module
task using cmd: load
)
Usage:
- hpc.module.load:
names: <MODULE_NAME | LIST_OF_MODULE_NAMES>
Examples:
- hpc.module.load:
names:
- gcc/10.2
- netcdf/4.3.0
If there is only a single module to be loaded, the name can be given without using a list:
- hpc.module.load:
names: git/2.19.3
The init
argument
As mentioned under Prerequisites, the hpc.module
tasks need Python3
initialisation scripts, usually provided by recent versions of the module
systems. Sometimes, however, older module versions are installed on some HPC
systems and Python3 support is missing. What the hpc.module
scripts really
need is an initialisation script, the rest of the implementation usually works
fine even with older module versions. Hence, it is possible to manually provide
the initialisation scripts.
In order to provide a user defined initialisation script, the init
argument
can be added to any of the module tasks (hpc.module
or hpc.module.load
).
Since the initialisation is only done once, the init
argument is only needed
at the first task executed. If the init
argument is present at any
subsequent task, it is ignored. If the init
argument is missing at the first
executed task (and default initialisation does not work) initialisation will
fail.
The init
argument must specify the path at which the initialisation script
can be found, for example:
- hpc.module.load:
init: /home/user/lmod/init/env_modules_python.py
names: gcc/10.2
- hpc.module:
cmd: list
In order to follow which task is initialising the module system and from what
location, run ScriptEngine with se --loglevel debug [...]
.