Using enroot on the HPC#

Introduction to enroot#

When setting up the correct versions of libraries, compilers and toolchains on the HPC is tedious and cumbersome, you might want to harness the power of containerization. You might be familiar with docker.

enroot is NVIDIA's implementation of a containerization environment that works natively with docker containers, including imports from dockerhub.

Loading enroot and creating containers#

To use enroot you need to load:
- codes/enroot-3.2.0

The syntax is quite simple:
- enroot import docker://DOCKERHUB_REPO_NAME
- enroot create --name CONTAINER_NAME DOCKERHUB_REPO_NAME.sqsh
- enroot start --rw --root CONTAINER_NAME

Please note that forward slashes (/) in DockerHub repo names are replaced by the (+) symbol when the SquashFS image is imported.

Creating a Job Script#

When using enroot containers, the same policies apply regarding NOT running computations on the master node, but instead submitting jobs to the scheduler to run computations on one of the compute nodes. In order to do this, we will write a job-script.

  1. Create a container as mentioned in the previous section, and start it
  2. Download all the required dependencies, packages and repositories as required by the tool of your choice
  3. If your tool also supports MPI, install the following packages (these are the names for Ubuntu, the exact name might differ for your container of choice):
  4. openmpi-bin
  5. libopenmpi-dev
  6. Having set up your container, exit out of it, back to your HPC user
  7. Write a job-script as follows:
#! /bin/bash
#PBS -N HelloWorld
#PBS -o out.log
#PBS -e err.log
#PBS -l ncpus=20
#PBS -q cpu

module load codes/enroot-3.2.0
enroot start --rw CONTAINER_NAME
cd /absolute/directory/to/computation/program
./program_executable --program-flags

If your program supports MPI, make sure you compile/install the correct MPI-enabled version on the container, and add the correct mpirun command in the job script. Currently, the enroot installation only supports MPI on a single node (utilizing all the cores), and not MPI across nodes.