PyFR

PyFR is a Python code for solving high-order computational fluid dynamics problems on unstructured grids. It leverages symbolic manipulation and runtime code generation in order to run different high-performance backends on a variety of hardware platforms. The characteristics of Flux Reconstruction make the method suitable for efficient execution on modern streaming architectures, and PyFR has been demonstrated to achieve good portability 1 and scalability on some of the world’s most powerful HPC systems: most notably, a contribution based on PyFR was selected as one of the finalists for the 2016 ACM Gordon Bell Prize 2.

Test case

As test case, we select the 2D Euler vortex example bundled with the PyFR source code. We run the example on 4 nodes with GPU acceleration enabled. We do not measure performance with this test and we do not make comparisons with a native implementation.

Running the container

We assume that a host scratchpad directory is configured to be mounted automatically inside the container by Sarus, and is accessible at the path defined by $SCRATCH. Documentation can be found on this dedicated page: Mounting custom files and directories into the container. For instance: --mount=type=bind,source=$SCRATCH,destination=$SCRATCH

First step is to use an interactive container to prepare the simulation data:

# Launch interactive container
srun -C gpu -N1 -t10 --pty sarus run --tty \
     ethcscs/pyfr:1.8.0-cuda9.2-ubuntu16.04 bash

then within the container, prepare the data:

 1#!/bin/bash
 2
 3# From the container:
 4## copy the example data to the scratch directory
 5mkdir $SCRATCH/pyfr/
 6cd PyFR-1.8.0/examples/
 7cp -r euler_vortex_2d/ $SCRATCH/pyfr/euler_vortex_2d
 8
 9## Convert mesh data to PyFR format
10cd $SCRATCH/pyfr/euler_vortex_2d
11pyfr import euler_vortex_2d.msh euler_vortex_2d.pyfrm
12
13## Partition the mesh and exit the container
14pyfr partition 4 euler_vortex_2d.pyfrm .

and terminate the interactive session:

exit

Now that the data is ready, we can launch the multi-node simulation. Notice that we use the --pty option to srun in order to visualize and update correctly PyFR’s progress bar (which we request with the -p option):

srun -C gpu -N4 -t1 --pty sarus run \
    --mpi \
    ethcscs/pyfr:1.8.0-cuda9.2-ubuntu16.04 \
    pyfr run -b cuda -p \
    $SCRATCH/pyfr/euler_vortex_2d/euler_vortex_2d.pyfrm \
    $SCRATCH/pyfr/euler_vortex_2d/euler_vortex_2d.ini

A typical output will look like:

100.0% [===========================>] 100.00/100.00 daint: 00:00:29 rem: 00:00:00

Container image and Dockerfile

The container image ethcscs/pyfr:1.8.0-cuda9.2-ubuntu16.04 (based on Nvidia cuda/9.2) used for this test case can be pulled from CSCS DockerHub or be rebuilt with this Dockerfile:

 1FROM nvidia/cuda:9.2-devel-ubuntu16.04
 2
 3LABEL com.pyfr.version="1.8.0"
 4LABEL com.python.version="3.5"
 5
 6# Install system dependencies
 7# Metis is a library for mesh partitioning:
 8# http://glaros.dtc.umn.edu/gkhome/metis/metis/overview
 9RUN apt-get update && apt-get install -y   \
10        unzip                       \
11        wget                        \
12        build-essential             \
13        gfortran-5                  \
14        strace                      \
15        realpath                    \
16        libopenblas-dev             \
17        liblapack-dev               \
18        python3-dev                 \
19        python3-setuptools          \
20        python3-pip                 \
21        libhdf5-dev                 \
22        libmetis-dev                \
23        --no-install-recommends  && \
24    rm -rf /var/lib/apt/lists/*
25
26# Install MPICH 3.1.4
27RUN wget -q http://www.mpich.org/static/downloads/3.1.4/mpich-3.1.4.tar.gz && \
28    tar xvf mpich-3.1.4.tar.gz                       && \
29    cd mpich-3.1.4                                   && \
30    ./configure --disable-fortran --prefix=/usr      && \
31    make -j$(nproc)                                  && \
32    make install                                     && \
33    cd ..                                            && \
34    rm -rf mpich-3.1.4.tar.gz mpich-3.1.4            && \
35    ldconfig
36
37# Create new user
38RUN useradd docker
39WORKDIR /home/docker
40
41# Install Python dependencies
42RUN pip3 install numpy>=1.8         \
43                 pytools>=2016.2.1  \
44                 mako>=1.0.0        \
45                 appdirs>=1.4.0     \
46                 mpi4py>=2.0     && \
47    pip3 install pycuda>=2015.1     \
48                 h5py>=2.6.0     && \
49    wget -q -O GiMMiK-2.1.tar.gz    \
50        https://github.com/vincentlab/GiMMiK/archive/v2.1.tar.gz && \
51    tar -xvzf GiMMiK-2.1.tar.gz  && \
52    cd GiMMiK-2.1                && \
53    python3 setup.py install     && \
54    cd ..                        && \
55    rm -rf GiMMiK-2.1.tar.gz GiMMiK-2.1
56
57# Set base directory for pyCUDA cache
58ENV XDG_CACHE_HOME /tmp
59
60# Install PyFR
61RUN wget -q -O PyFR-1.8.0.zip http://www.pyfr.org/download/PyFR-1.8.0.zip && \
62    unzip -qq PyFR-1.8.0.zip     && \
63    cd PyFR-1.8.0                && \
64    python3 setup.py install     && \
65    cd ..                        && \
66    rm -rf PyFR-1.8.0.zip
67
68CMD ["pyfr --help"]

Used OCI hooks

  • NVIDIA Container Runtime hook

  • Native MPI hook (MPICH-based)

References

1

F.D. Witherden, B.C. Vermeire, P.E. Vincent,Heterogeneous computing on mixed unstructured grids with PyFR, Computers & Fluids, Volume 120, 2015, Pages 173-186, ISSN 0045-7930, https://doi.org/10.1016/j.compfluid.2015.07.016

2

P. Vincent, F. Witherden, B. Vermeire, J. S. Park and A. Iyer, “Towards Green Aviation with Python at Petascale”, SC ‘16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Salt Lake City, UT, 2016, pp. 1-11. https://doi.org/10.1109/SC.2016.1