PyFR
PyFR is a Python code for solving high-order computational fluid dynamics problems on unstructured grids. It leverages symbolic manipulation and runtime code generation in order to run different high-performance backends on a variety of hardware platforms. The characteristics of Flux Reconstruction make the method suitable for efficient execution on modern streaming architectures, and PyFR has been demonstrated to achieve good portability [1] and scalability on some of the world's most powerful HPC systems: most notably, a contribution based on PyFR was selected as one of the finalists for the 2016 ACM Gordon Bell Prize [2].
Test case
As test case, we select the 2D Euler vortex example bundled with the PyFR source code. We run the example on 4 nodes with GPU acceleration enabled. We do not measure performance with this test and we do not make comparisons with a native implementation.
Running the container
We assume that a host scratchpad directory is configured to be mounted
automatically inside the container by Sarus, and is accessible at the path
defined by $SCRATCH
. Documentation can be found on this dedicated page:
Mounting custom files and directories into the container. For instance:
--mount=type=bind,source=$SCRATCH,destination=$SCRATCH
First step is to use an interactive container to prepare the simulation data:
# Launch interactive container
srun -C gpu -N1 -t10 --pty sarus run --tty \
ethcscs/pyfr:1.8.0-cuda9.2-ubuntu16.04 bash
then within the container, prepare the data:
1#!/bin/bash
2
3# From the container:
4## copy the example data to the scratch directory
5mkdir $SCRATCH/pyfr/
6cd PyFR-1.8.0/examples/
7cp -r euler_vortex_2d/ $SCRATCH/pyfr/euler_vortex_2d
8
9## Convert mesh data to PyFR format
10cd $SCRATCH/pyfr/euler_vortex_2d
11pyfr import euler_vortex_2d.msh euler_vortex_2d.pyfrm
12
13## Partition the mesh and exit the container
14pyfr partition 4 euler_vortex_2d.pyfrm .
and terminate the interactive session:
exit
Now that the data is ready, we can launch the multi-node simulation. Notice that
we use the --pty
option to srun
in order to visualize and update correctly
PyFR's progress bar (which we request with the -p
option):
srun -C gpu -N4 -t1 --pty sarus run \
--mpi \
ethcscs/pyfr:1.8.0-cuda9.2-ubuntu16.04 \
pyfr run -b cuda -p \
$SCRATCH/pyfr/euler_vortex_2d/euler_vortex_2d.pyfrm \
$SCRATCH/pyfr/euler_vortex_2d/euler_vortex_2d.ini
A typical output will look like:
100.0% [===========================>] 100.00/100.00 daint: 00:00:29 rem: 00:00:00
Container image and Dockerfile
The container image ethcscs/pyfr:1.8.0-cuda9.2-ubuntu16.04 (based on Nvidia
cuda/9.2) used for this test case can be pulled from CSCS DockerHub or be rebuilt with this
Dockerfile
:
1FROM nvidia/cuda:9.2-devel-ubuntu16.04
2
3LABEL com.pyfr.version="1.8.0"
4LABEL com.python.version="3.5"
5
6# Install system dependencies
7# Metis is a library for mesh partitioning:
8# http://glaros.dtc.umn.edu/gkhome/metis/metis/overview
9RUN apt-get update && apt-get install -y \
10 unzip \
11 wget \
12 build-essential \
13 gfortran-5 \
14 strace \
15 realpath \
16 libopenblas-dev \
17 liblapack-dev \
18 python3-dev \
19 python3-setuptools \
20 python3-pip \
21 libhdf5-dev \
22 libmetis-dev \
23 --no-install-recommends && \
24 rm -rf /var/lib/apt/lists/*
25
26# Install MPICH 3.1.4
27RUN wget -q http://www.mpich.org/static/downloads/3.1.4/mpich-3.1.4.tar.gz && \
28 tar xvf mpich-3.1.4.tar.gz && \
29 cd mpich-3.1.4 && \
30 ./configure --disable-fortran --prefix=/usr && \
31 make -j$(nproc) && \
32 make install && \
33 cd .. && \
34 rm -rf mpich-3.1.4.tar.gz mpich-3.1.4 && \
35 ldconfig
36
37# Create new user
38RUN useradd docker
39WORKDIR /home/docker
40
41# Install Python dependencies
42RUN pip3 install numpy>=1.8 \
43 pytools>=2016.2.1 \
44 mako>=1.0.0 \
45 appdirs>=1.4.0 \
46 mpi4py>=2.0 && \
47 pip3 install pycuda>=2015.1 \
48 h5py>=2.6.0 && \
49 wget -q -O GiMMiK-2.1.tar.gz \
50 https://github.com/vincentlab/GiMMiK/archive/v2.1.tar.gz && \
51 tar -xvzf GiMMiK-2.1.tar.gz && \
52 cd GiMMiK-2.1 && \
53 python3 setup.py install && \
54 cd .. && \
55 rm -rf GiMMiK-2.1.tar.gz GiMMiK-2.1
56
57# Set base directory for pyCUDA cache
58ENV XDG_CACHE_HOME /tmp
59
60# Install PyFR
61RUN wget -q -O PyFR-1.8.0.zip http://www.pyfr.org/download/PyFR-1.8.0.zip && \
62 unzip -qq PyFR-1.8.0.zip && \
63 cd PyFR-1.8.0 && \
64 python3 setup.py install && \
65 cd .. && \
66 rm -rf PyFR-1.8.0.zip
67
68CMD ["pyfr --help"]
Used OCI hooks
NVIDIA Container Runtime hook
Native MPI hook (MPICH-based)
References