PyFR¶
PyFR is a Python code for solving high-order computational fluid dynamics problems on unstructured grids. It leverages symbolic manipulation and runtime code generation in order to run different high-performance backends on a variety of hardware platforms. The characteristics of Flux Reconstruction make the method suitable for efficient execution on modern streaming architectures, and PyFR has been demonstrated to achieve good portability [1] and scalability on some of the world’s most powerful HPC systems: most notably, a contribution based on PyFR was selected as one of the finalists for the 2016 ACM Gordon Bell Prize [2].
Test case¶
As test case, we select the 2D Euler vortex example bundled with the PyFR source code. We run the example on 4 nodes with GPU acceleration enabled. We do not measure performance with this test and we do not make comparisons with a native implementation.
Running the container¶
We assume that a host scratchpad directory is configured to be mounted
automatically inside the container by Sarus, and is accessible at the path
defined by $SCRATCH
. Documentation can be found on this dedicated page:
Mounting custom directories into the container. For instance:
--mount=type=bind,source=$SCRATCH,destination=$SCRATCH
First step is to use an interactive container to prepare the simulation data:
# Launch interactive container
srun -C gpu -N1 -t10 --pty sarus run --tty \
ethcscs/pyfr:1.8.0-cuda9.2-ubuntu16.04 bash
then within the container, prepare the data:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | #!/bin/bash
# From the container:
## copy the example data to the scratch directory
mkdir $SCRATCH/pyfr/
cd PyFR-1.8.0/examples/
cp -r euler_vortex_2d/ $SCRATCH/pyfr/euler_vortex_2d
## Convert mesh data to PyFR format
cd $SCRATCH/pyfr/euler_vortex_2d
pyfr import euler_vortex_2d.msh euler_vortex_2d.pyfrm
## Partition the mesh and exit the container
pyfr partition 4 euler_vortex_2d.pyfrm .
|
and terminate the interactive session:
exit
Now that the data is ready, we can launch the multi-node simulation. Notice that
we use the --pty
option to srun
in order to visualize and update correctly
PyFR’s progress bar (which we request with the -p
option):
srun -C gpu -N4 -t1 --pty sarus run \
--mpi \
ethcscs/pyfr:1.8.0-cuda9.2-ubuntu16.04 \
pyfr run -b cuda -p \
$SCRATCH/pyfr/euler_vortex_2d/euler_vortex_2d.pyfrm \
$SCRATCH/pyfr/euler_vortex_2d/euler_vortex_2d.ini
A typical output will look like:
100.0% [===========================>] 100.00/100.00 daint: 00:00:29 rem: 00:00:00
Container image and Dockerfile¶
The container image ethcscs/pyfr:1.8.0-cuda9.2-ubuntu16.04 (based on Nvidia
cuda/9.2) used for this test case can be pulled from CSCS DockerHub or be rebuilt with this
Dockerfile
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | FROM nvidia/cuda:9.2-devel-ubuntu16.04
LABEL com.pyfr.version="1.8.0"
LABEL com.python.version="3.5"
# Install system dependencies
# Metis is a library for mesh partitioning:
# http://glaros.dtc.umn.edu/gkhome/metis/metis/overview
RUN apt-get update && apt-get install -y \
unzip \
wget \
build-essential \
gfortran-5 \
strace \
realpath \
libopenblas-dev \
liblapack-dev \
python3-dev \
python3-setuptools \
python3-pip \
libhdf5-dev \
libmetis-dev \
--no-install-recommends && \
rm -rf /var/lib/apt/lists/*
# Install MPICH 3.1.4
RUN wget -q http://www.mpich.org/static/downloads/3.1.4/mpich-3.1.4.tar.gz && \
tar xvf mpich-3.1.4.tar.gz && \
cd mpich-3.1.4 && \
./configure --disable-fortran --prefix=/usr && \
make -j$(nproc) && \
make install && \
cd .. && \
rm -rf mpich-3.1.4.tar.gz mpich-3.1.4 && \
ldconfig
# Create new user
RUN useradd docker
WORKDIR /home/docker
# Install Python dependencies
RUN pip3 install numpy>=1.8 \
pytools>=2016.2.1 \
mako>=1.0.0 \
appdirs>=1.4.0 \
mpi4py>=2.0 && \
pip3 install pycuda>=2015.1 \
h5py>=2.6.0 && \
wget -q -O GiMMiK-2.1.tar.gz \
https://github.com/vincentlab/GiMMiK/archive/v2.1.tar.gz && \
tar -xvzf GiMMiK-2.1.tar.gz && \
cd GiMMiK-2.1 && \
python3 setup.py install && \
cd .. && \
rm -rf GiMMiK-2.1.tar.gz GiMMiK-2.1
# Set base directory for pyCUDA cache
ENV XDG_CACHE_HOME /tmp
# Install PyFR
RUN wget -q -O PyFR-1.8.0.zip http://www.pyfr.org/download/PyFR-1.8.0.zip && \
unzip -qq PyFR-1.8.0.zip && \
cd PyFR-1.8.0 && \
python3 setup.py install && \
cd .. && \
rm -rf PyFR-1.8.0.zip
CMD ["pyfr --help"]
|
Used OCI hooks¶
- NVIDIA Container Runtime hook
- Native MPI hook (MPICH-based)
References
[1] | F.D. Witherden, B.C. Vermeire, P.E. Vincent,Heterogeneous computing on mixed unstructured grids with PyFR, Computers & Fluids, Volume 120, 2015, Pages 173-186, ISSN 0045-7930, https://doi.org/10.1016/j.compfluid.2015.07.016 |
[2] | P. Vincent, F. Witherden, B. Vermeire, J. S. Park and A. Iyer, “Towards Green Aviation with Python at Petascale”, SC ‘16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Salt Lake City, UT, 2016, pp. 1-11. https://doi.org/10.1109/SC.2016.1 |