3 Installation

Instructions for building and installing IPPL.

All the new developments of IPPL are merged into the master branch which can make it potentially unstable from time to time. So if you want a stable and more tested version please checkout the tagged branch corresponding to the latest release (e.g. git checkout tags/IPPL-x.x.x). Otherwise if you want the latest developments go with the master with the above caveat in mind.

3.1 Requirements

CMake
A C++ compilation toolchain (GPU-capable for GPU builds, e.g. nvcc, clang or rocmcc)
MPI (GPU-aware if building for GPUs)

3.1.1 Optional requirements

FFTW
CuFFT

3.2 Compilation

IPPL is a CMake Project and can be configured by passing options in CMake syntax:

cmake <src_dir> -D<option>=<value>

None of the options have to be set explicitly, all have a default.

The relevant options of IPPL are - IPPL_PLATFORMS, can be one of SERIAL, OPENMP, CUDA, "OPENMP;CUDA", default SERIAL - Kokkos_VERSION, default 5.2.0 - IPPL_ENABLE_KOKKOS_KERNELS, default OFF - Heffte_VERSION, default 2.4.0 - If set to MASTER, an additional flag Heffte_COMMIT_HASH can be set, default 9eab7c0eb18e86acaccc2b5699b30e85a9e7bdda - Currently, this is the only compatible commit of Heffte - IPPL_DYL, default OFF - IPPL_ENABLE_SOLVERS, default OFF - IPPL_ENABLE_FFT, default OFF - If IPPL_ENABLE_FFT is set, Heffte_ENABLE_CUDA will default to ON if IPPL_PLATFORMS contains cuda - Otherwise, Heffte_ENABLE_AVX2 is enabled. FFTW has to be enabled explicitly. - Heffte_ENABLE_FFTW, default OFF - IPPL_ENABLE_TESTS, default OFF - IPPL_ENABLE_UNIT_TESTS, default OFF - IPPL_ENABLE_ALPINE, default OFF - IPPL_USE_ALTERNATIVE_VARIANT, default OFF. Can be turned on for GPU builds where the use of the system-provided variant doesn’t work.
- IPPL_ENABLE_SANITIZER, default OFF - IPPL_ENABLE_SCRIPTS, default OFF

Kokkos and Heffte by default will try to use versions that are found on the system. If the system has kokkos@5.2.1 and you set Kokkos_VERSION=5.2.0, CMake’s find_package will consider the system version a match because it is newer than requested. The same applies for Heffte. If the requested Kokkos version is unavailable, or if the installed Kokkos does not provide every backend requested through IPPL_PLATFORMS, IPPL falls back to building Kokkos from source with FetchContent. You can override the variable to checkout any version by setting a git tag/sha/branch such as

cmake -DKokkos_VERSION=git.5.2.0 -DHeffte_VERSION=git.v2.4.1 ...
# or for a very specific version 
cmake -DHeffte_VERSION=git.9eab7c0eb18e86acaccc2b5699b30e85a9e7bdda ...

Note that by default, Kokkos release tags use a format such as 5.2.0 and Heffte git tags (extra v) are of the form vx.x.x (for example, v2.4.1).

3.2.1 Kokkos Kernels

Kokkos Kernels support is optional and disabled by default. Enable it with:

cmake -S <src_dir> -B <build_dir> \
    -DIPPL_ENABLE_KOKKOS_KERNELS=ON

IPPL first searches for an installed Kokkos Kernels package compatible with the selected Kokkos configuration. If none is found, it builds Kokkos Kernels with FetchContent. The following cache variables control the feature:

IPPL_ENABLE_KOKKOS_KERNELS, default OFF: enable Kokkos Kernels and the corresponding IPPL unit test.
KokkosKernels_VERSION, default 5.2.0: requested installed package version. A value such as git.5.2.0 or git.<commit> requests a source build.
IPPL_KOKKOS_KERNELS_HOST, default LAPACKE: select LAPACKE, MKL, or NONE as the host eigenanalysis provider.
IPPL_LAPACK_INTEGER_BYTES, default 4: select the 4-byte LP64 or 8-byte ILP64 integer interface.
IPPL_FETCH_LAPACKE, default ON: build reference BLAS, LAPACK, and LAPACKE when an installed LAPACKE provider cannot be found.
IPPL_LAPACKE_BUILD_JOBS, default 4: number of parallel jobs for the reference LAPACK build. This entry is created only when the fallback is used.
IPPL_LAPACKE_TOOLCHAIN_FILE, default empty: C/Fortran toolchain for the fallback. A suitable toolchain is required when cross-compiling.

The host provider is independent of IPPL_PLATFORMS. CUDA builds enable cuBLAS, cuSOLVER, and cuSPARSE in Kokkos Kernels; HIP builds enable rocBLAS, rocSOLVER, and rocSPARSE. These GPU libraries come from the selected CUDA or ROCm toolkit. Kokkos and Kokkos Kernels do not install them.

For an installed LAPACKE provider, set LAPACKE_ROOT or provide LAPACKE_INCLUDE_DIRS and LAPACKE_LIBRARIES. LAPACKE_LIBRARY_DIRS can be used to resolve non-absolute library names. LAPACKE_LIBRARIES must contain the complete semicolon-separated link line, including static transitive dependencies such as the Fortran runtime. Absolute paths, imported CMake targets, and linker items such as -lm are supported. For oneMKL, select MKL and provide MKL_DIR or add its installation prefix to CMAKE_PREFIX_PATH:

cmake -S <src_dir> -B <build_dir> \
    -DIPPL_ENABLE_KOKKOS_KERNELS=ON \
    -DIPPL_KOKKOS_KERNELS_HOST=MKL \
    -DMKL_DIR=<mkl-cmake-directory>

When LAPACKE is unavailable, the default fallback downloads the SHA256-verified reference LAPACK 3.12.1 sources and builds static position-independent BLAS, LAPACK, and LAPACKE libraries in an isolated host C/Fortran project. Select the Fortran compiler with CMAKE_Fortran_COMPILER or the FC environment variable; select the host C compiler with CC. Set IPPL_FETCH_LAPACKE=OFF to require an installed provider instead.

For offline builds, point FETCHCONTENT_SOURCE_DIR_KOKKOSKERNELS and FETCHCONTENT_SOURCE_DIR_IPPL_REFERENCE_LAPACK at unpacked source trees. An installed Kokkos Kernels package can be selected directly with KokkosKernels_DIR.

Set IPPL_KOKKOS_KERNELS_HOST=NONE when only portable or GPU kernels are needed. This omits host eigenanalysis and its LAPACKE or MKL dependency. Configure checks the selected host provider’s headers, integer size, link line, and, for native builds, a small eigenvalue calculation before generating the IPPL build.

Furthermore, be aware of CMAKE_BUILD_TYPE, which can be either - Release for optimized builds - RelWithDebInfo for optimized builds with debug info (default) - Debug for debug builds (with Sanitizers enabled)

3.2.2 Examples

Download and setup a build directory:

https://github.com/IPPL-framework/ippl
cd ippl
mkdir build
cd build

CMakeUserPresets

In the root IPPL source folder, there is a cmake user presets file which can be used to set some default cmake settings, they may be used in the following way

cmake --preset=release-testing ...

This will set the following variables automatically (exact values may change over time)

        "IPPL_ENABLE_TESTS": "ON",
        "IPPL_ENABLE_UNIT_TESTS": "ON"
        "BUILD_SHARED_LIBS": "ON",
        "CMAKE_BUILD_TYPE": "Release",
        "Kokkos_VERSION_DEFAULT": "5.2.0",
        "Heffte_VERSION_DEFAULT": "2.4.0",
        "IPPL_PLATFORMS": "OPENMP;CUDA",
        "IPPL_ENABLE_FFT": "ON",
        "IPPL_ENABLE_ALPINE": "ON",
        "IPPL_ENABLE_COSMOLOGY": "ON",
        "IPPL_USE_STANDARD_FOLDERS": "OFF"

Users are encouraged to define additional sets of flags and create presets for them.

Serial debug build with tests and a Kokkos version

cmake .. \
    -DCMAKE_BUILD_TYPE=Debug \
    -DCMAKE_CXX_STANDARD=20 \
    -DIPPL_ENABLE_TESTS=True \
    -DKokkos_VERSION=5.2.0

OpenMP release build with alpine and FFTW

cmake .. \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_CXX_STANDARD=20 \
    -DIPPL_ENABLE_FFT=ON \
    -DIPPL_ENABLE_SOLVERS=ON \
    -DIPPL_ENABLE_ALPINE=True \
    -DIPPL_ENABLE_TESTS=ON \
    -DIPPL_PLATFORMS=openmp \
    -DHeffte_ENABLE_FFTW=True

Cuda alpine release build

For Kokkos 5.2.0 builds, IPPL no longer needs to set Kokkos_ENABLE_CUDA_LAMBDA explicitly. If CMAKE_CUDA_ARCHITECTURES is not set, IPPL scans enabled Kokkos_ARCH_* cache entries and derives the corresponding CUDA architecture list for CMake and HeFFTe.

cmake .. \
    -DCMAKE_BUILD_TYPE=Release \
    -DKokkos_ARCH_[architecture]=ON \
    -DCMAKE_CUDA_ARCHITECTURES=<architecture compute capability> \
    -DCMAKE_CXX_STANDARD=20 \
    -DIPPL_ENABLE_FFT=ON \
    -DIPPL_ENABLE_TESTS=ON \
    -DIPPL_USE_ALTERNATIVE_VARIANT=ON \
    -DIPPL_ENABLE_SOLVERS=ON \
    -DIPPL_ENABLE_ALPINE=True \
    -DIPPL_PLATFORMS=cuda

HIP release build (LUMI)

cmake .. \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_CXX_STANDARD=20 \
      -DCMAKE_CXX_COMPILER=hipcc \
      -DBUILD_SHARED_LIBS=ON \
      -DCMAKE_HIP_ARCHITECTURES=gfx90a \
      -DCMAKE_HIP_FLAGS=--offload-arch=gfx90a \
      -DKokkos_ENABLE_DEBUG=OFF \
      -DKokkos_ARCH_ZEN3=ON \
      -DKokkos_ARCH_AMD_GFX90A=ON \
      -DKokkos_ENABLE_HIP=ON \
      -DIPPL_PLATFORMS="HIP;OPENMP" \
      -DIPPL_ENABLE_TESTS=ON \
      -DIPPL_ENABLE_FFT=ON  \
      -DIPPL_ENABLE_SOLVERS=ON \
      -DIPPL_ENABLE_ALPINE=ON \
      -DHeffte_ENABLE_ROCM=ON \
      -DCMAKE_EXE_LINKER_FLAGS="-L/opt/cray/pe/mpich/8.1.28/ofi/amd/5.0/lib -L/opt/cray/pe/mpich/8.1.28/gtl/lib -L/opt/cray/pe/libsci/24.03.0/AMD/5.0/x86_64/lib -L/opt/cray/pe/dsmml/0.3.0/dsmml
/lib -L/opt/cray/xpmem/2.8.2-1.0_5.1__g84a27a5.shasta/lib64 -lsci_amd_mpi -lsci_amd -ldl -lmpi_amd -lmpi_gtl_hsa -ldsmml -lxpmem -L/opt/rocm-6.0.3/lib/lib -L/opt/rocm-6.0.3/lib/lib64 -L/opt/roc
m-6.0.3/lib/llvm/lib"

[architecture] should be the target architecture, e.g. - PASCAL60 - PASCAL61 - VOLTA70 - VOLTA72 - TURING75 - AMPERE80 (PSI GWENDOLEN machine) - AMD_GFX90A (LUMI machine) - HOPPER90 (Merlin7 GPUs)

3.2.3 Enable Scripts

We add IPPL_ENABLE_SCRIPTS=ON/OFF and when enabled, cmake will use configure_file to copy the scripts to the build dir, and replace some strings in them with cmake generated ones with the correct paths/values in. This allows the user to

cmake -DIPPL_ENABLE_SCRIPTS=ON  ....
make LandauDamping
...
-- Scripts configured in /capstor/scratch/cscs/biddisco/build-santis/scripts

then

./scripts/landau/strong-scaling-alps/generate.sh

and the result will be something like

Generating job for node count 004 in /capstor/scratch/cscs/biddisco/build-santis/ippl/strongscaling_landau/nodes_004
Submitted batch job 396349
...
Generating job for node count 256 in /capstor/scratch/cscs/biddisco/build-santis/ippl/strongscaling_landau/nodes_256
Submitted batch job 396355

3.3 Build Instructions for specific sytems

Here we compile links to recipies for easy build on various HPC systems.

3.4 GWENDOLEN (PSI)

Gwendolen is a small Nvidia A100 cluster at PSI, accessible through Merlin 6. To build on this machine, load the following modules:

module load gcc/14.3.0 openmpi/5.0.10_slurm
module load cmake/3.26.3

The Cmake command you can use to build all unit tests, tests, and the alpine mini-apps including all solvers is:

cmake .. -DIPPL_PLATFORMS=CUDA -DCMAKE_BUILD_TYPE=Release -DKokkos_ARCH_AMPERE80=ON -DCMAKE_CXX_STANDARD=20 -DIPPL_ENABLE_FFT=ON -DIPPL_ENABLE_TESTS=ON -DIPPL_ENABLE_SOLVERS=ON -DIPPL_ENABLE_ALPINE=True -DIPPL_ENABLE_UNIT_TESTS=True

3.4.1 MERLIN 7 (PSI)

IPPL build for A100 and HG

3.4.2 ALPS (CSCS)

Start by loading a uenv that contains most of the tools we want. Note that in future an official uenv will be provided in the CSCS uenv repository, but until testing is complete, use the following …

uenv start --view=develop \
/capstor/store/cscs/cscs/csstaff/biddisco/uenvs/opal-x-gh200-mpich-gcc-2025-09-28.squashfs

or, look for a newer one and pick the one with the latest date in the name using

ls -al /capstor/store/cscs/cscs/csstaff/biddisco/uenvs/opal-x-gh200-*.squashfs

At the time of writing, the uenv provides (as well as many other packages)

cmake@4.1.1         ~doc+ncurses+ownlibs~qtgui 
cray-mpich@9.0.0    +cuda+cxi~rocm 
cuda@12.8.1         ~allow-unsupported-compilers~dev 
eigen@3.4.0         ~ipo~nightly~rocm 
fftw@3.3.10         +mpi~openmp~pfft_patches+shared 
gcc@13.4.0          ~binutils+bootstrap~graphite~mold~nvptx~piclibs+profiled+strip 
googletest@1.17.0   ~absl+gmock~ipo+pthreads+shared 
gsl@2.8             ~external-cblas+pic+shared 
h5hut@master        ~fortran+mpi 
hdf5@1.14.6         +cxx~fortran+hl~ipo~java~map+mpi+shared~subfiling+szip~threadsafe+tools api=default 
heffte@2.4.1        +cuda+fftw~fortran~ipo~magma~mkl~python~rocm+shared 
hpctoolkit@2025.0.1 +cuda~docs~level_zero~mpi~opencl+papi~python~rocm~strip+viewer 
hwloc@2.11.1        ~cairo~cuda~gl~level_zero~libudev~libxml2~nvml~opencl+pci~rocm 
kokkos@4.7.00       ~aggressive_vectorization~alloc_async~cmake_lang~compiler_warnings+complex_align+cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hip_relocatable_device_code~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl+openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning+wrapper build_system=cmake build_type=Release cuda_arch=90 cxxstd=20 generator=make intel_gpu_arch=none 
ninja@1.13.0        +re2c

You can check what is inside the uenv by executing the command

# This will show all packages installed by spack (including any ones you might have installed yourself outside of a uenv)
spack find -flv

# use this to only show packages inside the uenv (ie. not any you have installed elsewhere)
spack -C /user-environment/config find -flv

It is important to use the --view=develop when loading the uenv as this sets-up the paths to packages in the spack environment ready for you to use them (without needing to manually spack load xxx packages individually) (In fact it will also add /user-environment/env/develop/ to your CMAKE_PREFIX_PATH) which makes cmake-built packages ‘just work’.

To build, try the following which uses default CMake settings (release-testing) taken from CMakeUserPresets.json (in the root ippl source of the cmake-alps branch - it is not required to use this branch, but cmake support has been cleaned up)

ssh daint
uenv start --view=develop /capstor/scratch/cscs/biddisco/uenvs/gh200-opalxgccmpich-2025-07-23.squashfs

# clone IPPL
mkdir -p $HOME/src/ippl
cd $HOME/src
git clone https://github.com/IPPL-framework/ippl

# (optionally) checkout the cmake-alps branch since it is not yet merged to master
cd $HOME/src/ippl
git remote add biddisco https://github.com/biddisco/ippl.git
git fetch biddisco cmake-alps
git checkout cmake-alps

# create a build dir
mkdir -p $HOME/build/ippl
cd $HOME/build/ippl

# run cmake (note that ninja is available in the uenv and "cmake -G Ninja" can be used)
cmake --preset=release-testing -DCMAKE_INSTALL_PREFIX=$HOME/apps/ippl -DCMAKE_CUDA_ARCHITECTURES=90 $HOME/src/ippl/

3.5 EULER

!Ported from old Doxygen, review required!

This guide outlines the steps to install the IPPL library on the EULER cluster. Before beginning, ensure you are connected to the ETH-VPN to access the cluster.

3.5.1 Connecting to the EULER Cluster

Use SSH to connect to EULER. Replace <username> with your actual username.

ssh -Y <username>@euler.ethz.ch

The -Y flag enables trusted X11 forwarding, necessary for running graphical applications remotely.

3.5.2 Preparing the Environment

Load the New Software Stack: Transition to the new software stack to access the latest dependencies:

env2lmod

Clean the Environment: Ensure no previous modules are loaded to avoid conflicts:

module purge

Load Dependencies: Load the required modules for IPPL:

module load gcc/11.4.0 cmake/3.26.3 cuda/12.1.1 openmpi/4.1.4

3.5.3 Cloning the IPPL Library

Clone the IPPL library from its repository:

git clone https://github.com/IPPL-framework/ippl.git

3.5.4 Building IPPL

Setup a build directory:

cd ippl
mkdir build
cd build

Create build files

Choose from the following options based on your needs. If necessary, you can build multiple versions in separate directories.

Serial Version (for single-node computing)

cmake .. -DCMAKE_CXX_STANDARD=20 -DIPPL_PLATFORMS=SERIAL -DIPPL_ENABLE_SOLVERS=ON -DIPPL_ENABLE_FFT=ON -DIPPL_ENABLE_ALPINE=ON

OpenMP Version (for multi-threaded computing):

cmake .. -DCMAKE_CXX_STANDARD=20 -DIPPL_PLATFORMS=OPENMP -DIPPL_ENABLE_SOLVERS=ON -DIPPL_ENABLE_FFT=ON -DIPPL_ENABLE_ALPINE=ON

Cuda Version (for GPU computing):

cmake .. -DCMAKE_CXX_STANDARD=20 -DIPPL_PLATFORMS=CUDA -DIPPL_ENABLE_SOLVERS=ON -DIPPL_ENABLE_FFT=ON -DIPPL_ENABLE_ALPINE=ON -DIPPL_USE_ALTERNATIVE_VARIANT=OFF -DKokkos_ARCH_[architecture]=ON

[architecture] should be the target architecture, e.g. - PASCAL60 - VOLTA70 - TURING75 - AMPERE80

Compile

make

3.5.5 Testing Your Installation

Launch an interactive job on EULER to test your installation:

srun -n 1 --time=1:00:00 --mem-per-cpu=32g --pty bash

This command allocates one computing node with 32GB of RAM for 60 minutes.

Task : Execute a miniapp in the /alpine folder to verify the installation (don’t forget to compile with Make).