15  Performance Portability

IPPL’s portability model is based on Kokkos execution and memory spaces, MPI for distributed memory, and optional HeFFTe FFT backends. The same algorithmic code path is intended to run across CPU and GPU systems with only compile-time type selection and runtime parameter choices.

15.1 Themes

• Dimension-independent algorithms from 1D through 6D.
• Mixed precision with float and double.
• Mixed execution spaces for fields and particle attributes.
• GPU-aware MPI and backend-specific FFT support.
• Explicit synchronization with ippl::fence() when required.

15.2 Portability stack

flowchart TB
  App["User app / mini-app"] --> IPPL["IPPL abstractions"]
  IPPL --> Kernels["Kokkos kernels and views"]
  IPPL --> Comm["MPI communication"]
  IPPL --> FFT["HeFFTe FFT backend (optional)"]
  Kernels --> HW["CPU / GPU hardware"]
  Comm --> HW
  FFT --> HW

15.3 What is portable in IPPL

Layer	Portability contract
Domain and indexing	Index and NDIndex define dimension-independent ranges used identically on all backends.
Distributed ownership	FieldLayout and particle layouts encode decomposition and migration independent of target architecture.
Data containers	Field, ParticleAttrib, and solver vectors store data in Kokkos-backed memory spaces.
Kernel execution	Device/host execution is selected through Kokkos execution spaces and template instantiations.
FFT and communication	HeFFTe options (use_pencils, comm, use_gpu_aware) and MPI collectives tune reshapes and data motion.

15.4 Performance themes from production workflows

15.4.1 1) Dimension-independence

Most IPPL data structures and algorithms are parameterized by Dim and used from 1D through 6D. This reduces duplicated implementations and keeps feature behavior consistent between test and production geometries.

15.4.2 2) Mixed precision

IPPL supports float and double and mixed-precision workflows where memory-heavy arrays use lower precision while sensitive reductions or diagnostics remain in higher precision. In PIC workflows this can substantially reduce memory footprint for large runs.

15.4.3 3) Mixed execution spaces

Fields, particles, and solver operations can execute in different execution spaces where supported by the selected build and data movement contracts. The main requirement is that ownership and synchronization boundaries are explicit.

15.4.4 4) Communication and FFT tuning

The FFT-based solvers expose backend controls through ParameterList, including:

params.add("use_pencils", true);
params.add("use_reorder", false);
params.add("use_gpu_aware", true);
params.add("comm", ippl::a2av);
params.add("r2c_direction", 0);

These options map to different communication/reshape strategies in HeFFTe-based transforms and can change scaling behavior significantly on large systems.

15.5 Architecture-sensitive bottlenecks

For particle-mesh simulations, the dominant costs usually come from one or more of:

• particle migration and communication imbalance,
• scatter/gather atomics and memory traffic,
• FFT reshape communication and transpose cost,
• solver iteration count (for CG/PCG/FEM paths),
• host-device synchronization points.

Because bottlenecks shift by problem size and machine, the manual recommends workflow-level profiling first, then backend-specific tuning.

15.6 Practical tuning checklist

1. Start with correctness: verify conservation and solver residual behavior before any tuning.
2. Pick decomposition shape: evaluate pencil vs slab FFT decomposition for your rank topology.
3. Evaluate communication mode: compare a2a, a2av, and supported peer-to-peer reshape modes.
4. Check GPU-aware transfer path: keep use_gpu_aware aligned with system MPI capabilities.
5. Track memory high-water marks: memory pressure often dominates GPU runs before arithmetic throughput.
6. Tune solver tolerances/iterations: overly strict defaults can hide communication/kernel improvements.

15.7 Measuring performance

At minimum, report:

Metric	Why it matters
Time per PIC iteration	End-to-end throughput metric for production runs.
Solver-only time	Separates field solve performance from particle operations.
Strong scaling efficiency	Indicates communication and synchronization overhead growth.
Weak scaling efficiency	Indicates per-rank work consistency with growth.
Memory high-water mark	Critical on GPUs with limited memory capacity.
Iteration count / residual (iterative solvers)	Distinguishes numerical from implementation regressions.

For FFT-heavy solvers, include decomposition and communication settings alongside timing results so runs are reproducible.

15.8 Relationship to solver chapters

• Chapter 11 documents FFT and iterative Poisson paths and their parameters.
• Chapter 12 covers FDTD and FEM Maxwell diffusion workflows.
• Chapter 13 covers matrix-free finite-element spaces and operator application contracts.

This chapter focuses on portability and scaling behavior across those components rather than repeating solver derivations.

15.9 Benchmark references

The ALPINE/IPPL mini-app study documents particle-in-cell scaling and performance portability on pre-exascale systems [1]. The free-space spectral Poisson solver paper documents CPU/GPU scaling of solver kernels and FFT-heavy workloads [2]. The PhD thesis extends these themes with massively parallel free-space Poisson and matrix-free FEM solver development and analysis in IPPL [3].

[1]

S. Muralikrishnan et al., “Scaling and performance portability of the particle-in-cell scheme for plasma physics applications through mini-apps targeting exascale architectures,” in Proceedings of the 2024 SIAM conference on parallel processing for scientific computing (PP), Philadelphia, PA: Society for Industrial; Applied Mathematics, 2024, pp. 26–38. doi: 10.1137/1.9781611977967.3.

[2]

S. Mayani, V. Montanaro, A. Cerfon, M. Frey, S. Muralikrishnan, and A. Adelmann, “A massively parallel performance portable free-space spectral poisson solver,” ACM Transactions on Mathematical Software, vol. 51, no. 3, pp. 1–23, Sep. 2025, doi: 10.1145/3748815.

[3]

S. Mayani, “On massively parallel performance portable solvers for particle-in-cell,” PhD thesis, ETH Zurich, 2026.