Skip to main content

8  Particles

Managing particle data and layouts.

Particles in IPPL are treated as a particle container, which stores particle attributes (such as charge, position, and velocity). Furthermore, it contains layout information, which tells each rank which particles they own according to domain decomposition.

This is the Lagrangian half of IPPL’s particle-mesh model.

8.1 Main classes

Class Role
ParticleBase Base class for user particle containers.
ParticleAttrib Typed particle attribute stored in a Kokkos view.
ParticleAttribBase Common attribute interface used by the container.
ParticleLayout Base layout interface.
ParticleSpatialLayout Spatial decomposition.
ParticleSpatialOverlapLayout Spatial layout with ghost particles within a cutoff and search cells for short-range pair loops (Section 8.9).
ParticleBC Particle boundary condition support.
TruncatedGreenParticleInteraction Short-range (particle-particle) part of P3M on an overlap layout (Section 8.9.3).

8.2 Particle anatomy

classDiagram
  class ParticleBase {
    +R
    +ID
    +create(nLocal)
    +globalCreate(nTotal)
    +destroy(invalid, destroyNum)
    +update()
    +addAttribute(attribute)
    +forAllAttributes(functor)
    +getLocalNum()
    +getTotalNum()
  }
  class ParticleAttrib {
    +getView()
    +getHostMirror()
    +create(n)
    +destroy(...)
    +scatter(field, positions, policy, hash)
    +gather(field, positions, addToAttribute)
    +sum()
    +min()
    +max()
    +prod()
  }
  class ParticleLayout {
    +setParticleBC(bc)
    +applyBC(R, region)
  }
  class ParticleSpatialLayout {
    +update(container)
    +updateLayout(fieldLayout, mesh)
    +locateParticles(...)
    +getRegionLayout()
  }

  ParticleSpatialLayout --|> ParticleLayout
  ParticleBase --> ParticleSpatialLayout
  ParticleBase --> ParticleAttrib

8.3 User model

A user defines a particle container by deriving from ParticleBase and adding attributes such as position, velocity, mass, charge, or application-specific quantity. The layout determines which MPI rank owns each particle and how particles move when positions change.

The programming interface clearly defines the role of each attribute and method:

  • R stores particle positions.
  • ID stores particle IDs when IDs are enabled.
  • user attributes are registered with addAttribute.
  • getLocalNum() returns the rank-local particle count.
  • getTotalNum() returns the global particle count.
  • update() sends particles to the correct rank after positions change (which could result in particles leaving the local domain of the owning rank).

ParticleBase automatically registers the inherited position attribute R. If particle IDs are enabled through the IDProperties template parameter, it also registers ID. User attributes must be registered in the derived class constructor with addAttribute.

8.4 Minimal derived container

template <class PLayout>
class ChargedParticles : public ippl::ParticleBase<PLayout> {
public:
    using Base = ippl::ParticleBase<PLayout>;
    using Vector_t = typename PLayout::vector_type;

    ippl::ParticleAttrib<double> qm;
    typename Base::particle_position_type P;
    typename Base::particle_position_type E;

    ChargedParticles(PLayout& layout)
        : Base(layout) {
        this->addAttribute(qm);
        this->addAttribute(P);
        this->addAttribute(E);
        this->setParticleBC(ippl::BC::PERIODIC);
    }
};

This is the same pattern used by test/particle/PICnd.cpp: charge-to-mass ratio, velocity, and gathered electric field are particle attributes; the inherited R position attribute drives migration and interpolation.

8.5 Attribute storage and expressions

ParticleAttrib<T, Properties...> stores one value per local particle in a Kokkos view. Common types include scalar attributes such as charge and vector attributes such as velocity:

ippl::ParticleAttrib<double> q;
typename Base::particle_position_type velocity;

Useful operations:

Operation Example
scalar assignment bunch.qm = totalCharge / totalParticles;
expression assignment bunch.P = bunch.P + dt * bunch.qm * bunch.E;
device view access auto view = bunch.qm.getView();
host mirror auto host = bunch.R.getHostMirror();
reduction double total = bunch.qm.sum();
field deposit (particle-mesh) scatter(bunch.qm, rho, bunch.R);
field gather (particle-mesh) gather(bunch.E, efield, bunch.R);

To initialize an attribute on host (CPU), one should use a host mirror followed by Kokkos::deep_copy to copy back to the device view. Then call update() so the rank ownership of particles according to domain decomposition is enforced.

For example, to initialize particle positions:

auto R_host = bunch.R.getHostMirror();
for (size_t i = 0; i < bunch.getLocalNum(); ++i) {
    R_host(i) = initialPosition(i);
}
Kokkos::deep_copy(bunch.R.getView(), R_host);
bunch.update();

8.6 Creation and initialization

For the particle container creation, a ParticleSpatialLayout is needed, which uses the same domain decomposition description as for the fields. Therefore, a FieldLayout of an IPPL mesh is needed first (to obtain the domain decomposition, which is fully grid-based).

Object Role in particle setup
NDIndex<Dim> Global grid extent (number of cells per axis).
UniformCartesian<T, Dim> (mesh) Physical spacing, origin, and cell geometry.
FieldLayout<Dim> (fieldLayout) MPI decomposition: which rank owns which part of the domain.
ParticleSpatialLayout<T, Dim> Maps particle positions to rank-owned regions using fieldLayout and mesh.

End-to-end setup (same pattern as test/particle/PICnd.cpp):

constexpr unsigned Dim = 3;

// 1) Domain and mesh to obtain FieldLayout
ippl::Vector<int, Dim> nr = {64, 64, 64};
ippl::NDIndex<Dim> domain;
for (unsigned d = 0; d < Dim; ++d) {
    domain[d] = ippl::Index(nr[d]);
}

ippl::Vector<double, Dim> origin = {0.0, 0.0, 0.0};
ippl::Vector<double, Dim> hr = {1.0 / nr[0], 1.0 / nr[1], 1.0 / nr[2]};

using Mesh_t = ippl::UniformCartesian<double, Dim>;
Mesh_t mesh(domain, hr, origin);

// we parallely decompose all dimensions
std::array<bool, Dim> isParallel;
isParallel.fill(true);

using FieldLayout_t = ippl::FieldLayout<Dim>;
FieldLayout_t fieldLayout(MPI_COMM_WORLD, domain, isParallel);

// 2) Particle layout and container
using PLayout_t = ippl::ParticleSpatialLayout<double, Dim>;
PLayout_t particleLayout(fieldLayout, mesh);

ChargedParticles<PLayout_t> bunch(particleLayout);

// 3) Create local particles and set attributes
const size_t totalParticles = 10000;
size_t nloc = totalParticles / ippl::Comm->size();
bunch.create(nloc);
bunch.qm = totalCharge / totalParticles;
bunch.P  = 0.0;

// 4) Enforce ownership after positions are set
bunch.update();

globalCreate(totalParticles) is an alternative to create(nloc), which requires hand-splitting nloc. If using globalCreate(totalParticles), every rank calls it with the same global count, and IPPL distributes particles across ranks.

8.6.1 Collective operations

create, globalCreate, createWithID, destroy, and update are collective in the MPI sense: every rank in the communicator must execute the call, even if local arguments differ (for example each rank passes its own nloc).

Do not guard these calls so that only some ranks run them:

// Wrong: rank 0 skips create(); other ranks hang or corrupt state
if (ippl::Comm->rank() == 0) {
    bunch.create(nloc);
}

// Correct: every rank calls create; nloc may differ per rank
bunch.create(nloc);

The same rule applies after if branches: if one code path calls update(), every rank must take a path that also calls update() (or none must). Avoid return on one rank before a collective while others continue.

8.6.2 Interpolation between Grid and Particles

!Ported from old Doxygen, review required!

Methods for data transfer between particles and grid, supporting operations like scatter and gather for efficient particle-field interaction.

// Particles -> grid
// Can interpolate any particle attribute (both scalar and vectors) to the grid.
// Only linear (cloud-in-cell) interpolation is available at the moment.
// Interpolate a scalar particle attribute q (e.g. charge) onto the field rho.
scatter(bunch.q, rho, bunch.R);

// Grid -> particles
// Can interpolate any field (both scalar and vectors) to the particles.
// Only linear interpolation is available at the moment.
// Interpolate a vector field E (e.g. electric field) to the particles
gather(bunch.E, Efield, bunch.R);

8.7 Particle boundary conditions

Particle boundary conditions are selected with the ippl::BC enum:

BC Behavior
PERIODIC Wraps coordinates back into the domain.
REFLECTIVE Mirrors coordinates back into the domain.
SINK Clips coordinates to the boundary.
NO Leaves coordinates unchanged.

Use one boundary condition on all faces:

bunch.setParticleBC(ippl::BC::PERIODIC);

or provide one value per face:

typename Bunch::bc_container_type bcs;
bcs.fill(ippl::BC::PERIODIC);
bunch.setParticleBC(bcs);

The spatial layout applies particle boundary conditions at the start of update(), before migration of particles to appropriate ranks. Unit tests cover upper and lower faces for periodic, reflective, sink, and no-boundary behavior.

8.8 Particle update (spatial migration)

ParticleSpatialLayout maps particle positions to rank-owned spatial regions derived from a FieldLayout and a mesh. The update algorithm is:

  1. Apply particle boundary conditions to R.
  2. Locate each particle in the current rank, neighboring ranks, or the global region list.
  3. Count sends per destination rank.
  4. Pack and send particles that left the current rank.
  5. Delete invalid local particles.
  6. Receive incoming particles and unpack all registered attributes.

flowchart LR
  R["Updated positions R"]
  BC["applyBC"]
  Locate["locateParticles"]
  Pack["pack/send"]
  Delete["internalDestroy"]
  Recv["recv/unpack"]
  Valid["Rank-local valid particles"]

  R --> BC --> Locate --> Pack --> Delete --> Recv --> Valid

The ParticleSendRecv unit test verifies that after migration every remaining local particle has an expected rank equal to ippl::Comm->rank(), and the global particle count is conserved.

8.9 Short-range pair interactions

ParticleSpatialOverlapLayout adds a pair cutoff \(r_c\) to the spatial layout. Its update() migrates particles like ParticleSpatialLayout, then gives every rank copies of the particles within \(r_c\) of its region and sorts all particles into search cells. All partners within \(r_c\) of a particle can then be found in the neighbouring search cells. IPPL uses this for the particle-particle part of P3M (Section 8.9.3), but the pair loops accept any functor.

using PLayout_t = ippl::ParticleSpatialOverlapLayout<double, Dim>;
PLayout_t layout(fieldLayout, mesh, rc);  // rc: pair cutoff
ChargedParticles<PLayout_t> bunch(layout);
// create particles and set positions as above
bunch.update();                           // migrate, add ghosts, sort into search cells
classDiagram
  class ParticleSpatialLayout
  class ParticleSpatialOverlapLayout {
    +update(container)
    +updateLayout(fieldLayout, mesh, rcut)
    +setCutoff(rcut)
    +getCutoff()
    +forEachPair(f)
    +forEachOwnedParticle(pair, finalize)
  }
  class ParticleInteractionBase
  class TruncatedGreenParticleInteraction {
    +solve()
  }

  ParticleSpatialOverlapLayout --|> ParticleSpatialLayout
  TruncatedGreenParticleInteraction --|> ParticleInteractionBase
  TruncatedGreenParticleInteraction --> ParticleSpatialOverlapLayout : pair loops
Figure 8.1: Classes for short-range pair interactions. The interaction runs its pair sum through one of the two pair loops of the overlap layout.

8.9.1 Owned and ghost particles

After update(), the attribute views of a rank hold two kinds of particles:

  • Owned particles lie inside the rank’s region. They occupy the indices 0 to getLocalNum() - 1, sorted by search cell. The rank computes their fields and moves them.
  • Ghost particles are read-only copies of particles within \(r_c\) of the region boundary. They come from neighbouring ranks or, with PERIODIC particle boundary conditions, are periodic images, also on a single rank. They are stored after the owned particles, are not counted by getLocalNum(), and are rebuilt by every update(). They only act as interaction partners. Ghosts carry all attributes, but getView() covers only the owned particles; read a partner through the attribute itself, for example bunch.R(j).

Search cells are at least \(r_c\) wide, and wider when the number of cells on a rank would exceed a cap. All partners within \(r_c\) of an owned particle therefore lie in its own search cell or an adjacent one, \(3^d\) cells in total (27 in 3D). The cutoff must be positive and at most half the local region length in every dimension; the constructor, setCutoff(), and the three-argument updateLayout(fieldLayout, mesh, rc) throw otherwise. A new cutoff takes effect at the next update(), and the pair loops throw until then.

8.9.2 Pair loops

Both pair loops visit the same directed pairs: every owned particle \(i\) with every particle \(j \ne i\), owned or ghost, in the neighbouring search cells. These are candidates, so the functor has to test the distance itself. A pair of two owned particles is visited twice, once from each side, so a functor only writes to \(i\).

Table 8.1: Pair loops of ParticleSpatialOverlapLayout.
forEachPair(f) forEachOwnedParticle<ExecSpace, Accumulator>(pair, finalize)
Functor calls f(i, j) for every pair. pair(i, j, acc) for every pair, then finalize(i, acc) once per owned particle.
Parallelism One Kokkos team per search cell. Each team thread takes one neighbouring cell and runs all pairs of the two cells in a loop. One thread per owned particle, which loops over all neighbouring cells.
Writes to target \(i\) From the threads of all neighbouring cells at the same time, so the functor must use atomics. Only from the thread of \(i\), so no atomics are needed.
Timer PPInteractionTimer PPOwnedTimer

forEachOwnedParticle is the faster loop on GPUs. With forEachPair, a field update such as Kokkos::atomic_sub(&E(i), F_ij) on a Vector<double, 3> has no hardware instruction on CUDA, which has native atomics only for 4- and 8-byte types, so Kokkos protects every 24-byte update with a lock. In forEachOwnedParticle, the thread of \(i\) sums into the accumulator acc in registers and stores the result once in finalize. Owned particles are sorted by search cell, so neighbouring threads of a GPU warp usually belong to the same cell and read the same partner \(j\) in each step.

The following example counts the neighbours within \(r_c\) of every owned particle:

using Vector_t = ippl::Vector<double, 3>;
const auto& R = bunch.R;  // the attribute, not getView(): partners j include ghosts
Kokkos::View<int*> neighbours("neighbours", bunch.getLocalNum());
const double rc2 = rc * rc;

layout.forEachOwnedParticle<Kokkos::DefaultExecutionSpace, int>(
    KOKKOS_LAMBDA(const std::size_t i, const std::size_t j, int& count) {
        const Vector_t d = R(i) - R(j);
        if (d.dot(d) < rc2) {
            ++count;
        }
    },
    KOKKOS_LAMBDA(const std::size_t i, const int& count) { neighbours(i) = count; });

The accumulator type must be constructible from 0 and support +=, for example int, double, or Vector<double, 3>. The loop fences before it returns. finalize runs for all owned particles in parallel. To sum a value over all particles, store it per particle and reduce with Kokkos::parallel_reduce. Adding every value atomically to a single counter serializes all threads on one address, and for types without a native atomic each add becomes a compare-and-swap loop (for example unsigned long long with Kokkos 5.2 on CUDA under Linux), which can take much longer than the pair loop itself.

8.9.3 Short-range P3M field

TruncatedGreenParticleInteraction (header Interaction/TruncatedGreenParticleInteraction.h) computes the particle-particle part of P3M with these pair loops. The mesh part is computed by FFTTruncatedGreenPeriodicPoissonSolver (Chapter 16). For every owned particle \(i\), solve() adds \[ \mathbf{F}_i \mathrel{+}= k \sum_{\substack{j \ne i \\ 0 < r_{ij} < r_\mathrm{cut}}} q_j \, \nabla_i \frac{\operatorname{erfc}(\alpha r_{ij})}{r_{ij}}, \qquad r_{ij} = |\mathbf{R}_i - \mathbf{R}_j|, \] to the vector attribute \(\mathbf{F}\). Here \(q_j\) is the scalar attribute (charge or mass), \(k\) is force_constant, and \(\alpha\) must be the same as the alpha of the mesh solver. With charges and \(k = -1/(4\pi\varepsilon_0)\), \(\mathbf{F}\) receives the short-range part of the electric field. Below regularization_cutoff \(r_\mathrm{reg}\), the singular Coulomb part is replaced by a linear core that is continuous at \(r_\mathrm{reg}\). Distinct particles at the same position contribute nothing. solve() adds to \(\mathbf{F}\), so set \(\mathbf{F}\) to zero first if it should hold only this contribution.

Table 8.2: Parameters of TruncatedGreenParticleInteraction.
Parameter Default Meaning
rcut required Largest pair distance \(r_\mathrm{cut}\). solve() throws if it exceeds the cutoff of the particle layout.
alpha required Splitting parameter \(\alpha\), the same as in the mesh solver (for example \(2/r_\mathrm{cut}\)).
force_constant required Prefactor \(k\).
regularization_cutoff 1e-9 Radius \(r_\mathrm{reg}\) of the linear core. solve() throws if it is not positive.
pair_kernel OWNED_PARTICLE Pair loop. OWNED_PARTICLE uses forEachOwnedParticle; CELL_PAIRS uses forEachPair with one Kokkos::atomic_sub per pair. Any other value selects OWNED_PARTICLE.
speed_of_light required with moving sources Speed of light \(c\) in the units of the caller, so that \(\mathbf{B} = \boldsymbol{\beta} \times \mathbf{F} / c\). Only read in the moving-source mode; solve() throws if it is not positive.
using Bunch_t       = ChargedParticles<PLayout_t>;
using Interaction_t = ippl::TruncatedGreenParticleInteraction<
    Bunch_t, Bunch_t::particle_position_type, ippl::ParticleAttrib<double>>;

ippl::ParameterList params;
params.add("rcut", rc);
params.add("alpha", 2.0 / rc);
params.add("force_constant", 1.0);
// params.add("pair_kernel", static_cast<int>(Interaction_t::CELL_PAIRS));  // previous loop

Interaction_t interaction(bunch, bunch.E, bunch.R, bunch.qm, params);
bunch.E = Vector_t(0.0);
interaction.solve();

Moving sources

A second constructor adds the field of sources that move with their own velocity, such as the particles of a relativistic beam, and also accumulates a magnetic field:

params.add("speed_of_light", c);
Interaction_t interaction(bunch, bunch.E, bunch.B, bunch.R, bunch.P, bunch.qm, params);

Here P holds the normalized momentum \(\mathbf{u}_j = \gamma_j \boldsymbol{\beta}_j\) of every particle, and B receives the magnetic field. The truncated field of source \(j\) is evaluated at the separation in its rest frame and transformed to the lab: \[ \Delta\mathbf{F}_i = -k\,\gamma_j q_j \frac{g(\alpha r'_{ij})}{r'^{3}_{ij}}\,\mathbf{r}_{ij}, \qquad \Delta\mathbf{B}_i = \frac{\mathbf{u}_j \times \Delta\mathbf{F}_i}{\gamma_j c}, \qquad r'^{2}_{ij} = r^{2}_{ij} + (\mathbf{u}_j \cdot \mathbf{r}_{ij})^2, \] with \(\mathbf{r}_{ij} = \mathbf{R}_i - \mathbf{R}_j\), \(\gamma_j = \sqrt{1 + u_j^2}\), and \(g(x) = \operatorname{erfc}(x) + 2x e^{-x^2}/\sqrt{\pi}\). This is the short-range part of the field of a uniformly moving charge. A pair is skipped once \(r'_{ij}\) reaches \(r_\mathrm{cut}\); since \(r' \ge r\), the search cells still contain every remaining pair. With all \(\mathbf{u}_j = 0\) the result is the static field and \(\mathbf{B}\) stays unchanged. The mode needs speed_of_light and three dimensions, and works with both pair loops.

solve() chooses between the static and the moving-source pair term once per call, so static sources do not pay for the moving term. With all momenta zero, the moving term takes about 23 % longer than the static one on an A100 (\(2^{22}\) and \(2^{24}\) particles) and 16 to 30 % longer on one CPU core (TestP3MPairKernels). On the A100, OWNED_PARTICLE is 14 to 15 times faster than CELL_PAIRS in the moving-source mode.

Both pair loops give the same field up to the order of summation; CELL_PAIRS is kept for comparisons. Table 8.3 lists measured times. The GPU speedup comes from removing the locked atomics. It is largest in dense search cells, where the rate of CELL_PAIRS falls from \(8.4 \times 10^{9}\) to \(2.6 \times 10^{9}\) candidate pairs per second on a GH200, while OWNED_PARTICLE stays at about \(1.2 \times 10^{11}\). On one CPU thread both loops run the same nested loops and take the same time.

Table 8.3: Time per solve() measured with TestP3MPairKernels (Release builds, Kokkos 5.2.0). Particles fill the unit box, and \(r_c = 0.99999/m\) gives \(m^3\) search cells.
Hardware and case CELL_PAIRS OWNED_PARTICLE Speedup
One GH200, uniform, \(N = 2^{16}\) to \(2^{24}\), about 64 particles per search cell 42 ms to 2.96 s 1.4 ms to 232 ms 13 to 31
One GH200, uniform, \(N = 2^{20}\), 8 to 1049 particles per search cell 31 ms to 9.1 s 2.9 ms to 193 ms 11 to 47
One A100-40GB, uniform, \(N = 2^{24}\) and \(2^{25}\), about 64 particles per search cell 5.39 s and 10.7 s 524 ms and 1.04 s 10.3
One CPU core (SERIAL), uniform, \(N = 20000\), \(r_c = 0.08\) 20.7 ms 19.8 ms 1.05

TestP3MPairKernels compares the two pair loops, and TestTruncatedGreenParticleInteraction checks the default loop against known pair fields; both are described in Chapter 20.

8.10 Developer model

Particle data movement is a correctness-critical layer. Changes to particle layouts must be documented with:

  • communication pattern,
  • memory-space assumptions,
  • unit tests covering multiple dimensions and execution spaces.

The test suite instantiates particle tests across floating point precision, dimensions (1D to 3D), and execution spaces. When adding a particle feature, prefer tests that cover both host and device memory-space paths and more than one dimension.