classDiagram
class ParticleBase {
+R
+ID
+create(nLocal)
+globalCreate(nTotal)
+destroy(invalid, destroyNum)
+update()
+addAttribute(attribute)
+forAllAttributes(functor)
+getLocalNum()
+getTotalNum()
}
class ParticleAttrib {
+getView()
+getHostMirror()
+create(n)
+destroy(...)
+scatter(field, positions, policy, hash)
+gather(field, positions, addToAttribute)
+sum()
+min()
+max()
+prod()
}
class ParticleLayout {
+setParticleBC(bc)
+applyBC(R, region)
}
class ParticleSpatialLayout {
+update(container)
+updateLayout(fieldLayout, mesh)
+locateParticles(...)
+getRegionLayout()
}
ParticleSpatialLayout --|> ParticleLayout
ParticleBase --> ParticleSpatialLayout
ParticleBase --> ParticleAttrib
8 Particles
Managing particle data and layouts.
Particles in IPPL are treated as a particle container, which stores particle attributes (such as charge, position, and velocity). Furthermore, it contains layout information, which tells each rank which particles they own according to domain decomposition.
This is the Lagrangian half of IPPL’s particle-mesh model.
8.1 Main classes
| Class | Role |
|---|---|
ParticleBase |
Base class for user particle containers. |
ParticleAttrib |
Typed particle attribute stored in a Kokkos view. |
ParticleAttribBase |
Common attribute interface used by the container. |
ParticleLayout |
Base layout interface. |
ParticleSpatialLayout |
Spatial decomposition. |
ParticleSpatialOverlapLayout |
Spatial layout with ghost particles within a cutoff and search cells for short-range pair loops (Section 8.9). |
ParticleBC |
Particle boundary condition support. |
TruncatedGreenParticleInteraction |
Short-range (particle-particle) part of P3M on an overlap layout (Section 8.9.3). |
8.2 Particle anatomy
8.3 User model
A user defines a particle container by deriving from ParticleBase and adding attributes such as position, velocity, mass, charge, or application-specific quantity. The layout determines which MPI rank owns each particle and how particles move when positions change.
The programming interface clearly defines the role of each attribute and method:
Rstores particle positions.IDstores particle IDs when IDs are enabled.- user attributes are registered with
addAttribute. getLocalNum()returns the rank-local particle count.getTotalNum()returns the global particle count.update()sends particles to the correct rank after positions change (which could result in particles leaving the local domain of the owning rank).
ParticleBase automatically registers the inherited position attribute R. If particle IDs are enabled through the IDProperties template parameter, it also registers ID. User attributes must be registered in the derived class constructor with addAttribute.
8.4 Minimal derived container
template <class PLayout>
class ChargedParticles : public ippl::ParticleBase<PLayout> {
public:
using Base = ippl::ParticleBase<PLayout>;
using Vector_t = typename PLayout::vector_type;
ippl::ParticleAttrib<double> qm;
typename Base::particle_position_type P;
typename Base::particle_position_type E;
ChargedParticles(PLayout& layout)
: Base(layout) {
this->addAttribute(qm);
this->addAttribute(P);
this->addAttribute(E);
this->setParticleBC(ippl::BC::PERIODIC);
}
};This is the same pattern used by test/particle/PICnd.cpp: charge-to-mass ratio, velocity, and gathered electric field are particle attributes; the inherited R position attribute drives migration and interpolation.
8.5 Attribute storage and expressions
ParticleAttrib<T, Properties...> stores one value per local particle in a Kokkos view. Common types include scalar attributes such as charge and vector attributes such as velocity:
ippl::ParticleAttrib<double> q;
typename Base::particle_position_type velocity;Useful operations:
| Operation | Example |
|---|---|
| scalar assignment | bunch.qm = totalCharge / totalParticles; |
| expression assignment | bunch.P = bunch.P + dt * bunch.qm * bunch.E; |
| device view access | auto view = bunch.qm.getView(); |
| host mirror | auto host = bunch.R.getHostMirror(); |
| reduction | double total = bunch.qm.sum(); |
| field deposit (particle-mesh) | scatter(bunch.qm, rho, bunch.R); |
| field gather (particle-mesh) | gather(bunch.E, efield, bunch.R); |
To initialize an attribute on host (CPU), one should use a host mirror followed by Kokkos::deep_copy to copy back to the device view. Then call update() so the rank ownership of particles according to domain decomposition is enforced.
For example, to initialize particle positions:
auto R_host = bunch.R.getHostMirror();
for (size_t i = 0; i < bunch.getLocalNum(); ++i) {
R_host(i) = initialPosition(i);
}
Kokkos::deep_copy(bunch.R.getView(), R_host);
bunch.update();8.6 Creation and initialization
For the particle container creation, a ParticleSpatialLayout is needed, which uses the same domain decomposition description as for the fields. Therefore, a FieldLayout of an IPPL mesh is needed first (to obtain the domain decomposition, which is fully grid-based).
| Object | Role in particle setup |
|---|---|
NDIndex<Dim> |
Global grid extent (number of cells per axis). |
UniformCartesian<T, Dim> (mesh) |
Physical spacing, origin, and cell geometry. |
FieldLayout<Dim> (fieldLayout) |
MPI decomposition: which rank owns which part of the domain. |
ParticleSpatialLayout<T, Dim> |
Maps particle positions to rank-owned regions using fieldLayout and mesh. |
End-to-end setup (same pattern as test/particle/PICnd.cpp):
constexpr unsigned Dim = 3;
// 1) Domain and mesh to obtain FieldLayout
ippl::Vector<int, Dim> nr = {64, 64, 64};
ippl::NDIndex<Dim> domain;
for (unsigned d = 0; d < Dim; ++d) {
domain[d] = ippl::Index(nr[d]);
}
ippl::Vector<double, Dim> origin = {0.0, 0.0, 0.0};
ippl::Vector<double, Dim> hr = {1.0 / nr[0], 1.0 / nr[1], 1.0 / nr[2]};
using Mesh_t = ippl::UniformCartesian<double, Dim>;
Mesh_t mesh(domain, hr, origin);
// we parallely decompose all dimensions
std::array<bool, Dim> isParallel;
isParallel.fill(true);
using FieldLayout_t = ippl::FieldLayout<Dim>;
FieldLayout_t fieldLayout(MPI_COMM_WORLD, domain, isParallel);
// 2) Particle layout and container
using PLayout_t = ippl::ParticleSpatialLayout<double, Dim>;
PLayout_t particleLayout(fieldLayout, mesh);
ChargedParticles<PLayout_t> bunch(particleLayout);
// 3) Create local particles and set attributes
const size_t totalParticles = 10000;
size_t nloc = totalParticles / ippl::Comm->size();
bunch.create(nloc);
bunch.qm = totalCharge / totalParticles;
bunch.P = 0.0;
// 4) Enforce ownership after positions are set
bunch.update();globalCreate(totalParticles) is an alternative to create(nloc), which requires hand-splitting nloc. If using globalCreate(totalParticles), every rank calls it with the same global count, and IPPL distributes particles across ranks.
8.6.1 Collective operations
create, globalCreate, createWithID, destroy, and update are collective in the MPI sense: every rank in the communicator must execute the call, even if local arguments differ (for example each rank passes its own nloc).
Do not guard these calls so that only some ranks run them:
// Wrong: rank 0 skips create(); other ranks hang or corrupt state
if (ippl::Comm->rank() == 0) {
bunch.create(nloc);
}
// Correct: every rank calls create; nloc may differ per rank
bunch.create(nloc);The same rule applies after if branches: if one code path calls update(), every rank must take a path that also calls update() (or none must). Avoid return on one rank before a collective while others continue.
8.6.2 Interpolation between Grid and Particles
!Ported from old Doxygen, review required!
Methods for data transfer between particles and grid, supporting operations like scatter and gather for efficient particle-field interaction.
// Particles -> grid
// Can interpolate any particle attribute (both scalar and vectors) to the grid.
// Only linear (cloud-in-cell) interpolation is available at the moment.
// Interpolate a scalar particle attribute q (e.g. charge) onto the field rho.
scatter(bunch.q, rho, bunch.R);
// Grid -> particles
// Can interpolate any field (both scalar and vectors) to the particles.
// Only linear interpolation is available at the moment.
// Interpolate a vector field E (e.g. electric field) to the particles
gather(bunch.E, Efield, bunch.R);8.7 Particle boundary conditions
Particle boundary conditions are selected with the ippl::BC enum:
| BC | Behavior |
|---|---|
PERIODIC |
Wraps coordinates back into the domain. |
REFLECTIVE |
Mirrors coordinates back into the domain. |
SINK |
Clips coordinates to the boundary. |
NO |
Leaves coordinates unchanged. |
Use one boundary condition on all faces:
bunch.setParticleBC(ippl::BC::PERIODIC);or provide one value per face:
typename Bunch::bc_container_type bcs;
bcs.fill(ippl::BC::PERIODIC);
bunch.setParticleBC(bcs);The spatial layout applies particle boundary conditions at the start of update(), before migration of particles to appropriate ranks. Unit tests cover upper and lower faces for periodic, reflective, sink, and no-boundary behavior.
8.8 Particle update (spatial migration)
ParticleSpatialLayout maps particle positions to rank-owned spatial regions derived from a FieldLayout and a mesh. The update algorithm is:
- Apply particle boundary conditions to
R. - Locate each particle in the current rank, neighboring ranks, or the global region list.
- Count sends per destination rank.
- Pack and send particles that left the current rank.
- Delete invalid local particles.
- Receive incoming particles and unpack all registered attributes.
flowchart LR R["Updated positions R"] BC["applyBC"] Locate["locateParticles"] Pack["pack/send"] Delete["internalDestroy"] Recv["recv/unpack"] Valid["Rank-local valid particles"] R --> BC --> Locate --> Pack --> Delete --> Recv --> Valid
The ParticleSendRecv unit test verifies that after migration every remaining local particle has an expected rank equal to ippl::Comm->rank(), and the global particle count is conserved.
8.9 Short-range pair interactions
ParticleSpatialOverlapLayout adds a pair cutoff \(r_c\) to the spatial layout. Its update() migrates particles like ParticleSpatialLayout, then gives every rank copies of the particles within \(r_c\) of its region and sorts all particles into search cells. All partners within \(r_c\) of a particle can then be found in the neighbouring search cells. IPPL uses this for the particle-particle part of P3M (Section 8.9.3), but the pair loops accept any functor.
using PLayout_t = ippl::ParticleSpatialOverlapLayout<double, Dim>;
PLayout_t layout(fieldLayout, mesh, rc); // rc: pair cutoff
ChargedParticles<PLayout_t> bunch(layout);
// create particles and set positions as above
bunch.update(); // migrate, add ghosts, sort into search cellsclassDiagram
class ParticleSpatialLayout
class ParticleSpatialOverlapLayout {
+update(container)
+updateLayout(fieldLayout, mesh, rcut)
+setCutoff(rcut)
+getCutoff()
+forEachPair(f)
+forEachOwnedParticle(pair, finalize)
}
class ParticleInteractionBase
class TruncatedGreenParticleInteraction {
+solve()
}
ParticleSpatialOverlapLayout --|> ParticleSpatialLayout
TruncatedGreenParticleInteraction --|> ParticleInteractionBase
TruncatedGreenParticleInteraction --> ParticleSpatialOverlapLayout : pair loops
8.9.1 Owned and ghost particles
After update(), the attribute views of a rank hold two kinds of particles:
- Owned particles lie inside the rank’s region. They occupy the indices
0togetLocalNum() - 1, sorted by search cell. The rank computes their fields and moves them. - Ghost particles are read-only copies of particles within \(r_c\) of the region boundary. They come from neighbouring ranks or, with
PERIODICparticle boundary conditions, are periodic images, also on a single rank. They are stored after the owned particles, are not counted bygetLocalNum(), and are rebuilt by everyupdate(). They only act as interaction partners. Ghosts carry all attributes, butgetView()covers only the owned particles; read a partner through the attribute itself, for examplebunch.R(j).
Search cells are at least \(r_c\) wide, and wider when the number of cells on a rank would exceed a cap. All partners within \(r_c\) of an owned particle therefore lie in its own search cell or an adjacent one, \(3^d\) cells in total (27 in 3D). The cutoff must be positive and at most half the local region length in every dimension; the constructor, setCutoff(), and the three-argument updateLayout(fieldLayout, mesh, rc) throw otherwise. A new cutoff takes effect at the next update(), and the pair loops throw until then.
8.9.2 Pair loops
Both pair loops visit the same directed pairs: every owned particle \(i\) with every particle \(j \ne i\), owned or ghost, in the neighbouring search cells. These are candidates, so the functor has to test the distance itself. A pair of two owned particles is visited twice, once from each side, so a functor only writes to \(i\).
ParticleSpatialOverlapLayout.
forEachPair(f) |
forEachOwnedParticle<ExecSpace, Accumulator>(pair, finalize) |
|
|---|---|---|
| Functor calls | f(i, j) for every pair. |
pair(i, j, acc) for every pair, then finalize(i, acc) once per owned particle. |
| Parallelism | One Kokkos team per search cell. Each team thread takes one neighbouring cell and runs all pairs of the two cells in a loop. | One thread per owned particle, which loops over all neighbouring cells. |
| Writes to target \(i\) | From the threads of all neighbouring cells at the same time, so the functor must use atomics. | Only from the thread of \(i\), so no atomics are needed. |
| Timer | PPInteractionTimer |
PPOwnedTimer |
forEachOwnedParticle is the faster loop on GPUs. With forEachPair, a field update such as Kokkos::atomic_sub(&E(i), F_ij) on a Vector<double, 3> has no hardware instruction on CUDA, which has native atomics only for 4- and 8-byte types, so Kokkos protects every 24-byte update with a lock. In forEachOwnedParticle, the thread of \(i\) sums into the accumulator acc in registers and stores the result once in finalize. Owned particles are sorted by search cell, so neighbouring threads of a GPU warp usually belong to the same cell and read the same partner \(j\) in each step.
The following example counts the neighbours within \(r_c\) of every owned particle:
using Vector_t = ippl::Vector<double, 3>;
const auto& R = bunch.R; // the attribute, not getView(): partners j include ghosts
Kokkos::View<int*> neighbours("neighbours", bunch.getLocalNum());
const double rc2 = rc * rc;
layout.forEachOwnedParticle<Kokkos::DefaultExecutionSpace, int>(
KOKKOS_LAMBDA(const std::size_t i, const std::size_t j, int& count) {
const Vector_t d = R(i) - R(j);
if (d.dot(d) < rc2) {
++count;
}
},
KOKKOS_LAMBDA(const std::size_t i, const int& count) { neighbours(i) = count; });The accumulator type must be constructible from 0 and support +=, for example int, double, or Vector<double, 3>. The loop fences before it returns. finalize runs for all owned particles in parallel. To sum a value over all particles, store it per particle and reduce with Kokkos::parallel_reduce. Adding every value atomically to a single counter serializes all threads on one address, and for types without a native atomic each add becomes a compare-and-swap loop (for example unsigned long long with Kokkos 5.2 on CUDA under Linux), which can take much longer than the pair loop itself.
8.9.3 Short-range P3M field
TruncatedGreenParticleInteraction (header Interaction/TruncatedGreenParticleInteraction.h) computes the particle-particle part of P3M with these pair loops. The mesh part is computed by FFTTruncatedGreenPeriodicPoissonSolver (Chapter 16). For every owned particle \(i\), solve() adds \[
\mathbf{F}_i \mathrel{+}= k \sum_{\substack{j \ne i \\ 0 < r_{ij} < r_\mathrm{cut}}} q_j \, \nabla_i \frac{\operatorname{erfc}(\alpha r_{ij})}{r_{ij}},
\qquad r_{ij} = |\mathbf{R}_i - \mathbf{R}_j|,
\] to the vector attribute \(\mathbf{F}\). Here \(q_j\) is the scalar attribute (charge or mass), \(k\) is force_constant, and \(\alpha\) must be the same as the alpha of the mesh solver. With charges and \(k = -1/(4\pi\varepsilon_0)\), \(\mathbf{F}\) receives the short-range part of the electric field. Below regularization_cutoff \(r_\mathrm{reg}\), the singular Coulomb part is replaced by a linear core that is continuous at \(r_\mathrm{reg}\). Distinct particles at the same position contribute nothing. solve() adds to \(\mathbf{F}\), so set \(\mathbf{F}\) to zero first if it should hold only this contribution.
TruncatedGreenParticleInteraction.
| Parameter | Default | Meaning |
|---|---|---|
rcut |
required | Largest pair distance \(r_\mathrm{cut}\). solve() throws if it exceeds the cutoff of the particle layout. |
alpha |
required | Splitting parameter \(\alpha\), the same as in the mesh solver (for example \(2/r_\mathrm{cut}\)). |
force_constant |
required | Prefactor \(k\). |
regularization_cutoff |
1e-9 |
Radius \(r_\mathrm{reg}\) of the linear core. solve() throws if it is not positive. |
pair_kernel |
OWNED_PARTICLE |
Pair loop. OWNED_PARTICLE uses forEachOwnedParticle; CELL_PAIRS uses forEachPair with one Kokkos::atomic_sub per pair. Any other value selects OWNED_PARTICLE. |
speed_of_light |
required with moving sources | Speed of light \(c\) in the units of the caller, so that \(\mathbf{B} = \boldsymbol{\beta} \times \mathbf{F} / c\). Only read in the moving-source mode; solve() throws if it is not positive. |
using Bunch_t = ChargedParticles<PLayout_t>;
using Interaction_t = ippl::TruncatedGreenParticleInteraction<
Bunch_t, Bunch_t::particle_position_type, ippl::ParticleAttrib<double>>;
ippl::ParameterList params;
params.add("rcut", rc);
params.add("alpha", 2.0 / rc);
params.add("force_constant", 1.0);
// params.add("pair_kernel", static_cast<int>(Interaction_t::CELL_PAIRS)); // previous loop
Interaction_t interaction(bunch, bunch.E, bunch.R, bunch.qm, params);
bunch.E = Vector_t(0.0);
interaction.solve();Moving sources
A second constructor adds the field of sources that move with their own velocity, such as the particles of a relativistic beam, and also accumulates a magnetic field:
params.add("speed_of_light", c);
Interaction_t interaction(bunch, bunch.E, bunch.B, bunch.R, bunch.P, bunch.qm, params);Here P holds the normalized momentum \(\mathbf{u}_j = \gamma_j \boldsymbol{\beta}_j\) of every particle, and B receives the magnetic field. The truncated field of source \(j\) is evaluated at the separation in its rest frame and transformed to the lab: \[
\Delta\mathbf{F}_i = -k\,\gamma_j q_j \frac{g(\alpha r'_{ij})}{r'^{3}_{ij}}\,\mathbf{r}_{ij},
\qquad
\Delta\mathbf{B}_i = \frac{\mathbf{u}_j \times \Delta\mathbf{F}_i}{\gamma_j c},
\qquad
r'^{2}_{ij} = r^{2}_{ij} + (\mathbf{u}_j \cdot \mathbf{r}_{ij})^2,
\] with \(\mathbf{r}_{ij} = \mathbf{R}_i - \mathbf{R}_j\), \(\gamma_j = \sqrt{1 + u_j^2}\), and \(g(x) = \operatorname{erfc}(x) + 2x e^{-x^2}/\sqrt{\pi}\). This is the short-range part of the field of a uniformly moving charge. A pair is skipped once \(r'_{ij}\) reaches \(r_\mathrm{cut}\); since \(r' \ge r\), the search cells still contain every remaining pair. With all \(\mathbf{u}_j = 0\) the result is the static field and \(\mathbf{B}\) stays unchanged. The mode needs speed_of_light and three dimensions, and works with both pair loops.
solve() chooses between the static and the moving-source pair term once per call, so static sources do not pay for the moving term. With all momenta zero, the moving term takes about 23 % longer than the static one on an A100 (\(2^{22}\) and \(2^{24}\) particles) and 16 to 30 % longer on one CPU core (TestP3MPairKernels). On the A100, OWNED_PARTICLE is 14 to 15 times faster than CELL_PAIRS in the moving-source mode.
Both pair loops give the same field up to the order of summation; CELL_PAIRS is kept for comparisons. Table 8.3 lists measured times. The GPU speedup comes from removing the locked atomics. It is largest in dense search cells, where the rate of CELL_PAIRS falls from \(8.4 \times 10^{9}\) to \(2.6 \times 10^{9}\) candidate pairs per second on a GH200, while OWNED_PARTICLE stays at about \(1.2 \times 10^{11}\). On one CPU thread both loops run the same nested loops and take the same time.
solve() measured with TestP3MPairKernels (Release builds, Kokkos 5.2.0). Particles fill the unit box, and \(r_c = 0.99999/m\) gives \(m^3\) search cells.
| Hardware and case | CELL_PAIRS |
OWNED_PARTICLE |
Speedup |
|---|---|---|---|
| One GH200, uniform, \(N = 2^{16}\) to \(2^{24}\), about 64 particles per search cell | 42 ms to 2.96 s | 1.4 ms to 232 ms | 13 to 31 |
| One GH200, uniform, \(N = 2^{20}\), 8 to 1049 particles per search cell | 31 ms to 9.1 s | 2.9 ms to 193 ms | 11 to 47 |
| One A100-40GB, uniform, \(N = 2^{24}\) and \(2^{25}\), about 64 particles per search cell | 5.39 s and 10.7 s | 524 ms and 1.04 s | 10.3 |
| One CPU core (SERIAL), uniform, \(N = 20000\), \(r_c = 0.08\) | 20.7 ms | 19.8 ms | 1.05 |
TestP3MPairKernels compares the two pair loops, and TestTruncatedGreenParticleInteraction checks the default loop against known pair fields; both are described in Chapter 20.
8.10 Developer model
Particle data movement is a correctness-critical layer. Changes to particle layouts must be documented with:
- communication pattern,
- memory-space assumptions,
- unit tests covering multiple dimensions and execution spaces.
The test suite instantiates particle tests across floating point precision, dimensions (1D to 3D), and execution spaces. When adding a particle feature, prefer tests that cover both host and device memory-space paths and more than one dimension.