Mesh based particle methods are increasingly being applied to applications over geometrically complex domains that require accurate accounting for the interactions of particles with the domain boundary and that often have non-uniform distributions of particles. Unstructured mesh methods over highly graded and anisotropic meshes provide an attractive approach for use in such applications. Supporting the development of unstructured mesh-based particle applications requires an infrastructure that provides an appropriate set of data structures and APIs that support a full range of particle interactions with the mesh and domain boundary and that execute effectively on accelerator based parallel computers.
Primary Contacts: Mark Shephard and Jacob Merson
SCOREC Researchers: Merson, Sahni, Shephard, Smith, Tumuklu
Parallel Mesh-Based Particle Method Developments
The PUMIPic infrastructure, originally developed to support unstructured mesh-based PIC calculations, is being extended to support additional unstructured mesh particle simulations including the material point method, neutronics calculations, impurity transport including changed states and neutral particle modeling. Key PUMIPic features are:
- Distributed mesh approach that can scale with respect to both the particles and mesh.
- Extensible particle data structure support to meet application needs of particles per cell and frequency of particles transiting cells.
- Mesh distribution based on core plus buffer parts to ensure on-process movement of particles in a single particle motion step.
- Dynamic load balancing of meshes and particles.
- Execution of all operations on GPUs.
- Distributed mesh level equation solver on both CPUs and GPUs
- Performance portability using Kokkos.
- Supports XGCm gyrokinetic fusion, GITRm impurity tracking and PumiTally that integrates into OpenMC for neutronics applications.
- Adjacency search with support for particle-wall interactions to support GITRm.
PUMIPic Specifics:
Full support for distributed mesh and particle data structures on massively parallel HPC computing systems controlled through a set of distributed and data parallel methods.
Adjacency based search methods, including extensions to support near surface physics, are effective for unstructured mesh-based particle simulations.
Two data structures that associate particles to mesh elements (Sell-C-Sigma and AoSoA (using ECP Copa Cabana)) support high density particle applications. The DPS data structure supports lower density with highly variable particle distributions.
Dynamic load balancing is supported using EnGPar multi-graph procedure to account for evolving non-uniform particle distributions.
Initial applications addressed:
- GITRm which is a HPC unstructured mesh impurity transport code with the same physics as the original GITR code.
- XGCm that includes a subset of the physics of the XGC edge plasma code.
- PumiTally a GPU accelerated unstructured mesh tally system that integrates into OpenMC for neutronics simulations.
Mesh partitioned for parallel calculations
Adjacency based particle tracking
Particle data structures
Components of a push operation
Excellent performance for PIC applications:
- GITRm supporting large particle count simulations on HPC systems in support of on-going physics studies.
- XGCm demonstrated excellent scaling properties and, for the limited physics case addressed, showed substantial speed-up over XGC
XGCm weak scaling result on Summit supercomputer using the cyclone ITG case. [Zhang et. al, https://scorec.rpi.edu/REPORTS/2023-1.pdf]
XGCm time cost results on Summit supercomputer using the Cyclong ITG test case. The number of particles used in each GPU is changed, while the total number of GPUs used is kept constant.
Efforts are underway to extend PUMIPic to support unstructured mesh based neutronics calculations:
- Focused on excellent performance on GPU accelerated HPC systems
- Initial results indicate substantial performance gains for execution of unstructured mesh tallies.