18 Profiling
Profiling tools and memory events tracking.
18.1 MPI Calls
You can use the mpiP tool (https://github.com/LLNL/mpiP) to get statistics about the MPI calls in IPPL.
To use it, download it from Github and follow the instructions to install it. You may run into some issues while installing, here is a list of common issues and the solution: - On Cray systems “MPI_Init not defined”: This I fixed by passing the correct Cray wrappers for the compilers to the configure: ./configure CC=cc FC=ftn F77=ftn - If you have an issue with it not recognizing a function symbol in Fortran 77, you need to substitute the line echo "main(){ FF(); return 0; }" > flink.c (line 706) in the file configure.ac by the following line echo "extern void FF(); int main() { FF(); return 0; }" > flink.c - During the make all, if you run into an issue of some Testing file not recognizing mpi.h, then you need to add the following line CXX = CC in the file Testing/Makefile.
If the installation was successful, you should have the library libmpip.so in the mpiP directory.
To instument your application with the mpiP library, add the following line to your jobscript (or run it in your command line if you are running locally/on an interactive node): export LD_PRELOAD=$[path to mpip directory]/libmpiP.so To pass any options to mpiP, you can export the variable MPIP with the options you want. For example, if you would like to get a histogram of the data sent by MPI calls (option -y), you would need to add the following line to your jobscript: export MPIP="-y"
If you application has been correctly instrumented, you will see that mpiP has been found and its version is printed at the top of the standard output. At the end of the standard output, you will get the name of the file containing the MPI statistics: Storing mpiP output in ...
To get a total amount of bytes moved around by your application, you can use the python script mpiP.py (found in the top level IPPL directory) in the following form: python3 mpiP.py [path/to/directory] where path/to/directory refers to the place where the .mpiP output can be found. This python script will then print out the total amount of Bytes moved by MPI in your application.
18.2 Profiling on LUMI
18.2.1 rocprof
Analysis with: https://ui.perfetto.dev/
#!/bin/bash -l
#
#SBATCH --job-name=opalx1
#SBATCH --error=opalx-%j.error
#SBATCH --output=opalx-2-%j.out
#SBATCH --time=00:05:00
#SBATCH --partition=standard-g
#SBATCH --nodes 1
#SBATCH --ntasks-per-core=1
#SBATCH -c 56 --threads-per-core=1
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
#SBATCH --account=project_465001705
#SBATCH --hint=nomultithread
#SBATCH --hint=exclusive
CPU_BIND="map_cpu:49,57,17,25,1,9,33,41"
export MPICH_GPU_SUPPORT_ENABLED=1
ulimit -s unlimited
export EXE_DIR=/users/adelmann/sandbox/opalx/build/src/
module load cray-python/3.11.7
module use /appl/local/containers/test-modules
module load LUMI/24.03 partition/G cpeAMD rocm/6.1.3 buildtools/24.03
cat << EOF > select_gpu
#!/bin/bash
export HIP_VISIBLE_DEVICES=\$SLURM_LOCALID
exec \$*
EOF
chmod +x ./select_gpu
srun ./select_gpu rocprof --hip-trace ${EXE_DIR}/opalx input.in --info 5
rm -rf ./select_gpu
18.2.2 omniperf (do not use omnitrace)
doc url: https://rocm.docs.amd.com/projects/rocprofiler-compute/en/docs-6.2.4/how-to/profile/mode.html
#!/bin/bash -l
#
#SBATCH --job-name=opalx1
#SBATCH --error=opalx-%j.error
#SBATCH --output=opalx-2-%j.out
#SBATCH --time=00:05:00
#SBATCH --partition=standard-g
#SBATCH --nodes 1
#SBATCH --ntasks-per-core=1
#SBATCH -c 56 --threads-per-core=1
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
#SBATCH --account=project_465001705
#SBATCH --hint=nomultithread
#SBATCH --hint=exclusive
CPU_BIND="map_cpu:49,57,17,25,1,9,33,41"
export MPICH_GPU_SUPPORT_ENABLED=1
ulimit -s unlimited
export EXE_DIR=/users/adelmann/sandbox/opalx/build/src/
module load cray-python/3.11.7
module use /appl/local/containers/test-modules
module load LUMI/24.03 partition/G cpeAMD rocm/6.1.3 buildtools/24.03
module load omniperf
cat << EOF > select_gpu
#!/bin/bash
#export ROCR_VISIBLE_DEVICES=\$SLURM_LOCALID
export HIP_VISIBLE_DEVICES=\$SLURM_LOCALID
exec \$*
EOF
chmod +x ./select_gpu
srun ./select_gpu omniperf profile --name opalx --roof-only --kernel-names -- ${EXE_DIR}/opalx input.in --info 5
rm -rf ./select_gpu18.3 Kokkos Profiling Tools (MemoryEvents)
!Ported from old Doxygen, review required!
In certain applications, you might want to use profiling tools for debugging and testing. Since IPPL uses Kokkos as a backend, you can leverage Kokkos’ built-in profiling tools.
This guide explains how to use Kokkos’ profiling tools, using the MemoryEvents tool as an example.
18.3.1 Description of MemoryEvents
MemoryEvents tracks a timeline of allocation and deallocation events in Kokkos Memory Spaces. It records time, pointer, size, memory-space-name, and allocation-name. This is in particular useful for debugging purposes to understand where all the memory is going.
Additionally, the tool provides a timeline of memory usage for each individual Kokkos Memory Space.
The tool is located at: https://github.com/kokkos/kokkos-tools/tree/develop/profiling/memory-events
18.3.2 1. Clone the Kokkos Tools Repository
First, clone the Kokkos tools repository, which contains a variety of profiling tools:
git clone https://github.com/kokkos/kokkos-tools18.3.3 2. Build and Install the Tools
Navigate into the repository and build the tools using CMake:
cd kokkos-tools
cmake ..
make -j
sudo make install18.3.4 3. Set Up the Profiling Tool
Before running your application, export the Kokkos Tools environment variable to point to the kp_memory_events.so tool:
export KOKKOS_TOOLS_LIBS={PATH_TO_TOOL_DIRECTORY}/kp_memory_events.so Replace {PATH_TO_TOOL_DIRECTORY} with the actual path where the tool is located.
18.3.5 4. Run your Application
Execute your application normally. The MemoryEvents tool will automatically collect data during execution. For example:
./application COMMANDS18.3.6 5. Output Files
The MemoryEvents tool will generate the following files:
HOSTNAME-PROCESSID.mem_events:Lists memory events.HOSTNAME-PROCESSID-MEMSPACE.memspace_usage:Provides a utilization timeline for each active memory space.
18.3.7 6. Example on with SLURM
Here’s an example of how to run the profiling with a SLURM system using sbatch:
sbatch -n 2 --wrap="export KOKKOS_TOOLS_LIBS=$HOME/kokkos-tools/kp_memory_events.so; \
mpirun -n 2 LandauDamping 128 128 128 10000 10 FFT 0.01 LeapFrog --overallocate 2.0 --info 10"In this example:
sbatch -n 2specifies 2 nodes.- The Kokkos tool is exported and applied to the
LandauDampingapplication.
This guide provides the basic steps for integrating Kokkos profiling tools into your IPPL-based projects. You can adjust the commands as needed depending on your specific application and environment.
18.3.8 Example
Consider the following code:
#include <Kokkos_Core.hpp>
typedef Kokkos::View<int*,Kokkos::CudaSpace> a_type;
typedef Kokkos::View<int*,Kokkos::CudaUVMSpace> b_type;
typedef Kokkos::View<int*,Kokkos::CudaHostPinnedSpace> c_type;
int main() {
Kokkos::initialize();
{
int N = 10000000;
for(int i =0; i<2; i++) {
a_type a("A",N);
{
b_type b("B",N);
c_type c("C",N);
for(int j =0; j<N; j++) {
b(j)=2*j;
c(j)=3*j;
}
}
}
}
Kokkos::finalize();
}This will produce the following output:
HOSTNAME-PROCESSID.mem_events
# Memory Events
# Time Ptr Size MemSpace Op Name
0.311749 0x2048a0080 128 CudaHostPinned Allocate InternalScratchUnified
0.311913 0x2305ca0080 2048 Cuda Allocate InternalScratchFlags
0.312108 0x2305da0080 16384 Cuda Allocate InternalScratchSpace
0.312667 0x23060a0080 40000000 Cuda Allocate A
0.317260 0x23086e0080 40000000 CudaUVM Allocate B
0.335289 0x2049a0080 40000000 CudaHostPinned Allocate C
0.368485 0x2049a0080 -40000000 CudaHostPinned DeAllocate C
0.377285 0x23086e0080 -40000000 CudaUVM DeAllocate B
0.379795 0x23060a0080 -40000000 Cuda DeAllocate A
0.380185 0x23060a0080 40000000 Cuda Allocate A
0.384785 0x23086e0080 40000000 CudaUVM Allocate B
0.400073 0x2049a0080 40000000 CudaHostPinned Allocate C
0.433218 0x2049a0080 -40000000 CudaHostPinned DeAllocate C
0.441988 0x23086e0080 -40000000 CudaUVM DeAllocate B
0.444391 0x23060a0080 -40000000 Cuda DeAllocate AHOSTNAME-PROCESSID-Cuda.memspace_usage
# Space Cuda
# Time(s) Size(MB) HighWater(MB) HighWater-Process(MB)
0.311913 0.0 0.0 81.8
0.312108 0.0 0.0 81.8
0.312667 38.2 38.2 81.8
0.379795 0.0 38.2 158.1
0.380185 38.2 38.2 158.1
0.444391 0.0 38.2 158.1HOSTNAME-PROCESSID-CudaUVM.memspace_usage
# Space CudaUVM
# Time(s) Size(MB) HighWater(MB) HighWater-Process(MB)
0.317260 38.1 38.1 81.8
0.377285 0.0 38.1 158.1
0.384785 38.1 38.1 158.1
0.441988 0.0 38.1 158.1HOSTNAME-PROCESSID-CudaHostPinned.memspace_usage
# Space CudaHostPinned
# Time(s) Size(MB) HighWater(MB) HighWater-Process(MB)
0.311749 0.0 0.0 81.8
0.335289 38.1 38.1 120.0
0.368485 0.0 38.1 158.1
0.400073 38.1 38.1 158.1
0.433218 0.0 38.1 158.1Happy profiling!