Interactive debugger adds live controls for python gpu computing
Interactive Debugger for Performance Portable Python HPC Kernels
Distributed, Parallel, and Cluster Computing
Summary
Debugging high-performance, GPU-accelerated Python code is difficult because existing tools are limited or require changing the program, which can hide bugs. The authors present PKDB, an interactive debugger that works directly on GPUs and multithreaded CPUs without altering the source code. It supports usual debugging actions like breakpoints and variable checks, plus unique features like running any code while paused and updating running kernels on the fly. This lets developers find and fix problems more easily in complex Python HPC programs.
What this means in practice
- •For high performance computing developers: Debug and modify GPU and threaded Python kernels interactively without restarting or changing code, preserving real device behavior.
- •For parallel computing engineers: Update and recompile kernels in running applications seamlessly to expedite development cycles for parallel programs.
Authors
Ivan Grigorik, Gabriel Kosmacher, George Biros, Milos Gligoric
Abstract
We propose PKDB, the first interactive debugger for GPU and multithreaded low-level kernels written in Python. Python is widely used in high performance computing (HPC), with frameworks such as PyKokkos translating Python-embedded domain-specific languages to native code that runs across OpenMP-threaded CPUs and various GPUs. Yet interactive debugging support for such code is absent: developers resort to print statements, framework-specific assertions, or CPU-only execution, the last of which requires altering the program or its data and can mask device-specific bugs. PKDB enables standard interactive debugging like breakpoints, stepping, and variable inspection while preserving actual on-device execution without source modification. Beyond these fundamentals, PKDB introduces two advanced capabilities that exploit the dynamic nature of Python and PyKokkos: (i) Live code evaluation, which lets developers execute arbitrary Python expressions or entire kernels in the middle of a paused kernel without restarting the process; (ii) Kernel call site substitution, which allows an actively running kernel to be updated and reloaded on the fly, so only the kernel is recompiled and re-executed without restarting the application. Our performance evaluation on Intel, AMD, and NVIDIA CPUs, and NVIDIA and AMD GPUs shows that PKDB introduces limited overhead and is practical for everyday use while introducing critical debugging features to the Python HPC ecosystem.