SigmaTrace enables detailed monitoring of CPU and GPU RDMA communication
SigmaTrace: Library-Agnostic Observability for CPU/GPU-Initiated RDMA Communication
Distributed, Parallel, and Cluster Computing
Summary
Communication using RDMA (Remote Direct Memory Access) lets computers talk very quickly, but it's hard to see exactly what's happening during data transfers, especially when GPUs start the communication. The authors created SigmaTrace, a tool that can watch these data exchanges in fine detail without needing to change existing software. It works across different communication libraries on both CPUs and GPUs, adding only a small delay. SigmaTrace reveals hidden communication patterns that older tools miss, helping engineers understand and improve system performance.
What this means in practice
- •For high performance computing teams: Monitor detailed RDMA activity for complex CPU/GPU systems without modifying existing libraries.
- •For data center network operators: Identify hidden traffic patterns and imbalances in RDMA networks involving GPUs to optimize communication.
Authors
Javid Baydamirli, Didem Unat
Abstract
RDMA networking enables low-latency, high-bandwidth communication on large systems. Despite its adoption, however, observability is lacking. Existing tools target specific communication libraries or only report aggregated data, missing per-operation behavior. The gap is widest for GPU-initiated networking, whose in-kernel RDMA operations are invisible to the host. We present SigmaTrace, a library-agnostic RDMA profiling framework providing fine-grained, per-operation visibility for CPU and GPU-initiated networking. It traces operations at the libibverbs/mlx5dv layers, the common denominator beneath communication libraries on InfiniBand and RoCE, and employs a low-overhead memory interposition technique that, to our knowledge, gives the first observability into GPU-initiated RDMA without source patching or binary instrumentation. Host-side tracing adds 1.3% overhead on NCCL AllReduce at 128 GPUs and under 2% on RCCL at 1024 GPUs. On a persistent-kernel BFS and Mixture-of-Experts inference, SigmaTrace exposes an in-kernel barrier spread and 2.13x inter-node traffic skew invisible to existing profilers.