Summary
Kolmogorov-Arnold Networks (KANs) are a type of neural network that uses flexible functions instead of fixed ones to improve understanding and efficiency. This paper tests how well KANs can be trained on several GPUs working together in a high-performance supercomputer. The study found that KAN training scales reasonably well with more GPUs, showing similar performance gains to regular neural networks. Communication between GPUs and the size of the model affect training speed, but overall, KANs can be efficiently used in large distributed setups. The authors also offer advice on how to best set up these networks on supercomputers, though they note that their tests used a simplified problem.
Kolmogorov-Arnold NetworksMulti-Layer Perceptronactivation functionsdistributed trainingGPUsparallel efficiencydata-parallelismPyTorch Distributed Data Parallelscalabilitycommunication overhead
Abstract
Kolmogorov-Arnold Networks (KANs) replace the fixed activation functions and linear weights of Multi-Layer Perceptrons (MLPs) with learnable univariate functions on network edges, offering improved interpretability and, in some settings, competitive parameter efficiency. While the approximation properties of KANs have received considerable attention, their behavior under distributed, multi-GPU training has not been systematically characterized. This paper presents an empirical scalability study of data-parallel KAN training on multi-node, multi-GPU high-performance computing (HPC) infrastructure, evaluated along four dimensions: strong scaling, weak scaling, communication overhead, and model-size scaling. Experiments were conducted on the FinisTerrae III supercomputer using up to 8 NVIDIA A100 GPUs across 4 nodes with PyTorch Distributed Data Parallel (DDP). KAN training reaches 74.7% parallel efficiency at 8 GPUs with a 5.97x speedup, consistent with conventional deep learning workloads. Weak scaling shows an initial single-to-multi-GPU throughput drop followed by strong stability. Communication overhead follows a non-monotonic pattern (1.3%-6.1%), driven primarily by All-Reduce algorithm selection and inter-node latency rather than KAN's edge-wise gradient structure. The parameter-to-memory ratio improves with model size even as training time scales unfavorably. These results indicate that operator-level and data-parallel optimizations for KAN are complementary. We provide deployment guidelines for GPU topology and model-size selection, and discuss the limitations of a synthetic-regression evaluation.