Subspace Clifford Networks cut training time and parameters in image classification
Subspace Clifford Networks
Computer Vision and Pattern Recognition
Summary
Training neural networks to recognize images can take a lot of time and computing power. The authors propose a new way to organize how the network’s internal features interact, grouping them in small sets rather than mixing all together. This results in fewer parameters and faster training while maintaining or improving accuracy. They also design efficient code that runs these grouped interactions quickly on GPUs. Their experiments on image classification show that their method speeds up training and performs better than some existing approaches.
What this means in practice
- •For machine learning engineers: Reduce training time and model size for image classifiers by using structured subspace channel interactions.
- •For gpu kernel developers: Implement efficient fused CUDA kernels for grouped low-dimensional feature interactions to speed up deep learning workflows.
Tested on one dataset.
Authors
Zhongping Ji
Abstract
In this work, we propose Subspace Clifford Networks, a reformulation that replaces global cyclic rolling with grouped subspace interactions. Instead of sampling sparse cyclic diagonals across the entire $D$-dimensional channel space, we project the features into an aligned space $\R^{\tilde{D}} = \bigoplus_{g=1}^{G} V_g$ partitioned into $G$ low-dimensional groups $V_g \cong \R^k$ with $k \in \{2, 3, 4\}$. Within each $k$-dimensional group, we compute all $\binom{k}{2}$ wedge coordinates and concatenate them with $\operatorname{SiLU}$-activated coordinate-wise products, while sandwiching the grouped interaction between learnable $1\times 1$ linear projections to enable cross-group channel mixing. This design reduces intermediate feature expansion from $2|\mathcal{S}|D$ (typically $10D$ for 5 shifts) down to $1.5\tilde{D}$ ($k=2$), $2.0\tilde{D}$ ($k=3$), or $2.5\tilde{D}$ ($k=4$). Furthermore, exploiting the small, fixed group size $k \in \{2, 3, 4\}$, we implement a fused CUDA kernel that evaluates the forward interaction and a recomputation-based analytical backward pass using thread-local variables without allocating global-memory buffers for elementwise intermediates. Finally, we extend the backbone from an isotropic columnar layout to a multi-resolution Hierarchical Clifford Network with configurable per-stage subspace dimensions. Experiments on CIFAR-100 show that the proposed models substantially reduce parameter count and training time across both regimes, while the hierarchical variants also improve classification accuracy over the evaluated shift-based hierarchical baselines.