FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts
2026-06-30 • Machine Learning
Machine Learning
AI summaryⓘ
The authors propose a new method for fine-tuning large models called Fractional-Fourier Mixture of Experts. Instead of sticking to one fixed way of adjusting model weights, their method learns how to best combine adjustments between two domains: spatial and spectral (Fourier). This lets the model flexibly pick the most efficient way to fine-tune for each task or layer, reducing interference between tasks and improving performance. Their approach adds little extra computational cost and works better than previous similar methods on various benchmarks.
Parameter-efficient fine-tuningLow-rank adaptersFourier transformFractional Fourier transformMixture of ExpertsLoRAMulti-task learningSpectral methodsModel optimizationTransform domains
Authors
Tom Saliencro, Maya Lindqvist, Rohan Desai, Priya Nair, Daniel Whitmore
Abstract
Parameter-efficient fine-tuning (PEFT) reparameterizes weight updates in a fixed basis: low-rank adapters operate in the spatial domain, while a recent line of spectral methods operates in a fixed Fourier domain. We argue that the choice of domain is itself a design degree of freedom that should be learned, and that no single basis is optimal across tasks, layers, or tokens. We introduce Fractional-Fourier Mixture of Experts, a mixture-of-experts adapter in which every expert carries a learnable fractional-Fourier order that continuously interpolates between the spatial domain (recovering vanilla LoRA) and the Fourier domain (recovering a spectral adapter). Routing tokens through experts that occupy different points on this spatial-spectral continuum lets the model place each low-rank update in the domain where it is most compact, and -- because fractional-Fourier operators of different orders are mutually incoherent -- makes the experts naturally decorrelated, which reduces interference and improves multi-task composition. The order is a single scalar per expert, trained with a separate optimizer, and the transform is computed with an $\mathcal{O}(d\log d)$ chirp--FFT surrogate, so Fractional-Fourier Mixture of Experts adds negligible cost over standard MoE-LoRA. Across commonsense, mathematical, code, and knowledge benchmarks on LLaMA-3.1-8B and Qwen2.5-7B, Fractional-Fourier Mixture of Experts improves over strong MoE-LoRA and spectral baselines -- including FlyLoRA, FourierMoE, and HMoRA -- while keeping the active-parameter budget small, and analysis shows that the learned orders specialize by task and layer in interpretable ways.