Graph prompting improves frozen models by balancing representation signals
Spectral Reversal: Counteracting Singular Value Bias for Graph Prompting
Machine LearningArtificial IntelligenceSocial and Information Networks
Summary
Pre-trained graph neural networks often focus too much on the strongest signals in their data, ignoring weaker but useful information. The authors found this happens due to a 'spectral bias' during training. They created a method called Spectral Reverse Prompt (SRP) that adjusts these signals to bring out the hidden details without changing the original model. This helps the network adapt better when faced with new types of data, making it more useful in practice.
What this means in practice
- •For machine learning engineers: Improve pre-trained graph models on new data types without retraining full models by reweighting spectral components using SRP.
- •For biomedical data scientists: Better adapt graph models analyzing biological networks by extracting underutilized spectral information from frozen encoders.
Authors
Hanxu Yang, Yuhuan Zhao, Xiaodong He, Zhao Kang
Abstract
Pre-training Graph Neural Networks (GNNs) via self-supervised learning has become a dominant paradigm, yet efficiently adapting frozen encoders remains a challenge. Graph prompting offers a parameter-efficient alternative to fine-tuning, but existing methods largely treat pre-trained models as opaque feature extractors, ignoring their internal spectral structure. In this work, we identify a systematic phenomenon in pre-trained GNNs, which we term spectral bias: optimization during pre-training disproportionately aligns representations with directions associated with large singular values, leaving low-energy directions under-explored. We show that these underutilized directions can encode complementary information that is beneficial for downstream adaptation, especially under distribution shift. To leverage this insight, we propose Spectral Reverse Prompt (SRP), a prompting framework that rebalances the spectral contributions of frozen GNN encoders. SRP applies a learnable soft-thresholding mask in the spectral domain to down-weight dominant directions while amplifying weaker ones. In addition, SRP incorporates a null-space augmentation module that captures variation in directions with minimal activation under the frozen encoder. Extensive experiments across multiple benchmarks demonstrate that SRP achieves state-of-the-art performance with minimal additional parameters, highlighting that reweighting spectral components is a principled and effective strategy for parameter-efficient graph adaptation.