G2LoRA: Gradient Orthogonal Low-Rank Adaptation Framework for Graph Continual Learning on Text-Attributed Graphs
2026-06-01 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study how to improve large language models that link text and graphs (Text-Attributed Graphs) when these models learn tasks one after another, without forgetting old ones. They identify two main problems: tasks are different which makes learning tricky, and the text and graph parts of the model adapt unevenly, causing misalignment. To fix this, they propose G2LoRA, a method that aligns all kinds of graph-related tasks under one goal, carefully manages learning updates to avoid conflicts, and balances how text and graph encoders change during training. Experiments show their method helps the model remember old tasks better and learn new tasks more effectively than existing methods.
Large Language ModelsText-Attributed GraphsContinual LearningCatastrophic ForgettingContrastive LearningGraph EncodersText EncodersGradient ProjectionTransfer LearningParameter-Efficient Fine-Tuning
Authors
Yuhan Wang, Yibo Ding, Yutong Ye, Mufan Zhao, Wenbo Zhang, Ruijie Wang, Jianxin Li
Abstract
LLM-as-Aligner has emerged as a prevalent pre-training paradigm for Text-Attributed Graphs(TAGS), aligning graph and text modalities into a shared embedding space via CLIP-style contrastive learning. While effective on individual downstream tasks, we observe severe catastrophic forgetting when such models are sequentially fine-tuned on streaming tasks. Although parameter-efficient fine-tuning alleviates forgetting to some extent, it remains insufficient to resolve task interference and ineffective knowledge transfer. In this work, we study graph continual learning for LLM-as-Aligner models on TAGs, with the goal of mitigating interference while promoting positive transfer across tasks. This setting introduces two fundamental challenges: (1) heterogeneous downstream tasks induce shifting optimization objectives, hindering unified fine-tuning; and (2) graph and text encoders exhibit different sensitivities to adaptation, making uncoordinated updates prone to misalignment. To address these challenges, we propose G2LoRA, a continual learning framework for TAGs. G2LoRA unifies node-, link-, and graph-level tasks under a single graph--text alignment objective, and enables consistent optimization across domain/class/task incremental modes. To reduce task interference while encouraging positive transfer, G2LoRA performs category-aware gradient projection in structured subspaces, resolving conflicting updates and enabling conditional backward transfer to balance forward and backward knowledge flow. To further prevent cross-modal drift, G2LoRA introduces gradient magnitude modulation to coordinate update rates between graph and text encoders. Extensive experiments on benchmark datasets demonstrate that G2LoRA consistently outperforms strong baselines across different backbone architectures, achieving superior continual performance and transferability.