Long-Context Fine-Tuning with Limited VRAM
Artificial Intelligence
Summary
The authors address the problem of training large language models on long sequences without using too much memory. They propose a method called Hierarchical Global Attention (HGA), which stores and processes only small chunks of relevant past information in VRAM, while offloading older data to slower memory. This allows training on sequences much longer than usual, with similar accuracy to standard methods but using less GPU memory. Their tests show HGA can handle sequences up to 16,384 tokens during training and over 131,000 tokens during evaluation, making it more scalable for long-context tasks.
Authors
Vladimir Fedosov, Aleksandr Sazhin, Artemiy Grinenko, Frank Woernle
Abstract
Parameter-efficient fine-tuning reduces model and optimizer memory, but dense attention still makes long training sequences expensive. We combine Hierarchical Global Attention (HGA) with segment-wise backpropagation and tiered KV storage. Only the active segment remains differentiable in VRAM; older KV is detached into RAM or NVMe, and HGA loads a bounded set of exact historical tokens for each query block. On Qwen3-8B with 4-bit QLoRA and PG19, dense training on a 16 GB Quadro RTX 5000 fits 2,048 tokens but fails at 4,096, whereas HGA reaches 16,384 tokens with 15.28 GB peak VRAM. Under evaluation the same adapter runs through 131,072 tokens on this card; VRAM is not constant but grows gently with the resident chunk summaries, so RAM and NVMe capacity set the practical limit beyond these lengths. At the shared 2K training length, HGA-trained and dense-trained adapters obtain 2.7405 and 2.7383 nat under the same dense-attention readout, while the stock model obtains 2.9541. At this boundary HGA training is already marginally faster (217.75 vs. 207.02 tokens/s), and the HGA-to-dense throughput ratio improves from 1K to 2K; because HGA keeps the attended historical set per token approximately constant while dense work per token grows, we expect this lead to widen as context grows. Dense attention is used for the main quality and retrieval comparisons so that they measure the learned weights and remain compatible with standard generation frameworks. HGA can also be used for retrieval and generation; an optimized production-grade serving implementation is under development.