Internalizer maps context to parameters for huge language models
Internalizer: Portable Context-to-Parameter Mapping for Very Large Language Models
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Large language models have limited space to remember details from documents they read at once. The paper introduces the Internalizer, a tool that creates small extra pieces of memory for a very large language model by turning document context into model parameters. This lets the model better understand and use long documents without needing them all to fit in its usual memory space. The authors demonstrate this works on a model far bigger than those used in similar previous work.
What this means in practice
- •For language model engineers: Create context-specific adapters for extremely large frozen models to improve accuracy on long documents without retraining full model weights.
- •For cloud ai service providers: Serve user documents as adapters to large models for faster inference or improved accuracy without increasing context window size.$Commercial implications: Enables new products that personalize huge language model outputs efficiently by selling document-specific adapters as lightweight add-ons.
Authors
Peter Devine, Nick Ryan, Benjamin Sirb, Alex Chiocchi
Abstract
Hypernetworks that map a context directly to a LoRA adapter let a large language model carry that context in its weights, but prior work has demonstrated them only on base models of up to 14 billion parameters. We present the Internalizer, a state-of-the-art, portable Context-to-Parameter Mapping hypernetwork that generates document-specific LoRA adapters for the frozen 284B-parameter DeepSeek v4 Flash, a target two orders of magnitude larger than in any previous work. Most of its parameters live in a model-agnostic trunk with only thin entry and exit layers per base model, so it trains cheaply against small models before being ported to the large one. On unseen documents of up to 4096 tokens, the generated adapters reach 84.9% top-1 and 97.8% top-5 teacher-forced accuracy against 63.4% and 83.5% for the base model, with nothing in the context window but a three-word instruction. Once the hypernetwork is trained, a single forward pass turns any document into an adapter for such a model, which could be served alone for speed or alongside the document in the window to raise accuracy further.