Benchmark tests language models editing neural network weights
WeightBench: Evaluating Language Models as Weight-Space Programmers
Machine LearningArtificial Intelligence
Summary
It can be hard to understand or fix how one neural network works by looking inside it, especially for other AI systems. The authors created a test called WeightBench that sees how well large language models can read, fix, or write the parameters of other neural networks without retraining. They tested many tasks and found current models can handle small networks well but still struggle with bigger ones or harder tasks. This benchmark helps measure progress in having AI directly program other AI systems.
What this means in practice
- •For machine learning engineers: Assess and improve AI systems that can modify neural network weights directly without retraining to speed debugging and customization.
- •For ai model developers: Use the benchmark to evaluate tools that write or repair small neural networks from scratch by editing their parameters.
Authors
Aayush Marasini, Barsat Khadka, Nick Rahimi
Abstract
Recent work shows that large language models (LLMs) can operate directly on the parameters of other neural networks. They can read what a network was trained to do, fix a network that fails, and write working networks from scratch, all without gradient descent. This capability has been termed Weight-Space Programming (WSP). Exploring this new paradigm systematically requires a robust benchmark. We introduce WEIGHTBENCH, a benchmark that evaluates LLMs of increasing capability on 42 WSP tasks across five tracks: reading, fixing and writing weights, recognizing edits that are impossible, and data-free compression. The networks whose weights the LLMs read and edit range from 2K-parameter control policies to 7B-parameter language models. Experimental benchmark results show that frontier agents can already read small networks and write working control policies without ever running them, but cannot yet patch language models, compress them without data, or tell an impossible edit from a hard one. On all 42 tasks, Claude Opus 5.5 reaches a macro score of 0.535 (Sonnet 5.5 0.466, Haiku 5.5 0.402, DeepSeek v4-flash 0.215); on the 38 tasks it was run on, GPT-6.1-sol reaches 0.568 against Opus's 0.546. Code, the task registry, our OS-level sandbox harness, and the live leaderboard are publicly available at https://github.com/Aayush-Marasini/WeightBench.