Small-world patterns in AI models predict better reasoning skills
Looking Inside LLMs: Small-World Connectivity as a Signature of Reasoning Performance
Artificial Intelligence
Summary
Figuring out how large language models (LLMs) think is tricky, so these authors looked inside the models instead of just at their answers. They found that models whose internal connections look like 'small worlds'—lots of tight groups linked by short steps—tend to reason better. The researchers studied specific parts of the model, called heads, and saw that important ones stick closely in groups rather than connecting widely. Using this insight, they came up with a way to prune the model that keeps its reasoning ability strong while making it simpler.
What this means in practice
- •For ai system engineers: Preserve reasoning performance while simplifying LLMs by pruning based on small-world connectivity patterns.
- •For cloud ai service teams: Optimize resource use in language model hosting by applying small-world guided pruning to maintain quality at lower cost.
Authors
Zheng Huang, Sansheng Cao, Enpei Zhang, Weikang Qiu, Elynn Chen, Xiang Zhang, Yaoqing Yang, Rex Ying, Dawei Zhou, Yujun Yan
Abstract
Understanding large language model (LLM) reasoning requires looking beyond behavioral performance to examine how reasoning ability is reflected in internal organization. Inspired by neuroscience findings linking higher intelligence to stronger small-world organization in functional brain networks, we investigate small-world connectivity as a structural signature of LLM reasoning. We construct functional graphs from attention-head activation similarities and find that a higher small-world index (SWI), capturing local clustering and short global paths, consistently correlates with better fluid reasoning performance across models and training checkpoints. Since local clustering is central to small-world organization, we further examine how heads important for model performance connect within and across communities. We find that these heads tend to have a larger share of connection weight within their own communities (high core scores) and a more concentrated weight distribution across communities (low bridge scores). These observations motivate the hypothesis that high core and low bridge scores serve as structural indicators of head importance for reasoning capability. We validate this hypothesis through pruning, introducing Small-World Allocation (SWA), a hierarchical sparsity allocation method guided by these scores. Across six LLMs, SWA better preserves small-world organization and model performance than competing allocation strategies, reducing WikiText perplexity by up to 20%. Together, these findings identify small-world functional connectivity as a measurable signature of LLM reasoning performance, offering a structural perspective that complements behavioral evaluation.