Increasing chess skill boosts deeper attention use in transformer models
Increasing Skill Level Recruits Deeper Attention Layers in a Frozen Chess Transformer
Artificial IntelligenceMachine Learning
Summary
The paper looks at how a chess-playing AI changes its focus when it is told to play at different skill levels. By adjusting the AI to act like a player of varying skill, the researchers found that higher skill levels make the AI use more complex and deeper layers of its 'attention' mechanism. This effect is especially strong for certain tricky moves like knight forks. One shallow attention layer stays active regardless of skill, while others shift deeper for specialized tasks. These findings help us understand how AI changes its internal thinking depending on the skill context.
What this means in practice
- •For ai developers: Improve transformer models by conditioning on skill level to allocate computational focus dynamically based on task complexity.
- •For game ai designers: Create more human-like chess engines by adjusting internal attention depth to mimic different playing strengths and styles.
Authors
David Litman
Abstract
Chess involves complex reasoning in a deterministic environment, which makes it a useful setting for studying the mechanisms of computation inside transformers. The Maia-3 chess transformer takes Elo, a measure of competitive chess skill, as an input to the pre-trained network, so we can vary the skill the network is conditioned on with no change to its weights. Here we investigate how turning this skill dial affects self-attention. Ablating every attention head at every Elo from 700 to 2500, we find 1) increasing skill pushes the causal center of mass of the computation deeper, monotonically, for every chess piece and move type we measured; 2) the depth migration is much greater for specific tactics, especially knight forks, than for other move types; 3) the migration consists of deeper heads getting recruited for more specialized computations while one shared shallow head keeps a roughly constant contribution. These results may shed light on how conditioning inputs redistribute computation in larger transformers.