Language model weekday concepts behave as points not smooth circles
Spokes, not a Ring: Revisiting the Weekday Manifold in Language Models
Machine LearningArtificial Intelligence
Summary
The authors looked at how large language models represent the idea of weekdays. People thought weekdays might be stored as a smooth circle where you could slide gradually from Monday to Tuesday and so on. But the authors found weekdays are actually stored as seven separate points arranged in a circle, not a smooth ring. This means the model’s understanding jumps between distinct days instead of changing continuously.
What this means in practice
- •For natural language engineers: Improve techniques for guiding language model outputs by recognizing some concepts change discretely, not smoothly, affecting control methods.
- •For data visualization specialists: Design better visual tools for interpreting language model concepts by accounting for discrete states instead of continuous manifolds.
Authors
Diksha Gupta, Rieke Fruengel, Stefan Heimersheim
Abstract
Recent work has hypothesised that large language models may represent some concepts as multi-dimensional manifolds instead of single directions. Using these manifolds to interpret or steer representations often assumes that moving along them corresponds to a continuous change in the represented concept. In this work, we test this continuous manifold assumption on the prototypical weekday manifold. Despite its striking circular geometry, we find that the weekday representations behave more like seven discrete states arranged in a circle than a continuous ring. First, we show that local sensitivity has seven minima aligned with the weekdays, producing a step-like mapping between representation and behaviour under on-manifold steering. Second, we test whether any concepts are represented along the manifold outside the seven discrete states, and find neither time of day, uncertainty, nor natural text populate the intermediate space. Together, these results challenge the continuous interpretation of the weekday manifold and motivate testing whether movement along other representation manifolds correspond to meaningful changes in model representations and behaviour.