Continuous Thought Machine traces image classifications along a fixed pattern

Riding the Fixed-Point Curve: Image Classification in the Continuous Thought Machine

Machine LearningComputer Vision and Pattern RecognitionNeural and Evolutionary Computing

Summary

Classifying images is a complex task, and this paper looks at how a special type of neural network called the Continuous Thought Machine (CTM) does it. The network processes images step-by-step by moving through a series of stable states, like following a path of fixed points that change over time. Most of this processing path is shared for all images, which means the CTM uses a common strategy to understand different pictures. The authors also found that swapping parts of the network’s attention does not harm accuracy much, showing this process is very stable and generic.

What this means in practice

  • For deep learning engineers: Design image classifiers that follow stable fixed-point trajectories to improve interpretation of recurrent network predictions.
  • For neural network researchers: Develop analysis tools for recurrent models by examining non-normality and fixed-point movements during training and inference.

Authors

Thomas Hannagan

Abstract

The Continuous Thought Machine (CTM) is a recurrent network that attends to inputs and makes predictions based on the synchronization between its neurons. On tasks spanning many domains, it displays emergent thought processes that have not been explained. We report a number of findings about how the CTM classifies images, focusing on ImageNet. Each thinking step is an attracting but strongly non-normal map exhibiting balanced amplification (Murphy and Miller, 2009); while attraction is already present at CTM initialization, training highly promotes non-normality. By Banach's theorem, the neural state has a locally unique attracting fixed point, which moves at every step, and the thought trajectory is the curve these fixed points trace, followed one tick behind. Most of that curve's motion is shared across images, and visible as a previously reported traveling wave of activity across neurons. The attention schedule along the thought is largely generic: shuffling queries across images at every tick costs only 7 percent of classification accuracy, while image-independent increments predict the next query with R^2 = 0.98 (held out). Our results suggest that the CTM has found a single strategy to process all ImageNet images.