Transformer program inspection reveals hidden errors beyond token accuracy
What Accuracy Hides In Learned Transformer Programs
Machine Learning
Summary
Transformer Programs are designed so their learned computations can be turned into readable Python code. The authors found that looking only at token accuracy can hide mistakes in full answers. They showed how carefully inspecting these programs reveals errors missed by standard tests, which helped improve training by adding targeted examples. Sometimes programs scored perfectly on sample tests but failed on other valid inputs, a problem uncovered only by thorough program inspection. This extra insight can help make transformer learning more reliable on algorithmic tasks.
What this means in practice
- •For machine learning engineers: Use program inspection to detect and fix hidden errors in transformer models beyond token-level accuracy on algorithmic tasks.
- •For software testers: Incorporate exhaustive testing and program inspection to catch subtle failures in AI-generated outputs that sampled tests miss.
Authors
Jonathan Lee, Narmeen Oozeer, Philip Quirke
Abstract
Transformer Programs (TP) (Friedman et al., 2023) constrain a Transformer's architecture so that its learned computation can be extracted as readable Python code. TP expose learned computations, but token accuracy can hide incorrect complete answers. In this work, we examine how program inspection provides insights beyond accuracy scores and helps guide training. We first evaluated TP on nine algorithmic tasks. We compared token accuracy with accuracy on complete output sequences and checked for failures missed by sampled tests. Further, we assessed training data improvements informed by program inspection and failure analysis, finding gains in sequence accuracy on several tasks. We examined a counting task where a program achieved 100% accuracy on sampled test sequences but failed on another input using the symbols and lengths allowed during training. The program counted correctly, but the required answer was missing from its output labels. After adding training examples with larger counts, the retrained models passed exhaustive evaluation over the bounded domain. In the frequency ranking task, inspection showed how correct counts contributed to incorrect answers in programs trained before and after changes to the training data. Finally, we studied constraints imposed by TP training and operations. The TP training process can lead to accuracy losses late in training. Our reference constructions also show how composing the restricted TP operations can use many layers or large lookup tables.