Born-measured transformer shows exact word evidence on light and qubits
Reading Why: Born-Measured Attention with Exact Word Evidence on Light and Qubits
Computation and LanguageMachine Learning
Summary
Understanding why a language model makes a certain decision is often uncertain. The authors build a special transformer where every step uses quantum-inspired Born measurements, allowing them to exactly calculate how much each word contributes to the model's answer. This new model performs about as well as a normal transformer on a language test, but it can more precisely show which words influenced the decision. The same math can be run on optical devices or quantum computers, with some early tests run on a real quantum machine.
What this means in practice
- •For natural language engineers: Create interpretable language models whose decisions can be exactly decomposed into word contributions for clearer explanations.
- •For quantum algorithm developers: Implement natural language tasks on quantum hardware using models that compile exactly to quantum circuits with known complexity.
Authors
Teerawat Tongloy, Jiraporn Pongsopa, Rangsan Jomtarak
Abstract
When a language model answers, we can usually only estimate why. We build a transformer in which every step is a Born measurement - attention $|\langle q|k\rangle|^4$ normalised over each row, a measured product in place of the feed-forward nonlinearity, and a Born readout. With its denominators held at their values for the input, each class probability is a polynomial in the word amplitudes, so its integrated-gradient evidence is computed by a finite quadrature with no quadrature error: the words' contributions sum to the class probability above its no-information share, to $10^{-11}$ or better. On e-SNLI natural language inference, with about 98k parameters outside the embedding, the model reaches $76.98 \pm 0.17$% test accuracy, level with a softmax transformer of the same size ($76.62 \pm 0.36$%). Deleting the word its evidence ranks first changes the answer in 45% of test pairs, out of the 53% any single deletion can change, against 32% for the softmax model explained the same way and 40% by its attention x gradient. Attention can look more like the words annotators highlighted, yet accounts for the answer less well: deleting the word it ranks first changes the Born Transformer's answer in 32% of pairs. The same weights compile exactly to optoelectronic optics and to qubits, and a coherent attention layer becomes one circuit equal to the model (852 CNOTs on 23 qubits). On IBM's ibm_kingston, reading only the attention overlaps by Hadamard tests, the device gave the model's answers on two short test pairs and kept or changed them under deletion as the model does; the evidence itself did not yet survive the device, and we work out the shots and calibration that measuring it on hardware would need.