Benchmark measures reasoning across limited instruction set languages
ISA-Bench: A Benchmark for Computational Reasoning Across Instruction Set Architectures
Artificial Intelligence
Summary
Language models usually do well when writing code in common programming languages like Python. This paper points out that they struggle with unusual or very basic instruction sets, like those with just subtraction or logic gates. The authors created ISA-Bench, a set of programming games with small instruction sets and tools to test how well models can write and run programs in these environments. They found that models reason better than just copy code but still fail with unfamiliar syntax, especially when needing to write correct, complex programs. The benchmark helps show where models break down in reasoning about programming in new settings.
What this means in practice
- •For ai developers: Test and improve language models’ ability to reason about unusual instruction sets beyond common programming languages.
- •For compiler engineers: Use the benchmark to evaluate how well tools convert computational ideas into correct instructions on uncommon architectures.
Authors
Aditya Pola, Arkaprava Majumdar, Vineeth N. Balasubramanian
Abstract
Large language model code generation benchmarks primarily evaluate well-resourced languages like Python and Java, where models benefit from abundant training data. They provide limited evidence about reasoning in unfamiliar computational models: deriving arithmetic from a single subtract instruction, coordinating parallel programs across communicating nodes, or wiring logic gates into circuits. We present ISA-Bench, a benchmark of programming games with constrained instruction sets. For each game we provide a full execution stack (parser, VM, and verifier), enabling automated evaluation with structured feedback for iterative refinement. Reasoning models achieve higher average solve rates than code-specialized and general-purpose models, but unfamiliar syntax remains a major source of failure. Models solve more tasks with iterative feedback, though the gains vary substantially across architectures. We introduce a reasoning--execution gap (REG) analysis that reveals a recurring disconnect between identifying a plausible computational strategy and expressing it as a correct program in the target ISA. Code is open-sourced.