Benchy creates a universal language to benchmark AI tasks easily
Benchy: towards a universal language for task-oriented AI benchmarks
Artificial Intelligence
Summary
Benchy is a new way to describe and run tests that check how well AI programs work on specific tasks. It uses a simple and clear language that is the same no matter who writes the test or what AI system takes it. The language is written in a neat file format and then turned into a standard code form that a special engine understands to run the test. This setup keeps the test descriptions separate from the AI systems, making it easier to compare different AI fairly and consistently.
What this means in practice
- •For ai developers: Create and run standardized tests on AI programs to compare them reliably across different tasks using a single language.
- •For software testers: Integrate AI benchmark tests into software pipelines with a clear input-output format that separates test logic from AI code.
Authors
Francis F Daniel, Mauro Ibañez, Francis Perelman, Marian Basti
Abstract
Benchy is a semantic language and execution engine for benchmarking AI programs. A benchmark is completely specified by a program, a scoring function, and a dataset, B=(P,S,D), and is separate from the AI-system taking it; a run binds the two, R=(B,AI). Benchmarks are authored as canonical YAML in which each semantic concept has one valid syntax, classified by a shared task/domain/language ontology, and deterministically compiled into a canonical JSON intermediate representation that the engine executes. Compilation changes representation, not meaning: it does not repair invalid definitions or inject hidden defaults. Programs use fixed schemas of named input and output fields, the leaf output fields are the scoring dimensions, and the engine exposes one universal runtime contract --- a named-field input object in, a named-field output object out --- to which external AI-systems adapt at the boundary, so integration mechanics never propagate into benchmark semantics. This paper gives the semantic object model, the ontology and task-to-program validation rule, the scoring and failure semantics, the compilation and execution architecture, and the scope of the current language. An appendix fixes the normative engineering contract for the first engine implementation.