Benchmark tests large language models on software version rules

SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics

Artificial IntelligenceSoftware Engineering

Summary

Large language models (LLMs) like GPT often need to decide if software version numbers match certain rules, but it was never checked how well they understand these rules. The authors made a new benchmark called SemVerBench with questions from three major coding systems and tested six top LLMs. They found common predictable mistakes related to specific version rules, and showed some models do much better than others. They also showed that giving models a little extra hint fixes many mistakes, suggesting the problem is using knowledge, not lacking it. They recommend that coding tools should let specialized software handle version checking instead of relying on LLMs alone.

What this means in practice

  • For software development teams: Use SemVerBench to test and improve LLM-based tools that manage software dependencies and version constraints automatically.
  • For tool developers for package managers: Integrate SemVerBench results to avoid errors in LLM-enhanced version resolution and improve delegation to accurate resolvers.
  • For ai coding assistant makers: Build commercial coding assistants that delegate version constraint checks to verified resolvers to achieve near-perfect accuracy.$Commercial implications: This enables AI assistants to reliably handle version management tasks, making the product more trustworthy and useful.

Authors

Qibai Chen, Zeming Liu

Abstract

Large language model (LLM) coding agents constantly decide whether a version satisfies a constraint such as ^1.2.3 or >=2.0,<3, yet their grasp of version-constraint semantics has never been measured directly. We introduce SemVerBench, the first benchmark of LLM version-constraint resolution semantics across three ecosystems (npm, PEP 440, Cargo): 240 machine-checkable items with unique answers, built author-neutrally from four balanced sources (each ecosystem's official test suite plus three frontier LLM proposers) and labeled by a non-circular two-implementation oracle. Evaluating six frontier models, we find systematic, predictable per-mechanism blind spots: a partial-comparator carry rule (>1.2 means >=1.3.0) traps every model on Cargo (near 60%), and although standard PEP 440 prefix matching is universal, on zero-pad/post-release corner cases GPT-5.1 collapses (0/26) while Claude stays at 97-100% (verified on a 67-item oracle-validated set). Opus significantly outperforms all other models, and Sonnet outperforms the OpenAI models (McNemar). The failures look more like an activation/application gap than a knowledge gap: injecting the rule or a light correct hint recovers most errors, whereas interval decomposition does not, and models are at ceiling on the basic forms of the same rules. An author-stratified analysis finds no statistically significant self-favoritism. Because the task is verifiable and a free, 100%-correct resolver exists, tool delegation reaches ~100%: coding agents should delegate version resolution to a resolver rather than reason about versions in-head.
View PDFOpen arXiv