Summary
Public rankings of language models don't help engineers pick the best one for their specific needs. The authors propose a way to test models in the exact context of an application, like database migration or chatbots, by running each as separate containers with their own data and evaluation methods. This method tracks detailed results over time, ensuring fair comparisons and checking for noise in measurements. Their tests show that model choice matters more than testing setup, and that success reported by systems can be misleading compared to independent checks. The approach helps engineers understand how well models work in their own use cases instead of relying on general rankings.
What this means in practice
- •For software engineers: Evaluate and compare language models tailored to specific enterprise automation tasks to choose the best fit under cost and architecture constraints.
- •For database migration teams: Use the containerized testing approach to verify real-world Oracle-to-PostgreSQL migration agents for correctness before deployment.
Authors
Oleg Grynets, Vasyl Lyashkevych, Andrii Salyk, Dmytro Koziuberda, Dmytro Kostetskyi, Oleh Kaskun, Olena Pochernina, Alona Seletska, Oleksii Ilchuk, Viktoriia Tsymbal, Daryna Tukalo, Bohdan Saukh, Ivan Fedorchuk, Hryhorii Petrenko
Abstract
Public leaderboards rank large language models (LLMs) in general, but an engineer has to choose a model for a specific application, architecture and cost. This paper presents a method for application-specific capability profiling of LLMs and its realization as a containerized testbed. The method treats each application as a unit of extension: an Oracle-to-PostgreSQL migration agent, a documentation-grounded chatbot or an enterprise-support automation agent runs in its own container behind a small contract, brings its own dataset, evaluator and owned attributes, and is onboarded together with its dashboards. Measurements accumulate in a versioned, sparse metamodel of every model. The method separates the model under test from judges and agents, freezes and hashes plans, refuses to compare values from different contexts, diagnoses the instrument before the system, and applies a null-agent check and a noise rule whose operating characteristics are quantified by simulation. In the evaluated automation conditions, model choices produced larger differences on selected observable outcomes than framework choices, although the experimental design does not establish a general causal ranking of these factors. The experiments also show that instruction tuning is deployment-specific and that system-reported success can exceed independently observed success: of 216 migration units checked against a real Oracle database, 108 were demonstrably correct. These measurements establish application-specific observations, not a universal model ranking.