Large language models generate reliable quantified inferences from tables
From Tables to Quantified Statements: Evaluating LLM Inference Generation through Executable Verification
Computation and Language
Summary
It can be hard for AI to make statements about data in tables that are actually true. The authors created a task where AI models write sentences using words like all, some, or most to describe stats from tables. They check if these sentences are true by writing and running Python code that tests the facts against the table. Bigger and newer AI models did better at making accurate and varied statements. Human reviewers agreed with the computer's checks, showing the method works well.
What this means in practice
- •For data analysts: Automatically generate trustworthy summaries with quantified claims directly from statistical tables to aid decision making.
- •For business intelligence teams: Create natural language inference explanations of tabular reports verified by executable checks for improved report validation.
Authors
Mai Mohamed Eida, Gunjan Anand, Ayush Singh, Aleksandre Maskharashvili
Abstract
LLMs can generate fluent descriptions from tables, but their outputs may remain logically unsupported by the structured data. We introduce STAT-TO-TEXT, a controlled task in which LLMs generate quantified natural language inferences from statistical tables using quantified constructions such as all, some, no, and most. To evaluate these inferences, we use an LLM generated Python checker code which when executed verifies the corresponding truth conditions against the table. We compare four open-weight LLMs across model families and scales, evaluating faithfulness, logical accuracy, table coverage, and diversity. Our results show that model scale and family matter, with the largest model (GPT-OSS-120B) consistently producing the most faithful inferences without sacrificing greater table coverage and quantifier diversity, as opposed to smaller models. These findings are supported by human annotation, which shows that the automated checker closely aligns with human judgments.