Typed decision models match large language models on tabular data tasks
Typed Decision Models on Tabular Data
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
This paper studies special language models that choose from fixed answer options rather than generating free text, focusing on data organized in tables. The authors compare four such decision models to three popular large language models across tasks like classifying rows and finding anomalies. They find decision models perform just as well but cost less and are often more stable when questions are phrased differently. However, none of the models handle changes in table layout well, causing many answers to shift. This means these decision models work well but need improvements in handling table formats for reliable use.
What this means in practice
- •For data analysts: Use typed decision models to classify and verify tabular data rows more cost-effectively than large language models.
- •For quality assurance teams: Deploy decision models to detect anomalies in tabular datasets while monitoring for robustness to data and question changes.
Authors
Baowen Zhang, Hangting Ye, Hongyuan Zha
Abstract
Typed decision models are language models that answer a question with a fixed set of possible answers by returning a probability for each answer instead of generating text. We study them on tabular data and compare four decision models with three large language models (LLMs) on row classification, fact verification and anomaly detection. Every request is sent in its original form and under 21 types of rewrite that keep its content, such as renaming the options or writing the table in another format. The decision models perform as well as the LLMs at a lower cost. Three of them are also far more robust to how the question is formulated. No model we test is robust to how the table is organized. For every model, at least a third of the row decisions change under some rewrite of the table or the question. The decisions to which a decision model gives low probability are the ones most likely to change. Decision models are a practical choice for tabular decisions, but their robustness remains an open problem.