Papers for

multimodal analytics developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Calibrating language model ordinal outputs with few labels improves accuracy

CORDIAL: Calibrating Ordinal LLM Outputs from Few Labels

Abstract: A large language model (LLM) can turn a text into a distribution over an ordered scale, but that distribution is a noisy measurement: saturated, compressed or exaggerated, and biased in a consistent direction. We propose CORDIAL, which treats the model's output as a noisy reading of the true label and corrects it with a channel of five interpretable parameters. The channel is small enough for its posterior to be averaged from a handful of labels, and we prove that the resulting calibration preserves first-order stochastic order. On Amazon reviews and CMU-MOSEI transcripts with four LLMs, CORDIAL has the lowest log loss among nine calibrators in 76 of 80 settings with 5 to 100 labels; with 20 labels and the main 7B reader, it matches the strongest baseline using 28-54 labels. The same posterior lets us learn priors from other tasks and fuse several LLMs. Unrestricted calibrators such as Dirichlet calibration overtake it only as the calibration set grows into the hundreds or thousands.

Thu 24 SeptComputation and LanguageMachine Learning

The gist

Large language models can predict ratings or scores on an ordered scale from text, but their raw predictions are often biased or noisy. The authors propose CORDIAL, a way to adjust these predictions using only a few labeled examples to make them more accurate and reliable. Their method works well even with just 5 to 100 labels and outperforms many other calibration techniques on Amazon reviews and spoken transcript datasets. This calibration also allows combining outputs from different models or learning from other tasks to improve performance.

Open → 2609.29807v1