Small language model quickly detects scam calls with calibrated confidence
Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Model
Computation and Language
Summary
Phone calls can sometimes be scams, and it's important to know quickly if a call is risky. The authors tested a way to use a small language model to decide if a call is a scam right after each part of the conversation. This method gives a trustworthy probability instead of a generated text answer, which makes decisions faster. They found it works about as well as bigger models but much faster and with fewer false alarms on safe calls.
What this means in practice
- •For call center operators: Improve fraud detection in phone calls by integrating fast, calibrated scam likelihood scores per caller turn.
- •For customer support platform developers: Add efficient scam screening with reliable probability estimates to reduce false alarms and speed up hang-up decisions.
Tested on simulated data.
Authors
Simiao Ren, Kidus Zewde, Xingyu Shen, Yuchen Zhou, Dennis Ng, Ankit Raj, Tommy Duong, Yuxin Zhang, Neo Tiangratanakul
Abstract
Screening a phone call for fraud needs a trustworthy probability after every caller turn, in milliseconds. Jev-style typed decisions promise exactly that: declared options go in, one calibrated probability per option comes out of a single forward pass, with no generated text. We test an open implementation of this readout, JevLite, on scam-call screening: Qwen3-4B is LoRA-tuned so that the temperature-scaled softmax over two answer-label logits is P(scam). On 41 held-out CallScreenBench scenarios (577 per-turn decisions) a three-seed ensemble reaches AUROC .974 with calibration error .052, non-inferior to an LLM judge (MiniMax-M3) at a pre-registered .02 margin, with no false alarms on legitimate calls, decisions 1.14 turns earlier under the same hang-up rule, and 64.5 ms per decision on one consumer GPU, 4.9x lower than the same backbone fine-tuned to generate its answer. The gain is in the readout and calibration, not accuracy: a fine-tuned ModernBERT encoder is not significantly worse, the recipe was selected with test-set exposure, and all callers are synthetic. We claim no architectural novelty; the contribution is the application and an evaluation reporting calibration, false alarms and decision timing alongside AUROC.