TellTale: Blending Multi-Instance LoRA Text Encoders and a Zero-Shot LLM Judge for Ambivalence/Hesitancy Recognition in Videos
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
The authors developed TellTale, a method that detects hesitation or mixed feelings in interview videos using only the text transcripts, not the video or audio. They combined scores from two fine-tuned language models and a large language model prompted without training to rate each transcript chunk. Their approach learns from video-level labels without needing detailed annotation for each transcript part. Tested on a new dataset, TellTale performed much better than the previous video-based baseline. This shows text alone can be very effective for recognizing ambivalence or hesitancy in interviews.
Authors
Abdel-Karim Al-Tamimi, Ali Rodan
Abstract
We present TellTale, a text-only approach to ambivalence/hesitancy (A/H) recognition in interview videos, evaluated on the BAH dataset as part of the 3rd A/H Video Recognition Challenge (11th ABAW Workshop, ECCV 2026). Although the dataset provides video, audio, facial crops, and transcripts, TellTale relies on the transcript alone and combines three probability streams. Two text encoders, multilingual-e5-large and mDeBERTa-v3-base, are fine-tuned with parameter-efficient LoRA adapters under a multiple-instance learning (MIL) objective, in which transcript chunks are scored individually and pooled with a smooth maximum so that only the video-level label is needed for supervision. The third stream requires no training: a quantized 14B instruction LLM is prompted, zero-shot, to rate each transcript for A/H. The three probabilities are combined by a weighted average and a single decision threshold, both selected on participant-grouped cross-validated predictions. On the organizer-scored private test set of 152 videos from unseen participants, TellTale achieves a Macro-F1 of 0.7364 and an average precision of 0.7940, compared with 0.2827 Macro-F1 for the official vision-based baseline.