Jev model matches some large language models on stance detection tasks
Can Decision Models Understand Stance? Evaluating Jev Against General-Purpose LLMs
Computation and Language
Summary
Figuring out how someone feels about a topic in a conversation can be tricky. The paper looks at Jev, a special kind of model made to make clear decisions, and checks how well it understands people’s opinions compared to big general AI models. Jev does about as well as the top general AI on one English dataset but struggles more on a Chinese conversation dataset. The authors think Jev has trouble with knowing how replies connect and if opinions are for or against, not just with how long the conversation is.
What this means in practice
- •For social media monitoring teams: Use Jev to identify public attitudes toward topics in English text streams with accuracy comparable to advanced LLMs.
- •For multilingual chatbot developers: Incorporate Jev to improve stance understanding in English but rely on stronger LLMs for complex Chinese conversation dynamics.
Authors
Xing Li, Jinzhong Ning, Yijia Zhang, Liang Yang, Hongfei Lin
Abstract
Stance detection requires identifying an author's attitude toward a given target, sometimes based on conversational context. Jev, a specialized decision model designed for structured decision-making, offers an alternative to general-purpose large language models (LLMs). In this work, we evaluate Jev on two stance detection datasets, VAST (English texts) and ZS-CSD (Chinese conversations), comparing it with four general-purpose LLMs and two fine-tuned models. Results show that Jev achieves competitive performance on VAST, matching GPT-5.6 and outperforming the other general-purpose LLMs. However, it falls behind stronger LLMs on ZS-CSD, particularly in distinguishing favor from against. Further analysis suggests that this limitation may be related to understanding reply relationships and stance direction rather than conversation length alone. These findings highlight both the potential and limitations of Jev for stance detection.