Modular prompt optimization improves stance detection in language models
MoPrO-SD: Modular Prompt Optimization for Stance Detection
Computation and Language
Summary
Stance detection, or figuring out a person's opinion in text, is tough for large AI language models because opinions are often hidden or expressed indirectly. The paper presents a new method called MoPrO-SD that breaks down prompts into smaller parts and improves them step-by-step by asking the model for feedback. The authors tested this method on several datasets, including a new vaccine-related stance dataset, and found it often worked better than human-written instructions. This approach helps AI better identify opinions in tricky language without changing the model itself.
What this means in practice
- •For social media analytics teams: Improve detection of opinions and stances in social media posts using modular prompt refinements optimized for class-specific errors.
- •For public health monitoring teams: Better identify public stances on vaccines for real-time sentiment tracking with a specialized vaccine stance detection dataset and modular prompt tuning.
Authors
Meysam Sabbaghan, Nikesh Gyawali, Arman Zareian Jahromi, Sebastian Loera-Lozano, Doina Caragea, Cornelia Caragea
Abstract
Stance detection remains challenging for large language models (LLMs) due to implicit opinions, sarcasm, and context-dependent expressions of stance. Although automatic prompt optimization offers an alternative to fine-tuning without updating model parameters, existing structured approaches are not specifically designed around stance-label semantics and class-specific ambiguities. We introduce MoPrO-SD, a stance-specific prompt optimization framework that decomposes prompts into functional semantic modules and refines them sequentially using LLM-based feedback. We study three refinement strategies: instruction-only optimization (I), instruction with reasoning (I+R), and instruction with per-example reasoning (I+R+E), using class-conditioned errors for stance-label refinement. We also introduce VaccineStance, a new human-annotated benchmark for vaccine stance detection. Experiments across four datasets and five open-weight LLMs show that MoPrO-SD improves upon the human-written Base prompt in many settings. On SemEval-2016, class-conditioned feedback achieves higher mean Macro-F1 than pooled feedback for four of five backbones. Both reasoning-based strategies, I+R and I+R+E, achieve strong performance, with their relative effectiveness varying across models and datasets, while remaining competitive with recent prompt optimization methods. The MoPrO-SD code and VaccineStance dataset are available on GitHub.