Long social media posts measured more accurately with new fusion method
Count Evidence, Not Sentences: Tempered Evidence Fusion of LLM Judgments for Long-Text Value Measurement
Computation and LanguageArtificial Intelligence
Summary
Measuring public opinions from long social media posts is tricky because not all sentences clearly show the writer’s stance. The authors show that simply counting sentences or guessing the overall label can be misleading. They introduced a new way to weigh sentences by how much clear evidence they carry, ignoring uncertain ones. This method works better than previous ways when tested on thousands of posts in Chinese and English.
What this means in practice
- •For social media analytics teams: Improve accuracy of public value measurement from long social media posts by weighting decisive sentences more than uncertain ones.
- •For public opinion monitoring services: Develop better tools to track changing public value dimensions over time using the proposed tempering fusion technique on multi-lingual data.$Commercial implications: Enables creation of subscription analytics platforms offering more precise insights on public attitudes by leveraging improved long-text judgment fusion.
Authors
Yuhe Wu, Rui Qian, Guangyu Wang, Yuran Chen, Yuanchao Zhu, Junjie Yang, Zhengheng Li, Jiulin Cai, Tianyi Zhang, Zihan Dong, Jiaxin Liu, Yujie Chen, Guang Zhang
Abstract
Large language models (LLMs) are increasingly used to measure public value orientations from long social media posts, yet such posts often mix background, quotations, concessions, and only a few stance-bearing sentences. Existing approaches either ask the model to predict a document-level label directly, which can be overconfident, or aggregate sentence-level predictions by majority or soft voting, which treat uncertain and decisive sentences as equally informative. We formulate long-text value measurement as a decision-fusion problem and propose Tempered Evidence Fusion (TEF), a training-free rule that weights each sentence's log-odds by its normalized information gain, as derived from a generalized Bayesian posterior. This makes the fused score nearly vanish for uncertain sentences while preserving the Bayes-optimal weight of decisive evidence. We further introduce Multi-event Insight Network Dimensions (MIND), a benchmark of 8,358 Chinese and English posts spanning five years of public events and six value dimensions. On MIND, TEF outperforms the strongest baseline among Direct, Majority Vote, and Soft Vote by an average of 4.5 accuracy points and 4.6 macro-F1 points across five LLMs and two languages. MIND dataset and code are available at https://github.com/Kzczc/ICASSP2027-TEF.