Financial sentiment tools show different signals same day versus next day
Same Day, Same Story; One Day Ahead, a Different Signal: The Dual Validity of Financial Sentiment
Artificial IntelligenceComputation and LanguageSocial and Information Networks
Summary
Sentiment analysis tools are often checked by comparing them to human judgments and then used to predict stock market moves. This paper finds that these two steps don't always measure the same thing. The researchers compared five sentiment tools to human labels and stock returns on the same day and the next day. They found that while human agreement matches same-day signals better, it doesn't reliably predict which sentiment scores forecast future price changes. Also, a lot of online messages were spam, and simply counting messages didn't predict stock losses or settlement sizes.
What this means in practice
- •For quantitative traders: Choose and validate sentiment tools by distinguishing same-day versus next-day market signals to improve trading strategies.
- •For financial data engineers: Implement sentiment pipelines that consider sample selection and score representation to better align sentiment extraction with market moves.
Authors
AS Aravinthkakshan, Laven Srivastava, Harsh Nandwani
Abstract
Financial NLP has a standard workflow: validate a sentiment tool against human labels, then trust it to extract market signal. This assumes the two evaluations measure the same thing. We test that assumption in a setting where both can be measured at once: a corpus of securities class actions (2002-2025) linking 70,500 X messages to abnormal stock returns, with a single-annotator human labelled gold sample. Running five instruments (VADER, Loughran-McDonald, FinBERT, Twitter-RoBERTa, and an LLM annotator) through one identical pipeline, we find that the relationship between construct and predictive validity depends on the sampling convention and score representation. Under conventional method-specific sampling, human agreement aligns more closely with graded same-day associations than with one-day leads. On a fixed-n panel, however, agreement has similar graded rank correlations at both horizons, while the coarse ordering remains weak. Benchmark agreement therefore establishes semantic validity but does not by itself determine predictive rankings. In a conversation that is 17.6% spam, message volume predicts neither market damage nor settlement size.