Ensemble model predicts investor post profit with moderate accuracy
BAIBAICHUCHU at the NTCIR-19 FinArg-3 Task: When Is Maximum Possible Profit Predictable from Investor Text?
Computation and Language
Summary
Predicting how much money investors can make based on their social media posts is a difficult problem. The authors combined three methods, including language models and text features, to rank posts by their maximum possible profit but found mixed results. They discovered that market volatility and the timing window play important roles in making these predictions, while the text from posts alone was not strongly predictive. The study suggests more careful evaluation considering market conditions and investment timing is needed to understand these predictions better.
What this means in practice
- •For financial analytics teams: Use combined text and market volatility features to partially rank investor posts by potential profit under specific market regimes and horizons.
- •For social media monitoring teams: Filter and prioritize investor sentiment posts by potential future market impact considering stance, volatility, and investment time frames.
Authors
Zong-Han Bai, Po-Yen Chu
Abstract
The BAIBAICHUCHU team participated in the Social Media Subtask of NTCIR-19 FinArg-3, ranking Chinese investor posts by Maximum Possible Profit (MPP). A three-track ensemble of lexical features, a FinArg-2-pre-finetuned MacBERT ranker, and an LLM judge reaches 0.734 in post-grouped development evaluation, but our best official run scores 0.517. All twelve submitted runs lie between 0.4598 and 0.5402, and our 26 unanimous three-track pairs score 0.500. A post-hoc audit finds that the submitted judge applied a long-only rule to bearish posts although MPP is stance-aware. Correcting it changes 28 of 87 official predictions without changing accuracy, yet lowers development accuracy from 0.680 to 0.622: a regime-specific semantic shortcut improved validation fit. We decompose ranking into directional text, volatility and horizon, and pairwise margin. A running-extremum model predicts $σ\sqrt{T}$ scaling, observed ex post in 502 price-aligned posts from a separate July 2026 collection. Pre-posting volatility scaled by the post-specific observation horizon, $σ_{\mathrm{pre},20}\sqrt{N_i}$, is associated with a later truncated-horizon MPP proxy (Spearman $ρ=0.320$; ticker-cluster 95% CI [0.133,0.466]) and correctly orders 61.4% of unequal-outcome pairs. Transferred text is nearly uncorrelated with the July outcome ($ρ=0.055$) and adds little conditional on this score and stance. Development reliability rises from about 0.60 to 0.93 as the labeled MPP gap widens. An earlier ERAI result of 0.6207 rules out universal unpredictability as a simple explanation. Evaluation should jointly consider text, market regime, historical volatility, horizon, and pair composition.