AI Trading: Evaluating Large Language Models for Technical Market Analysis
Machine LearningArtificial Intelligence
Summary
The authors tested five large language models (LLMs) to see how well they understand and analyze financial market data. They looked at tasks like recognizing trading patterns, making buy/sell/hold signals, simulating investment results, and understanding financial reports. GPT-4 Turbo performed best among general models, while FinGPT did well because it was trained specially for finance. All models showed some common problems, like making up numbers and struggling in flat market conditions. The authors suggest careful testing and fine-tuning are needed for practical use in trading.
Authors
Geofrey Ntale
Abstract
Large Language Models (LLMs) have emerged as powerful tools for processing the heterogeneous information environments of modern financial markets. This paper presents a systematic, comparative evaluation of five prominent LLMs: GPT-4 Turbo, Claude 3 Opus, Gemini 1.5 Pro, Llama 3 70B, and the domain-specialized FinGPT, with respect to their capacity for technical market analysis. The evaluation spans four structured tasks: candlestick pattern recognition from OHLCV data, directional signal generation (BUY/SELL/HOLD), backtesting of signal quality through a simulated execution pipeline, and financial report comprehension. Our experimental framework employs rigorous quantitative metrics, including Sharpe ratio, maximum drawdown, Sortino ratio, information coefficient, F1-score, and BLEU score. Findings from simulated backtesting indicate that GPT-4 Turbo achieves the highest annualized return and Sharpe ratio among general-purpose models, while FinGPT demonstrates competitive risk-adjusted performance due to domain-specific fine-tuning. Both models outperform a passive S&P 500 benchmark under the tested conditions. The study identifies persistent failure modes across all evaluated models, including numerical hallucination, context-window limitations, and inconsistent performance in sideways market regimes. We conclude that while LLMs hold genuine promise within AI trading systems, robust deployment requires careful task decomposition, rigorous backtesting protocols, and domain-aware fine-tuning strategies.