Language models choose cost-effective forecasts to improve decisions
Forecast Workflow Bench: Evaluating Language-Model Decisions with Budgeted Forecast Tools
Machine LearningArtificial Intelligence
Summary
Making good predictions for things like electricity use is important but using every forecast can be expensive. This paper presents FWBench, a tool that tests how well language models decide which forecasts to buy and use under budget limits. The authors show that some language models can pick cheaper, short-term forecasts and still make better decisions than fixed methods. This helps understand how language models balance forecast costs and benefits in real-world decisions.
What this means in practice
- •For energy grid operators: Select economical forecasting models under budget limits to improve electricity capacity planning decisions.
- •For public transit managers: Manage cycle-hire resources effectively by choosing forecast horizons that balance cost and accuracy in demand predictions.
Authors
Shunya Nagashima
Abstract
Time-series foundation models (TSFMs) provide forecasts for operational decisions, but accuracy alone does not determine their value. Evaluating agents that use these models requires measuring decision quality and forecast cost. FWBench evaluates this capability on 1,251 electricity and cycle-hire cases using fixed forecast tools and simulated capacity contracts. Agents select models, histories and horizons, then submit capacities to minimize a stated loss-cost objective. We evaluated two hosted and eight local configurations, including small language models, and tested local models with and without TSFMs. GPT-6 Astra bought inexpensive short-horizon forecasts selectively, using 2.5% of the budget, and outperformed fixed policies when the saved decisions were scored with three loss-cost weightings. FWBench enables reproducible evaluation of how language models select and use time-series forecasts to make decisions under cost constraints.