Training LLMs for Event Forecasting and Prediction Markets

What The Literature Has Done

1. Static and Dynamic Forecasting Benchmarks

Autocast established the original problem: real forecasting questions, a dated news corpus, and human crowd baselines. The key result was negative for LLMs: the best ML baseline was far below expert human aggregates, though larger models and retrieval helped. Autocast also set the methodological norm that backtests must simulate what was knowable at the forecast date.

Schoenegger and Park tested GPT-4 in a live Metaculus tournament and found it underperformed the median human crowd. Halawi et al. then showed the first serious system improvement: retrieval, reasoning, and aggregation brought an LM system close to the human crowd. Their data scale also matters: 48,754 questions and over 7 million user forecasts, with a human crowd Brier around 0.149 and the system around 0.179 on all questions; simple system+crowd aggregation was even stronger.

ForecastBench moves evaluation from static backtests to dynamic unresolved questions. This is the right direction because any resolved test set becomes contaminated as model cutoffs advance. FutureX, BTF/BTF-2, MIRAI, and Prophet Arena extend this line by adding live updates, frozen corpora, agentic tool environments, and multi-horizon market events.

2. Forecasting Agents and Retrieval

The agent literature converges on a simple pattern: search, reason, update, aggregate. ReAct and Toolformer provide the general tool-use template. Search-R1 shows how to train search use directly with RL and, importantly, masks retrieved tokens so the policy loss is computed only on model-generated tokens. That matters for any forecasting model trained with retrieved context.

The newest systems go beyond appending search results. Bayesian Linguistic Forecaster maintains a structured belief state with probability, evidence for/against, and open questions, runs multiple independent trials, aggregates in logit space, and calibrates with hierarchical Platt scaling. TimeSeek shows search is helpful on average but not uniformly: web search improved pooled Brier Skill Score for all models overall, but hurt in 6 of 50 model-checkpoint conditions. This argues for a selective search/defer/predict gate, not uniform tool use.

3. Training LLMs To Forecast

The training literature has three useful recipes and one caution.

• Self-play/DPO: generate pairs of reasoning traces, rank them by realized outcome distance, then DPO fine-tune. Turtel et al. report 7-10% accuracy gains on 14B models. The risk is reinforcing lucky reasoning traces.
• Outcome RL: train on Brier-style rewards using GRPO/ReMax variants. The strongest practical lesson is to avoid per-question standard-deviation normalization when it erases useful calibration gradients. Guardrails and strict output parsing are not optional.
• Accuracy+Brier rewards: RLCR and OpenForesight show why binary correctness alone is dangerous. Calibration needs to be part of the reward. OpenForesight adds an accuracy term because Brier-only can encourage "Unknown" at near-zero confidence.
• OpenForesight scale: 250,000 articles, heavy filtering, 52,183 final samples, Qwen3-8B training, offline retrieval, and GRPO. This is the closest reproducible recipe.

The caution is that outcome rewards do not prove the model learned good reasoning. A well-reasoned miss on a tail event still receives a poor outcome reward, and a lucky guess still receives a good one. Any training result must be paired with calibration, reasoning-quality, and leakage audits.

4. Calibration and Proper Scoring

Forecasting is probability elicitation. Gneiting and Raftery formalize why strictly proper scoring rules incentivize honest probabilities. Hanson connects scoring rules to markets: market scoring rules elicit group consensus, and LMSR gives the bridge between probabilistic forecasts and market prices. LLM-specific calibration papers show that models can verbalize confidence, but scaling and reasoning do not automatically produce calibrated probabilities. KalshiBench is the clearest recent warning: five frontier models were overconfident, and only Claude Opus 4.5 had positive Brier Skill Score in that sample.

5. Temporal Leakage

Leakage is not a side issue. It is the credibility issue. Autocast and Halawi use dated retrieval. ForecastBench avoids leakage with unresolved questions. Pitfalls in Evaluating LM Forecasters argues that many claims still overstate real-world performance. Shapley-DCLR and TimeSPEC are the most useful recent framework: decompose rationales into atomic claims, determine temporal provenance, and weight leaked claims by decision impact. Our project should implement a lighter version first and a full claim-level audit for paper-grade results.

Hosted PDF Library

I downloaded and uploaded the open PDFs used for this review. Most are arXiv papers; the non-arXiv files are public PDFs from ACL, author/institution pages, or NBER. If a paper is not openly mirrorable, it should be linked rather than rehosted.