Archive / Prediction markets / E550
A free weather model does not beat temperature contract prices
Trading daily-high temperature contracts against a bias-corrected 2-day-old forecast lost 3.26 c per contract net (95% interval -5.21 to -1.40 c); the price was the better forecast (0.1125 vs 0.1466, lower wins).
Daily-high temperature contracts pay out on whether a city's high lands in a given degree range. The belief: recreational bettors anchor on rounded phone-app forecasts, so a station-corrected public weather model should price the ranges better. We built that forecast without look-ahead for seven US cities and bought whichever side it favoured by at least 5 c after fees. Across 1,654 trades the average result was -3.26 c per contract, and none of the seven cities was positive. On all 2,617 quoted contracts the market price had the lower Brier score (a standard accuracy score for probabilities, lower is better): 0.1125 against 0.1466 for the model.
Temperature contracts are priced by amateurs who read a rounded phone-app forecast; a station-specific, bias-corrected model forecast knows better and can be traded against the price.
Why people believe it
People argue that casual bettors take the headline forecast from an app, rounded to the whole degree and not corrected for how the official weather station tends to read against the model. A forecast corrected for that station bias should assign different probabilities to the one-degree and two-degree contract ranges than the crowd does.
Why it would persist The argument, as stated before testing
The forecast is free and public, but turning it into probabilities for each range needs station-level calibration that casual bettors do not do. Weather bots that could do it exist, but the argument is that they may not cover every city-day or range.
For each of seven US cities we took a public weather-model forecast issued about two days before the day in question, so it was available before the trading decision. We corrected it with that city's recent forecast errors against the official station, turned it into a probability for every temperature range, and looked at the contract prices at 16:00 UTC on the day before. If the model's probability beat the price by at least 5 c after the fee, we bought that side and held to settlement. We also scored the model and the market price against the actual outcomes on every quoted contract.
Not stated in advance as a separate pass line; the rejection rule below was pre-registered.
Market-mid Brier score at or below the model's; or net mean at or below 0; or 95% interval includes 0; or fewer than 4 of 7 cities with a positive mean.
Fee 0.07 x p x (1 - p) per contract at the traded price, plus crossing the spread (median quoted spread 1 c).
The market mid price as the benchmark forecast: Brier score of the mid versus the model on all quoted contracts, whether or not traded. Bootstrap by city-day cluster (4,000 resamples); first versus second half of the trades by date; sign of the mean in each city.
The result met three of the four pre-registered rejection criteria: the market's Brier score was lower than the model's, the net mean was below zero, and no city had a positive mean. The fourth, an interval that includes zero, was not met because the whole interval is negative.
- On all 2,617 quoted contracts (465 city-days) the market mid price had a Brier score of 0.1125 and the model 0.1466; a lower score is better, so the price was closer to the outcomes.
- The model disagreed with the price by at least 5 c after fees on 1,654 contracts. Net per contract was -3.26 c (95% interval -5.21 to -1.40 c); the median trade was -4.3 c and the worst lost 92.5 c.
- Before the exchange fee the same trades averaged -2.27 c, so the model's disagreements were already unprofitable at the quote, and the fee took a further 0.99 c.
- The first half of the trades by date averaged -4.23 c and the second half -2.28 c: negative in both.
- No city had a positive mean. Per-city means ranged from -5.93 c (Chicago) to -0.26 c (Miami).
The price scored better than the model
Market mid priceBias-corrected model
Mean squared error between the probability and the 0/1 outcome over all 2,617 quoted contracts. The market mid price has the lower error, so the model was the weaker forecast on the same contracts.
Chart data (Brier score (lower is better))
| Market mid price | Bias-corrected model | |
|---|---|---|
| All quoted contracts | 0.11 | 0.15 |
Trades against the price lost money
Mean net result per contract across the 1,654 trades, with a city-day cluster bootstrap 95% interval. The whole interval is below zero.
Chart data (c per contract)
| Mean | 95% low | 95% high | |
|---|---|---|---|
| All trades, net of fee (n 1,654) | -3.26 | -5.21 | -1.4 |
No city was positive
Mean net result per contract in each city. All seven are negative, against a pre-registered requirement of at least four positive cities. Per-city means are noisy: none, including Miami (-0.26 c), is distinguishable from zero.
Chart data (c per contract)
| Net per contract | |
|---|---|
| New York (n 215) | -3.84 |
| Chicago (n 250) | -5.93 |
| Miami (n 212) | -0.26 |
| Austin (n 244) | -2.18 |
| Denver (n 247) | -3.13 |
| Los Angeles (n 224) | -4.37 |
| Philadelphia (n 262) | -2.83 |
Why it failed Interpretation · not established
A likely reason is that traders use fresher model runs than a forecast issued two days earlier, which would explain why the price scores better than the model. The test did not observe what information traders used, so this is a hypothesis.
Lesson
A forecast built without look-ahead still has to be judged against the market's own forecast, not only against the outcomes. When a model's probabilities differ from the price by 5 c or more, the first thing to test is whether the model or the price scores better on the same contracts; here it was the price, and the trades lost money.
Scope of this result
Covers seven US cities' daily-high contracts, one forecast model run about two days old, one entry time (16:00 UTC on the day before), a 5 c entry threshold, and about two months of quotes.
A rejected experiment applies to the tested conditions and does not prove an idea can never work. bp = basis point, 0.01%.
1. Test design
Pre-registeredDataUniverseSignalExecutionCostsControlsSample splitPass criterionKill criterionPower2. Result
ObservationsInstrumentsNet mean95% intervalMedianWorstFinding 1Finding 2Finding 3Finding 4Finding 53. Interpretation
Why it failedLessonScope4. Reproducibility
StepsGrade