What a Win Probability Actually Means (and How to Judge One)

When the model says a team has a 60% chance to win, it is making a statement about many games, not one. If you collected every game it called 60/40, the favoured side should win about six in ten. It will lose the other four — and those losses are part of the forecast being right, not a sign that it failed.

Why baseball probabilities stay close to 50%

Baseball has the smallest gap between good and bad teams of the major sports. Even a 100-win team loses 38% of its games. A single game turns on a handful of balls in play, and the best pitcher in the league can give up four runs on a windy afternoon. That is why the model's probabilities rarely go past 70%, and why the best public baseball models pick only about 57–59% of winners over a season. Our walk-forward backtest came in at 57.3%.

Calibration: the test that matters

Picking winners is a blunt test. A model that says 51% for every home team would pick about 53% of winners and still be useless. A better test is calibration: group projections by their probability and compare with what happened. In our backtest the games the model put in the 65–70% band were won by the favoured side 66.0% of the time (predicted 67.1%), and the 70%+ band came in at 73.1% (predicted 72.6%). The middle bands ran one to two points too confident, which the calibration step now corrects for.

Brier score and log loss

Two numbers summarise how good a set of probabilities is. The Brier score is the average squared gap between the probability and the result (1 for a win, 0 for a loss): a coin flip scores 0.25 and lower is better. Log loss punishes confident misses much harder: a coin flip scores 0.693. Our backtest scores were 0.2421 and 0.677. Those look like small improvements over a coin flip because baseball is noisy — on this scale a few thousandths is a large difference.

How to use a win probability