Reading Brier Scores and Calibration Curves

Two friends are guessing whether it will rain tomorrow. Wang says 90%, Li says 60%, and it rains. Both leaned the right way, but who did better? Intuitively Wang — he committed more and was right. But if it hadn’t rained, Wang would have been more wrong. The Brier score turns that intuition into a number.

Brier score: lower is better

The math is simple. Call your probability p, score the outcome as 1 if it happened and 0 if it didn’t, and compute (p − outcome)².

• Say 90%, it happens: (0.9 − 1)² = 0.01 — nearly perfect.
• Say 90%, it doesn’t: (0.9 − 0)² = 0.81 — a heavy penalty.
• Say 60%, it happens: (0.6 − 1)² = 0.16.
• Say 50%: 0.25 no matter what.

Average the scores across all your resolved forecasts and you have your Brier score. 0 is perfect; 0.25 is what you’d get by always saying 50%. Anything above 0.25 is worse than a coin flip.

Notice the trade-off: the more certain you sound, the bigger the reward when you’re right and the bigger the penalty when you’re wrong. So the best strategy is neither bold nor timid — it’s saying how sure you really are. Drag the probability slider below and watch how the score changes when the event happens and when it doesn’t.

Calibration curve: do your 70%s happen 70% of the time?

The Brier score is one overall number; the calibration curve shows where the problem lies. Group your forecasts by probability — say, everything you called around 70% — and check how often those events actually happened.

• Points on the diagonal: your 70% calls happen about 70% of the time. Well calibrated.
• Points below the diagonal: things happen less often than you said. Overconfident.
• Points above the diagonal: things happen more often than you said. Underconfident.

Each bucket needs at least a dozen or so forecasts before the curve means much, so keep writing and resolving.