Outcome Bias

A friend drives home after drinking, gets there safely, and says the next day, "See, I told you it'd be fine." Was that a good decision?

Of course not. Getting home safely was luck this time; the decision itself was still bad.

This is outcome bias: judging the quality of a decision or judgment by how things turned out.

Outcome bias in forecasting

You forecast "it will rain tomorrow" at 70%, and it doesn't rain. Was the forecast wrong?

Not necessarily. 70% means that in situations like this, it won't rain about three times in ten. One dry day is entirely expected. To judge whether someone forecasts well, look at many forecasts: did the things they called 70% happen about 70% of the time? That's exactly what the Brier score and calibration curve measure.

Four combinations

Separate the quality of the reasoning from the outcome and you get four cases:

• Good reasoning, good outcome: a deserved success.
• Good reasoning, bad outcome: bad luck; the method doesn't need to change.
• Bad reasoning, good outcome: a fluke, and the most dangerous case, because it encourages you to repeat the mistake.
• Bad reasoning, bad outcome: the real thing to improve.

Outcome bias makes us treat the second case as failure and the third as success, so we abandon good methods and keep bad habits.

How to guard against it when forecasting

• Write out your full reasoning: base rate, reasons for and against, key uncertainties. That way there's something to evaluate in the review.
• Accept that good probabilities sometimes miss: giving 70% means being ready for the 30% of times it doesn't happen.

How to spot it in a review

In a review, judge the reasoning first, then look at the outcome. If you catch yourself rejecting a well-reasoned forecast just because it missed, or praising a gut-feel forecast just because it hit, that's outcome bias at work.