Counterfactuals: What If There Were No X?

You catch a cold, take cold medicine for a week, and get better. Did the medicine work?

The catch: a common cold usually clears up in about a week even without medicine. To judge the medicine, you shouldn't compare "before" with "after." You should compare "you with the medicine" against "you without it."

What is a counterfactual?

A counterfactual is "what would have happened without X." The real effect of X = the outcome with X − the outcome without X.

The hard part is that we never get to see the world without X; we can only estimate it. Common approaches:

• Look at the trend: which way were things already heading before X? If a city restricts home purchases and prices fall, but prices had already been falling for six months before that, the policy deserves less credit.
• Find a comparison: is there a similar person or place that didn't get X?
• Think through the mechanism: through which concrete steps does X affect the outcome? Does each step hold up?

Write out the causal chain

When writing your forecast's reasoning, instead of "because of X, Y," spell out the steps in between:

X → step 1 → step 2 → Y

For example: "the new subway line opens → I no longer need to transfer to a bus → my commute drops from 60 minutes to under 40." Once it's written down, you can check each link: how likely is this step to hold? If several steps all need to hold, the overall probability is their product (see lesson 1.3).

Then add one line of counterfactual: without X, what would probably happen? If the answer is "about the same," X isn't a good reason for your forecast.