Causal inferences aren't necessarily Lucas-robust

2/06/2014 12:00:00 PM
Here's a Lucas critique: George Lucas should not be in a post about the Lucas Critique.
Noah Smith has a nice post on how the Chicago school put the Austrian school of economics out of business. The part about "theory ahead of measurement" reminds me of a point I was thinking about recently about causality. Statistical causality is a topic I've written about before. In general, if you want to provide policy advice based on statistical evidence, the critical thing to determine is whether the effects you estimated are truly causal as opposed to mere correlation. We have several tools to do this, including randomized-controled-trials, difference-in-differences, instrumental variables, and regression discontinuity. Each of these seeks to do the same thing--identify some source of exogenous variation from which to estimate a causal effect--and in most micro papers this is essentially the only thing researchers worry about.

But that is misguided. Although the Lucas critique is usually applied to macro papers, the fact is that the exact same problem arises in micro, and it is quite possible to detect a causal effect that is not Lucas-robust.

The Lucas critique says that the ex-ante estimated effects of policies may not remain constant once that policy is implemented, because the parameter is not policy-invariant. Suppose we want to estimate the effects of inflation on the unemployment rate. Suppose that we have a datasample in which several countries chose (exogenously) to increase their inflation rates, and several countries that held steady. Provided there are no confounding factors regarding the timing of this event, we could use a difference-in-differences model to estimate the causal effect of raising the inflation rate on unemployment. In principle, the only way we could end up with a correlation other than the true causal effect is by pure random chance--and we can push that chance to near-zero by gathering enough data. In such a model, we are likely to find that raising inflation by 1 percent reduces unemployment by 0.5 percent. (Or maybe not. I've not estimated such a model, and it seems possible that our estimates would be overwhelmed by the wage curve effect, even if the phillips curve relationship exists. I refer you to Campbell 2008 for an explanation of why the wage curve and phillips curve effects can both be valid at the same time. The bottom line is that the assumptions of the difference-in-differences model aren't actually satisfied here, but that's beyond the scope of this post.)

Unfortunately, that effect never lasts. If we are at full employment already, then if we really raised the inflation rate what we'd see is a temporary decrease in unemployment, after which it would rise again--the policy parameter would shift merely in response to the fact that we implemented the policy. The intuition here is that the effect we estimated was due in large part to the inattentiveness of price-setters to government policy--if we actually pursue such a policy to any non-trivial degree, then price-setters will wise-up and start anticipating the government's policy, thereby eliminating the effects we previously observed. This does not mean that the original estimate, measured before price-setters caught on to what the government was doing, was not truly causal; it was. What it means is that causal effects needn't be policy invariant.

Though I rarely see microeconomists mention this except in passing, the fact is that the same failure happens all the time to microeconomic parameter estimates. For example, many, many health economists have estimated the causal effects of raising the cigarette tax on smoking rates. While I don't doubt that the effects they estimate are truly causal, there's no reason to suppose that they are policy invariant. In fact, theory says that in general, they won't be: the causal effect they estimated is a "local average treatment effect" meaning that they measure the elasticity at the old cigarette price, and there is no particular reason to suppose that the elasticity would be the same at a higher cigarette price. Hence, it doesn't quite make sense to argue that an additional increase in the tax will have the predicted effect, even though that is the advice that microeconomists usually offer.

It gets worse. Even at the old price, the estimated effect of a cigarette tax on smoking rates may not be policy-invariant. Although in the short run people will pay the tax and reduce smoking rates, in the long-run it's possible underground networks will form to smuggle tax-free cigarettes to people--a model of the cigarette market that does not include an underground market is not actually Lucas-robust. New York is the example that comes to mind. At Cornell I knew tons of people who smoked, but none who paid taxes on those cigarettes.

I remain evidence-over-theory, but I do think that any econometric papers, micro or macro, that lack a persuasive theory section should automatically be suspect.