The Reinhart-Rogoff Fiasco

4/18/2013 07:34:00 PM
If you've been following the econosphere at all, then surely you've already read a lot about Reinhart's and Rogoff's "death by Excel" (Krugman's words).

The fiasco has really revealed a lot about the economics academia. A couple, such as Paul Krugman and Aaron Carroll, have moved to partially indict the punditocracy for accepting the R&R result on reputation alone, without thinking critically about their methods and claims. To a certain extent, I agree. Tim Fernholz has an article showing how influential the R&R paper was on policy makers everywhere--here is some of the language top policy makers used to describe R&R's key assertion that gross government debt becomes harmful above the 90% of GDP threshold:
  1. "Economists...tell us ..."--Congressman Paul Ryan
  2. "Economists... have argued" --Lord Lamont of Lerwick, advisor to UK Chancellor of the Exchequer
  3. "widely acknowledged"--EU Commissioner Oli Rehn
  4. "The canonical work of Carmen Reinhart and Kenneth Rogoff "--Doug Holtz-Eakin
Ok, what's wrong with the first two quotes? For one thing, it gives the impression that this view is the economics consensus, which it isn't. But more importantly, the economists they are referring to, Reinhart and Rogoff, didn't "tell us" or "argue" that in their paper at all! As Neil Irwin points out, "The original paper didn't argue either way." In their paper, R&R claim only that slow growth and high debt are correlated--it is devoid of any causal claim or policy relevance, so to say that the R&R paper "argues" that high debt causes slow growth is completely false. In case you don't believe me, here's R&R in their own words:
"By the way, we are very careful in all our papers to speak of "association" and not "causality" since of course our 2009 book THIS TIME IS DIFFERENT showed that debt explodes in the immediate aftermath of financial crises"
In short, when pressed, "economists" concede that it's likely that causality runs in the opposite direction--bad economies cause high debt.

OK, lets go back to quotes 3 and 4 above. How can a paper that was NOT published in a peer-reviewed journal and was rejected by most economists for fundamental methodological flaws qualify as a "widely acknowledged" result and "canonical work?" How is it that it took a hugely embarrassing excel spreadsheet error for public policy wonks to realize that this was a conference paper from the May "papers and proceedings" issue, not a peer-reviewed article published in the AER.

This is not the first time that ideologues have latched onto a screw-ball study in the informal, non-peer-reviewed conference paper issue of AER and paraded it around as the scientific consensus. Not too long ago, there was a discussion in the econosphere about whether we still need the peer-review process in the age of the Internet. The argument was basically that research moves at a much faster pace than publication anyway, so in essence peer review journals these days are little more than where research papers go to die. But two studies in recent history, the R&R paper as well as the infamous Regnerus paper have taught me that a robust peer-review process is essential, because while any scientist can critique a scholar's work, only a journal's referees can demand changes when there are deep methodological flaws likely to get misinterpreted by the public.

Had Reinhart and Rogoff gone through the normal channels, in addition to catching the excel error, I strongly think the referees would have demanded substantial revisions in methodology along the lines of what others have done here, here, here and here. Unfortunately that process did not happen.

I will pile on one additional critique that I don't think has gotten enough mention. Reinhart and Rogoff have really bad data for the kind of analysis they want to do. Horrible data. Take for examples the US and UK. In their US data, there is literally only one episode where we had debt over 90% of GDP--in the post-war demobilization from World War II. In the UK, there are exactly two--in the post-war demobilization after the napoleonic wars, and the post-war demobilization from World War II. In fact, this is a systemic feature across most of their sample, the 90% threshold has usually only been breached during post-war demobilization episodes. Why does this matter? Because those periods have been associated with rapid, severe cuts in government spending. It is therefore unclear whether R&R found that high debt causes slow growth, supporting the austerity thesis, or that large government spending cuts causes slow growth, contradicting the austerity thesis.

The main thing you look for in data for reduced-form empirical papers is a source of exogenous variation in the treatment variable (government debt, in this case). Reinhart and Rogoff haven't got it.
Metatone 4/19/2013 02:32:00 PM
This is very pertinent to the debate:

Essentially two of the biggest newspaper column writing, Congress policy advising pundits who took this paper and need to be indicted for blowing it's results out of proportion are.... R&R....
Matthew Martin 4/19/2013 06:57:00 PM
Very true. I think that perhaps R&R mistook their personal credibility as evidence that their causal claim was credible.