The Reinhart-Rogoff Fiasco
Matthew Martin
4/18/2013 07:34:00 PM
Tweetable
If you've been following the econosphere at all, then surely you've already read a lot about Reinhart's and Rogoff's "death by Excel" (Krugman's words).
The fiasco has really revealed a lot about the economics academia. A couple, such as Paul Krugman and Aaron Carroll, have moved to partially indict the punditocracy for accepting the R&R result on reputation alone, without thinking critically about their methods and claims. To a certain extent, I agree. Tim Fernholz has an article showing how influential the R&R paper was on policy makers everywhere--here is some of the language top policy makers used to describe R&R's key assertion that gross government debt becomes harmful above the 90% of GDP threshold:
- "Economists...tell us ..."--Congressman Paul Ryan
- "Economists... have argued" --Lord Lamont of Lerwick, advisor to UK Chancellor of the Exchequer
- "widely acknowledged"--EU Commissioner Oli Rehn
- "The canonical work of Carmen Reinhart and Kenneth Rogoff "--Doug Holtz-Eakin
"By the way, we are very careful in all our papers to speak of "association" and not "causality" since of course our 2009 book THIS TIME IS DIFFERENT showed that debt explodes in the immediate aftermath of financial crises"In short, when pressed, "economists" concede that it's likely that causality runs in the opposite direction--bad economies cause high debt.
OK, lets go back to quotes 3 and 4 above. How can a paper that was NOT published in a peer-reviewed journal and was rejected by most economists for fundamental methodological flaws qualify as a "widely acknowledged" result and "canonical work?" How is it that it took a hugely embarrassing excel spreadsheet error for public policy wonks to realize that this was a conference paper from the May "papers and proceedings" issue, not a peer-reviewed article published in the AER.
This is not the first time that ideologues have latched onto a screw-ball study in the informal, non-peer-reviewed conference paper issue of AER and paraded it around as the scientific consensus. Not too long ago, there was a discussion in the econosphere about whether we still need the peer-review process in the age of the Internet. The argument was basically that research moves at a much faster pace than publication anyway, so in essence peer review journals these days are little more than where research papers go to die. But two studies in recent history, the R&R paper as well as the infamous Regnerus paper have taught me that a robust peer-review process is essential, because while any scientist can critique a scholar's work, only a journal's referees can demand changes when there are deep methodological flaws likely to get misinterpreted by the public.
Had Reinhart and Rogoff gone through the normal channels, in addition to catching the excel error, I strongly think the referees would have demanded substantial revisions in methodology along the lines of what others have done here, here, here and here. Unfortunately that process did not happen.
I will pile on one additional critique that I don't think has gotten enough mention. Reinhart and Rogoff have really bad data for the kind of analysis they want to do. Horrible data. Take for examples the US and UK. In their US data, there is literally only one episode where we had debt over 90% of GDP--in the post-war demobilization from World War II. In the UK, there are exactly two--in the post-war demobilization after the napoleonic wars, and the post-war demobilization from World War II. In fact, this is a systemic feature across most of their sample, the 90% threshold has usually only been breached during post-war demobilization episodes. Why does this matter? Because those periods have been associated with rapid, severe cuts in government spending. It is therefore unclear whether R&R found that high debt causes slow growth, supporting the austerity thesis, or that large government spending cuts causes slow growth, contradicting the austerity thesis.
The main thing you look for in data for reduced-form empirical papers is a source of exogenous variation in the treatment variable (government debt, in this case). Reinhart and Rogoff haven't got it.
This is not the first time that ideologues have latched onto a screw-ball study in the informal, non-peer-reviewed conference paper issue of AER and paraded it around as the scientific consensus. Not too long ago, there was a discussion in the econosphere about whether we still need the peer-review process in the age of the Internet. The argument was basically that research moves at a much faster pace than publication anyway, so in essence peer review journals these days are little more than where research papers go to die. But two studies in recent history, the R&R paper as well as the infamous Regnerus paper have taught me that a robust peer-review process is essential, because while any scientist can critique a scholar's work, only a journal's referees can demand changes when there are deep methodological flaws likely to get misinterpreted by the public.
Had Reinhart and Rogoff gone through the normal channels, in addition to catching the excel error, I strongly think the referees would have demanded substantial revisions in methodology along the lines of what others have done here, here, here and here. Unfortunately that process did not happen.
I will pile on one additional critique that I don't think has gotten enough mention. Reinhart and Rogoff have really bad data for the kind of analysis they want to do. Horrible data. Take for examples the US and UK. In their US data, there is literally only one episode where we had debt over 90% of GDP--in the post-war demobilization from World War II. In the UK, there are exactly two--in the post-war demobilization after the napoleonic wars, and the post-war demobilization from World War II. In fact, this is a systemic feature across most of their sample, the 90% threshold has usually only been breached during post-war demobilization episodes. Why does this matter? Because those periods have been associated with rapid, severe cuts in government spending. It is therefore unclear whether R&R found that high debt causes slow growth, supporting the austerity thesis, or that large government spending cuts causes slow growth, contradicting the austerity thesis.
The main thing you look for in data for reduced-form empirical papers is a source of exogenous variation in the treatment variable (government debt, in this case). Reinhart and Rogoff haven't got it.
http://www.theatlantic.com/business/archive/2013/04/forget-excel-this-was-reinhart-and-rogoffs-biggest-mistake/275088/
Essentially two of the biggest newspaper column writing, Congress policy advising pundits who took this paper and need to be indicted for blowing it's results out of proportion are.... R&R....