The controversy about the mistakes in the Reinhart and Rogoff 2010 paper has left their reputation somewhat in tatters and has provided ample scope for expressions of Schadenfreude among economists. But there are actually also some interesting lessons in this sad development for those of us who are interested in economic and financial research.
If by any chance, you should not have not heard about this story, you may want to go here, if you are on the political left, or here, if you are on the political right, or here for the new paper by Herndon, Ash and Pollin that is responsible for the saga.
Read the paper not just about the paper
I finally read the Reinhart-Rogoff paper only last night. I suspect that I am not the only one who became aware of their findings through various reports on the paper in the media and, in my case, by listening to presentations by each of the authors on two separate occasions. Their results were always presented, by themselves and others, as meaning that if a country’s public debt to GDP ratio passed the ninety percent boundary, its growth prospects would be dramatically impaired. As a result, every politician and central banker advocating austerity has cited that ninety percent figure ever since.
But the paper itself is actually a lot less clear on this point. It presents two sets of results and those based on medians are much less dramatic than those based on the averages that have received all the attention. I may be engaging in self-delusion but I am reasonably certain that I would have been much less likely to believe their dramatic story if I had read the paper.
Serious empirical research in economics tends to devote a lot of space to the assumptions that underlay the results and any limitations on their relevance. As a result, it is sometimes impenetrable for the non-expert. So the Reinhart-Rogoff paper must have been a godsend to the various commentators who picked up the story: a simple, clear result with important public policy implications and not an “if” or “but” in sight. Consequently, they must bear much of the blame. Some of them undoubtedly wanted to believe the story for political reasons. But Reinhart and Rogoff are also to blame; they encouraged the world to believe the dramatic version of their results not just by not correcting what was said by others about their study but also by focussing their own presentations on the dramatic part of their results as well – that which now has been put very much into question. This is perhaps not surprising since their newly found fame on the lecture circuit depended critically on the existence of those dramatic results.
But no excuses, I should have read the paper…
Correlation and causality
It is easy to assign causality to an established relationship. We do it all the time in daily life. But in economics, we are supposed to be a bit more careful. “Common sense” is not enough as an argument. There are reasons why excessive public debt may impact growth but there are also arguments for lack of growth impacting public debt.
It is surprising that, other than a few economists like Paul Krugman, nobody questioned this casual imposition of causality going in a particular way. Once again, this would have been more obvious if I had read the paper.
On grace in adversity
Mistakes can happen and will happen. But how researchers deal with them when they have happened is crucial for their reputation. Unfortunately, Reinhart and Rogoff have chosen to respond in a way that combines defensiveness and condescension in a rather unattractive package (here). I had actually found them very knowledgeable and rather interesting to listen to and I have no axe to grind. But their claim that their mistakes happened despite “their best efforts to be consistently careful” unfortunately suggests that perhaps their best is simply not good enough. Their further claim that they do not believe that “this regrettable slip affects in any significant way the central message of the paper” suggests that they do not fully understand their problems. While they claim that they “will redouble their efforts in the future”, I wonder if anybody will care.
Rolf Banz spent his career in the investment industry in the US, the UK and, most recently, in Switzerland. To older people, he is known as the "father of the small firm effect". This weblog consists of a series of essays and shorter pieces on a range of issues at the intersection of institutional investment and investment theory. Please see this post for a description of the objectives of the weblog and the About page for further information on the author and the site.
Reader beware
By Rolf on 18 April 2013
Lessons from the Reinhart-Rogoff saga
The controversy about the mistakes in the Reinhart and Rogoff 2010 paper has left their reputation somewhat in tatters and has provided ample scope for expressions of Schadenfreude among economists. But there are actually also some interesting lessons in this sad development for those of us who are interested in economic and financial research.
If by any chance, you should not have not heard about this story, you may want to go here, if you are on the political left, or here, if you are on the political right, or here for the new paper by Herndon, Ash and Pollin that is responsible for the saga.
Read the paper not just about the paper
I finally read the Reinhart-Rogoff paper only last night. I suspect that I am not the only one who became aware of their findings through various reports on the paper in the media and, in my case, by listening to presentations by each of the authors on two separate occasions. Their results were always presented, by themselves and others, as meaning that if a country’s public debt to GDP ratio passed the ninety percent boundary, its growth prospects would be dramatically impaired. As a result, every politician and central banker advocating austerity has cited that ninety percent figure ever since.
But the paper itself is actually a lot less clear on this point. It presents two sets of results and those based on medians are much less dramatic than those based on the averages that have received all the attention. I may be engaging in self-delusion but I am reasonably certain that I would have been much less likely to believe their dramatic story if I had read the paper.
Serious empirical research in economics tends to devote a lot of space to the assumptions that underlay the results and any limitations on their relevance. As a result, it is sometimes impenetrable for the non-expert. So the Reinhart-Rogoff paper must have been a godsend to the various commentators who picked up the story: a simple, clear result with important public policy implications and not an “if” or “but” in sight. Consequently, they must bear much of the blame. Some of them undoubtedly wanted to believe the story for political reasons. But Reinhart and Rogoff are also to blame; they encouraged the world to believe the dramatic version of their results not just by not correcting what was said by others about their study but also by focussing their own presentations on the dramatic part of their results as well – that which now has been put very much into question. This is perhaps not surprising since their newly found fame on the lecture circuit depended critically on the existence of those dramatic results.
But no excuses, I should have read the paper…
Correlation and causality
It is easy to assign causality to an established relationship. We do it all the time in daily life. But in economics, we are supposed to be a bit more careful. “Common sense” is not enough as an argument. There are reasons why excessive public debt may impact growth but there are also arguments for lack of growth impacting public debt.
It is surprising that, other than a few economists like Paul Krugman, nobody questioned this casual imposition of causality going in a particular way. Once again, this would have been more obvious if I had read the paper.
On grace in adversity
Mistakes can happen and will happen. But how researchers deal with them when they have happened is crucial for their reputation. Unfortunately, Reinhart and Rogoff have chosen to respond in a way that combines defensiveness and condescension in a rather unattractive package (here). I had actually found them very knowledgeable and rather interesting to listen to and I have no axe to grind. But their claim that their mistakes happened despite “their best efforts to be consistently careful” unfortunately suggests that perhaps their best is simply not good enough. Their further claim that they do not believe that “this regrettable slip affects in any significant way the central message of the paper” suggests that they do not fully understand their problems. While they claim that they “will redouble their efforts in the future”, I wonder if anybody will care.
Posted in Comments/ramblings | Tagged Empirical research