The brouhaha over Carmen Reinhart’s and Kenneth Rogoff’s article “Growth in a Time of Debt” may be the most conspicuous and incendiary scholarly controversy since 1974, when two earlier economists, Robert Fogel and Stanley Engerman, published a notorious book, Time on the Cross, defending the efficiency of American plantation slavery.
As with Time on the Cross, the Reinhart/Rogoff controversy, while ostensibly stemming from the authors’ statistical procedures, is actually rooted in the purposes to which others put their study.
Some of the results reported by Fogel and Engerman were used – not by the authors themselves, it should be noted – to challenge affirmative action and question the civil-rights movement. Similarly, some of the results reported by Reinhart and Rogoff have been used by politicians and others to justify fiscal austerity.
When the problems with the Reinhart/Rogoff analysis came to light, the critics were aghast. The authors had inadvertently omitted data, used a questionable weighting scheme, and employed an erroneous observation on GDP growth.
This raised uncomfortable questions not only about the efficacy of austerity, but also about the reliability of economic analysis. How could a flawed study have appeared first in the prestigious working-paper series of the National Bureau of Economic Research (NBER) and then in a journal of the American Economic Association? And, if this was possible, why should policymakers and a discerning public vest any credibility in economic research?
It was possible because economists are not obliged to make their data and programs publicly available when publishing scientific research. It is said that NBER working papers are even more prestigious than publication in refereed journals. Yet the Bureau does not require scholars to post their data and programs to its website as a condition for working-paper publication.
Independent scholars seeking to replicate these studies’ findings must first replicate the data and then replicate the programs. And, as empirical economics has progressed, the difficulty of doing so has grown. Reinhart and Rogoff may have used a relatively small set of mostly publicly available data, but the profession as a whole is using ever-larger tailor-made data sets.
Big data promises big progress. But large data sets also make replication impossible without the author’s cooperation. And the incentive for authors to cooperate is, at best, mixed. It is therefore the responsibility of editorial boards and the directors of organizations like the NBER to make open access obligatory.
Moreover, in a discipline that regards ingenuity as the ultimate virtue, those who engage in the grunt work of data cleaning and replication receive few rewards. Nobel prizes are not awarded for constructing new historical estimates of GDP that allow policy analysis to be extended back in time.