Evidence for a limit to effective peer review

I missed it first time around but apparently, back in October, Nature published a somewhat-controversial article: Evidence for a limit to human lifespan. It came to my attention in a recent tweet:

The source: a fact-check article from Dutch news organisation NRC titled “Nature article is wrong about 115 year limit on human lifespan“. NRC seem rather interested in this research article. They have published another more recent critique of the work, titled “Statistical problems, but not enough to warrant a rejection” and a discussion of that critique, Peer review post-mortem: how a flawed aging study was published in Nature.

Unfortunately, the first NRC article does itself no favours by using non-comparable x-axis scales for its charts and not really explaining very well how the different datasets (IDL and GRG) were used. Data nerds everywhere then, are wondering whether to repeat the analysis themselves and perhaps fire off a letter to Nature.

My advice: don’t waste your time since Philipp Berens and Tom Wallis already did a great job, which is described here and documented (in a manner that you wish applied to Nature articles) at Github.

Maximum reported age at death (n = 33) showing fitted segmented regression

Maximum reported age at death (n = 33) showing fitted segmented regression

That said: I have wasted a little time and the results are in this Github repository. My code illustrates one way to obtain the data, and apply some regression models (linear, segmented and LOESS). One contentious aspect of the original paper was the choice of breakpoint (1995) for segmented regression, which seemed rather arbitrary. In fact it is possible to fit something very similar (albeit with a breakpoint in 1999) to the “maximum age at death by year” data using the R segmented package. Thanks to Nathan Lemoine for his tutorial on that topic.

As to whether segmented linear regression should have been applied, the arguments continue. According to Berens and Wallis, it is supported by the GRG dataset but not by the data from the IDL. All of the articles linked to in this post are worth a read, in particular the third of the NRC articles which contains some telling quotes regarding how articles come to be published. This one is my favourite:

Until then, I had thought focusing on statistics was a major task for a reviewer. But at Nature, that is apparently not a problem at all. In the eleven primary questions drawn up by the journal for an “ideal review”, statistics and methodology are not explicitly addressed. They feature in the second list of questions, for “if time is available”.

3 thoughts on “Evidence for a limit to effective peer review

  1. I find this one odd:

    «Statisticians criticise the fact that the authors assumed in advance that a turning point occurred around 1995.»

    If anything, he should be praised for that — if he indeed has come up with that year before looking at the data.

    If you are given a dataset of completely random data, it is likely that you will be able to find *some* pattern in it. But if you know in advance *which* pattern you expect, and you indeed find it, that makes it more credible.

    If he picked that year in advance, I wish he wrote *how* he did it in his paper.

  2. Pingback: 100 is about it | Future Yada Yada Yada

  3. Hi Neil,

    thanks for the post and the praise. Actually, I think you added an important bit. Our regression model allows for an offset at 1995, which in hindsight is an even stranger model than one with a change in slope. Why would there be a sudden jump? So the breakpoint model with change in slope you implemented is a valuable addition. I will add this to our analysis.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s