This week in Retraction Watch: Hypertension retracts paper over data glitch.
The retraction notice describes the “data glitch” in question (bold emphasis added by me):
…the authors discovered an error in the code for analyzing the data. The National Health and Nutrition
Examination Survey (NHANES) medication data file had multiple observations per participant and
was merged incorrectly with the demographic and other data files. Consequently, the sample size was
twice as large as it should have been (24989 instead of 10198). Therefore, the corrected estimates of
the total number of US adults with hypertension, uncontrolled hypertension, and so on, are significantly
different and the percentages are slightly different.
Let’s leave aside the observation that 24989 is not 2 x 10198. I tweeted:
"an error in the code for analyzing the data" - retractionwatch.wordpress.com/2012/08/16/hyp…. Entirely avoidable if methods were published in full.
—
Neil Saunders (@neilfws) August 17, 2012Not that simple though, is it? Read on for the Twitter discussion.
Bosco replied:
@neilfws do you really think one can do a complete code review in the average time it takes to review a paper?
—
Bosco Ho (@boscoh) August 17, 2012Good point. The best response I could manage at the time was that errors may not be detected during review, but stand a better chance of eventual detection if code is available. Or as Bosco says, this is not really what review is for:
@boscoh 1/2 No. We need multiple changes in how things are done, of which publishing reproducible code is just one.
—
Neil Saunders (@neilfws) August 17, 2012@boscoh 2/2 But if the code were "out there", there's a chance someone may have spotted the problem.
—
Neil Saunders (@neilfws) August 17, 2012@neilfws agreed. what needs to happen is outside of the actual review process itself. it's more of a meta-requirement to deposit code.
—
Bosco Ho (@boscoh) August 17, 2012Chris made a similar point:
@neilfws you will be able to find errors in published code, yes. But people will still make them and they are not found automatically.
—
Chris Evelo (@Chris_Evelo) August 17, 2012Summary
It’s easy to make the mistake of thinking that making code available is the solution to reproducibility and error detection. In fact, it’s just a small part of the solution, since someone (or something) then has to (1) look at the code and (2) notice the errors.
While we wait for that change to sweep aside traditional scientific publishing, I suggest that if you want to publish research that uses code to generate the results – at least get your colleagues to take a look before submission.
What You're Doing Is Rather Desperate 
Required code submission could have made a meaningful difference. Since authors do worry at least a bit about how their own code would stand up under the harsh light of review, if code submission were required, then authors would probably give their code another quality check or two before submitting the whole batch to a journal. That could have turned up the error. We already check the article text for typos before submission, right?
Absolutely agree; code checking should be like typo checking. One “problem”, I guess, is that journals fear submissions will drop if they enforce standards. Personally, I don’t see how “less crap” is a problem.