Just a brief note: the title of this post is taken from the cover of today’s Nature. It contains several very good feature articles on the challenges of dealing with peta- (and more) byte size datasets, grouped under the heading “Big data”.
Nature contents Sep 4 2008.
Nature News Big Data special.
By far the best of the articles is The future of biocuration: it offers practical recommendations, as opposed to the “gee whizz, what a lot of data” approach. Not least of which: “curators, researchers, academic institutions and funding agencies should, in the next ten years, increase the visibility and support of scientific curation as a professional career.”
Almost as good are Wikiomics, which tackles the lack of participation issue and Welcome to the petacentre, in which Boing-Boing’s Cory Doctorow explores, amongst other places, the Sanger Institute data centre.
So far as I can tell from the Nature News link, these articles are freely-available.