Software availability
A BioStar user asks: where is the software for the method described in a Science article, “A Composite of Multiple Signals Distinguishes Causal Variants in Regions of Positive Selection.”

No-one can find it on the Web; the best we can do is a press release from 2010 stating that the software “should soon be available.” Why do the so-called flagship journals and their reviewers have so little interest in the methods used to generate data shown in papers? It’s beyond my comprehension.

Retractions on retractions
I was reviewing PubMed retraction data for 2011 (the “year of retractions”) using my PMRetract application, when I noticed:

Retraction: An Integrated Approach to the Prediction of Chemotherapeutic Response in Patients with Breast Cancer

It caught my eye for a couple of reasons. First, the authors decided to retract because they based their analyses on methods described in – another retracted article. Second, although the authors of that article don’t mention it in the retraction notice, their retraction is the result of a reproducibility study in one of my all-time favourite articles: “Deriving chemosensitivity from cell lines: Forensic bioinformatics and reproducible research in high-throughput biology.”

I note that the PLoS ONE retraction occurred about 8 months after the Nat. Med. retraction, which in turn occurred over 4 years after the original publication and just over a year after the Ann. Appl. Stat. re-analysis. This suggests to me that the PLoS ONE authors were alerted by the Nat. Med. retraction, not the excellent Baggerley et al. article. If only the latter article were published in Nat. Med., as opposed to a less widely-read applied statistics journal.

Regardless, credit to the PLoS ONE authors for a courageous retraction based on the flawed work of others.

Org-mode published
I was somewhat surprised (and pleased) to see that org-mode, a tool for Emacs aimed at generating reproducible code, data analysis and documentation, has been written up and published in the Journal of Statistical Software.

And then cynicism set in. Is software that life scientists don’t use, published in a journal that they don’t read, really going to address the problem?

Perhaps one day, all journals will demand appropriate standards for reproducible research. Until then I’ll just have to continue feeling amazed, bemused and troubled.

Click here to see the complete report.

Sequencing in the good old days

By the time I started my first postdoc, technology had moved on a little. We still did Sanger sequencing but the radioactive label had been replaced with coloured dyes and a laser scanner, which allowed automated reading of the sequence. During my second postdoc, this same technology was being applied to the shotgun sequencing of complete bacterial genomes. Assembling the sequence reads into contigs was pretty straightforward: there were a few software packages around, but most people used a pipeline of Phred (to call base qualities), Phrap (to assemble the reads) and Consed (for manual editing and gap-filling strategy).

The last time I worked directly on a project with sequencing data was around 2005. Jump forward 5 years to the BioStar bioinformatics Q&A forum and you’ll find many questions related to sequencing. But not sequencing as I knew it. No, this is so-called next-generation sequencing, or NGS. Suddenly, I realised that I am no longer a sequencing expert. In fact:

I am a relic from the Sanger era

I resolved to improve this state of affairs. There is plenty of publicly-available NGS data, some of it relevant to my current work and my organisation is predicting a move away from microarrays and towards NGS in 2012. So I figured: what better way to educate myself than to blog about it as I go along?

This is part 1 of a 4-part series and in this installment, we’ll look at how to get hold of public NGS data.

For these blog posts, I thought we’d stay with the theme of #arseniclife. A draft genome sequence for the organism in question, Halomonas sp. GFAJ-1, was recently assembled from NGS data and made publicly available.

My starting point is this page at the NCBI Short Read Archive (SRA), a database saved from closure this year when the curators secured extra funding. There’s a lot of useful information on this page, some of which is hidden under “more” links.

First, details of the DNA library:

Strategy:  WGS
Source:    GENOMIC
Selection: RANDOM
Layout:    PAIRED, Orientation: , Nominal length: 150, Nominal Std Dev: 30

Second, details of the sequencing platform:

Instrument model: Illumina HiSeq 2000
Spot descriptor:  -> 1  forward 102  reverse <-
Total:            1 run, 65.1M spots, 13.2G bases
    Download reads for this experiment in sra(7.0G) or sra-lite(7.0G) formats 
#       Run          # of Spots    # of Bases
1.      SRR385952    65,107,412    13.2G

Data are available in 2 formats: sra and sra-lite. There isn’t much to choose between them, except that sra contains some extra information such as intensity scores. Note that both are quite large files: after processing they’ll become even larger. So the first rule of NGS work: you need powerful hardware with plenty of storage and memory. I’m currently working on a shared server with access to 2 TB of storage, 96 GB RAM and 24 CPUs – and it’s just about enough.

There are software packages which handle SRA format, but most people convert to a widely-used NGS format called FASTQ. To do this we use the SRA toolkit provided by NCBI. There’s no pre-compiled 64-bit version, but it was easy enough to compile on my machine, just by following the instructions in README-build:

cd
wget http://trace.ncbi.nlm.nih.gov/Traces/sra/static/sra_sdk-2.1.8.tar.gz
tar zxvf sra_sdk-2.1.8.tar.gz
cd sra_sdk-2.1.8
OUTDIR=/home/neil/sra
make OUTDIR="$OUTDIR" out
make static
make GCC debug
make

This provides the program ~/sra/bin64/fastq-dump. We get the SRA file from the NCBI FTP site:

mkdir -p ~/projects/gfaj1
cd ~/projects/gfaj1
wget ftp://ftp-trace.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByExp/sra/SRX/SRX109/SRX109792/SRR385952/SRR385952.sra

Now the fun begins. First, don’t do this:

~/sra/bin64/fastq-dump SRR385952.sra

If you do and then examine the resulting file SRR385952.fastq, you’ll see sequences with length = 202. However, you’ll see from the sequencing platform details earlier that there should be forward and reverse sequences, each with length = 101. These are joined in the SRA file and need to be split, which you can do in one of two ways:

# 1 file
~/sra/bin64/fastq-dump --split-spot SRR385952.sra
# 2 files
~/sra/bin64/fastq-dump --split-files SRR385952.sra

Note: the documentation for –split-files is confusing. It reads “Dump each read into separate file.” What that actually means is: dump into forward and reverse files. Note also: very poor documentation is a hallmark of NGS software; we’ll see plenty more of it later.

I ran fastq-dump with the –split-spot option, which generates a single file with the reverse read of each pair below the forward read for that pair. Whichever way you do it, there’s a problem:

head -8 SRR385952.fastq

@SRR385952.1 HWI-ST484_0123:8:1101:1154:2066 length=101
ATNCGTCCTTTATTCTGCCAGGGAATTGGCCCGTTCCGCTGGGCAGCGCTTTCCGGCGACCCGGAAGATATTTACAAAACCGACCAGAAGGTCAAAGAGCT
+SRR385952.1 HWI-ST484_0123:8:1101:1154:2066 length=101
BC#4ADDFHHHHHJHIJJJJJJJIJJIJJJJJJJJHGGHIJJJJIJJJIJJHHHHFEDDDDDDDDDDDDDDEECDCCCBDBD>DDDDDDDDCCCCCDDDDA
@SRR385952.1 HWI-ST484_0123:8:1101:1154:2066 length=101
GGTCGTCGGGGATCAGCTCTTTGACCTTCTGGTCGGTTTTGTAAATATCTTCCGGGTCGCCGGAAAGCGCTGCCCAGCGGAACGGGCCAATTCCCTGGCAG
+SRR385952.1 HWI-ST484_0123:8:1101:1154:2066 length=101
CCBFFFFFHHHHHJJJJJJJJJJJJIIJJJJIEHIJHIJJIHHIJJIIGIIIHHHFDDDDDDDDDDDDDDBDDDDDDDDDBDDDDDDDDBDDCDDDDBDDD

Each sequence in a FASTQ file is described using 4 lines: a header, the sequence, a header for quality scores and the scores themselves. So by using head -8 we see the first pair of forward + reverse reads. Unfortunately, they have the same ID in the header line.

One convention for identifying paired reads is to suffix the ID using either /1 or /2. We can do that using sed:

sed -i '1~8s/^@SRR385952\.[0-9]*/&\/1/' SRR385952.fastq
sed -i '5~8s/^@SRR385952\.[0-9]*/&\/2/' SRR385952.fastq

To explain that: headers for forward reads are on lines 1, 9, 17… and those for reverse on lines 5, 13, 21… So using sed, we append “/1″ to the ID on lines 1, 9, 17… and “/2″ to the ID on lines 5, 13, 21… Result:

head -8 SRR385952.fastq

@SRR385952.1/1 HWI-ST484_0123:8:1101:1154:2066 length=101
ATNCGTCCTTTATTCTGCCAGGGAATTGGCCCGTTCCGCTGGGCAGCGCTTTCCGGCGACCCGGAAGATATTTACAAAACCGACCAGAAGGTCAAAGAGCT
+SRR385952.1 HWI-ST484_0123:8:1101:1154:2066 length=101
BC#4ADDFHHHHHJHIJJJJJJJIJJIJJJJJJJJHGGHIJJJJIJJJIJJHHHHFEDDDDDDDDDDDDDDEECDCCCBDBD>DDDDDDDDCCCCCDDDDA
@SRR385952.1/2 HWI-ST484_0123:8:1101:1154:2066 length=101
GGTCGTCGGGGATCAGCTCTTTGACCTTCTGGTCGGTTTTGTAAATATCTTCCGGGTCGCCGGAAAGCGCTGCCCAGCGGAACGGGCCAATTCCCTGGCAG
+SRR385952.1 HWI-ST484_0123:8:1101:1154:2066 length=101
CCBFFFFFHHHHHJJJJJJJJJJJJIIJJJJIEHIJHIJJIHHIJJIIGIIIHHHFDDDDDDDDDDDDDDBDDDDDDDDDBDDDDDDDDBDDCDDDDBDDD

There’s one more thing to note about fastq-dump. By default, the quality score offset is 33. What does that mean? Let’s look at the first five characters in the quality line for the first sequence:

character    ASCII code (decimal)    ASCII code - 33
B            66                      33
C            67                      34
#            35                       2
4            52                      19
A            65                      32

SRA sequence view with quality scores

So to get the quality score, just subtract 33 from the decimal ASCII code of the character. We can check our results by looking at the read back at the SRA website; see image, left. Schemes for FASTQ encoding of quality scores are, apparently, a bit of a mess. We’ll see in later posts that some software tools expect certain encodings (and even have completely undocumented features to deal with other encoding schemes). For now though, it’s enough to know that our FASTQ file is in agreement with the SRA data and move on.

So what have we achieved?

ls -l SRR385952.fastq

-rw-r--r-- 1 neil neil 43719072816 2011-12-22 14:56 SRR385952.fastq

We’ve generated a large (~ 44 GB) FASTQ file, containing paired reads with unique header IDs. It was quite a bit of work simply to fetch and format the raw data.

In the next installment: manipulation of FASTQ files and sequence quality assessment.

As Iddo pointed out on Twitter, sequencing the DNA from GFAJ-1 is itself strong evidence against arsenate in the DNA backbone, since the sequencing chemistry would be highly unlikely to work in that case. However, if like me you think that a new microbial genome provides the most fun to be had in bioinformatics [*], you’ll be excited by the availability of the data.

In this post then: where to get it, some very preliminary analysis and some things that you might like to to with it. Projects for your students, perhaps.

[*] note to self: why, then, am I working on colorectal cancer?

1. The data
Here’s your starting point: the NCBI Whole Genome Shotgun Sequencing Project page for GFAJ-1. It provides a nice summary of the data including: the sequencing technology used (Illumina GAIIx), coverage (170x), assembly software (Mira) and number of contigs (103).

Files for download: annotated contigs in Genbank format (wgs.AHBC.1.gbff.gz), the WGS “master” Genbank file (wgs.AHBC.mstr.gbff.gz – not really necessary) and contig sequences in fasta format (wgs.AHBC.1.fsa_nt.gz).

You can also fetch raw sequencing data from the Short Read Archive, but more about that a little later.

Be aware that this is a draft genome, annotated using an automated computational pipeline. This means that there will be errors in the sequence, the genes and the annotation. It also means plenty of potential improvements can be made by smart, enthusiastic bioinformaticians.

2. Arsenic metabolism
Of course your first instinct is to see whether any of the putative protein products have been annotated as having an “arsenic-related” function. Since this might involve arsenic, arsenate or arsenite, the simplest approach is:

grep arsen wgs.AHBC.1.gbff

And the result:

                     /product="arsenite-activated ATPase ArsA"
                     /note="COG1055 Na+/H+ antiporter NhaD and related arsenite
                     /product="arsenical-resistance protein"

Digging around in the file a little more reveals more information about these two putative products:

CDS             90659..91666
                     /locus_tag="MOY_00685"
                     /note="COG0003 Oxyanion-translocating ATPase"
                     /codon_start=1
                     /transl_table=11
                     /product="arsenite-activated ATPase ArsA"
                     /protein_id="EHK62524.1"
                     /db_xref="GI:359298308"
                     /translation="MQAVLNRRLLWVGGKGGVGKTTVAASLAVLAARRGKRVLVVSTD
                     PAHSLGDVFDRALSDIPRRLLPNLDAMEIDPDIEVEAHLARVVKQMRRYAAPEMMQEL
                     ERQMRLTRQSPGTQEAALLERLARLMVDDSAPYDLIIFDTAPTGHTLRLLTLPEAMAA
                     WTDGLLAHNRKSAELGKVLEHLTPKRGRDVATPFDDPTVDPLDDLDERTRDVAKTLID
                     RRRLFHQARRRIEDSKACSFLFVMTPERLPILETDRAVKALEEVHIPVAGVLINRLIP
                     IEADGDFLQARREQEATYLTRIDELFERLPRPTLPWLPTDVQGIEVLEMLAQKLEQQG
                     F"

CDS             8835..9863
                     /locus_tag="MOY_16088"
                     /note="COG0798 Arsenite efflux pump ACR3 and related
                     permeases"
                     /codon_start=1
                     /transl_table=11
                     /product="arsenical-resistance protein"
                     /protein_id="EHK59503.1"
                     /db_xref="GI:359295220"
                     /translation="MGLFERYLSIWVAIAIVAGIALGQFAPAVPEVLSRFEYAQVSIP
                     IAVLIWAMIFPMMAQIDFSAIAGVRRQPKGLTITTTVNWLIKPFTMFALAWLFFMVIF
                     KPFIPEELASQYLAGAILLGAAPCTAMVFVWSYLTRGDAAYTLVQVALNDLIMLFAFA
                     PLVVFLLGISNIQVPWDTVILSVVLYIVIPLAAGYFTRKTLIAKYGTEWYDNVFMKRV
                     GPITPIGLIITLVLLFAFQGEVILNNPLHIVLIAIPLIIQTFLIFFIAYGWAKAWRVP
                     HNIAAPGAMIGASNFFELAVAAAIALFGLQSGAALATVVGVLVEVPLMLALVRIANKT
                     RQHFPENT"

So it seems we may have a system for pumping arsenite out of the cell. Not unexpected, for an organism known to have high arsenic tolerance. Presumably, if the form of arsenic in the environment is arsenate, there’s a mechanism for reducing it to arsenite (if these annotations are correct). Some interesting bioenergetic possibilities there.

By the way, these findings seem to be at odds with the statement in this article from USA Today that “none of them [the genes] look like identified arsenic resistance genes found in other bugs.”

If we were to postulate that this organism incorporated arsenate into the DNA backbone (I do not believe it does), how might we use the genome sequence data? I guess there are two main possibilities:

1. The genes for nucleotide metabolism encode modified proteins which can catalyse reactions using arsenate-containing compounds instead of, or as well as, phosphates
2. The genome contains novel genes for nucleotide metabolism that we might not even recognise as such

Exploring idea (1) a little further – nucleotide biosynthesis is a complicated business. See, for example, this KEGG pathway map for purine metabolism. I suppose in principle, it’s possible to retrieve the sequences for some of the key enzymes, search for GFAJ-1 orthologs using BLAST, then further investigate the GFAJ-1 protein sequences (domains? homology modelling?) to see if any of them are “unusual”. However, this entails all manner of unfounded assumptions about a hypothetical arsenic-based biochemistry: for example, the existence of analogous mono-, di- and tri-arsenate compounds. I wouldn’t go there myself.

3. Phylogeny and genome alignment
How “Halomonas-like” is Halomonas GFAJ-1 anyway? Let’s start by looking for 16S rRNA genes:

grep -P "^LOCUS|ribosomal RNA" wgs.AHBC.1.gbff

Like many bacteria, GFAJ-1 appears to have multiple rRNA operons (unless the assembly has failed to merge them). One of these is on contig AHBC01000086:

LOCUS       AHBC01000086            5885 bp    DNA     linear   BCT 02-DEC-2011                     /product="16S ribosomal RNA"                     
 /product="23S ribosomal RNA"                     
 /product="5S ribosomal RNA"

It’s easy enough to copy/paste the 16S rDNA sequence from the file, or you can use the handy bp_extract_feature_seq script which comes with Bioperl to grab all rDNA sequences. The 16S sequence can be identified by its length (not the shortest which is 5S, or the longest which is 23S):

bp_extract_feature_seq -i wgs.AHBC.1.gbff --format genbank --feature=rRNA -o rrna.fa

NCBI BLAST GFAJ-1 vs. Bacteria

Halomonas appears to be a rather diverse genus but indeed, GFAJ-1 seems most similar to other Halomonas. Is there genomic information for any of the related organisms? A quick search suggests not; 7 organisms, only 2 of which have available genome sequence, neither of which feature in the top 100 BLAST hits for GFAJ-1 16S rDNA.

Regardless, it might be fun to try aligning the GFAJ-1 contigs with a reference genome, in this case from Halomonas elongata. We can save the chromosome sequence in Fasta format, install MUMmer and run:

nucmer -maxmatch -p nucmer helongata.fa wgs.AHBC.1.fsa_nt
show-coords -r -c -l nucmer.delta > nucmer.coords
mapview -n 1 -p mapview nucmer.coords
xfig mapview_0.fig

And sure, there are a few nicely-aligned segments (click image, right), but nothing spectacular in terms of a good reference genome which we could use to order large numbers of contigs.

Part of MUMmer alignment: GFAJ-1 contigs to H. elongata chromosome

4. Assembly quality and raw data
First, a quick look at the contig statistics, using grep and a little R:

grep ^LOCUS wgs.AHBC.1.gbff > locus.txt library(ggplot2) loc <- read.table("locus.txt", header = F, stringsAsFactors = F) fivenum(loc$V3) # [1] 545 1385 9163 48993 233806 ggplot(loc) + geom_density(aes(V3), fill = "cornsilk") + theme_bw() + opts(title = "Density plot GFAJ-1 contig lengths")

Density plot of GFAJ-1 contig length

This tells us that many of the contigs are rather short: 50% are 9 163 bp or less (the median) and 12 contigs are 100 000 bp or more in length. The long contigs add up to around 1/3 of the total genome size, which seems like a good result.

Good, but might other assembly software perform better? Happily, you’re in a position to find out because the raw sequencing data can be found in the Short Read Archive (SRA). You can grab them in SRA format (danger! 7.5 GB!) like so:

wget ftp://ftp-trace.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR385/SRR385952/SRR385952.sra

You’ll then need to grab the SRA toolkit and dump the SRA to FASTQ format. Things become complex here because the toolkit documentation is not great and we need to figure out what (a) kind of reads we have and (b) how to restore them from the SRA file.

To cut a long story short, this line tells us that we have a paired-end library:

Layout: PAIRED, Orientation: , Nominal length: 150, Nominal Std Dev: 30

and so we need to dump that into 2 fastq files:

fastq-dump --split-files SRR385952.sra

but this has the unfortunate effect of generating identical fastq headers in both files:

head -4 SRR385952_*.fastq

==> SRR385952_1.fastq <==
@SRR385952.1 HWI-ST484_0123:8:1101:1154:2066 length=101
ATNCGTCCTTTATTCTGCCAGGGAATTGGCCCGTTCCGCTGGGCAGCGCTTTCCGGCGACCCGGAAGATATTTACAAAACCGACCAGAAGGTCAAAGAGCT
+SRR385952.1 HWI-ST484_0123:8:1101:1154:2066 length=101
BC#4ADDFHHHHHJHIJJJJJJJIJJIJJJJJJJJHGGHIJJJJIJJJIJJHHHHFEDDDDDDDDDDDDDDEECDCCCBDBD>DDDDDDDDCCCCCDDDDA

==> SRR385952_2.fastq <==
@SRR385952.1 HWI-ST484_0123:8:1101:1154:2066 length=101
GGTCGTCGGGGATCAGCTCTTTGACCTTCTGGTCGGTTTTGTAAATATCTTCCGGGTCGCCGGAAAGCGCTGCCCAGCGGAACGGGCCAATTCCCTGGCAG
+SRR385952.1 HWI-ST484_0123:8:1101:1154:2066 length=101
CCBFFFFFHHHHHJJJJJJJJJJJJIIJJJJIEHIJHIJJIHHIJJIIGIIIHHHFDDDDDDDDDDDDDDBDDDDDDDDDBDDDDDDDDBDDCDDDDBDDD

and so, some sed is required to generate the more standard “/1 or /2″ fastq header suffix:

sed -i -e 's/^@SRR385952\.[0-9]*/&\/1/' SRR385952_1.fastq
sed -i -e 's/^@SRR385952\.[0-9]*/&\/2/' SRR385952_2.fastq

After which, you can experiment with the de novo assembler of your choice. I’ll leave the appalling quality and documentation of NGS software in general for another post.

Enjoy – but don’t spend too much time on it. You won’t find the key to “arsenic life” in this genome.

1. R/statistics

• ggbio

A new Bioconductor package which builds on the excellent ggplot graphics library, for the visualization of biological data.

• R development master class

Hadley Wickham recently presented this course on R package development for my organisation. I was on parental leave at the time, otherwise I would have attended for sure.

2. Bioinformatics in the media
DNA Sequencing Caught in Deluge of Data

I described this NYT article as a “surprisingly-good intro article“. Michael Eisen described it as “kind of silly“.

I think we’re both right. Michael’s perspective is that of an expert in high-throughput sequencing data; I’m just pleased to see an introduction to bioinformatics for non-specialists in a mainstream newspaper. And I note that they have corrected the figure caption which offended Michael.

As to the “deluge”: yes, there are other sciences that generate more data and yes, we probably don’t need to archive/analyse a lot of the raw data. However, I’d contend that the basic premise of the article is correct: we are sequencing faster than we can analyse. The solution, obviously, is more bioinformaticians.

Despite the fact that there is a voluminous literature on thinking and problem solving, including intensive case-history studies of the process of invention, I could find nothing comparable to a time-and-motion-study analysis of the mental work of a person engaged in a scientific or technical enterprise. In the spring and summer of 1957, therefore, I tried to keep track of what one moderately technical person actually did during the hours he regarded as devoted to work. Although I was aware of the inadequacy of the sampling, I served as my own subject.

It soon became apparent that the main thing I did was to keep records, and the project would have become an infinite regress if the keeping of records had been carried through in the detail envisaged in the initial plan. It was not. Nevertheless, I obtained a picture of my activities that gave me pause. Perhaps my spectrum is not typical–I hope it is not, but I fear it is.

About 85 per cent of my “thinking” time was spent getting into a position to think, to make a decision, to learn something I needed to know. Much more time went into finding or obtaining information than into digesting it. Hours went into the plotting of graphs, and other hours into instructing an assistant how to plot. When the graphs were finished, the relations were obvious at once, but the plotting had to be done in order to make them so. At one point, it was necessary to compare six experimental determinations of a function relating speech-intelligibility to speech-to-noise ratio. No two experimenters had used the same definition or measure of speech-to-noise ratio. Several hours of calculating were required to get the data into comparable form. When they were in comparable form, it took only a few seconds to determine what I needed to know.

Throughout the period I examined, in short, my “thinking” time was devoted mainly to activities that were essentially clerical or mechanical: searching, calculating, plotting, transforming, determining the logical or dynamic consequences of a set of assumptions or hypotheses, preparing the way for a decision or an insight. Moreover, my choices of what to attempt and what not to attempt were determined to an embarrassingly great extent by considerations of clerical feasibility, not intellectual capability.

Normally, I’d use something like Ruby’s Mechanize library for this purpose. However, where the purpose is to retrieve delimited data for analysis using R, I figured it was time to try and achieve the entire process within R. So here’s how I used the RCurl and XML packages to interact with the WHAT IF server, which provides tools for the analysis of protein structure.

1. A note of caution
The administrators of some servers don’t like robots; particularly those which fire off thousands of queries per second to a server designed to cope only with a small number of requests. So use common sense; check whether the administrators have made their policy public and respect the limits.

2. Know your HTML
Step 1 in automating form submission is understanding how the web form works. So: head to the WHAT IF server, click “Atomic contacts” in the left menu, then right-click “Salt bridges” and open that link in a new tab. Examination of the HTML shows this:

<FORM METHOD="POST" ACTION="/wiw-cgi/GenericCGI.py" ENCTYPE="multipart/form-data"> 
<INPUT TYPE="hidden" NAME="request" VALUE="shosbr" > 
<TABLE border=0 cellpadding=4 cellspacing=1 width="100%"> 
<TR><TD Align=left >Either Choose a pdb-file,</TD> 
<TD Align=left ><INPUT TYPE="TEXT" NAME="&PDB1" SIZE=8 MAXLENGTH=4 > 
</TD> 
</TR> 
<TR><TD Align=left >Or Choose your own file</TD> 
<TD Align=left ><INPUT TYPE="file" NAME="&FIL1" > 
</TD> 
</TR> 
</TABLE><P> 
<INPUT TYPE="submit" NAME="SubmitButton" VALUE="Send" > 
<INPUT TYPE="reset" NAME="Reset" VALUE="Clear Form" > 
</FORM>

What this tells you is that the server runs a Python script, named GenericCGI.py, to which you can submit a set of name=value parameters: request=shosbr, &PDB1=NNNN (where NNNN is a PDB identifier) or &FIL1=FILE, where FILE is a PDB file on your local machine.

3. Submission of query
Let’s choose a small protein – 3N2J, a bacterial azurin and submit it to WHAT IF using R:

library(RCurl)
salt <- postForm("http://swift.cmbi.ru.nl/wiw-cgi/GenericCGI.py", "&PDB1" = "3N2J", "request" = "shosbr", "submitButton" = "Send")

That takes a little while to run and when finished, the variable salt contains a bunch of HTML which looks like this:

<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2//EN\">\n<HTML>\n\n<!-- This file generated using Python HTMLgen module. -->\n<HEAD>\n  <META NAME=\"GENERATOR\" CONTENT=\"HTMLgen 2.0.6\">\n        <TITLE>Salt bridges.</TITLE>\n </HEAD>\n<BODY BGCOLOR=\"WHITE\">\n<H1>Salt bridges.</H1>\n\n\n            Your job has been passed to WHAT IF and is processed.\n            Your request might take a while, please wait.\n            \n\n</BODY> </HTML>\n<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2//EN\">\n<HTML>\n\n<!-- This file generated using Python HTMLgen module. -->\n<HEAD>\n  <META NAME=\"GENERATOR\" CONTENT=\"HTMLgen 2.0.6\">\n        <TITLE>Salt bridges.</TITLE>\n </HEAD>\n<BODY BGCOLOR=\"WHITE\">\n\n<P>\n\nAccess your result.\n\n<FORM METHOD=\"POST\" ACTION=\"/wiw-cgi//GenericCGI.py\">\n<INPUT TYPE=\"hidden\" NAME=\"ID\" VALUE=\"/tmprM7rDq/\" >\n<INPUT TYPE=\"hidden\" NAME=\"refresh\" VALUE=\"shosbr\" >\n<INPUT TYPE=\"submit\" NAME=\"submit\" VALUE=\"Results\" >\n\n</FORM>\n\n    <HR> \n    If you have detected any error, or have any question or suggestion,\n    please send an Email to Gert Vriend.<BR>\n    Roland Krause, Jens Erik Nielsen, \n    <a href=\"mailto: Vriend@CMBI.ru.nl\">Gert Vriend</a>.<P>\n    <HR>\n    Last modified Thu Sep  8 12:07:53 2011\n    \n\n</BODY> </HTML>\n

4. Retrieval of results
The WHAT IF server is a bit of a tease. Examination of the HTML in variable salt, above, reveals that instead of returning results, we have an intermediate page containing another form, which we have to submit to retrieve the final data. The key form parameters are: ID=/tmprQzd2o/, submit=Results and refresh=shosbr. The parameter ID refers to a temporary location on the server which holds the results and will change with each submitted query.

We can submit this form in the same way:

library(RCurl)
salt <- postForm("http://swift.cmbi.ru.nl/wiw-cgi/GenericCGI.py", "ID" = "/tmprQzd2o/", "request" = "shosbr", "submitButton" = "Send")

That returns a bunch more HTML – just looking at the first part:

<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2//EN\">\n<HTML>\n\n<!-- This file generated using Python HTMLgen module. -->\n<HEAD>\n  <META NAME=\"GENERATOR\" CONTENT=\"HTMLgen 2.0.6\">\n        <TITLE>Salt bridges.</TITLE>\n </HEAD>\n<BODY BGCOLOR=\"white\">\n<H1>Salt bridges.</H1>\n\nThe list of salt bridges:\n<PRE>Date= 2011-09-08 12:07:52\nYou are prompted for the first range\nYou are prompted for the second range\n   1   6 ASP  (   6 ) A      OD1 -    1 ALA  (   1 ) A      N     6.78\n   2   6 ASP  (   6 ) A      OD2 -    1 ALA  (   1 ) A      N     5.98\n   3  11 ASP  (  11 ) A      OD1 -   35 HIS  (  35 ) A      ND1   4.35\n   4  11 ASP  (  11 ) A      OD1 -   35 HIS  (  35 ) A      NE2   4.23\n
...

Success. We have results and if you squint, you might be able to see that they are wrapped up in PRE tags. Now, how to get them out?

5. Parsing of results
Here, we bring the XML package into play. First, parse the returned HTML and pull out the text between the PRE tags:

library(XML)
sb  <- htmlTreeParse(salt, useInternalNodes = T)
sb1 <- xpathApply(sb, "//pre", xmlValue)
class(sb1)
# "list"

That gives us a list, containing one element which is a character vector containing the data as preformatted text.

6. Conversion to a data frame
Now for the tricky part. It looks as though each line of the data is space-delimited. So, you might think that we could read it into a data frame by treating the text as a connection. Question is: how consistent is the data output? Do all lines contain the same number of fields, for example? What about fields that themselves contain spaces?

Well, all we can do is give it a try. First, a quick look at the start and end of the data:

head(readLines(textConnection(sb1[[1]])))
# [1] "Date= 2011-09-08 12:07:52"                                             
# [2] "You are prompted for the first range"                                  
# [3] "You are prompted for the second range"                                 
# [4] "   1   6 ASP  (   6 ) A      OD1 -    1 ALA  (   1 ) A      N     6.78"
# [5] "   2   6 ASP  (   6 ) A      OD2 -    1 ALA  (   1 ) A      N     5.98"
# [6] "   3  11 ASP  (  11 ) A      OD1 -   35 HIS  (  35 ) A      ND1   4.35"

tail(readLines(textConnection(sb1[[1]])))
# [1] " 5871506 ASP  (  98 ) L      OD2 - 1509 LYS  ( 101 ) L      NZ    6.02"
# [2] " 5881512 GLU  ( 104 ) L      OE1 - 1432 LYS  (  24 ) L      NZ    2.84"
# [3] " 5891512 GLU  ( 104 ) L      OE2 - 1432 LYS  (  24 ) L      NZ    4.42"
# [4] " 5901514 GLU  ( 106 ) L      OE2 - 1511 LYS  ( 103 ) L      NZ    5.43"
# [5] " 5911572 LYS  ( 128 ) L      O'' - 1432 LYS  (  24 ) L      NZ    6.43"
# [6] ""

OK, looks like we can lose the first 3 lines and perhaps split on whitespace?

sbl <- readLines(textConnection(sb1[[1]]))
sbl <- sbl[4:length(sbl)]
sbt <- read.table(textConnection(sbl), fill = T)

head(sbt)
#   V1 V2  V3 V4 V5 V6 V7  V8 V9 V10 V11 V12 V13 V14 V15 V16  V17
# 1  1  6 ASP  (  6  )  A OD1  -   1 ALA   (   1   )   A   N 6.78
# 2  2  6 ASP  (  6  )  A OD2  -   1 ALA   (   1   )   A   N 5.98
# 3  3 11 ASP  ( 11  )  A OD1  -  35 HIS   (  35   )   A ND1 4.35
# 4  4 11 ASP  ( 11  )  A OD1  -  35 HIS   (  35   )   A NE2 4.23
# 5  5 11 ASP  ( 11  )  A OD2  -  35 HIS   (  35   )   A ND1 6.01
# 6  6 11 ASP  ( 11  )  A OD2  -  35 HIS   (  35   )   A NE2 6.33

tail(sbt)
#          V1  V2 V3  V4 V5 V6  V7 V8   V9 V10 V11 V12 V13 V14 V15  V16 V17
# 586 5861506 ASP  (  98  )  L OD2  - 1435 LYS   (  27   )   L  NZ 5.09  NA
# 587 5871506 ASP  (  98  )  L OD2  - 1509 LYS   ( 101   )   L  NZ 6.02  NA
# 588 5881512 GLU  ( 104  )  L OE1  - 1432 LYS   (  24   )   L  NZ 2.84  NA
# 589 5891512 GLU  ( 104  )  L OE2  - 1432 LYS   (  24   )   L  NZ 4.42  NA
# 590 5901514 GLU  ( 106  )  L OE2  - 1511 LYS   ( 103   )   L  NZ 5.43  NA
# 591 5911572 LYS  ( 128  )  L O''  - 1432 LYS   (  24   )   L  NZ 6.43  NA

That almost worked – except…

7. …we have a glitch

Rows in the data frame sbt which look like this:

5911572 LYS  ( 128  )  L O''  - 1432 LYS   (  24   )   L  NZ 6.43  NA

should actually look like this:

591 1572 LYS  ( 128  )  L O''  - 1432 LYS   (  24   )   L  NZ 6.43

The problem lies with the original output from WHAT IF. There needs to be a space between column 1 (contact number) and column 2 (residue number of first interacting molecule). In fact, I noticed this problem for the first time today, in the course of this R investigation.

Well you win some, you lose some. At least I learned a thing or two about RCurl and XML.

One source of data is the tags used for questions. Tags are somewhat arbitrary of course, but fortunately BioStar has quite an active community, so “bad” tags are usually edited to improve them. Hint: if your question is “How to find SNPs”, then tagging it with “how, to, find, snps” won’t win you any admirers.

OK: we’re going to grab the tags then use a bunch of R packages (XML, wordcloud and ggplot2) to take a quick look.

1. Fetch the tags
Fortunately, I enjoy sufficient privileges at BioStar to obtain a dump of the database. It contains a file named “Tags.xml”, with this simple structure:

<Tags>
  <row>
    <Id>3</Id>
    <Name>bed</Name>
    <Count>20</Count>
    <UserId>2</UserId>
    <CreationDate>2009-09-30T14:55:00.167</CreationDate>
  </row>
  ...
</Tags>

A hint for people who write XML parsing documentation. Most of us just want to get the values from between the tags. Just tell us how to do that. OK?

Thanks to this StackOverflow thread, I discovered the incredibly-useful xmlToDataFrame() function in the R XML package:

library(XML)
tags <- xmlToDataFrame("Tags.xml")
head(tags)
#   Id       Name Count UserId            CreationDate
# 1  3        bed    20      2 2009-09-30T14:55:00.167
# 2  4        gff    12      2 2009-09-30T14:55:00.167
# 3  5     galaxy    11      2 2009-09-30T15:09:43.417
# 4  6      yeast     5      3 2009-09-30T16:09:06.723
# 5  7      motif    19      3  2009-09-30T16:09:06.74
# 6  8 microarray    96      2 2009-09-30T16:44:22.677

Too easy. However, class(tags$Count) = “character”, which is not quite not we want. So let’s change that to numeric, then sort the data frame on Count, decreasing:

tags$Count <- as.numeric(tags$Count)
tags <- tags[sort.list(tags$Count, decreasing = T),]

2. For those who like a “top N” plot Next, we’ll grab the top 20 tags by Count. To plot them in decreasing order, we need to reorder the tag Name by Count. With thanks again to a StackOverflow thread. library(ggplot2) tags.20 <- head(tags, 20) tags.20 <- transform(tags.20, Name = reorder(Name, Count)) ggplot(tags.20) + geom_bar(aes(Name, Count), fill = "coral") + coord_flip() + theme_bw() + opts(title = "Top 20 BioStar Tags") Click image, right, for full-size version.

Top 20 Biostar Tags

3. For those who like word/tag clouds

Here, we look at tags which occur 10 or more times and display a maximum of 1000 tags in the cloud. Again, click image for the full-size version. library(wordcloud) library(RColorBrewer) png(file = "tags.png", width = 1024, height = 1024) wordcloud(tags$Name, tags$Count, scale = c(8,.2), min.freq = 10, max.words = 1000, random.order = F, rot.per = .15, colors = brewer.pal(8, "Dark2")) dev.off()

BioStar tag cloud

Conclusions? XML, ggplot2 and wordcloud are all great packages. And whilst so-called “next-generation-sequencing” might be all the rage, it’s good to see the old stalwarts of bioinformatics hanging in there: BLAST, alignment, phylogenetics, Python and Perl. It will be interesting to see how tags change over time.

PubMed cumulative retractions 1977-present

Several of these sources cite data from my humble web application, PMRetract. So now seems like a good time to mention that:

• The application is still going strong and is updated regularly
• I’ve added a few enhancements to the UI; you can follow development at GitHub
• I’ve also added a long-overdue about page with some extra information, including the fact that I wrote it :)

Now I just need to fix up my Git repositories. Currently there’s one which pushes to GitHub and a second, with a copy of the Sinatra code for pushing to Heroku, which isn’t too smart.

@vlandham points out that when the main BioRuby repository updates, you’ll want to update your local repository. Using git, you do that by adding a remote which points to the original repository, from which you can fetch updates and merge with your local version:

git remote add upstream https://github.com/bioruby/bioruby.git
# fetch/merge only when main repo updates
git fetch upstream
git merge upstream master

This is described at the GitHub help page Fork A Repo.

Michael points to an article titled A successful Git branching model. It suggests that when developing new features you create a feature branch (also called topic branch). This can help with the management of new features and creates a more complete commit history if/when the new feature is merged back into your development repository. The article also suggests a main branch for development named develop, rather than the default master.

I haven’t quite got my head around all the ins-and-outs of the article yet, but it’s well worth a read.