Coincidentally, someone asked the same question at Biostar. Pierre recommended the TogoWS REST service, which provides an API to multiple biological data sources. An article describing TogoWS was published in 2010.

An excellent suggestion – and one which, I later discovered, I had bookmarked. Twice. As long ago as 2008. This “rediscovery of things I once knew” happens to me with increasing frequency now, which makes me wonder whether (1) we really are drowning in information, (2) my online curation tools/methods require improvement or (3) my mind is not what it was. Perhaps some combination of all three.

Anyway – using Ruby (1.8.7), a list of HGNC symbols given a KEGG pathway, e.g. MAPK signaling, is as simple as:

require 'rubygems'
require 'open-uri'
require 'json/pure'

j = JSON.parse(open("http://togows.dbcls.jp/entry/pathway/hsa04010/genes.json").read)
g = j.first.values.map {|v| /^(.*?);/.match(v)[1] }
# first 5 genes
g[0..4]
# ["MAP3K14", "FGF17", "FGF6", "DUSP9", "MAP3K6"]

This code parses the JSON returned from TogoWS into an array with one element; the element is a hash with key/value pairs of the form:

"9020"=>"MAP3K14; mitogen-activated protein kinase kinase kinase 14 [KO:K04466] [EC:2.7.11.25]"

Values for all keys that I’ve seen to date begin with the HGNC symbol followed by a semicolon, making extraction quite straightforward with a simple regular expression.

Version 3.0.0 is great news for bioinformaticians, due to the introduction of long vectors. What does that mean? Well, several months ago, I was using the simpleaffy package from Bioconductor to normalize Affymetrix exon microarrays. I began as usual by reading the CEL files:

f <- list.files(path = "data/affyexon", pattern = ".CEL.gz", full.names = T, recursive = T)
cel <- ReadAffy(filenames = f)

When this happened:

Error in read.affybatch(filenames = l$filenames, phenoData = l$phenoData,  : 
  allocMatrix: too many elements specified

I had a relatively-large number of samples (337), but figured a 64-bit machine with ~ 100 GB RAM should be able to cope. I was wrong: due to a hard-coded limit to vector length in R, my matrix had become too large regardless of available memory. See this post and this StackOverflow question for the computational details.

My solution at the time was to resort to Affymetrix Power Tools. Hopefully, the introduction of the LONG vector will make Bioconductor even more capable and useful.

I had the pleasure of giving a brief presentation on the use of Git in bioinformatics. Nothing startling; aimed squarely at bioinformaticians who may have heard of version control in general and Git in particular but who are yet to employ either. I’m excited because for once I am free to share, resulting in my first upload to Slideshare in almost 4.5 years. You can view it here, or at the Australian Bioinformatics Network Slideshare, or in the embed below.

If you’re reading this, you’re probably aware that Google Reader will cease to exist from July 1 2013. Others have ranted, railed against the corporate machine and expressed their sadness. I thought I’d try to explain why, for this working scientist at least, RSS and feed readers are incredibly useful tools which I think should be valued highly.

Some feeds, yesterday

RSS: a primer
When I first discovered the concept of RSS, it was one of those moments that made me think: “this is so brilliant, simple and obvious – why isn’t everyone using this?”

In fact even today, very few of my immediate peers know what RSS is or why it’s useful. This may be an issue specific to Australian science, which is not exactly renowned for being at the cutting edge of the web revolution. However, for anyone else struggling with the concept, let’s spell it out:

The point of RSS is that:

• you can monitor multiple, diverse sources of information in one location (aggregation)
• you don’t have to visit those sources until their content updates and your feed reader tells you when that happens

What are these multiple, diverse sources of information? For a scientist they could include:

• Tables of contents from journals
• Alerts and searches at key research-related websites e.g. NCBI PubMed
• Science blogs
• Saved job searches
• Activity monitoring at personal websites

Brilliant, simple, obvious. I wonder how scientists keep up to date in their field without RSS.

The Rise of Google Reader

Soon after launch, Google Reader rose to become the predominant feed reader. Undoubtedly, this was due in part to the brand. However, GReader does boast several key features which I believe contributed to its adoption:

• It’s part of the “Google suite”; one login, multiple applications; in other words it’s “just there”
• It’s “in the cloud” and so available and synched on all your machines; no local setup
• No need to read everything immediately; it’s a searchable archive (they did take their time implementing search though, didn’t they)
• Sharing to multiple networks is relatively easy via the “send to” function (forget about the now-defunct sharing button)
• Intelligent keyboard shortcuts and simple layout enable rapid click-through, allowing focus on interesting items and discarding of irrelevant ones

RSS Is Not Dead (even if you’d like it to be)

Permit me a brief rant?

I’m tired of twenty-something hipster data scientists telling me that RSS is dead and has been supplanted by Twitter, Google+ and so on. Or as someone put it at the Google Operating System blog: “not everyone likes seeing 1% of their news as it scrolls by”. It seems that there are those who would like RSS to be dead and who believe that if they repeat the phrase often enough, it will come true. They may be right.

I speculate that the popularity of RSS among (enlightened) scientists and librarians is an indication that it’s a tool for people who like to read things properly, slowly and in-depth.

The Future

Google can do what they like of course, none of us paid for the product and there are alternatives available. I’m currently trying Feedly, who assure us that their product will continue to work after July 1. My fear is that there are those who equate RSS with Google Reader and who see the demise of the latter as further evidence of the death of the former. And again, they may be right as a self-fulfilling prophecy takes hold.

However the Web evolves, I just hope there will always be tools and protocols to provide information for people with attention spans longer than a gnat and a requirement for serious research.

Imagine that you have data consisting of observations for several variables – let’s say A, B, C – where each observation is from one of two groups – call them X and Y:

df1 <- data.frame(A = rnorm(50), B = rnorm(50), 
                  C = rnorm(50), group = rep(LETTERS[24:25], 25))
head(df1)
#           A          B           C group
# 1 0.2748922 -0.4805635 -1.80242191     X
# 2 0.0060852 -1.2972077  0.64262069     Y
# 3 0.1994655 -0.4628783  0.07670911     X
# 4 0.5416900  0.3853958  0.50193895     Y
# 5 0.3118773  0.9488503 -0.55855749     X
# 6 2.0924626  0.3027878 -0.03000122     Y

If you were interested in the distribution of variable A by group, you might generate a boxplot like so:

png("A.png", width = 800, height = 600) print(ggplot(df1) + geom_boxplot(aes(group, A, fill = group)) + theme_bw()) dev.off()

Boxplot of A by group

Here, the arguments to aes() are expressions (group, A) which ggplot interprets as column names from the data frame.

What if you wanted to generate plots for each of variable A, B and C using a loop? You might start like this:

for(i in names(df1)[1:3])
# oh wait, these are characters not expressions
# [1] "A" "B" "C"

You see the problem. How do we pass the column names which are characters, not expressions, to aes()?

The answer: use aes_string() instead.

Description:
     Aesthetic mappings describe how variables in the data are mapped
     to visual properties (aesthetics) of geoms.  Compared to aes this
     function operates on strings rather than expressions.

And so:

for(i in names(df1)[1:3]) {
  png(paste(i, "png", sep = "."), width = 800, height = 600)
  df2 <- df1[, c(i, "group")]
  print(ggplot(df2) + geom_boxplot(aes_string(x = "group", y = i, fill = "group")) + theme_bw())
  dev.off()
}

It’s a little ugly as it stands (better to write a function using one of the apply family). However, the key point is: you can pass data frame column names as expressions to aes() or as characters to aes_string().

With thanks to Hadley’s contribution to this mailing list thread.

Let’s create a data frame which contains five variables, vars, named A – E, each of which appears twice, along with some measurements:

df.orig <- data.frame(vars = rep(LETTERS[1:5], 2), obs1 = c(1:10), obs2 = c(11:20))
df.orig
#    vars obs1 obs2
# 1     A    1   11
# 2     B    2   12
# 3     C    3   13
# 4     D    4   14
# 5     E    5   15
# 6     A    6   16
# 7     B    7   17
# 8     C    8   18
# 9     D    9   19
# 10    E   10   20

Now, let’s say we want only the rows that contain the maximum values of obs1 for A – E. In bioinformatics, for example, we might be interested in selecting the microarray probeset with the highest sample variance from multiple probesets per gene. The answer is obvious in this trivial example (6 – 10), but one procedure looks like this:

# use aggregate to create new data frame with the maxima
df.agg <- aggregate(obs1 ~ vars, df.orig, max)
# then simply merge with the original
df.max <- merge(df.agg, df.orig)
df.max
#   vars obs1 obs2
# 1    A    6   16
# 2    B    7   17
# 3    C    8   18
# 4    D    9   19
# 5    E   10   20

This also works using min() and, I guess, using any function that returns a single value per variable mapping to a value in the original data frame.

With thanks to this mailing list thread.

Inputs to CNAmet are three m x n matrices, where m is the number of genes and n the number samples

What we’re looking at here is the hot, but poorly-defined topic of data integration, in which biological measurements from two or more different platforms are somehow combined in a way that provides more information than each platform separately. Read any paper on this topic, download the software and you’ll find example datasets containing two or more matched matrices, with rows where measurements have been summarized to a “gene”. What you won’t find, typically, is a detailed explanation of the summarization procedure that you could implement yourself.

To their credit the authors of CNAmet are quite clear that the procedure used to generate these matrices is not their problem:

Since the three microarray platforms contain non-overlapping probes, the m dimension of the input matrices must match. This is because the problem of mapping measurements (probe to probe mapping) between different array types is not dealt with by CNAmet.

Two problems.

First, let’s face it, the very concept of an object called a “gene” is flawed; what we have in reality are fuzzy locations of transcriptional activity.

Second, some measurements summarize more readily than others. Exon expression arrays, for example, are frequently summarized to “gene level” by taking the median measurement of probesets in a transcript cluster. For copy number arrays, we might typically segment the measurements over each chromosome, then assign a number to a “gene” by determining overlap between gene and segment. However, something like a methylation array is more difficult; probesets map to different transcript-associated features (islands, shores, shelves) – which do we use?

Our group recently looked at several publications which tried to integrate measurements of methylation and gene expression. We found at least half a dozen ways of generating the “genes x samples” matrices, from selecting one probe per gene using particular criteria (e.g. highest variance) to complex clustering procedures based on chromosome coordinates. In one horror show of a study, the authors decided that it was fine to combine methylation data from their study with completely-unrelated publicly-available expression data. Why the reviewers and editors agreed is anyone’s guess.

My second law of bioinformatics, then:

On no account must the data pre-processing steps required to summarize multi-platform measurements to gene level be revealed

Seriously, if you have a great idea about the best way to combine, for example, measurements from the Affymetrix Human Exon 1.0 ST and the Illumina Infinium HumanMethylation450 beadchip – go for it in the comments.

First, we’ve enjoyed a speaking tour by Sir Tim Berners-Lee, during which he discussed the benefits of open data several times. I was able to attend two events in Sydney in person and a third, linux.conf.au, by video stream. The events were the work of many people but in particular, Pia Waugh. Go follow her on Twitter, now.

Next – I wish I had been able to get to this one – the Open Research Conference on February 6-7, University of Auckland. I’m enjoying the high-quality live stream right now. Flying the flag for Sydney are Mat and Alex.

Not strictly under the “open” umbrella but worth a mention anyway: software carpentry is in town, February 7-8, just up the road from me at Macquarie University. Looking forward to hearing some reports from that.

I was not surprised to learn that the rate of public deposition of data is low, nor that most deposited data ignores standards and much of it is low quality. What did catch my eye though, was a retraction notice for one of the articles from the study, in which the authors explain the reason for retraction.

Two phrases in particular stand out:

we discovered an error in the data fed into the software

This decision was based on the instructions from the software during the initial data feed process

The language used strongly suggests a process whereby data was blindly “fed” into software, with little or no understanding of either how the software worked or the statistical methods employed. To quote Bill in our Twitter discussion:

@neilfws "decision was based on the instructions from the software" What the…? It's #overlyhonestmethods come to life!—
Bill Hooker (@sennoma) January 30, 2013

If you are in this situation, seek help. Talk to a friendly local statistician. Or if there isn’t one, do your research on the Web before publishing. At the very least, try to ensure that what you’re doing corresponds broadly with what most other people in the field would consider “best practice” – even if figuring out what that is, from the literature, is not always easy.

In 1996, you may recall, the Web was just a few years old. Amusingly (sadly?), it seems that Brenner predicted many of the topics in science publishing that we’re still discussing in 2013. It’s just that he thought they would be implemented in no time at all.

For example, open refereeing:

It is incidents such as this that have led me to question whether the anonymity of referees needs to be guarded so closely

Self-publishing/archiving and post-publication peer review:

The electronic pre-print with open discussion (not refereeing) will soon become commonplace; in fact, labs could go into the publication business by themselves

Demise of the journal impact factor, publishing economics and altmetrics:

We will need something to substitute for the present ratings given to papers appearing in ‘superior, peer-reviewed publications’ (and commercial publishers will find ways of making people pay for this)

Perhaps we should have a readership index; it should not be beyond the wit of man to devise a way of recording whenever a paper is read, hard-copied or cited

As Ethan said:

@neilfws what's taking so long?!

—
Ethan Perlstein (@eperlste) January 10, 2013