Has your knowledge stopped updating?

Some years ago I read an article – I forget where – describing how our general knowledge often becomes frozen in time. Asked to name the tallest building in the world you confidently proclaim “the Sears Tower!”, because for most of your childhood that was the case – never mind that the record was surpassed long ago and it isn’t even called the Sears Tower anymore. From memory the example in the article was of a middle-aged speaker who constantly referred to a figure of 4 billion for the human population – again, because that’s what he learned in school and had never mentally updated.

Is this the case with programming too? Oh yes – as I learned today when performing the simplest of tasks: reading CSV files using R.

Editing metadata in trail camera images using R, magick and exiftool

I have a new hobby: camera traps, also known as trail cameras. Strapped to trees in my local bushland they sit in wait, firing automatically when triggered by a passing animal. Once in a while, something quite magical happens.

The camera model I chose is the Campark T85 which for me, had the right combination of features and price point. One useful feature is the ability to transfer images and video to a phone wirelessly (albeit through a rather clunky phone app). Unfortunately, images retrieved in this way have one major flaw: an almost-complete absence of metadata. There is no GPS in the camera of course, but the EXIF data does not include the date/time of the image, nor the camera make.

With a little research, I found a way to add this information to the images later using R and some additional software named exiftool. Here’s how I did it.

Using R to detect the pressure wave from the 2022 Hunga Tonga eruption in personal weather station data

It seems like an age ago, but in fact it was only mid-January 2022 when this happened:

The satellite imagery from the Hunga Tonga eruption is unreal. Direct your attention to the lower right. The eruption then shock wave is simply incredible. pic.twitter.com/OTLCgyEozQ

— Taylor Trogdon (@TTrogdon) January 15, 2022

Wow. Now, pause for a moment and try to recall the last time you read any news about Tonga since the event.
The eruption sent an atmospheric pressure wave, clearly visible in this imagery, around the world. Friends online reported that this was detected by their personal weather stations (PWS) which made me wonder: was the wave apparent in online weather station data and can it be visualized using R?

The answers are yes and yes again.

Using R/fitzRoy to ask: how many times a V/AFL team with the same lineup has played together?

If you sit in the intersection of “likes Australian Rules football / finds sport statistics interesting / is on Twitter”, you’ve probably come across Swamp. One of his recent tweets tells us that:

No V/@AFL premiership winning lineup have all played together in another V/@AFL match, there has always been at least one person missing

All MELB 2021 premiership players are still at the club in 2022

@melbournefc

— Swamp (@sirswampthing) March 16, 2022

You may go on to ask: has any team lineup from one of the almost 16 000 recorded games played together again in another game? And if so, how often?

The answer to that question is at once surprising, less surprising when you think about it, and quite easy to figure out using the ever-helpful fitzRoy package.

Enhancement of old colour photographs using Generative Adversarial Networks

It’s almost Christmas, I haven’t posted anything in a while and I see that WordPress has an Image Compare feature, so let’s have some colourful fun.

When I’m not at the computer writing R code, I can often be found at the computer processing photographs. Or at the computer browsing Twitter, which is how I came across Stuart Humphryes, a digital artist who enhances autochromes. Autochromes are early colour photographs, generated using a process patented by the Lumière brothers in 1903. You can find and download many examples of them online. Stuart uses a variety of software tools to clean, enhance and balance the colours, resulting in bright vivid images that often have a contemporary feel, whilst at the same time retaining the somewhat “dreamy” quality of the original.

Having read that one of his tools uses neural networks, I was keen to discover how easy it is to achieve something similar using freely-available software found online. The answer is “quite easy” – although achieving results as good as Stuart’s is somewhat more difficult. Here’s how I went about it.

Gene names, data corruption and Excel: a 2021 update

It’s an old favourite of this blog, isn’t it. We had Gene name errors and Excel: lessons not learned (2012). Followed by Data corruption using Excel: 12+ years and counting (2016). Perhaps most depressingly of all, the conclusion of the trilogy, When your tools are broken, just change the data (2019-20).

Well, I’m happy (?) to see the publication of the latest instalment, inspired in part by the title of my first post: Gene name errors: Lessons not learned, from Mark Ziemann’s group. Here’s the accompanying Twitter thread. Summary: it’s even worse than we thought.

Tagging this one with the R tag, because the group are publishing monthly RMarkdown reports here. Congratulations Nature Communications!

As a footnote: you don’t escape this kind of thing when you leave bioinformatics. I listened to a colleague in a data science meeting yesterday declare that “we won’t be putting anything into production that relies on data supplied to us as spreadsheets”.

How I resurrected my ancient PhD thesis using R/bookdown (and some other tools)

An ancient thesis

I’ve long admired the look of publications generated using the R bookdown package, and thought it would be fun and educational to publish one myself. The problem is that I am not writing a book and have no plans to do so any time soon.

Then I remembered that I’ve already written a book. There it is on the right. It’s called “Cloning, sequence analysis and studies on the expression of the nirS gene, encoding cytochrome cd₁ nitrite reductase, from Thiosphaera pantotropha“. Catchy title, hey. It’s from my former life, as a biochemistry graduate turned reluctant molecular microbiologist. I believe there are 3 printed copies in existence: mine, one for the lab and one deposited in the university library.

That’s simple enough then Neil, you say, you just grab your digital files, copy/paste into RMarkdown files, do a bit of editing and you’re set. Here’s the thing.

There are no digital files.

There were, once. A collection of documents: Word, Powerpoint and JPEGs. I think they lived on a 100 MB zip drive for a while. At some point they were burned onto a CD. And at some other point, that CD became corrupted. And that was that. Like many (most?) people, I’d barely looked at the thesis since depositing a copy in the library anyway. It didn’t seem to matter much.

And then I grew older, and started looking at some of the documents in our family, and realising that in the event of accident or disaster, they’d be lost forever. So I started working on ways to digitally archive some of them. At some point my thoughts turned to that thesis, which took 4 years of my life. I wondered whether the university library had digitised it and if so, whether it might be available online. So far as I can tell, the answer is no. That seemed a shame.

So here, briefly, is the story of how I used R/bookdown and some other tools to resurrect that thesis.

Florence Nightingale’s “rose charts” (and others) in ggplot2

It’s been a while. I hope you are all well. Shall we make some charts?

When your tools are broken, just change the data

Update August 7 2020
The gene symbol renaming is now official. Here’s the publication (not open access, should be), coverage at The Verge and more coverage at The Register. The latter with quotes from me.

It’s been 3 years since we last visited that old favourite recurring topic, data corruption by Excel. Specifically, the unwanted auto-conversion of identifiers that look like dates, e.g. SEPT1, to – well, dates.

Here’s a new twist – well, a two year-old twist in fact, as I don’t keep up to date with this field any longer:

TIL that SEPT genes were renamed in 2017 to SEPTIN genes by the HGNC https://t.co/2UadZUMLCS pic.twitter.com/jCo0Hcf6sf

— mdziemann (@mdziemann) October 8, 2019

Yes, in 2017 the HGNC decided that the solution to this long-standing issue is to rename the offending genes to prevent the auto-conversion. I’m yet to determine whether anything more came of the proposal.

It is I suppose a practical suggestion that will work. The newsletter states that:

Our initial consultation with the research community publishing on these genes had very mixed results

I bet it did. However, given that ongoing consultation with the research community about the inappropriate use of software has had essentially no results in 15+ years, perhaps it is the most effective solution to the problem.

Debuting in a VFL/AFL Grand Final is rare

When Marlion Pickett runs onto the M.C.G for Richmond in the AFL Grand Final this Saturday, he’ll be only the sixth player in 124 finals to debut on the big day.

The sole purpose of this blog post is to illustrate how incredibly easy it is to figure this out, thanks to the dplyr and fitzRoy packages.

library(dplyr)
library(fitzRoy)

afldata <- get_afltables_stats()

afldata %>% 
  select(Season, Round, Date, ID, First.name, Surname, Playing.for, 
         Home.team, Home.score, Away.team, Away.score) %>% 
  group_by(ID) %>% 
  arrange(Date) %>%
  # a player's first game 
  slice(1) %>% 
  ungroup() %>% 
  # grand finals only
  filter(Round == "GF") %>%
  # get the winning/losing margin 
  mutate(Margin = case_when(Playing.for == Home.team ~ Home.score - Away.score,
                            TRUE ~ Away.score - Home.score)) %>% 
  select(-Home.team, -Away.team, -Home.score, -Away.score)

Season	Round	Date	ID	First.name	Surname	Playing.for	Margin
1908	GF	1908-09-26	5573	Harry	Prout	Essendon	-9
1920	GF	1920-10-02	6677	Billy	James	Richmond	17
1923	GF	1923-10-20	6915	George	Rawle	Essendon	17
1926	GF	1926-10-09	3824	Francis	Vine	Melbourne	57
1952	GF	1952-09-27	9361	Keith	Batchelor	Collingwood	-46