Organise your bioinformatics projects using Subversion and Trac: part 2

So – you’ve installed Subversion and Trac on your server, everything works and you’re ready to start managing your projects. In this post, I’d like to focus less on the technical details of using SVN/Trac and more on how they can be useful to bioinformaticians. Trac in particular is highly configurable and I think there are some exciting ways to use it. These include managing your data, sharing it with collaborators and even using it as an open notebook system.

1. Preamble

I’ll assume that you’ve read part 1. As before, some of the material is Ubuntu/Fiesty-specific. I’m also assuming that your SVN repository is in /home/svn, Trac is in /var/lib/trac and you access each of them through a browser at “https://localhost/svn” and “https://localhost/projects”, respectively.

2. Using Subversion

There are numerous guides to using Subversion. The best starting point is the official book “Version Control with Subversion” which is freely-available and online. Don’t be scared by its size! If you’re coming from CVS, you’ll find that SVN commands are frequently as simple as substituting “svn” for “cvs”. You can also run “svn help” or “svn help <command>” on the command line.

To illustrate, I’m going to use a toy bioinformatics project named “iep”. The aim of the project is to investigate the distribution of isoelectric points in various datasets of randomly-generated protein sequences. Currently, it lives in my home directory with the following organisation:

/home/neil/iep           # root
/home/neil/iep/doc       # documentation
/home/neil/iep/fasta     # fasta files
/home/neil/iep/iep       # output from EMBOSS iep
/home/neil/iep/pepstats  # output from EMBOSS pepstats
/home/neil/iep/perl      # perl scripts
/home/neil/iep/png       # PNG files from gnuplot

If you’re impatient, you can have a look at the Trac site for this project.
The first thing to bear in mind is that for each project SVN repository, there’s a corresponding Trac directory. You might be tempted to have a single SVN repository named “projects” into which you import everything under ~/projects and a corresponding single Trac directory. However, this will mean that any changes you make in ~/projects will result in an updated revision number across all of your projects – probably not what you want. A good rule is: one project, one SVN repository, one Trac environment.

Assuming that you set up SVN to use SSL as described in part 1, you’d load the iep project into SVN like so:

sudo svnadmin create /home/svn/iep
sudo chown -R www-data.www-data /home/svn/iep
cd ~/projects
svn import iep https://localhost/svn/iep

Obviously, substitute your full server name for localhost if you’re not working on the server itself. The first time that you import a project, Apache will ask you to authenticate yourself and accept the SSL certificate. If all goes well, you can now “ls /home/svn/iep”. Again, if you’re coming from CVS, you might expect to find versions of your files in there. You won’t under Ubuntu/Feisty- SVN exerts its magic via BerkeleyDB databases. Don’t worry about it – the details are in the SVN documentation.

Note here that SVN is quite happy at dealing with directories – that’s one improvement over CVS. It will also happily detect and import binary files (no need for the CVS -k switch). You can organise the SVN hierarchy any way that you like. The recommended way (/trunk, /branches, /tags – don’t worry if that makes no sense) is more suited to teams of people working on code; use it if you like or just choose whatever suits your projects.

Your project isn’t under version control yet – you have to checkout a working copy. If you’re doing this on the machine that hosts the SVN repository then go back to your projects directory, move the original projects files somewhere else and checkout. If all goes well, you can delete the originals:

cd ~/projects
mv iep /tmp
svn co https://localhost/svn/iep
rm -rf /tmp/iep

On another machine, you’d change to the location where you want to keep the iep project and substitute the full server name for “localhost”.

Well, congratulations. The iep project is now under SVN version control. You can now work on your files throughout the day. Use “svn add” to add newly-created files to the repository, “svn remove” to delete and at the end of the day, just remember to “svn commit” and your repository is updated. Once you’ve checked out a copy to another machine, use “svn update” at the start of the day to get the latest versions of all the files.

OK – how do we use Trac to organise the iep project?

3. Using Trac

First, create a Trac environment that corresponds to your project in SVN:

sudo mkdir /var/lib/trac/iep
sudo trac-admin /var/lib/trac/iep initenv
sudo trac-admin /var/lib/trac/iep permission add yourname TRAC_ADMIN
sudo chown -R www-data:www-data /var/lib/trac/iep

If you need to this for every project directory in ~/projects, it’s going to get tedious. Luckily, Ariejan de Vroom has written a small shell script to automate the procedure. Edit it to your taste, make it executable, supply it with the name of your project directory and it will setup the appropriate SVN and Trac environments:

sudo ./create_dev_env.sh myproject

Navigate to “https://localhost/projects” and you should see a page that says “Available projects” and a link to iep. Clicking the link takes us to the Trac page for the iep project. At this point, you can login with whatever name/password you created when you configured Apache. Once logged in, an “admin” tab will appear, where you’ll find many options to configure the Trac project pages. Options can also be set using trac-admin and by editing either /etc/trac/trac.ini (global) or /var/lib/trac/iep/conf/trac.ini (iep project only). The first thing to notice is that the front page is a wiki page. By default, the user “anonymous” has full access to the site. Again, this can be altered using trac-admin or the website admin tool. You might edit the front page to say “Welcome to the iep project” and provide some brief informative notes. The “Browse Source” tab takes you to a view of your SVN repository. From here you can view your files and revisions. If you have enscript installed, Trac provides syntax highlighting for certain file types. Other files (such as images) may be viewed directly. Click the thumbnails to see a screenshot of Trac pages for the iep project. From the top: default wiki front page; SVN browser view; Perl code with syntax highlighting; image generated in gnuplot.

The other navigation tabs – timeline, roadmap and tickets – are aimed more at software developers and are probably less important for most bioinformatics researchers. However, you may well find uses for them in fixing bugs, planning project goals or allowing collaborators to comment on your work.

That’s the basics of Trac. You’re now free to edit wiki pages and play around with permissions and configuration. There’s one more aspect to consider: plugins.

4. Trac Plugins

There are numerous plugins for Trac of varying quality, stability and age. Installing them is relatively easy; locate the appropriate repository URL, then run easy_install (in the Ubuntu python-setuptools package). For instance, to install the popular tags and blog plugins:

sudo easy_install https://trachacks.coderanger.net/svn/tagsplugin/trunk
sudo easy_install https://trachacks.coderanger.net/svn/tracblogplugin/0.10

Plugins are enabled and configured by editing either the global or project-specific trac.ini file – refer to the plugin documentation for the details. Your Trac pages might start spewing strange errors involving the location of something named PYTHON_EGG_CACHE. This environment variable is a temporary directory to which Apache must have write access. You can set it in /etc/apache2/sites-enabled/001-ssl, between the <location> tags:

SetEnv PYTHON_EGG_CACHE /tmp

You might now use the blog plugin as a project blog, where you record brief notes on project progress for yourself or other people. Note the useful RSS feed.

5. Ways to use Trac

I hope you can see where this could lead. The flexibility in Trac allows for all kinds of uses. For instance:

• A completely private website for your project files, using the wiki and/or blog plugin as an electronic notebook
• A lab intranet for a research team to collaborate
• A semi-open site with restricted permissions so as the world can see what you’re working on, but not all of your data
• Let’s be radical – a completely open research site where the world can view your data, edit the wiki, leave comments and discuss your results

For that matter, why restrict ourselves to bioinformatics? All research generates data and files – Trac + SVN could easily be the basis for open notebook science of any kind.

I hope you found these posts useful – let me know if you spot any errors.