Programming for Scientists

I used to co-write a blog called 'Programming for Scientists'.  All the posts from that blog can now be found here, under the tag Programming for Scientists.  Go take a look!

My co-author on that blog was my good friend Ben, who's a professional programmer, so between the two of us (programmer-interested-in-science and scientist-interested-in-programming), there should be some articles there that have stood the test of time :-)

The 'About' for this blog was this:

Just another blog about programming?

Well, not so much.  This is a blog about building better (scientific) software.  We’re making two important distinctions here.  The first is that developing a piece of software is about much more than just writing the code, important though that is.  The second is that for many people, scientists for example, building software is something they do in order to achieve their primary goal.  What we mean by this is that scientists’ primary goal is to do great science.  But to do this, they sometimes need to write some software.  So while they need as many software development skills as possible, they really need the ones that will allow them to build the tools they need and, critically, do a good job as quickly as possible so they can produce more good science.  We suspect there are lots of people for whom this is true, not just scientists.  

The aim of this blog is to explore ways of building better software, but to do so bearing in mind that the ultimate goal is to produce as much great science as possible.  This doesn’t mean coding as quickly as possible, which produces buggy, horrible software; it means drawing from the huge body of software development knowledge for minimising the amount of time it takes to build really reliable software that you can then use confidently to do your science.  We won’t be telling you the “best” way to do anything (there’s usually more than one) and we won’t be handing down stone tablets saying that you must do something without good reason.  What we will do is present a range of techniques, thoughts and considerations and try to explain why we think they’re helpful.  You should use somewhere between none and all of these, depending on what you find most useful.  

We’re not here to tell you how you should go about developing your software, we’d just like to suggest a few things that other people, ourselves included, have found useful. 

We hope you enjoy the blog and find it useful!  

How to...optimise Matlab

[caption id="" align="alignleft" width="300" caption="image from wikipedia"][/caption]

Matlab is a language that's used a lot in science and for good reason.  It's quick to code in, plus there are loads of built-in functions and packages available for many of the tasks that Programmer-Scientists might find themselves involved in.  The main issue that a language like Matlab faces is that it's slower than languages like C++ and FORTRAN, which can be inconvenient when dealing with resource-intensive tasks like data analysis, statistical modelling or numerical simulation.  But fear not!  We proudly present our guide to how to optimise your Matlab code.

Scripting for science papers

[caption id="" align="alignleft" width="300" caption="image by marco annunziata"][/caption]

Scientist-Programmers write a lot of scripts.  It's part-and-parcel of "trying stuff out", it's a quick way to get some number crunching done on those data, and it's very useful for generating the figures and tables that you need for that paper you're writing.  In this article, I give a quick once-over of some of the things I've learned over the years about using scripts as a scientific tool.

"Should I switch to Python?"

[caption id="" align="alignleft" width="140" caption="Logo owned by the Python Software Foundation"][/caption]

Rich has recently been considering switching to the Python programming language.  Currently, Matlab is the language of choice in his department for rapid development and prototyping of code.  It's very good at this, but Mathworks (the company who produces Matlab) have been tinkering with the licencing terms, leading to hassles where none should exist.  This is very frustrating and leads to the thought that it might be nice to use a free language where this will no longer be an issue.

But of course things are not quite that straightforward.  Matlab is used for good reason - it's very good at what it does.  So is it worth the effort to stop using Matlab and instead learn to use Python?  In this article we discuss some of the things that'll need to be considered.

Writing code for a big scientific collaboration

[caption id="" align="alignleft" width="300" caption="Photo by ralphbijker"][/caption]

One of the striking thing about scientific software is the range of different contexts in which it's needed.  Scientists need quick-and-dirty scripts to process their data and plot their results; they need prototypes so that they can experiment with new statistical techniques; and they sometimes need to build new software tools that they'll use again and again in their research.  While a lot of this will be for their own personal research, sometimes the Scientist-Programmer finds themselves developing software as part of a large scientific collaboration.  This has some particular requirements.

The 'programming for scientists' link montage

Image via Wikipedia

Continuing our theme of gathering together useful stuff for the scientists-programmer we present a list of interesting, useful, informative and possibly fun links. Enjoy!

The 'Programming for Scientists' training montage

[caption id="" align="alignleft" width="300" caption="photo by Iwan Gabovitch"][/caption]

As any attentive movie-goer knows, self-improvement can be a great way for the hero to overcome the obstacles in their path and generally do great stuff.  As it is with Rocky Balboa, so it is with programmers, and particularly with Scientist-Programmers.  Finding the time to develop your skills means that everything you do subsequently, you'll do a little better, and over time you become much better at what you do.  We think every Scientist-Programmer should be a little obsessive about self-improvement and here are a few articles that show you how.

The 'Programming for Scientists' software project primer

[caption id="" align="alignleft" width="300" caption="photo by vali..."][/caption]

The purpose of this blog is to help scientists program more effectively and efficiently, so they can get more science done.  For those of you who'd like to be systematic in your learning, we present the prog4sci software project primer! 

How to explain programming to your Mum

Image via Wikipedia

This article will hopefully be helpful for those times when you have to explain to your mum/dad/girlfriend/person on the bus what it is you do all day. I can't help you explain the intricacies of your particular scientific field but I can help explain how all the computers get involved.

Building scientific tools that are actually useful

[caption id="" align="alignleft" width="300" caption="Photo by flattop341"][/caption]

Lots of scientists write bits of software to get things done.  Sometimes they offer to give someone else (a collaborator, student, postdoc etc) a copy of some of their code, to help that person out.  Sometimes a given piece of code is useful enough that it gets handed out multiple times, and so starts to look a lot like a publicly-available scientific software tool.

That's great, but think about what could have just happened, back in the first sentence of this post.  A scientist wrote a bit of software to get something done.  Not "a scientist developed a robust, well-tested software tool".  Maybe it was, but maybe it was a knocked-together, prototype-y little chunk of code that was only meant to be used once.  And now suddenly that prototype is in widespread use.  We hope this fills you with horror!

The problem here is that this is a way that a prototype can end up being distributed as if it was a finished product.  But it doesn't have to be this way; in this post, we're going to discuss the considerations of producing good scientific software tools.

The basics of ... Python

Image via Wikipedia

Python (named after the TV series Monty Python's Flying Circus, not the snake) is a high level programming language that aims to have a clear syntax and only one correct way of doing something. This post will look at how it can be used for scientific computing.

10,000 hours and the Scientist-Programmer

[caption id="" align="alignleft" width="240" caption="Photo by ** rosa **"][/caption]

The concept of 10,000 hours effort as a benchmark to become an expert has recently become pretty well known.  The idea is this:  experts are made and not born, and the way that they're made is to accrue 10,000 hours of hard work at the subject in question.  This sounds to us like somewhat of a rule-of-thumb, but it's interesting in how many areas of human endeavour it seems to crop up.  Sportsmen, musicians, certainly scientists and programmers (and Scientist-Programmers) and many others seem to require of order 10,000 hours experience to reach the top of their game.

In this article, we're going to consider how the idea of 10,000 hours relates to you, the Scientist-Programmer.

What is an expert?
The circular definition is "someone who has at least 10,000 hours experience in a given field".  But obviously, that's not very helpful :-)  We suspect you can define "expert" in all sorts of different ways, but for the Scientist-Programmer it is someone who who routinely produces high quality code and does so efficiently.  We're not aware of any studies of programming productivity in science, we suspect that expertise helps a lot and for the Scientist-Programmer, this translates directly into generating more and better science.

Doing it for yourself - deciding whether to use someone else's code

[caption id="" align="alignleft" width="300" caption="Photo by jurvetson"][/caption]

Using someone else's code can be great.  Or it can be horrible.  If it does exactly what you need it to, doing so with no bugs and no ambiguity, then this is awesome.  If you need to make changes to buggy, uncommented code written by someone who thinks the GOTO statement "really isn't that bad", then we don't envy you one bit.

So, here are some of the things you need to consider when deciding whether or not to use someone else's code.

Stay on target! Ways to help yourself work

Image via Wikipedia

In a previous post we talked about how to keep your brain in tip top condition and staying in the 'zone'. We recommended some simple techniques like removing distracting email, IM or twitter traffic but sometimes the problem isn't staying in the zone, it's getting into the zone in the first place. In this post we recommend some tools and techniques to help you get going. Everybody has different techniques so we'd also like to hear from you about how you beat procrastination and get working.

The 4 levels of making code work

[caption id="" align="alignleft" width="300" caption="Photo by dbking"][/caption]

You want to make your code work.  Anything else would be dumb, right?  Right.  But as we've discussed before, deciding whether your code is good enough to count as "working" isn't always straightforward.  On way of looking at this is to categorise four stages of relative completion of a software project.  Any one of these four can count as "good enough" if that's what your project requires and in many ways they represent a sequence of levels of technological development.

The basics of SWIG

[caption id="attachment_535" align="alignleft" width="240" caption="Image from sketch22"][/caption]

Simplified Wrapper and Interface Generator, or SWIG for short, is a tool that provides a way to interface C/C++ with a variety of high level programming languages (notably Python, R but not Matlab - which has it's own way of linking to C). It generates wrapper methods that allow the two languages to talk to each other. If you have legacy C code that you want to use in your shiny new R program or if some of your code needs to be re-written in C++ for a speed boost then SWIG can help you out.

Fine-tuning your programming brain

[caption id="" align="alignleft" width="240" caption="Photo by Gaetan Lee"][/caption]

Even the most genius programmer has bad days.  But have you ever stopped to consider how much of a better programmer you'd be if you had as many good days as possible?  We think it's not so much about how great you are on your best day; it's about bringing your A-game to the project each and every day.

Feature-creep in scientific code

[caption id="" align="alignleft" width="300" caption="Photo by blmurch"][/caption]

(with thanks to Joe Z for the comment that inspired this post )

Feature-creep can be a real killer for your carefully crafted code.  Scientific code can be especially prone to it, simply because planning is hard when you're working on the edge of human knowledge.  This article explores the topic in a bit more detail.

Doing several things at once: types of parallel computing

Image via Wikipedia

For decades computers have got faster by increasing the power of a single processing unit but recently chip makers are hitting limits, mainly cost and heat. The results: instead of one processor getting faster, more are being added. At the same time the advent of cheap commodity computers and fast networking has given rise to various clusters, grids and clouds. This article gives a quick overview of what is available when you need lots and lots of power but don't have lots and lots of money.

Testing and the scientist-programmer

[caption id="" align="alignleft" width="300" caption="Photo by dullhunk"][/caption]

You do test your code, right?

Well?  We certainly hope so.

Testing can be a pain.  But it should also be a habit.  And we think it's pretty uncontroversial to say that testing your code is a Good Thing.  The Scientist-Programmer often finds a tension between allocating effort to testing their code and effort to getting on with the science, but not only would skimping on testing be a mistake, but (if you write them correctly) the tests themselves can have scientific value.