21st Century Scientist

An algorithm for selecting research projects

2011-11-18T11:40:00.001Z

I spend a lot of time thinking about my research projects. I revise and tweak the ones I'm working on, but I especially spend time planning my possible future projects, working out which ones I should most prioritise.

This is important. Choosing which projects to work on will fundamentally affect your research career, and given that a project might last months or years, spending hours or days making good choices seems pretty sensible.

I once read a suggested approach that I've come to realise has a lot of merit. Annoyingly, I can't track down the original reference (I think it may have been by Daniel Lemire or Study Hacks; even if not, you should go and read those blogs as they're very good). But the idea is this:

Identify your two best/most valuable pieces of research.
Your new project should better at least one of these.

That's it.

It's a really simple approach choosing new projects, and it's kind of obvious that this should lead to good projects (provided your assessment of value is reasonable). But there's actually an interesting alternative way of describing it.

It's a hill-climbing algorithm for optimising the scientific value of your new projects over time.

Think about it. We can imagine that there is some abstract quantity, "value", associated with each of the research projects that we work on. Different people may disagree on the precise value of any given project (and even the definition), but overall we would like to be working on progressively more valuable projects. By using the above algorithm as a criterion for deciding which projects to work on, we are aiming to always increase the value of our two best pieces of research (by progressively replacing each one with something better).

Why not the best one or three projects? Well of course, it should still work if you change the number. Two is probably a good number, as having a couple of headline grabbing projects is useful for writing grants, giving talks and the like.

It occurs to me that it would be very easy to always use the same scale for assessing how good/bad a potential project will be. But by using the Top-Two algorithm, you're always pushing to do more and more valuable work. And this seems like a thoroughly good idea to me.

Some Great Advice on an Academic Career

2011-10-21T10:49:00.003+01:00

Twitter has just directed me to a very good article giving advice about academic careers. Well worth a read!

Steve Jobs

2011-10-06T14:35:00.003+01:00

Someone who achieved a world-changing amount of stuff during his life. I'd thoroughly recommend giving Steve Job's 2005 Stanford Commencement address a watch. Totally worth it.

Scaling your research productivity

2011-10-05T13:47:00.000+01:00

Productivity is important in research. Ultimately, you'll be judged on the importance, quality (and number) of papers that you publish. This got me thinking about whether there was a simple way of encapsulating this and I came up with the following:

Aim to do more and more important research per unit-time.

It's sort of obvious, but the key quantity is the rate at which you produce important, high-quality research. If you hardly ever publish anything, that's a bad thing. If you publish plenty, but all your papers are low-grade rubbish, that's a bad thing too. And even if you produce large numbers of high (technical) quality papers, but they're all studying unimportant problems, that's not great.

Cognitive Science of Rationality

2011-09-27T16:30:00.000+01:00

Here's an interesting post on the cognitive science of rationality. I've only just read it so need some time to digest it, but it seems to me anyone wanting to improve themselves as a scientist should be paying attention to how they think rationally. As with many other things, it's a skill that you can learn and improve upon.

A mulling layer

2011-09-26T16:15:00.002+01:00

After my previous post on my approach to research, I realised I might have missed out a layer in the process (or at least undervalued it).

The idea of a pool-of-memes approach is to collect in your mind a set of useful/relevant memes, then let them mull. However, what I actually do is a bit more structured than that. The pool actually has a second layer, containing ideas that have occurred to me as particularly promising. I'll tend to actually write down these ideas and I'll come back to them periodically and consciously work on improving them.

This has the advantage of letting me refine my most promising ideas, as well as sometimes combining them into 'research arcs' (single, coherent research projects that will lead to multiple, related outcomes). It also makes it easier to develop ideas that might have some merit but be undercooked, currently.

Having this intermediate layer is also quite useful in terms of time management. It allows me to focus a bit more time on the ideas I think are promising, but without committing the significant chunk of full-on work time that a "small bet" requires.

a 'Pool-of-Memes' approach to research

2011-08-23T11:30:00.005+01:00

I tend to spend a fair amount of time thinking about how best to go about research. My rationale is that simply working harder isn't scalable - you can't work more than 24 hours a day (and indeed only a lot less than that in a sustainable way). Therefore to be a better scientific researcher, I need to find ways to improve my approach to research.

Study Hack's excellent article on his research system got me thinking about how I'd describe my own system of research. I think the phrase "pool of memes" fits pretty well.

I try to fill my mind with as many interesting/relevant ideas and concepts as possible. This means both being well read in my own subjects, and also hunting out other subject areas that might add something. For example, for the last year or two I've been becoming increasingly interested in computer science. I've found my most productive phases correspond to learning a new set of relevant ideas.

To this pool, I also try to add clear ideas about what questions are important in various areas of research.

Then I just sort of let all these ideas mull. I might think about something in an idle moment at the gym, or I might head to a coffee shop with my log book and tinker with some thoughts.

What I get from this is a list of possible projects on which to work. This list tends to be pretty organic and it evolves over time. I rank the list in terms of how good/important I think they are. And then I try out the top ones.

What I've not had previously, but what I'm just starting to add, is a stage like Study Hack's "small bets". The idea here is to try out the possible good projects for a month or so, with the aim of producing some concrete evidence as to whether or not to take the any further. I'm not very comfortable on a personal level with the idea of discarding projects like this (I don't like the waste), but objectively it makes a great deal of sense, so I may just need to get over myself

:-)

As much as anything, this approach works well for me because I really enjoy learning new things, so giving myself the justification for doing that during work time is nice :-)

The Heritage Health Prize

2011-06-13T14:31:00.003+01:00

There's been an interesting trend in recent years: the comeback of public competitions to solve problems of various types. From the X-Prizes to the now-finished Netflix prize, competing for the kudos (and cash prizes) of solving a given problem is very much of the moment.

Of particular interest to me personally is the Heritage Health Prize. Run by the website Kaggle, on behalf of the Heritage Provider Network (a Californian healthcare provider), the aim of the competition is to predict the likely number of days of hospitalisation for a range of clients, based on their historical (anonymised) medical claims data. Being someone who researches statistical machine learning and is interested in applying ML methods to medical data sets, this is a pretty cool challenge.

There are some complications to using such a competition data set to research new machine learning methods. HPN and Kaggle are rightly very cautious about the data, and there are various ethical considerations about mining medical data in this way. My understanding is that they're willing to consider case-by-case whether research arising from these data can be published. We'll see how that pans out in practice.

I'm hopeful that I can do reasonably well in the HHP, but it's also an interesting area in which to research (provided publishing papers proves possible) and I'm finding that I'm getting some useful experience with new approaches and methods that I'd not used before.

And perhaps this is the strength of these competitions. The cash prize is nice, but there are also a lot of other things the competitors can get out of it.

Predictive power and explanatory power

2010-11-30T14:01:00.006Z

Recently, I've been thinking about the goals of science. Science isn't about discovering Truth. This is for the very simple, practical reason that Truth is very hard to positively identify - how would you establish that a prospective Truth wasn't just a very good approximation? Not very helpful...

There are all sorts of philosophical discussions to be had here, but we can sidestep these and take a more practical approach. Specifically, we can choose to care about Predictive Power and Explanatory Power.

Predictive Power is the ability of a given theory to allow us to make predictions about the natural world. We know that Newtonian gravity is an approximation (to General Relativity, at the very least), but it's very good at predicting where the planets in our solar system will be. This is really a flat-out practical consideration - if a theory can't make predictions, it's not very useful (and some would argue it's not even science).

Explanatory Power is the quality of a theory that gives us some deeper understanding of what's going on in a physical system. For example, knowing about atomic electron orbitals allows us to make sense of the periodic table and chemical interactions. It gives us ways to develop other theories.

So, what we're looking for from a scientific theory is the ability to make predictions and for some explanatory insight as to why something happens, so that we can use that insight to develop further theories.

This is also relevant for statistical modeling (and hence data-intensive science), because we can build our models to address either or both of these. Predictive algorithms such as neural networks can perform very well, but the structure of the model is often hard to interpret in any kind of explanatory way. Conversely, a linear regression model might tell us a lot about which variables are important, but may not make good predictions. Ideally, it would be nice to build models that are useful for both prediction and explanation.

Data-intensive Science

2010-10-06T16:00:00.004+01:00

I've recently been noticing a number of articles and blog posts on "data science" (or "data-intensive science") and a speculated 4th paradigm for doing science (the first three being experimentation, theory and more recently simulation).

While there is debate about precise terminologies, there is definitely a new approach to science that's being used in various fields. It looks something like this:

1 - design a big, powerful experiment that will make measurements over a whole region of scientific parameter space
2 - run the experiment, reduce/process the data and make it available to people
3 - mine the data for interesting science
4 - (optionally) follow up these discoveries with new experiments

This is already the norm in big astrophysics and particle physics projects and works very well. There's also a number of medical/biological projects going in the same direction.

So why is this happening? Short answer: because it's a good way of making new discoveries.

Longer answer: it's the result of two key drivers. One is that (in the area of concern), someone has invented one or more measurement technique that's capable of generating huge volumes of useful data. The second is that we have computers and machine learning/statistics algorithms that are capable of extracting useful information from such large data sets.

This approach has several advantages.

1 - it can be systematic (astrophysicists have discovered whole new classes of celestial object simply by surveying the whole sky to a given sensitivity)
2 - it can be statistically very powerful (sheer volume of data can give small error bars and good signal-to-noise)

3 - there's a wisdom-of-the-crowds aspect to having many scientists working on the same data set (and if the data set is rich enough, it's worth having many people working on it)

Some interesting links on this sort of thing:

Data science Venn diagram

A taxonomy of data science

Skills of data geeks

Rise of the data scientist

What is data science?

Data-intensive science

Links (philosophy of science)

2010-08-06T15:11:00.002+01:00

I’ve recently read a few interesting posts over on the Science 2.0 blog site concerning probability theory. Well worth a read!

Can science be justified?

You, a Bayesian

Probability and induction: The very foundations of science

Science and Sherlock

2010-07-27T15:50:00.003+01:00

t’s not often that I write a post based on a TV show. Bear with me on this.

The BBC have just started showing ‘Sherlock’, a contemporary (and very good, so far) update of the Sherlock Holmes stories of Arthur Conan Doyle. And it got me thinking about the inspirations that originally made me want to be a scientist.

Like many people who end up being scientists, I was inspired by stories of the great scientists and their discoveries. I admired Einstein and Feynmann, I used to have undergraduate lectures near where Crick and Watson figured out the structure of DNA - the list goes on.

But my primary inspiration in how to think like a scientist wasn’t a scientist. And he wasn’t even a real person. Holmes’ deductive reasoning has always struck a chord with me and it’s the best written description I know of concerning how to think like a scientist. The focus and precision of it, the attention to detail and the fact Holmes treats it as a craft to be honed.

There are many very good science texts that the aspiring scientist should read. I’d suggest that it’s also worth spending some time reading the Sherlock Holmes stories, for the simple reason that in order to be a scientist you need to think like a scientist!

The memetics of science

2010-06-18T15:13:00.002+01:00

I've become fascinated recently by the concept of memes and the dynamics of how they evolve (memetics). I'm particularly curious about what memetics might have to tell us about how science works.

Full disclosure here is that I'm very much an interested amateur at this point - there are people who have spent many years thinking deeply about memes and memetics, and I'm not one of them. But the basic concept is kind of beautiful and not very hard to grasp, and it does give us some insights into how science works.

We should start with some definitions. A meme is a unit of information (such as an idea or concept) that's copied from person to person. The idea of memetics is that memes are subject to an evolutionary process because they are copied, a range of variants exist, and they are subject to selection pressures (some memes spread more effectively than others). So what we then have is a way of thinking about the dynamics of how (scientific) ideas evolve and develop.

It strikes me that science in particular is a memetic process where we have one additional concept: we subject our memes to the selection pressure that they must be confirmed empirically. This is an important difference. Memetics per se does not require any given meme to be true; it just has to be good at propagating. This explains why rumours that are false but appealing can spread so readily. By adding the additional constraint of empirical confirmation, we are adapting memetics in order to learn about the universe.

Thinking in this way, we can define a list of general scientific processes in which we can be involved.

Validating an existing meme or memeplex (empirically or via theoretical proof or computational analysis)
Improving an existing meme or memeplex (someone had a good idea, then you're able to refine it)
Making a new memeplex (a new combination of memes)
Creating a new meme

There may well also be others that this interested amateur hasn't yet thought of :-)

One additional thought is that this also gives us some insight as to how important is it to fill your brain full of relevant memes, so that you've got more to work with.

Rational inference in science

2010-06-07T15:57:00.002+01:00

One of the side-effects of working in science is that you end up thinking a lot about how we reason and learn. What rules do we choose in order to do this? Do we have a range of options? This is pretty important, as it underpins everything we do as scientists (or thinkers of any kind, come to that).

The term 'Rational Inference' is a good way to describe how we do this. It covers both logical deduction (for when we have definite facts: if A and B are true, then C must also be true) and induction (for when there is uncertainty: D and E are true probably, which implies that F is also likely to be true). Rational inference is how we reason in science, whether using experimental results, observations of the universe or even (the most obvious example) mathematics, and how we learn more about laws that describe how the world works.

The fascination for me is that we have a mathematical theory that tells us how rational inference must work - probability theory. If you're comfortable with a Bayesian perspective on the subject (and are therefore happy to allow probabilities to represent states of knowledge), then probability theory contains logical deduction as a special case when all probabilities are either zero (false) or one (true). It also extends this to allow for different degrees of plausibility (0 < probability < 1), giving us the capability to represent any state of knowledge from false, through 'maybe', to true.

This also gives us an answer to the question "am I using the right set of rules to make my inferences?". The physicist Richard Cox provided a proof (using only very general starting assumptions) that any consistent mathematical rules for handling degrees of plausibility must be those of probability theory (or equivalent to them). Meaning the answer to our question is "yes!" - and we realise we are using the only set of mathematical rules for rational inference that make any sense.

So we find ourselves with a uniquely correct approach to reasoning in science (and anywhere else), along with a set of mathematical rules to use. There are philosophical considerations along the way (such as whether to adopt a Bayesian or Frequentist viewpoint) but if you're comfortable with the idea that your degrees of plausibility can include states of knowledge, you've got a unique set of mathematical rules that tell you how to make rational inferences about the world.

How to get a research fellowship

2010-04-26T16:36:00.003+01:00

I've recently been through the process of applying for (and getting, happily) a research fellowship. While I've applied for fellowships before, this time I was a lot more focused and I think it really helped. To that end, I thought I would share some of my experiences and insights on the process.

Disclaimer: I'm going to write from the perspective of statistics/machine learning research, because that's what I know.

Planning
If you want to apply for a fellowship, you should be planning it three years in advance of when you'd like to start it. This may see like a lot, but imagine you decide your CV needs a particular type of extra research project to round it out for your application. Starting from scratch, you might need a year to produce enough original research for a good paper in the area. Getting it published could easily take 6 months, from submission to the actual journal issue coming out (or the paper appearing online in its journal-accepted form). And fellowship application deadlines can easily be 9 months before the eventual start date. 12 months + 6 months + 9 months = 2 years 3 months. So give yourself three years, to be on the safe side.

Preparation
This one is easy to screw up. So don't. Start writing your fellowship application three months before the deadline. Sure, that sounds like loads, but it isn't. Once you've planned and written the first draft, you'll need to allow time for proof-readers to get back to you, to incorporate their advice, to get the financial details sorted out, to get signatures from people like heads of department who are very busy and to give your core collaborators time to write reference letters for you (they're doing you a favour so the least you can do is give them a month's notice).

Also consider this: is there any reason not to start three months before the deadline? Really? If you get everything finished a fortnight ahead of time, you'll have had plenty of time to craft and polish your application. This isn't as important as the content, but an application that looks rushed and ill-considered does you no favours whatsoever.

Get advice from people
I guarantee there are people at your university that can give you useful advice about applying for a fellowship.

Research Support Services
People who have held fellowships
People who have sat on grants panels
People who are currently on the grants panel of interest
Senior academics who have a lot of experience writing grants

After plenty of planning, the single most important thing I did was to seek out and listen to as much advice as I possibly could. It's incredible the things you won't have thought about.

And remember that even criticism is useful to you. If a senior academic is unimpressed with part of your application, ask yourself why. It doesn't matter whether you agree with them, the point is that if one person can be unimpressed then so can another. And that other might be on the panel. So be happy you've gotten some advanced warning and see what you can do to fix the problem.

Be so good they can't ignore you...
Steve Martin gave the above answer when he was asked for advice on how to succeed and it also applies in academia. For a fellowship it means take your professional development very seriously, learn new skills, work with very good people, make your papers as good as they can possibly be, give really good talks (by working hard on the content and learning how to deliver it well). "Be so good..." doesn't mean you need to be born brilliant; it means you have to do your best research and try to improve a little bit each and every day.

A large part in relation to your application is demonstrating your calibre as a researcher, which means writing top-rate papers, giving good conference talks and generally building up a strong CV. But remember that we care about all those things because they can be indicators of a good researcher; they're not important in of themselves. They should be a consequence of your becoming better and better researcher.

Collaborations and data sets
If you're applying for a fellowship to do statistics/machine learning research, you may well need some good data sets and collaborators. These can both be excellent drivers for your research programme, so put some effort into developing these areas. Statistics/machine learning is good for this because the researchers who produce interesting scientific data sets are often very happy to have someone offer to help with their data analysis, which then gives you access to good data on which to develop your new algorithms.

It's very easy to skip over this area, comfortable in the knowledge that it's the algorithms that are the important research outcomes of your work. But your proposal (and your research programme) will be a lot stronger if you put the effort into building some good collaborations and negotiating access to some genuinely interesting data sets.

Don't try to do too much
More good research is better, right? Well yes, but only if you manage to do it all. One of the common failings of fellowship applications written by junior academics is that they say they'll do vastly more than is feasible in the time allowed. Don't fall into this trap! The referees and grant panelists who read your application will be experienced enough to know you can't deliver all of it, and it just makes it look like you're no good at planning your research. Better instead to pick only your most interesting and important research ideas.

Be specific!
Specific details are key to making a good proposal. What are your research aims? Why is each one important? What classes of statistical model will you be working with (and why is that a good idea). What data sets do you have access to and what are their details (size etc). Not only do all of these help to sell your proposal, they also help clarify in your own mind what you're doing and why. If you can't write down in a couple of sentences why a research aim is important, maybe it isn't...

Keep trying!
My fellowship plan always involved a contingency for being unsuccessful the first time around, simply because the majority of applications in any given round won't get funded. It's a fact of life that in modern academia there are more good researchers than there are fellowships to go round. So be persistent. Even if an application fails, it's helped you develop your ideas about your research programme and the feedback you've gotten along the way (and hopefully from the panel itself) means your next application will be even better. Not to mention the fact that your CV should have improved a bit in the interim.

In conclusion...
Getting a fellowship is hard simply because there are a lot of good, smart researchers out there applying for them. But there are things you can work on that increase your chances and will these efforts tend to make you a better, more productive researcher as well. Win-win, really...

The Information Hierarchy

2010-02-09T16:02:00.004Z

Rands In Repose posted an interesting article which included a concept called the Information Hierarchy (also known as Wisdom or Knowledge Hierarchy) which I'd not previously encountered.

The idea is this: information can be classified in a 4-level hierarchy.

Data - the raw material of knowledge
Information - data that have been organised/presented
Knowledge - information that has been acquired and understood
Wisdom - distilled and integrated knowledge and understanding

It strikes me that this gives us some insights into the process of science, especially nowadays. At the most basic scientific level, we're simply trying to gather data about the world and progress through the hierarchy to build up information, knowledge and ultimately wisdom about the world/universe/multiverse/whatever in which we live.

In the original version of this process, every stage was carried out by people. This no longer has to be the case, however. Much data gathering is now automated to at least some degree. Even if scientists are ultimately responsible for building and running the experiments/instruments, a lot of the heavy lifting is now carried out by automated or semi-automated systems, with data reduction carried out by software pipelines.

I would argue that we are also able to automate aspects of the second level of the hierarchy, the production of information. Specifically, I think one can regard statistical modeling and machine learning as doing just that. We live in an era of phenomenal scientific data production, so we now routinely use (and create) statistical methods for extracting the useful information from these giant data-sets.

So I think this begs an interesting question: I wonder how much of this process we might ultimately be able to automate, and in what ways? (and what would the implications be of automated systems capable of the Knowledge and Wisdom levels?)

Fishing for significance

2010-02-05T16:00:00.002Z

I recently read a very thought-provoking article (sorry, I think a subscription to the journal 'Bioinformatics' is required for this link) by Anne-Laure Boulesteix. The paper's title is 'Over-optimism in bioinformatics research' and one of the points the author makes is that there can be an effect called 'fishing for significance' when developing (for example) new statistical methods.

This problem is a version of what happens when you make multiple hypothesis tests. Imagine that you test 100 different genes to see whether or not they're differentially expressed between two different experiments and that for each gene you compute a regular, single-test p-value. If you decide to keep the genes that have a p-value <0.05, then you would expect to keep about 5 genes that aren't differentially expressed at all, but crop up by chance (false positives). This leads statisticians to make p-value corrections when making multiple hypothesis tests, to avoid getting (so many) false positives.

With that idea in mind, consider the typical development process for a new statistical method. Imagine you have a good idea for a new type of statistical method; it's clever, it should be really useful scientifically, and you're very keen to spend some time working on it. You build some software to implement your idea, analyse an example data-set and produce some results and, while it works quite quite well, the results give you several ideas about how to improve it.

You apply each of these ideas in turn, keep the good ones, discarding the bad ones and this improves the results. It also leads to even more good ideas, which you try out in a similar fashion. And eventually you have a method that's producing pretty impressive results on the test data-set. You write up the results, publish them and move on with the next project.

This is all well and good, and people certainly use this approach to produce genuinely good statistical methods. But there is an element of multiple testing in what I've just described. By trying out a range of ideas on our test data-set, then keeping the 'good' ones, we're optimising our approach to do well on the test data.

What's really happening in this process is two forms of apparent improvement are going on. The first is due to our developing genuinely improved statistical models that simply work better. The second is that we're over-fitting to our test data-set, finding models that just happen to work well on these particular data. This second one is a problem because it's an illusion; it won't help us with any other data (it will generalise poorly).

So, what's the solution? Probably the most robust way is to validate any new method on independent data once the model has been finalised, measuring performance using metrics that were decided upon in advance. This isn't always easy, because suitable data aren't always easy to obtain, but I think this has to be the goal to aspire to.

Interestingly, this highlights a merit in turning a clever new statistical method into a tool for people to use: this is a great way to test said method on a wide range of different data-sets. Of course, it can be substantially more effort to develop such a tool, and it can be a bit intimidating to subject your new method to such vigorous scrutiny. But maybe this is the only way to find new methods that really are improvements over the current state of the art.

Science and the Internet

2010-01-12T11:35:00.003Z

There are some interesting points in a recent online article by Martin Rees (president of the Royal Society and a very well-regarded astrophysicist). The article as a whole is very interesting and well worth a read, but a few ideas particularly grabbed me.

the Internet enables wider participation in front-line science
it allows new styles of research (for example, mining large publicly available data-sets)
scientific discoveries can now be made by 'brute force' number crunching (e.g. exhaustive computational searches), as well as the more traditional methods of experiment, insight (and I would add theoretical calculation to the list)

I would also add another point to this list.

the Internet gives us faster access to resources, so we can get science done more quickly. For example, literature searches are much easier and faster to do when the papers are online and can be found via Google Scholar or similar.

Science achieved so much in the 20th century, but all of the above makes me really enthused about how much (more?) we're doing right now and what we'll achieve in the next couple of decades...

Deliberate Practice

2010-01-11T13:53:00.006Z

There's an excellent post on Deliberate Practice/Serious Study, over at the Study Hacks blog. I've been interested for a while now in the idea that it takes humans about 10,000 hours to become an expert at something, but this post goes more into specific detail about the hows and whys.

The key idea is a thing called "Deliberate practice", which are activities specifically designed to improve your performance at something. I was aware of this idea in a sporting context, so it's cool to discover that it's a more general psychological principle. The idea is that to improve at something, you not only need to practice, that practice has to have certain characteristics. For example, it needs to push the boundaries of what you're capable of, and it needs to provide feedback so that you know whether or not you did well in a given instance. 10,000 hours of coasting in your comfort zone won't do much for you.

It seems to me a Very Good Thing to focus on improving how well you do your research, but I suspect that most scientific researchers don't really do this (not explicitly, anyway). Maybe they should...

The Study Hacks article looks well-researched and links to a range of other materials, so it's well worth a look!

E. W. Dijkstra's three golden rules for scientific research

2010-01-06T15:45:00.003Z

Very interesting post up over at the Successful Researcher blog, based on some original text by the man himself which gives a bit more discussion about the reasoning behind them. In short:

Raise your quality standards as high as you can and try to always work at the boundary of your abilities.
Ideally, your work should be socially relevant and scientifically sound. If it can't be both, scientific soundness should prevail.
Never tackle problems that are (or will soon be) addressed by people who are equal/better equipped than you to do so.

These all seem very good advice to me. The first one means that not only are you always producing work of the highest possible quality (that you're capable of), you're also pushing the boundaries of what you're capable of. In other words, the way to improve is to push yourself.

The second one is nicely explained in E. W. Dijkstra's original post. Scientific rigour is all-important, because if you don't have that then social relevance (or anything else) isn't going to be useful. Being scientifically sound is a foundation.

The third one is the consideration, "If I didn't work on this, would my efforts be missed". If the answer is no, then go and find something else to work on. Ideally, we should all be making contributions that we're uniquely well-suited to make.

Research flexibility

2009-12-17T16:09:00.003Z

(image by FeatheredTar)

How flexible should you be in your research?

This is one of those questions that I suspect many researchers never (or rarely) ask themselves; often, one can simply progress incrementally through a research career, going where the interesting work is. And that's fine, but I for one think tthat there are benefits to be gained from some strategic thinking in this area.

What I mean by flexible is how willing should you be to move on to new projects, areas of research and entirely new subjects.

By being flexible, there are a number of advantages.

You can follow the latest hot topics
Your interests may develop over time
The interdisciplinary effect (wider skill-set, acting as a vector to transfer ideas from one subject to another)

Of course there are also some downsides.

You'll need to build up new domain-specific knowledge
You'll need to build new collaboration networks
You'll need to build a reputation in th new subject area

I imagine there isn't a unique answer to this question, but I'm also sure that it's valuable to spend some time thinking about thius: Is your research optimally flexible?

Validating your statistical methods

2009-12-07T17:38:00.003Z

(image by kevindooley)

I'm in the business of creating 'clever' new statistical methods. It's what I most enjoy, research-wise. But I'm quite 'applied' about it, because I'm interested in using these 'clever' methods to do something useful.

I've recently been thinking a lot about how to identify the methods that work well and/or improve on the best existing methods. And I've been coming to the conclusion that it's very easy to do this poorly.
The problems I see time and again (and to be fair, I struggle to avoid myself sometimes) are that new 'clever' methods are tested on one or two standard test data-sets (which are real data if you're lucky, and synthetic if you're not), shown to improve some chosen test metric over the existing methods and then left at that.

This is fine, as far as it goes. But it sometimes doesn't go very far. Will the method work well more generally? Are the metrics measuring anything useful? What about types of data that have different noise characteristics? And how long will it take to run if you have a data-set one hundred times bigger? This is not to say that the results as given have no merit; it's just to highlight that this first wave of testing is far from the be-all and end-all.

So, what's the solution. Validation. On new, interesting data-sets. For which someone wants/needs to know the answer. In ways where the performance will matter. This last part is crucial. If the result matters, you instantly have a good metric for how well your 'clever' method is doing. As well as ensuring that you're developing something that will be useful.

A really good example of this (full disclosure: with which I have no experience whatsoever :-) ) is the use of statistical methods in financial trading. In this case there is a well-defined metric of success (how much money you make) and it's straightforward to generate new validating data - you just have to use the method to do some trading, then look at how well it performs. Ambiguous result? Just rinse and repeat until you've tested enough to convince even the most robust skeptic.

My suspicion is that a lot of this is terribly obvious in other, more applied disciplines. For example, Google's search engine algorithm has huge numbers of people (both Google engineers and volunteer testers) using it in real, creative ways to find the flaws, so they can be fixed. This is a whole iterative loop that doesn't occur that much in an academic context. Of course, we don't have anywhere near that level of resources, and we often don't have the luxury of a quick supply of new data (as new experiments will need to be performed first etc). Nevertheless, pushing on beyond the first paper's worth of work (and testing), applying your method to new data-sets, then using what you learn to make further improvements, can really make a difference to just how clever your 'clever' methods are.

How to pick the projects you work on

2009-11-27T15:39:00.004Z

(image by Jan Tik)
I recently read a very thought-provoking article about the process of picking research projects to work on. This is a topic that's very important, yet is easily overlooked. So I thought I would post my thoughts.

What should our aim be in picking research projects?

It's important (perhaps even vital) to pick projects that are interesting to you. Not only is this sensible on a personal level (why would you want to work on things you don't find interesting?), but it's also hugely important on an academic level; if you can't enthuse, immerse yourself, even obssess about a particular project, you'll struggle to gain the very deepest levels of insight and your results will be less good because of it.

Research that addresses important questions should be our second aim. Imagine that you have a miracle year of research where everything you work on turns to metaphorical gold. Wouldn't you rather this effort went into curing cancer, producing a working theory of quantum gravity or solving climate change, rather than some minutae of an obscure branch of your subject? I've put interesting and important in this order, but I think the key point here is that you want to work on projects that are both.

These two considerations ought to be enough. Sadly, there are also practical considerations because of the realities of building a research career. It's probably prudent to work on at least some projects that will help you secure future funding and/or jobs. This is a tricky topic, especially if you're on fixed term funding (such as a postdoc) and you have a very limited amount of time before you need to find more funding from somewhere (and you might be employed to work on a specific project). Of course, if you're working on important areas of research then it should be a lot easier to sell yourself. But you need to make sure that the projects you're working on will produce some publishable results and material on which you can talk at conferences. Even one great piece of interesting work can have a huge impact here.

At the risk of a sweeping generalisation, many researchers end up working on safe-but-slightly-uninspiring projects. These types of projects can produce a steady stream of publications and to be fair they do often have some incremental scientific value, but I think it's a huge mistake to only work in this way. Our profession is one of creativity and knowledge discovery, so we should spend a proportion of our time working on ideas that are speculative, exploring new intellectual territory. Of course, many of these won't come to anything, but the occasional one that does might have a huge impact. There are scientists who have built stellar careers and created whole new disciplines with one (really, really good) idea.

And what do I think? I think that a deep fascination with your research is vital. Within that, pick the projects that are likely to be important (in both your and other people's opinions) - you might as well work on things that might have some impact. And beyond that, try to build a good CV but if you're doing the first two things well, this shouldn't be a problem.

The pressure to publish...

2009-11-02T16:58:00.002Z

The modern academic faces a lot of pressure to be productive, especially to publish papers.

There are pros and cons to this. In the "good old days" (I'm told), academics gained a faculty position and then were left to their own research devices for the next few decades. This is great for truly creative research (so-called "blue sky" thinking), because you can focus exclusively on the problem, letting it develop and exploring its various facets without spending time/effort producing incremental publications. Or course, it may also have allowed some academics to coast.

I'm a strong believer that there's a lot of benefit to academic creativity. What we do is intrinsically creative and creativity needs a bit of scope to explore new ideas, without having to worry if they'll turn into a paper or a grant proposal. But I think there's also a pretty good case for a balance. After all, if an academic spends their whole career deep in thought and never writes a word of it down, their research hasn't been useful to anyone. So, the question becomes this: what balance should we strike between productivity and creativity? Between writing papers and trying out new ideas.

To some degree, this is a trade-off between quality and quantity. The academic that publishes all the time runs the risk of writing papers that have very little important content. There are lots of academic papers that get churned out that have some limited merit, but that exist mainly because the authors felt the pressure to publish. On the other hand, the academic who hardly ever publishes should (hopefully) write papers with lots of great content. Just not very many of them.

There is also a subtlety to this trade-off. While publishing more frequently will tend to mean less research goes into each paper, it does mean that you'll get more rapid feedback on your work (from referees and readers). This is important because it crowd-sources your research, getting a whole range of suggestions and criticisms that will help improve and inform the next stage of your work. Research is actually very incremental (think about how your projects progress on a day-to-day basis), so this can be really beneficial.

And of course it can be argued that the funder (the UK tax payer, in my case) has the right to expect some kind of return for their investment. I think this is fair enough, but I think a lot of care has to be taken in how one defines this return. Number of papers is almost certainly a terrible measure (who cares if an academic writes 50 papers if none of them have any lasting impact). Maybe there has to be a degree of trust between funder and academic?

My gut feeling is that one awesome lead-author paper per year is what we should be aiming for. If you generate enough research for more, great. But one really great paper per year where you're the lead researcher seems to me to be a good level. This should give you enough time to try ideas out and develop new projects, while also building a good publication record over time. If you're like me, you'll also spend a fair amount of time contributing to projects where someone else will be lead author on the papers; this is valuable and you should end up being a co-author on papers as a result.

So the message of this post is to strike a balance. Whatever the rights and wrongs of productivity versus creativity, you need to publish papers to build an academic career. And I really do mean it about the 'awesome' bit. Would you rather be known as the researcher who's produced half a dozen fantastic lead-author papers, or the one who has written fifteen that are deeply uninteresting?

Sleeping on the problem...

2009-10-15T14:56:00.001+01:00

Some problems are difficult to solve. You can spend all day trying to find a solution and only end up with a list of approaches that don't work, plus a strong need for the alcoholic beverage of your choice. One possible way forward is strangely counter-intuitive: stop working on the problem.

Not for ever (obviously). But long enough to take a break and give your brain a rest. If you've been beating your head against a metaphorical brick wall all day (or week. or month...) and still haven't solved your problem, it's unlikely that a bit of a break will slow you down much. And it might just help. It can be as short a break as stopping for a cup of tea/coffee. You could leave the task until tomorrow, so your brain gets the night off. Or you could even leave the project for weeks (or more), if you really want some separation.

This has a number of advantages. Firstly, it gives your poor overworked (and often frustrated) brain a rest. But there's also a more subtle effect. There are psychological studies (and apologies that I don't have the references to hand) that suggest that problems can become more easily solved if you take a break. The suggestion is that your brain continues to work on the problem subconsciously while you are doing something else, so that when you return to it you might have new insights that your (now-rested) brain can work with. Anecdotally, I can relate to this. I hate leaving a problem unsolved, but sometimes when I've forced myself to leave it until tomorrow, I find a fresh approach and a good night's sleep lead to me solving the problem very quickly the next day.

There are lots of examples where this can be useful. Trying to understand the meaning of your latest set of experimental results. Solving a particularly knotty piece of mathematics. Finding that invisible bug in your code. Or how best to write that troublesome paragraph in your latest paper.

Happily, this strategy is easy to try (provided you can exercise a little willpower to let go temporarily of the problem that's been bugging you). So the next time you're wrestling with a problem and find you're not winning, consider giving up for a while. Take a break. Eat lunch. Get a good night's sleep. Then come back and see if you can't solve the problem.

Changing disciplines

2009-08-17T10:42:00.003+01:00

You don't have to work in the same research discipline for your entire career. Indeed, at least some drift is probably the norm, if only because most problems don't warrant 40 years of attention. But researchers tend to be very focused and dedicated to their subject, spending many thousands of hours exploring it and communicating their findings to the world. They become known for it - "Susan the cosmologist", "Steve the mouse guy".

But it doesn't have to be that way. You could change disciplines.

There are a number of pros to this. You learn a lot of new things when you change discipline. Not just the facts and figures of the new subject (although there are many of these), but also you find out how people in that discipline approach scientific research. How do they collaborate? What are their conferences like? How do they write their papers? All these things can be surprisingly different from subject to subject, and you can learn a lot by seeing different ways of making these things work.

All this new knowledge is stimulating. Not only is this a great experience for the knowledge-hungry academic, but it's a great way of generating new ideas. Perhaps there are problems in the new subject that are much more tractable using the old subject's mindset. Or perhaps you had a general problem you'd been thinking about, and an idea known about in the new subject gives you a "Eureka!" moment.

You can also act as a vector for good ideas flowing between subjects. This is why interdisciplinary work can be so valuable. When the good ideas of two different disciplines mix, sometimes you get important new discoveries. Being the cause of this is obviously a Good Thing.

Sometimes, moving allows you to find a better niche for yourself. There are all sorts of reasons that determine how well you 'fit' in any job and research is no different. Do the day-to-day tasks suit your temperament? Are you happy working on decade-long projects, or is six months a better timescale for you? How well do you gel with the culture in a given discipline? How well does your particular set of research interests fit into a particular discipline.

Depending on the specifics of the change, the practicalities of funding might become easier. One good reason for moving disciplines is to move from one where there's little funding to another that has much better support. I don't think this should ever be the primary reason for moving (and I don't think most scientists do research for the money!), but all other things being equal, wouldn't it be nice to not have to worry quite so much about where the funding is going to come from?

It can also be the case that changing disciplines freshens things up for you. Working on the same small set of problems for a decade or more can get quite same-ish, so moving on to a new set of challenges can open a wellspring of enthusiasm.

A well-judged change of discipline can be a great move, but of course it's not without downsides. It take a big commitment of time and effort, because there will be many new things you need to learn and you'll likely be moving jobs at the same time, so there will be the normal upheavals that this implies (you might even be moving house or moving to a different country). And you'll stay junior in your career for longer. In your new subject, you'll be a rookie even if you bring a lot of relevant skills, plus you won't have a network of contacts and collaborators yet, so these will need building. And your publication list in the new subject will need time to develop, before you can start applying for fellowships etc. None of this is to say that change is bad; simply that there are costs associated with a change this big.

In conclusion, there's a lot to consider if you're thinking about changing disciplines. And rightly so - it's a huge commitment. But if you make the right change, there can be a lot of benefit. And here's one more thought: Maybe the modern scientist should always be looking to diversify?

Why do you work on the science you work on?

2009-08-14T10:17:00.002+01:00

What was the decision-making process? How did you make the choice?

Early in my research career, I discovered that I loved working with statistical inference and building the software to do it. I had chosen my PhD because I knew I was interested in astrophysics and I was offered a place at a good department to work on observational cosmology (which I found and still find fascinating). Gradually over the years my horizons have broadened, to the point where I now work mainly on medical and biological data. But the key point is that I'm using statistical inference and programming to do science.

I certainly didn't see this path coming - it evolved.

I suspect this is true for a lot (maybe even most) scientists. Maybe the subject was something that interested them during their degree. Or perhaps it's what they're trained for, given the choice of undergraduate degree they made. That's sobering - the choice you made aged 17 can define your entire career. Anyone else want a 17-year-old picking your career for you? Thought not...

My guess is that many people aren't sure what to do next. They enjoyed their undergraduate degree and therefore (quite reasonably) decided to do a masters in the same or a similar subject. That also turned out to be interesting and, still lacking inspiration as to a career direction, a PhD beckoned (perhaps they were even offered a place by their Msc supervisor, making it an easy option). Suddenly, they're in their mid-twenties, have a doctorate and almost a decade of training in an academic subject. Sounds like a good basis for academia, so off they trot.

Some people want to stay at the same university and this affects their choice of subject (I'll admit to this a little bit). Given a choice of several interesting topics, they take the one that also allows them to stay where they want to be.

Perhaps the subject in question was/is an up-and-coming area with the prospect of lots of interesting science to work on and important problems to tackle. This seems like a not unreasonable consideration.

And of course some people have a burning passion for the subject (something which strikes me as a very good reason indeed!).

As you progress through your career, you'll learn more about what your chosen subject is really like. Are the scientific challenges important? Is it well-funded? What do you really enjoy doing on a day-to-day basis? And do you try to change things in response to this knowledge? What if after five years in one field, you realise that another field might suit you better for whatever reason. Would you change?

I once read a suggestion that in life you shouldn't pick a good destination, but rather focus on a good direction in which to go. The point is that as you live and experience life and learn from it, you'll be better able to make future decisions about where to go. If you pick your path through life now and stick to it, you'll have to turn down all those unexpected opportunities that occur tomorrow. And the You of 5 years time is probably a better judge of where you should be going at that point than you are right now; so why not defer to your (future) superior judgement?

And the point of all this? Think about why you work on the things you work on. And be willing to be flexible. Even if your current plan is a good one, you might happen upon an even better one tomorrow!

Turning data into knowledge

2009-07-21T17:35:00.001+01:00

At some level, science is about turning data into useful knowledge. When the number of data is small (and especially when the signal is strong), just looking at the data can be enough to gain new understanding. The essence of modern science however, is making this transformation with very large amounts of data. And this requires a particular set of approaches.

The blindingly obvious...
Sometimes you'll get lucky and the knowledge will be obvious from the available data. For example, you might have a scientific image with 108 pixels, but the object you're looking at is imaged to high resolution and to huge signal-to-noise ratio. From this, you can probably learn a lot without doing anything more than looking at the image (and perhaps making a few basic measurements). But it's rare that you can't learn more by more fully exploring all those pixels, and to do that you'll need some better tools.

Vanilla methods
If you do need to do something to your data in order to extract some useful knowledge, your first port of call might be "vanilla" methods. These are the bulk-standard, well-understood tools of the data analysis trade. Taking an average, finding a p-value, fitting a regression line, clustering using k-means etc. You are now into the regime where your data cannot all fit into your brain at once, so you have to start using tools to help you extract useful scientific knowledge. Vanilla methods are by definition widely used and tend to be well-understood and easy to interpret. Your aim here is to use these tools to spot the patterns in large amounts of data. Do your genes group into distinct clusters? Are there significant sources in your astronomical image? What's the most likely curve for your measurements, given the noisy measurements you've made? If you can reduce a billion data-points to a hundred clusters, a thousand point sources or a curve defined by ten parameters, you have already made a lot of progress in understanding what your data are telling you.

Clever methods
Of course, you can also try to be more clever than that. If you have a good idea of the sort of general structure you expect in your data then you can build a method that can target that type of structure. Perhaps you have a good physical model of what's going on? The power spectrum of the Cosmic Microwave Background (CMB) radiation is a good example - the major structures in this curve are well-defined by the underlying physics.

You can try to build clever methods that do more of the donkey work for you. Are you clustering your data? Into how many clusters should you be dividing the data? The right clever method can apply a robust principle to determine this optimally, leaving you free to consider the results in greater detail.

You can also build very general clever methods that are capable of spotting a large range of different types of structure (for example, Bayesian non-parametric techniques, and splines for curve fitting). Care must be taken to not simply identify every noise spike as structure, but this can in prinicpal be a great way to spot the unexpected.

Statistical inference
All of this can be viewed as statistical inference. Inference is the extension of logical deduction to include uncertainty (because probability theory extends mathematical logic to include degrees of uncertainty). Indeed, there's a view that the scientific method is all statistical inference. With very certain observations (the sun rises every morning), we are just left with logical deduction. With uncertain/noisy observations, we are left with statistics, maximum likelihood techniques, Bayesian methods, the need for repeated experiments and the like. And consider Occam's razor (often held up as an important part of the scientific method): probability theory actually provides a mathematical derivation of Occam's razor, via Bayesian model selection (if two models fit equally well, the simpler model will have a higher Evidence value, meaning it's more likely given the data).

Science as data compression...?
While we've drifted into philosophy-of-science territory, there's another interesting idea that's highlighted by the advent of data-sets too large for a human brain:

One could consider science as a series of attempts at data compression.

Think about this for a moment.

What are we looking for, as scientists? We're looking for generalisations about the area in which we're working. We want to know how metals behave as we heat them. We want to know how the universe expands over time. We want to know how our favourite set of genes interact with one another in different conditions. We can make a vast number of observations about any one of these, but what we're after is a set of rules that tells us how these things behave and we want those rules to be as general as possible.

Once we find (and test) such a rule, we've encoded the essence of all those observations into one (often simple) rule. Think about Newton's law of gravity; it goes a long way to describing the the motion of a hundred billion stars in our galaxy, but it's just an inverse-square law with a couple of masses and a gravitational constant. In terms of bits of information, that's a pretty awesome compression factor.

So what? Well, we're talking about the need for methods for converting large data-sets into something more interpretable by a human brain. These are compressions in themselves. So we're using algorithms/statistical methods to partially automate the scientific method. Which leads me to wonder how much more of it we could automate, if we really put our minds to it....

The Scientist-Progammer

2009-07-15T13:47:00.002+01:00

I also co-run a blog called Programming for Scientists. A theme that's developed over the time we've been blogging there is the idea of the Scientist-Programmer.

I think a big part of the reason for the development of this theme is that I identify myself with it. I've always been a scientist first and foremost, but since starting my research career I've come to have a real love for the craft of coding and it's always been a big part of the work I've done (for reasons you can see below in "The need to get it right!"). I also think it applies to a lot of other scientists nowadays, and our numbers are growing :-)

The nature of the Scientist-Programmer
The Scientist-Programmer is a scientist first and foremost, but one who spends a lot of time coding and (crucially), takes a professional approach to the programming part of their work. Nowadays, there are areas of science that are critically dependent on software (anything to do with simulation of analysis of large data-sets, for example), so it's an aspect that scientists must take seriously and work hard at.

The programming skill-set is as important to modern science as 'wet' laboratory skills, electrical engineering (for building those big physics machines) and maybe even mathematics. Because some areas of science rely on software, it's a huge advantage to have researchers who are expert at both the science and programming, because they understand both the problem domain (i.e. the science) and also how to go about building the required software.

The need to get it right!
Here is why programming is important: computers are inevitable at some level in modern science because of the large data-sets we need to work with. Computers imply the necessity of programming. And if we're programming to do science, our science relies critically on the fact that our code works properly!

Consider the Conference Test: Imagine you're presenting your work at a major international conference. Do you really want to be stood in front of 200 eminent scientists at a conference when the world-leader in your field spots that your graph must be off by 10% because of a numerical error in your code? The way to avoid this is to take a professional approach to the software that you need to write.

Scientist-Programmer or Programmer-Scientist
Just as you can be a scientist who programs, you can also be a programmer who writes code for science. So, what's the difference? I think there's actually continuum that runs all the way from from "scientist who avoids programming" to "programmer who doesn't work in science" and that both the Scientist-Programmer and Programmer-Scientist are somewhere in the middle. It probably comes down to similar skill-sets but slightly different core interests.

In conclusion...
I've sometimes found it difficult to communicate to other scientists exactly what I am (professionally-speaking :-) ). I think the trouble is that while programming used to be another handy skill that a scientist might pick up in passing, it's grown in significance in recent years and is now much more important. The Scientist-Programmer is the result of this process, a scientist who spends a lot of their time doing science via a computer and strives to be (more or less) a professional-level programmer.

The point of clever methods

2009-06-29T16:30:00.002+01:00

The phrase "clever methods" is my label for statistical methods and/or algorithms that go beyond basic and/or standard approaches (which I think of as "vanilla methods"). Those of us whose research involves methodological work aim to write papers that detail new clever methods. Clever methods aim to go beyond the capabilities of the relevant vanilla methods in some meaningful way, as well as hopefully doing so without becoming intractably complicated or slow to run. In the same way that software engineers might be trying to craft better software for a given task, in researching clever methods we're trying to find better mathematical/statistical/algorithmic ways of doing something.

(By way of full disclosure, I should mention that I'm a fan of clever methods in that I really enjoy working on them and finding cunning and sneaky ways to make a method work better. This is great for motivation, but comes with the health warning to be careful to not make something more complicated just for the sake of it :-) )

Performance vs. complexity
Clever methods tend to be more complex than the equivalent simple methods. So, our goal in researching clever methods is often an attempt to trade off complexity for performance. The trick then becomes to minimise the increase in complexity, while maximising the improvement in performance. This is the benefit that most vanilla method possess; they provide reasonable performance in a very uncomplicated way.

In many case where reasonable performance is all we need, this is a very good solution. For example, if you're trying to detect local stars in an astronomical image, the signal-to-noise ratio of your image might be very high. In which case, even a basic method should be able to detect them all with little problem, meaning that such a choice will do everything you require and do so in a simple and easy-to-understand way (which is often a hidden benefit of vanilla methods).

When creating clever methods, it's very easy to ignore the complexity aspect and simply go all-out for performance (there are many, many papers for which this is true). While this can be okay if performance is so vital (relative to handling the complexity), usually this leads to methods that are so narrow in their application that they're not very useful.

Happily, there are also many cases where a little extra complexity gives you a significantly better method with which to work. And there can be other benefits; for example, if you generalise a vanilla method, the resulting clever method will probably be more complex but it may also be more reliable or allow the automation of some parts of its use. Consider a clustering method that has a well-defined, automated way for choosing the number of clusters into which to partition the data; the user no longer has to worry about doing this, thus saving them time. They probably don't care that the underlying maths is more complicated.

And of course very occasionally, you'll manage to create a clever method that's no more complex (or in extreme cases, less complex) than the simple method/s. Congratulations, you've probably discovered something genuinely importance!

The 10% improvement
Often, clever methods can provide and order 10% improvement (in whatever metric is important) over the simple method. The question then becomes, "is this worth the effort?"

The answer is "it depends". If you're trying to extract a signal from some noise, but the signal-to-noise ratio (SNR) is already 105 then increasing it by 10% may well be irrelevant. If on the other hand you're trying to detect signals right at the detection limit of your data, then it might be vital in uncovering that Nobel-winning new class of whatever. I've worked on astronomical source extraction where the data-set has had an effective cost of tens of millions of pounds (actually quite common when the data come from a space telescope). In this case (and assuming Gaussian noise), a 10% improvement in SNR using the simple extraction methods would require 20% extra data, at a cost of millions of pounds. Or you can just use the clever extraction algorithm.

One very important consideration in all of this is that if you develop a clever method that is reasonably general, then that 10% improvement will be a benefit many, many times.

The undiscovered country
So far I've focused on the mundane benefits of clever methods. There is also another aspect to consider. If your method of analysis is too simple, you might miss something important.

Think about a very rich, complex data-set where it's not obvious how to model the data. Gene expression measurements of whole genomes are a good example. We can certainly use simple methods to analyse this and to get some useful scientific results. But what if there is structure in the data to which our choice of method is insensitive? Imagine what would happen if you only fitted your data with straight lines! You'd miss peaks, troughs, oscillations and all manner of other interesting structure in your data. Your methods need to be able to account for all the interesting structure in given data-set and if that structure is complex, a vanilla method may well miss it.

A related point is that a good way of spotting complex patterns can be to use your eyes to look at the data. There are many examples where the best signal detection method is a person (eg. objects in an image, CAPTCHAs). But this doesn't work if your data-set is too big for a person to meaningfully do this. In this case, you need a clever method.

Clever methods as their own research discipline
There is justification for creating clever methods simply because doing so adds to the sum total of human knowledge. This is especially useful when that clever method extends an existing method and/or when it can be further built upon by you or other people. Whole new areas of methodology can be uncovered in this way, whether through being created or through making some existing ideas more widely known. And often reading a clever idea in one context can spark a thought in someone's mind about their own area of research (this is why it's very important to be well-read as a researcher).

If, like me, your research involves creating new clever methods, a burden of proof falls to you. Because there are infinitely many clever methods one could create, it's important to find the ones that are actually useful (defined as having superior performance to the vanilla methods, at the very least). And this means that you need to test your methods and compare them to other existing ones. This is actually one of the real tricks of methodological research; figuring out as many ways as you can to test a new method, to see if it's worth using. A few things I think are really important for this include:

Test in many different ways
Test on many different data-sets
Test using many different metrics
Testing on synthetic data can be good because you know the right answer
Testing on real data is very important; real data will always contain more junk than synthetic data
Real data where you know the right answer (eg. from some other source of information) are wonderful to have
Realistically simulated data can be very useful. But it takes a lot of effort to build a software/hardware simulation of most types of data
Use your methods. Do some science with them (or help other people to do so), because in the process you'll learn more about how the methods work and how to improve them

In conclusion...
The creation of clever methods is a craft, a balancing act between performance and complexity. But the right method in the right context can be a powerful solution and even open up whole new areas of research.

Do you work on important problems?

2009-06-17T15:00:00.002+01:00

I recently read the transcript of a talk given by Richard Hamming and realised many of us may have been missing a trick. He makes the point that we should aim to work on the important problems in our field. This is one of those ideas that seems obvious once you read it, but is somehow easy to overlook during the bustle of day-to-day research.

Why is it important to work on important problems?
Imagine you have a miracle year of work. You're in the zone 100% of the time, every hunch you have turns out to be right and every project you touch turns to gold. Now consider the difference in the impact of your work depending on whether you had been working on important problems or unimportant ones. In the first case, you might have done truly great, maybe Nobel-worthy work. In the second case, you've still done good work but it's not going to change the world. Which would you rather have happen?

What if you work on unimportant problems?
I'm not suggesting that you should exclusively identify important problems and only work on them. After all, you can't be 100% sure your list of "important" is complete and/or completely accurate. However, if all you work on are problems that are unimportant, by definition you limit the impact and value that your work will ever have. Unimportant problems are just that. By all means spend a bit of time tinkering with such problems if they really interest you, but don't waste your career on them.

What are the important problems in your field?
This really boils down to being able to identify what the important problems are in your field/s. This is more difficult than one might imagine, but with some time and thought you can make some headway. Take time to think about what you consider the important problems to be. Read some articles for inspiration. See if any other academics in your area have posted on this topic on the Web. And ask people! A great question over coffee or at a conference dinner is, "What do you think the important problems are in our field?".

Some of these problems will be intractable. Finding an exact O(n) solution to the travelling salesman problem would be awesome, but seems unlikely. And formulating a Grand Unified Theory of physics would be nice, but many people have tried and no-one has just succeeded. It can be tricky to distinguish between problems that are difficult and ones that are (or are likely to be) impossible to solve. There are judgement calls to be made here as to how much time to spend on each problem.

And then there are the problems about which we're uncertain. Is it important or not? Again, this is a judgement call. If the problem is interesting to you, plus you think you can make good progress on it quite quickly, then it's probably worth working on just in case it's important.

Nowhere so far have I mentioned a couple of other considerations that I think are also important. You should probably work on problems that inspire/enthuse you. And you should work on problems to which you're suited, in terms of abilities, skills andtemperament. If you're no mathematician, you shouldn't be trying to work on the Reimann hypothesis. And if you thrive on a sense of rapid progress and individuality, perhaps it's not such a great idea to work on that huge physics project that won't begin generating data for 5 more years.

In conclusion...
It's not the only consideration, but deciding on the important problems in your field and working on them is a pretty good starting point for scientific research.

Why start a research blog?

2009-06-15T16:09:00.001+01:00

As I've just started a research blog, I thought "Why do this?" was a sensible question to contemplate. (actually, I thought about it before I started, which seemed even more sensible...)

Sharing some thoughts...
Sometimes the best way to learn or come up with good new ideas is to talk to a fellow scientist. This is why chats over coffee and meeting up a conferences is so important (and why a few glasses of wine at a conference dinner can lead to valuable conversations). This works because of the ideas that are being shared.

There's no reason why talking should be the only medium through which to communicate in this way. And while a blog is less two-way (although please feel free to leave a comment!), it has the great advantage of being able to potentially reach a huge audience. It's difficult to have a conversation with more than a few people, and even a lecture/conference talk is unlikely to have an audience of more than a couple of hundred people (web-casts excepted). But there's nothing to stop thousands upon thousands of people reading a blog post!

Knowing your own mind
There's also a second benefit to writing one's ideas down in a blog post. It helps to clarify them. Writing something for an audience forces you to thing about what you're writing, to work it into a form that will be understandable by the reader, even to challenge your own assumptions. This can develop your own ideas and trains-of-thought in ways that simply thinking about them won't manage.

Returning the favour
Another very important reason is that I've found other people's research blogs very helpful in learning more about how to be a scientist (see myblogroll for some excellent examples). Not every blog will be useful for every reader, but it strikes me that if every researcher kept a blog where they posted their thoughts, observations and experiences, collectively that would form a great body of knowledge for other people to explore.

In conclusion...
There are several great reasons why a scientific researcher should consider keeping a research blog. And there may well be others that I've not thought of yet, but that will occur as I post on more topics. Which is itself an illustration of why research blogging is a good idea :-)

Science in the 21st century

2009-06-13T14:01:00.004+01:00

The nature of scientific research changes with time. Each epoch has its own particular characteristics, the result of a blend of factors ranging from our current state of knowledge and available technologies to the particular needs of our society and the world at that time. The early 21st century (ie. "right now") is no different.

Big data
The sum total of human data is growing more-or-less exponentially and scientific data are no exception. A decade ago, gigabyte-scale data-sets were the sort of thing the large physics experiments were producing. Currently, scientists talk about terabytes quite happily and it's far from stupid to be talking aboutpeta- and exa -scale data-sets. After all, we'll be able to handle routinely that size of data in the next decade of two (organisations like Google probably already do so).

The key point about big data is this: modern scientific data-sets are typically too large to fit in a human brain.

Think about that for a moment. As soon as you can't fit all your data in your brain at once, you need to start doing something new or you're going to have to start throwing data (and hence information) away. This leads to whole new areas of research into how to handle any given type of big data.

Computers
Computers can do things that people cannot. Ever tried adding a million numbers together in a faction of a second? Exactly.

There are two ways of looking at this. The first is that we need computers because otherwise we would be unable to handle the Big Data we are now generating. The other is that computers give us possibilities that didn't exist before, for example there are many applications of Bayesian statistical inference nowadays that were always technically possible, but were simply impractical in terms of the amount of computation required. That is often no longer a problem.

Computational science (ie. doing science using a computer) has become a whole distinct area of scientific research, which means that computing skill-sets have become valuable in a scientific context. In much the same way that scientists with specialist lab skills, mathematical skills, electrical engineering skills (eg. building the big physics experiments) and the like are vital to modern science, this is also true of scientists with specialist programming and other computer skills.

The flow of knowledge...
How rapidly scientific knowledge flows is key to the rate at which science advances. A world-changing idea will likely only do so once it's reached a substantial number of people. The Internet has become a game-changer for this. 20 years ago, a new paper would only typically become available when the hard copy of the journal reached your university library. Now I can scan the abstracts of a hundredpre-prints a day over a cup of coffee, via an RSS feed, months before they appear in the journals. A literature search that might occupy days of library time can now get under way in seconds via Google Scholar and the websites of other academics, and be completed in short order via downloadedPDFs. And even if I can't make it to a conference, there's a good chance I can access the slides online (or even see a webcast of the talk) and email the speaker if I have any questions.

All of this removes overheads from the process of learning about new scientific knowledge. And that makes a big difference to the amount of science you can get done.

Interdisciplinarity
In some sense, science has always featured what we now called "interdisciplinarity". Some of the best new ideas simply span more than one discipline. However, it seems to me that this is particularly true right now. The body of scientific knowledge has become large enough that no one person can know even a moderate proportion of it. This means that 'Eureka!' moments involving ideas from different disciplines are harder to find. So it's become very important to have people who are expert in one discipline who go and talk to people in other disciplines. This even extends to interdisciplinary centres, which have the benefit of putting people from different disciplines in the same office/meeting/seminar on a daily basis. A lot of science is driven by the conversations you have over coffee...

Advancing rapidly...
If I had to pick one thing to characterise modern science, it would be rapidity of its advance. Driven both by the speed of communication and by the rate of improvement of underlying technologies (eg. computers, the cost-per-bit of to generate useful data), we're making new discoveries at an amazing rate. And one of the most striking features is the speed with which new discoveries can be applied to, for example, new technologies - consider Moore's law for an obvious example.

The need for multiple skill-sets
Science is a large field, nowadays. Gone are the days when a gentleman scientist could be the master of all disciplines. Today there are many distinct specialisms, each of which benefits from (often requires) professional-level skill-sets. For example:

chemical/biological/physics lab skills
software engineering
electrical/mechanical engineering (eg. building the big physics machines such as telescopes, particle accelerators)

This leads to there being real value in multi-skilled scientists. For example, not just a scientist who can write a bit of code, but a scientist who is also a professional (or near-professional) level software engineer. Or not just a physicist who knows some things about electrical circuits, but one who could just as easily earn a living as an electrical engineer.

Weight of numbers....
I don't have any concrete numbers for this, but my guess is that we have more scientists now than at any point in history. There are several reasons for this intuition. Firstly, the world's population is bigger that it's ever been (and growing...). Secondly, more countries have developed economies to the point where they can afford significant programmes of scientific research. Thirdly, there are big private companies that have programmes of scientific research.

I would love to see some properly researched numbers on this. And I wonder what a graph of total-science-budget versus time would look like for the world as a whole...

In conclusion...
In the 20th century, science discovered some of the fundamental laws of nature (eg relativity and quantum theory), developed the standard models of cosmology and particle physics, unravelled the secrets of life (DNA) and beat the majority of infectious diseases (antibiotics). We know how to turn lead into gold (in a nuclear reactor) and while eternal life is trickier, theUK's life expectancy has risen by 30 years over the course of the last century. And science is now progressing faster than it did in the last century (maybe a lot faster).

Anyone else excited by the possibilities...?

Welcome!

2009-06-08T16:23:00.002+01:00

This is my research blog about conducting scientific research in the 21st century. I'm going to be writing on any and all topics that I think are interesting/relevant to modern scientific research. I hope you'll find some articles that interest you!