The point of clever (statistical) methods

What's the point of clever statistical methods?

As someone who spends a significant proportion of their working life creating new statistical machine learning methods, it's a question that I dwell on.  Possibly even brood about.  And with good reason.

The subtext to this question is the suggestion that maybe for a given data analysis task, 'vanilla' methods might be perfectly adequate and that there's no reason to go to all the effort of inventing something new.  Want to find sub-groups in your data?  Why not just use k-means clustering?  Want to make some predictions?  Surely a support vector machine or a neural net would suffice.  Feature selection?  Random Forest.  Dimensionality reduction?  Principal component analysis is pretty good.

This is potentially vexing for someone (like me) who enjoys developing clever new ways of analysing data.  But it's entirely a valid point.  And for many of the tasks I find myself doing day-to-day analysing scientific data, these standard methods are indeed perfectly adequate.

So that begs the question, is there actually any point to developing clever statistical methods?

Happily, I think the answer is still (sometimes) yes, for two reasons.

The first is that novel statistical methods can constitute research in their own right.  There is always merit to adding to the pool of human knowledge.

The second is much more central to the scientific side of my research.  It's not always true, but sometimes there is something that the 'vanilla' methods simply do not do.  A trick that they miss.  Some quality of the data to which they're insensitive.

I'm not talking about some incremental gain where your super-clever classifier outperforms a support vector machine by 2.7% in prediction.  Who cares, right?  I'm talking about a novel method that infers the most likely number of clusters in the data, rather than relying on user input.  Or the classifier that learns how to optimally combine information from an arbitrary number of different data sets.  Or the latent Dirichlet allocation model for data where we know there are clusters, but we also know that they overlap.

I think this second reason is the key.  For some types of data, we know more about the underlying structure than a standard method can encode.  When this is the case, by developing a more advanced statistical model we can better analyse and understand our data.  We have to be careful to identify these cases.  Sometimes, the standard existing method really is good enough.  But sometimes that extra level of complexity will pay for itself with an extra level of understanding.  And just occasionally, that's vital.

(Update)

I realised that I wrote a similar post a few years ago (!).  Interesting to see how my thoughts have developed :-)

Predictive power and explanatory power

Recently, I've been thinking about the goals of science. Science isn't about discovering Truth. This is for the very simple, practical reason that Truth is very hard to positively identify - how would you establish that a prospective Truth wasn't just a very good approximation? Not very helpful...

There are all sorts of philosophical discussions to be had here, but we can sidestep these and take a more practical approach. Specifically, we can choose to care about Predictive Power and Explanatory Power.

Predictive Power is the ability of a given theory to allow us to make predictions about the natural world. We know that Newtonian gravity is an approximation (to General Relativity, at the very least), but it's very good at predicting where the planets in our solar system will be. This is really a flat-out practical consideration - if a theory can't make predictions, it's not very useful (and some would argue it's not even science).

Explanatory Power is the quality of a theory that gives us some deeper understanding of what's going on in a physical system. For example, knowing about atomic electron orbitals allows us to make sense of the periodic table and chemical interactions. It gives us ways to develop other theories.

So, what we're looking for from a scientific theory is the ability to make predictions and for some explanatory insight as to why something happens, so that we can use that insight to develop further theories.

This is also relevant for statistical modeling (and hence data-intensive science), because we can build our models to address either or both of these. Predictive algorithms such as neural networks can perform very well, but the structure of the model is often hard to interpret in any kind of explanatory way. Conversely, a linear regression model might tell us a lot about which variables are important, but may not make good predictions. Ideally, it would be nice to build models that are useful for both prediction and explanation.

Fishing for significance

I recently read a very thought-provoking article (sorry, I think a subscription to the journal 'Bioinformatics' is required for this link) by Anne-Laure Boulesteix. The paper's title is 'Over-optimism in bioinformatics research' and one of the points the author makes is that there can be an effect called 'fishing for significance' when developing (for example) new statistical methods.

This problem is a version of what happens when you make multiple hypothesis tests. Imagine that you test 100 different genes to see whether or not they're differentially expressed between two different experiments and that for each gene you compute a regular, single-test p-value. If you decide to keep the genes that have a p-value <0.05, then you would expect to keep about 5 genes that aren't differentially expressed at all, but crop up by chance (false positives). This leads statisticians to make p-value corrections when making multiple hypothesis tests, to avoid getting (so many) false positives.

With that idea in mind, consider the typical development process for a new statistical method. Imagine you have a good idea for a new type of statistical method; it's clever, it should be really useful scientifically, and you're very keen to spend some time working on it. You build some software to implement your idea, analyse an example data-set and produce some results and, while it works quite quite well, the results give you several ideas about how to improve it.

You apply each of these ideas in turn, keep the good ones, discarding the bad ones and this improves the results. It also leads to even more good ideas, which you try out in a similar fashion. And eventually you have a method that's producing pretty impressive results on the test data-set. You write up the results, publish them and move on with the next project.

This is all well and good, and people certainly use this approach to produce genuinely good statistical methods. But there is an element of multiple testing in what I've just described. By trying out a range of ideas on our test data-set, then keeping the 'good' ones, we're optimising our approach to do well on the test data.

What's really happening in this process is two forms of apparent improvement are going on. The first is due to our developing genuinely improved statistical models that simply work better. The second is that we're over-fitting to our test data-set, finding models that just happen to work well on these particular data. This second one is a problem because it's an illusion; it won't help us with any other data (it will generalise poorly).

So, what's the solution? Probably the most robust way is to validate any new method on independent data once the model has been finalised, measuring performance using metrics that were decided upon in advance. This isn't always easy, because suitable data aren't always easy to obtain, but I think this has to be the goal to aspire to.

Interestingly, this highlights a merit in turning a clever new statistical method into a tool for people to use: this is a great way to test said method on a wide range of different data-sets. Of course, it can be substantially more effort to develop such a tool, and it can be a bit intimidating to subject your new method to such vigorous scrutiny. But maybe this is the only way to find new methods that really are improvements over the current state of the art.

Validating your statistical methods

(image by kevindooley)

I'm in the business of creating 'clever' new statistical methods. It's what I most enjoy, research-wise. But I'm quite 'applied' about it, because I'm interested in using these 'clever' methods to do something useful.

I've recently been thinking a lot about how to identify the methods that work well and/or improve on the best existing methods. And I've been coming to the conclusion that it's very easy to do this poorly.
The problems I see time and again (and to be fair, I struggle to avoid myself sometimes) are that new 'clever' methods are tested on one or two standard test data-sets (which are real data if you're lucky, and synthetic if you're not), shown to improve some chosen test metric over the existing methods and then left at that.

This is fine, as far as it goes. But it sometimes doesn't go very far. Will the method work well more generally? Are the metrics measuring anything useful? What about types of data that have different noise characteristics? And how long will it take to run if you have a data-set one hundred times bigger? This is not to say that the results as given have no merit; it's just to highlight that this first wave of testing is far from the be-all and end-all.

So, what's the solution. Validation. On new, interesting data-sets. For which someone wants/needs to know the answer. In ways where the performance will matter. This last part is crucial. If the result matters, you instantly have a good metric for how well your 'clever' method is doing. As well as ensuring that you're developing something that will be useful.

A really good example of this (full disclosure: with which I have no experience whatsoever :-) ) is the use of statistical methods in financial trading. In this case there is a well-defined metric of success (how much money you make) and it's straightforward to generate new validating data - you just have to use the method to do some trading, then look at how well it performs. Ambiguous result? Just rinse and repeat until you've tested enough to convince even the most robust skeptic.

My suspicion is that a lot of this is terribly obvious in other, more applied disciplines. For example, Google's search engine algorithm has huge numbers of people (both Google engineers and volunteer testers) using it in real, creative ways to find the flaws, so they can be fixed. This is a whole iterative loop that doesn't occur that much in an academic context. Of course, we don't have anywhere near that level of resources, and we often don't have the luxury of a quick supply of new data (as new experiments will need to be performed first etc). Nevertheless, pushing on beyond the first paper's worth of work (and testing), applying your method to new data-sets, then using what you learn to make further improvements, can really make a difference to just how clever your 'clever' methods are.