The Sage Bionetworks - DREAM Breast Cancer Prognosis Challenge

I've been competing in the  Sage Bionetworks/DREAM Breast Cancer Prognosis Challenge.   The submission deadline was a few hours ago (early start for those of us in the UK!), so I thought now was a good time to share some of my thoughts on what has been a very interesting experience.  I enjoyed it a lot and I think the folks at Sage Bionetworks are really onto something with this as a concept.

The goal of the challenge is to develop machine learning models that can predict survival in breast cancer.  We've been given access to a remotely-hosted R system on which to develop our models, and (on said system) use of molecular and clinical data from the Metabric study of breast cancer.  We run our models on this remote system and they're scored using concordance index, a nonparametric statistic for survival analysis that is sensitive to the ranking of predictions.

What makes this challenge a bit different is that it's both competitive and also collaborative.  Not only are we competing against one another to get the best-performing model, but once someone has submitted a model, I can download it and inspect their code to see how it works!  This is very ambitious (and certainly not without its issues), but aims to create a hybrid competition/crowdsourcing approach that can produce very strong solutions to the scientific goal of interest.

Having put a lot of hours into working on this challenge over the last few months, I have developed some opinions on it.  So, in no particular order, here are my thoughts on the challenge:

1. Incentives for academics.  In addition to some small financial prizes along the way, the big prize on offer is co-authorship on a journal paper.  This is a very big incentive for academics (such as myself) who want to compete in a challenge like this.  I'm very enthusiastic about the whole concept and would love to join in with future challenges.  However, in order to justify spending my work time on it, there needs to be some kind of academic return.  Co-authorship fits the bill nicely!  Currently, I think the top couple of teams (?) get this prize, but I think extending this would be a good plan.  Certainly, my experience in this challenge is that there are many more than 2 academic teams who have contributed significantly to the success of the challenge.
2. Incentives for non-academics.  Of course, it's also important to have rewards on offer for non-academics.  The real strength of such a challenge comes from having a diverse community of competitors.  I presume the small financial prizes are nice in this regard; it'd be really interesting to hear from some of the non-academic competitors what their views are on this.
3. Sharing of code.  This has been a very innovative (and brave!) aspect of the challenge.  I don't think the organisers quite nailed every aspect of it, but I think the general approach is very powerful and certainly worth persisting with.  I wonder if the sharing should be constrained in some way - perhaps code can only be accessed 48 hours after it is submitted?
4. Blitzing the leaderboard?  In this challenge we could make as many submissions as we liked to the leaderboard (of which I'm as guilty as anyone :-) ).  This worries me as it could lead to a lot of over-fitting.  Maybe in future challenges there should be a limit - say 5 submissions per day?
5. Challenge length.  In total it was approx 3 months long.  2 - 3 months feels about right to me.
6. Competitive vs. collaborative.  Another research model that's relevant here is the Polymath Project.  Essentially, one can imagine a sliding scale between competition and collaboration.  Polymath lives at one end, with things like the Netflix Prize and Kaggle competitions at the other.  This challenge lives somewhere in the middle.   I like the idea of blending the two concepts.
7. Populations of ideas vs. monoculture.  A competition is great for generating a wide range of ideas.  Once people start sharing, I expect (as happened in this challenge) the pool of ideas tends towards more of a monoculture.  
8. Building an ongoing community.  This challenge has been a great way of starting up a research community (a smart mob :-) ).  It would be great to harness this community in an ongoing basis.      

Sharing code

Sharing code means sharing ideas, and this has allowed us to benefit from each others ideas during the challenge.  I'm sure this has led to better overall results.  However, it has also has some quirks that might need tweaking.

First is the phenomenon of 'sniping'.  Someone else can spend a month developing an awesome model, but once it's been submitted to the leaderboard I can download it straight away, spend 30 minutes applying my favourite tweak and then resubmit the (possibly improved) new model, jumping above the hardworking other competitor on the leaderboard.  Of course, overall this leads to better models, which is the collective aim of the challenge.  But I think care needs to be taken to ensure that credit (and reward in general) is given where it's due.  It can be a bit dissatisfying when this happens to you!

The other consideration is that after a while of sharing models, we end up with a monoculture.  Examinations of the high-ranking models over the last couple of weeks show that almost all the models are based on those of the Attractor Team (with some chunks of my own code scattered around, I was gratified to see!).  This is probably not surprising, as the Attractor Team won both the monthly incremental prizes, but it's probably an indication that we've got about as far as we can with the challenge when this happens.  Now is probably a good time to stop :-)

So, what would I change?  I might suggest something a bit more like the following:

A possible model for future challenges

The 21st Century Scientist Speculative Future Challenge (21SFC) would look like this:

Stage 1 (initial competition) - a month long competition to top the leaderboard.  No-one can access other people's code and at the end of the month, a prize is awarded on the basis of a held-out validation set.  After the deadline, all code for Stage 1 is made available.

Stage 2 (competition/code sharing) - another month long competition to top a new leaderboard.  Everyone has access to the Stage 1 models, but Stage 2 code is either unavailable or only accessible 48 hours after is has been submitted.  At the end of the month, a prize is awarded on the basis of a held-out validation set. 

(it might be worth re-randomising the training, test, validation sets for stage 2)

Stage 3 (collaboration) - A non-competitive stage.  The aim here is to work as a team to pull together everything that has been learned, produce 1 (or a small number) of good, well-written models and to publish a paper of the results.

The author for the paper is "21SFC collaboration", with an alphabetised list of people given.  There can be different ways to qualify for authorship:

• Placing in the top-n in either stage 1 or 2
• Making significant contributions in stage 3 (the criteria for this would need to be established)

This structure uses initial competition to generate a lot of good ideas, then uses a second stage of competition to combine/evolve those ideas.  It then has a final, collaborative phase where everyone who wants to pulls everything together to produce, publish and release the code for the challenge's solution to the problem.  The challenge doesn't take too long to complete and the contributors get rewarded for their efforts in various ways.

--------

This post has turned into a long one and I hope I've communicated the intended positive tone.  I enjoyed the Sage/DREAM BCC a great deal and I think this is a hugely powerful way of getting answers to scientific problems.  I'm certainly going to take a look at whatever the next challenge is (I know there are some in the pipeline) and I would certainly recommend you doing the same.

a 'Pool-of-Memes' approach to research

I tend to spend a fair amount of time thinking about how best to go about research. My rationale is that simply working harder isn't scalable - you can't work more than 24 hours a day (and indeed only a lot less than that in a sustainable way). Therefore to be a better scientific researcher, I need to find ways to improve my approach to research.

Study Hack's excellent article on his research system got me thinking about how I'd describe my own system of research. I think the phrase "pool of memes" fits pretty well.

I try to fill my mind with as many interesting/relevant ideas and concepts as possible. This means both being well read in my own subjects, and also hunting out other subject areas that might add something. For example, for the last year or two I've been becoming increasingly interested in computer science. I've found my most productive phases correspond to learning a new set of relevant ideas.

To this pool, I also try to add clear ideas about what questions are important in various areas of research.

Then I just sort of let all these ideas mull. I might think about something in an idle moment at the gym, or I might head to a coffee shop with my log book and tinker with some thoughts.

What I get from this is a list of possible projects on which to work. This list tends to be pretty organic and it evolves over time. I rank the list in terms of how good/important I think they are. And then I try out the top ones.

What I've not had previously, but what I'm just starting to add, is a stage like Study Hack's "small bets". The idea here is to try out the possible good projects for a month or so, with the aim of producing some concrete evidence as to whether or not to take the any further. I'm not very comfortable on a personal level with the idea of discarding projects like this (I don't like the waste), but objectively it makes a great deal of sense, so I may just need to get over myself

:-)

As much as anything, this approach works well for me because I really enjoy learning new things, so giving myself the justification for doing that during work time is nice :-)

The memetics of science

I've become fascinated recently by the concept of memes and the dynamics of how they evolve (memetics). I'm particularly curious about what memetics might have to tell us about how science works.

Full disclosure here is that I'm very much an interested amateur at this point - there are people who have spent many years thinking deeply about memes and memetics, and I'm not one of them. But the basic concept is kind of beautiful and not very hard to grasp, and it does give us some insights into how science works.

We should start with some definitions. A meme is a unit of information (such as an idea or concept) that's copied from person to person. The idea of memetics is that memes are subject to an evolutionary process because they are copied, a range of variants exist, and they are subject to selection pressures (some memes spread more effectively than others). So what we then have is a way of thinking about the dynamics of how (scientific) ideas evolve and develop.

It strikes me that science in particular is a memetic process where we have one additional concept: we subject our memes to the selection pressure that they must be confirmed empirically. This is an important difference. Memetics per se does not require any given meme to be true; it just has to be good at propagating. This explains why rumours that are false but appealing can spread so readily. By adding the additional constraint of empirical confirmation, we are adapting memetics in order to learn about the universe.

Thinking in this way, we can define a list of general scientific processes in which we can be involved.

• Validating an existing meme or memeplex (empirically or via theoretical proof or computational analysis)
• Improving an existing meme or memeplex (someone had a good idea, then you're able to refine it)
• Making a new memeplex (a new combination of memes)
• Creating a new meme

There may well also be others that this interested amateur hasn't yet thought of :-)

One additional thought is that this also gives us some insight as to how important is it to fill your brain full of relevant memes, so that you've got more to work with.