Translational Medicine with Data Science

I've been thinking a lot recently about how to use all the medical data that are being generated to actually help patients.  We can (and do) measure a lot of medically-relevant things about a given patient, but that doesn't automatically lead to an improvement in care or outcome.

So, the question is:  What are the challenges to using data science to improve medical treatment?

Small Data

Many medical data sets end up  being small to the point of being underpowered.  Reasons for this include the cost of collecting samples and making the measurements, and practical considerations of recruiting patients to the study.  This misses a trick, both because they can be underpowered, and also because of the 'unreasonable effectiveness of data' (the observation that larger training sets can lead to significant improvement in predictive ability).  This is even more of an issue with many new data types, many of which are very high-dimensional.

The Right Patient Cohort?

Does the cohort of your training data match the cohort of people who you'll actually want to help, clinically?  The answer to this is often 'no', and this may seriously affect the generalisation of your model. Is it possible to use transfer learning to help offset this effect?

NHS-Quality Data?

Data from research labs is not enough!  The quality can be too low, and will typically not be certified to NHS standards.  Trust in data quality is vital, but also your clever data science method will not be
allowed into clinical practice if the data aren't NHS-certified.

Changes in Measurement Technology

We are now in an era where medical measurement technologies are developing very rapidly.  For example, if you have a method for cancer diagnosis that uses next-generation sequencing technology, will it still work with whatever platform is being used in 5 years time?  We need to learn how to future-proof, as much as possible.

Do You Trust Your Models?

If one is using a data science method to help treat someone medically, the price for the method failing could be very high (possibly even death). Therefore, do you *really* trust your model?  Is the parameter fitting robust?  It the model robust to poor data (outliers etc).

Do You Trust Your Code?

Similarly, do you trust your code?  Really?  Because a bug in your code in this context could kill someone.  There are other disciplines that have expertise in this kind of thing (e.g. control systems for
aircraft), but this is a huge challenge for medical data science.

Data Integration

An open methodological question:  how best to combine multiple sources of information, in e.g. statistically principled ways.  Medicine is now routinely producing multiple data types for many patients, so we need reliable (automated) ways to get the most out of these.

Covariate Drift

Our goalposts will move over time.  For example, if we're screening for the early detection of a disease, the more successful we are, the more the at-risk cohort will change over time.  Can our models adapt to account for this?

Ongoing Data Aggregation

As we use a medical data science system, it will aggregate more data (i.e. old test examples can subsequently be added to the training set). Can we use these data to improve e.g. the predictive ability of our models?

Are You Solving The Right Problems?

Are we solving problems that will actually improve clinical practice. It's all very well to develop a statistical model that can predict which cancer patients will die sooner, but if one has no treatments
available which can change this, there is no value to the prediction.


There are a lot of challenges that we've barely begun to address, when it comes to the idea of getting data science methods into clinical practice.  But at the same time, there are so many potential benefits
that it is well worth the effort.  There's much to be done...

Big Data in Cancer

We recently organised a workshop on 'Big Data in Cancer', as part of a year-long programme to launch Warwick's new Data Science Institute.  It was a fascinating day, with four brilliant speakers (Florian Markowetz, Andrew Teschendorff, Paul Moss, and Sean Grimmond) and covered a lot of important topics.

Florian was first to speak, and led off with some conceptual and philosophical points about Big Data, which I think are hugely important.  There is, of course, a lot of hype surrounding Big Data and what it's capable of.  The more 'enthusiastic' supporters even argue that it makes the scientific method obsolete.  I share Florian's view that this is kind of silly; no matter how much data you have, there is still a difference between correlation and establishing a causal link, for example.  And patterns in data are not the same as establishing underlying general rules.  Yes, you can map out a rule using data if you have enough of it, but wouldn't you rather just have a simple formula?  Rather, Big Data is a technical challenge (how do we handle it), with the pay-off being smaller variance in our estimates, more detail about the things we are studying and the like.  For example, getting a much more detailed description of the heterogeneity and evolutionary history of a given tumour.

Andrew told us about some of his work in epigenomics.  This is a fascinating topic that I'm trying to learn more about, but is seems that tumours come with an explosion of epigenetic modification throughout the genome, which contains potentially all sorts of information that may allow us to diagnose cancer, its type, and its likely progression.  The early detection of cancer is a hugely important area of translational research, which makes this doubly exciting.

Paul gave us a whistle-stop tour of some areas of cancer research.  He talked a lot about the potential for electronic health records and Health Episode Statistics data, something in which the Queen Elizabeth Hospital in Birmingham is really leading the way, with a serious informatics infrastucture. To my mind, this is part of a coming revolution in healthcare where all relevant medical data are stored electronically in integrated databases, which turns medical research into a software/algorithms problem.  Evidence from other areas of human endeavour suggest that this will be a huge driver of the pace of innovation.

Finally, we were treated to a great talk by Sean, who's heavily involved in the International Cancer Genome Consortium and has recently moved to the UK to pursue the translational aspects of genomic medicine (note: the NHS and the UK's science base makes us world-leading in this area, and this means we can attract the world's best researchers).  I was really grabbed by just how rapidly genomic medicine is scaling, in terms of data volume.  The cost to sequence a whole human genome has dropped from ~$100m (2001) to ~$1000 (2014), an incredible 5 orders of magnitude in 13 years (think about that for a moment...).  With the UK's 100,000 genome project already underway, the US recently announcing a 1,000,000 genome project, and China undertaking a similar scale project, we are just about to be awash in genomic data.

The limiting factor in translational cancer research is about to become our ability to effectively handle and use the flood of data that we're just seeing the first trickles of.  This is a hugely exciting time to be involved in cancer research, but we all might need to buy bigger computers...

Big Data - in the process of growing up

A very good article was published in the FT online a few days ago.  Its title is 'Big Data: are we making a big mistake', and it's a commendably well thought out discussion of some of the challenges of Big Data.  And perhaps a bit of a warning to not get too carried away :-)

You should go and read the full article, because it's got loads of great stuff in it.  A few of the concepts that leapt out at me in particular were the following.

One of the interesting things about modeling data in order to make predictions (as opposed to explain the data), is that features that are correlated to (but not causal for) the outcome of interest are still useful. But, the FT article makes the really good point that even in this case, correlations can be more fragile than genuinely causal features.  This is because while a cause is likely to remain a cause, correlations can more easily change over time (covariate drift).  This doesn't mean we can't use correlation as a way to inform predictions, but it does mean that we need to be much more careful and be aware that the correlations may change.

The article also discusses sample variance and sample bias.  This is in many ways the crux of the matter for Big Data.  In principle, very large data sets offer us the chance to drive sample variance towards zero.  But it really has much less to offer in terms of sample bias, and indeed (as the article points out) many of largest data sets, because of the way they're generated, are actually very vulnerable to high levels of sample bias.  This is not to say that one can't have both (the large particle physics and astrophysics data sets are great examples where both sample variance and sample bias are addressed very seriously), but it is a warning that just because your data set is huge, it doesn't mean that is is free from sample bias.  Far from it.

I've felt for a while that 'Big Data' (and data science) are currently going through a rapid phase of growing up, which I suppose is pretty inevitable because they're in many ways new disciplines.  They've gotten up to speed on the algorithmic/computer science side of things very rapidly, but are still very much in the process of learning many of the lessons that are well-known to the statistics community.  I think this is just a transition phase (these things take a bit of time), but it seems clear that a big part of the maturation of Big Data/data science lies in getting fully up to speed on the knowledge and skills of statistics.

The DREAM Toxicogenetics Challenge

So, we entered a team (WarwickDataScience) into the DREAM Toxicogenetics Challenge and we've managed to win the leaderboard stage!  It's a really interesting (and challenging) problem, so we're really pleased :-)

I wrote a blog post detailing our approach to the modelling, which can be found here.

Fixing modern healthcare?

(caveat: this post post will be fairly UK-centric, because that's the healthcare system with which I'm familiar!)

Modern healthcare is the victim of its own success.  We've been so successful over the last few decades at developing amazing new medical treatments that healthcare systems can no longer afford to give everyone every single treatment that might help their ailments.  

Now, at some level this is a glass-is-half-empty view of the situation.  Just because we can't afford to use every single viable treatment in every single case doesn't mean we're in a bad way.  Quite the contrary.  We live in an age where people are routinely cured of diseases that were fatal a few short decades ago, and modern treatments improve the lives of millions who would previously simply have to suffer their ongoing conditions.

However, this is no reason to not be ambitious.  We also live in an age of huge innovation, where information technologies such as computers and the internet are radically changing and improving many areas of life.  Why should healthcare be any different?

Here in the UK, the NHS has within it a body called the National Institute for Clinical Excellence (NICE).  Part of NICE's job is to assess the cost effectiveness of different medical treatments.  Because the NHS has finite resources to be shared across the UK population, this is very important - if money is wasted in one area, it's to the detriment of people elsewhere.  Getting the best overall medical return per unit cost is therefore key.  But this begs an interesting question:

What if certain medical costs dropped to zero?

There would be an immediate knock-on effect that the saved money could be used elsewhere in the NHS to treat people (in an insurance-based healthcare system, I suppose what should happen is that insurance costs come down, so more people can afford it).  There's also a more subtle consequence, depending on what the cost in question is - for example, if the cost to test for a given disease drops to zero, we can test everyone for it, as often as we like.  If the disease in question is cancer, say, early detection will also hugely improve survival rates and make the required treatments a lot cheaper (you need a lot less chemo- and radiotherapy if your tumour is small and unmetastasised to begin with).

That would be great, but why on Earth might we expect the cost of anything to drop to zero?  Well, by analogy to what's happened in other industries that have been radically changed by information technologies. The minimum costs to effectively market and distribute books, music and video have dropped essentially to zero - this is why people can self-publish books, release albums without needing a record label, or run entire TV channels over YouTube.  Scientific journals can now be run entirely online, eliminating the need to spend money on marketing or distributing hard copies, leading to new journals that can be run at greatly reduced costs.  And modern search engines let us, for free, efficiently access a reasonable approximation of the sum total of human knowledge.

In medicine, there are a couple of obvious trends that are likely to radically reduce certain costs.  The first is genome sequencing.  The first time we sequenced a human genome, it cost several billion dollars and took a decade and a half.  Currently, it costs about $1000 and takes maybe a week.  Not bad progress considering the (more or less) complete first human genome was only announced 12 years ago!  Post-genomic medicine is taking a little time to arrive, but arriving it is.  And who knows how huge the impact will be when we can routinely sequence the DNA of not only every person, but every pathogen that they have.  Immediately.  For minimal cost.

The second trend is machine learning.  Medical diagnostics and prognostics can be very expensive.  But what if we can use all the easily available information that we have on a given patient (from their history, previous medical notes, and cheap-to-administer measurements) to spot diseases early and to identify which treatments are likely to be most effective?  There may be a whole range of diseases for which we can test at essentially no cost, if we marshal the information appropriately and build effective algorithms to analyse the data.  And many diseases have a range of possible treatments available - what if we can use the information we know about a given patient to accurately predict which treatment will work best for them.   This is personalised medicine, but it could also be very cost-effective medicine.

Of course, this is all pretty speculative.  But maybe it's something on which we should be focusing.  Anything medical that has negligible cost is something from which everyone in the world can benefit, as well as freeing up resources to help elsewhere.  And this is a pretty worthwhile goal.

The Sage Bionetworks - DREAM Breast Cancer Prognosis Challenge

I've been competing in the  Sage Bionetworks/DREAM Breast Cancer Prognosis Challenge.   The submission deadline was a few hours ago (early start for those of us in the UK!), so I thought now was a good time to share some of my thoughts on what has been a very interesting experience.  I enjoyed it a lot and I think the folks at Sage Bionetworks are really onto something with this as a concept.

The goal of the challenge is to develop machine learning models that can predict survival in breast cancer.  We've been given access to a remotely-hosted R system on which to develop our models, and (on said system) use of molecular and clinical data from the Metabric study of breast cancer.  We run our models on this remote system and they're scored using concordance index, a nonparametric statistic for survival analysis that is sensitive to the ranking of predictions.

What makes this challenge a bit different is that it's both competitive and also collaborative.  Not only are we competing against one another to get the best-performing model, but once someone has submitted a model, I can download it and inspect their code to see how it works!  This is very ambitious (and certainly not without its issues), but aims to create a hybrid competition/crowdsourcing approach that can produce very strong solutions to the scientific goal of interest.

Having put a lot of hours into working on this challenge over the last few months, I have developed some opinions on it.  So, in no particular order, here are my thoughts on the challenge:

1. Incentives for academics.  In addition to some small financial prizes along the way, the big prize on offer is co-authorship on a journal paper.  This is a very big incentive for academics (such as myself) who want to compete in a challenge like this.  I'm very enthusiastic about the whole concept and would love to join in with future challenges.  However, in order to justify spending my work time on it, there needs to be some kind of academic return.  Co-authorship fits the bill nicely!  Currently, I think the top couple of teams (?) get this prize, but I think extending this would be a good plan.  Certainly, my experience in this challenge is that there are many more than 2 academic teams who have contributed significantly to the success of the challenge.
2. Incentives for non-academics.  Of course, it's also important to have rewards on offer for non-academics.  The real strength of such a challenge comes from having a diverse community of competitors.  I presume the small financial prizes are nice in this regard; it'd be really interesting to hear from some of the non-academic competitors what their views are on this.
3. Sharing of code.  This has been a very innovative (and brave!) aspect of the challenge.  I don't think the organisers quite nailed every aspect of it, but I think the general approach is very powerful and certainly worth persisting with.  I wonder if the sharing should be constrained in some way - perhaps code can only be accessed 48 hours after it is submitted?
4. Blitzing the leaderboard?  In this challenge we could make as many submissions as we liked to the leaderboard (of which I'm as guilty as anyone :-) ).  This worries me as it could lead to a lot of over-fitting.  Maybe in future challenges there should be a limit - say 5 submissions per day?
5. Challenge length.  In total it was approx 3 months long.  2 - 3 months feels about right to me.
6. Competitive vs. collaborative.  Another research model that's relevant here is the Polymath Project.  Essentially, one can imagine a sliding scale between competition and collaboration.  Polymath lives at one end, with things like the Netflix Prize and Kaggle competitions at the other.  This challenge lives somewhere in the middle.   I like the idea of blending the two concepts.
7. Populations of ideas vs. monoculture.  A competition is great for generating a wide range of ideas.  Once people start sharing, I expect (as happened in this challenge) the pool of ideas tends towards more of a monoculture.  
8. Building an ongoing community.  This challenge has been a great way of starting up a research community (a smart mob :-) ).  It would be great to harness this community in an ongoing basis.      

Sharing code

Sharing code means sharing ideas, and this has allowed us to benefit from each others ideas during the challenge.  I'm sure this has led to better overall results.  However, it has also has some quirks that might need tweaking.

First is the phenomenon of 'sniping'.  Someone else can spend a month developing an awesome model, but once it's been submitted to the leaderboard I can download it straight away, spend 30 minutes applying my favourite tweak and then resubmit the (possibly improved) new model, jumping above the hardworking other competitor on the leaderboard.  Of course, overall this leads to better models, which is the collective aim of the challenge.  But I think care needs to be taken to ensure that credit (and reward in general) is given where it's due.  It can be a bit dissatisfying when this happens to you!

The other consideration is that after a while of sharing models, we end up with a monoculture.  Examinations of the high-ranking models over the last couple of weeks show that almost all the models are based on those of the Attractor Team (with some chunks of my own code scattered around, I was gratified to see!).  This is probably not surprising, as the Attractor Team won both the monthly incremental prizes, but it's probably an indication that we've got about as far as we can with the challenge when this happens.  Now is probably a good time to stop :-)

So, what would I change?  I might suggest something a bit more like the following:

A possible model for future challenges

The 21st Century Scientist Speculative Future Challenge (21SFC) would look like this:

Stage 1 (initial competition) - a month long competition to top the leaderboard.  No-one can access other people's code and at the end of the month, a prize is awarded on the basis of a held-out validation set.  After the deadline, all code for Stage 1 is made available.

Stage 2 (competition/code sharing) - another month long competition to top a new leaderboard.  Everyone has access to the Stage 1 models, but Stage 2 code is either unavailable or only accessible 48 hours after is has been submitted.  At the end of the month, a prize is awarded on the basis of a held-out validation set. 

(it might be worth re-randomising the training, test, validation sets for stage 2)

Stage 3 (collaboration) - A non-competitive stage.  The aim here is to work as a team to pull together everything that has been learned, produce 1 (or a small number) of good, well-written models and to publish a paper of the results.

The author for the paper is "21SFC collaboration", with an alphabetised list of people given.  There can be different ways to qualify for authorship:

• Placing in the top-n in either stage 1 or 2
• Making significant contributions in stage 3 (the criteria for this would need to be established)

This structure uses initial competition to generate a lot of good ideas, then uses a second stage of competition to combine/evolve those ideas.  It then has a final, collaborative phase where everyone who wants to pulls everything together to produce, publish and release the code for the challenge's solution to the problem.  The challenge doesn't take too long to complete and the contributors get rewarded for their efforts in various ways.

--------

This post has turned into a long one and I hope I've communicated the intended positive tone.  I enjoyed the Sage/DREAM BCC a great deal and I think this is a hugely powerful way of getting answers to scientific problems.  I'm certainly going to take a look at whatever the next challenge is (I know there are some in the pipeline) and I would certainly recommend you doing the same.

Machine Learning that Matters

The ICML closing talk was this:

Machine learning that matters

Kind of inspiring and probably required reading for machine learners.

The Heritage Health Prize

There's been an interesting trend in recent years: the comeback of public competitions to solve problems of various types. From the X-Prizes to the now-finished Netflix prize, competing for the kudos (and cash prizes) of solving a given problem is very much of the moment.

Of particular interest to me personally is the Heritage Health Prize. Run by the website Kaggle, on behalf of the Heritage Provider Network (a Californian healthcare provider), the aim of the competition is to predict the likely number of days of hospitalisation for a range of clients, based on their historical (anonymised) medical claims data. Being someone who researches statistical machine learning and is interested in applying ML methods to medical data sets, this is a pretty cool challenge.

There are some complications to using such a competition data set to research new machine learning methods. HPN and Kaggle are rightly very cautious about the data, and there are various ethical considerations about mining medical data in this way. My understanding is that they're willing to consider case-by-case whether research arising from these data can be published. We'll see how that pans out in practice.

I'm hopeful that I can do reasonably well in the HHP, but it's also an interesting area in which to research (provided publishing papers proves possible) and I'm finding that I'm getting some useful experience with new approaches and methods that I'd not used before.

And perhaps this is the strength of these competitions. The cash prize is nice, but there are also a lot of other things the competitors can get out of it.

Data-intensive Science

I've recently been noticing a number of articles and blog posts on "data science" (or "data-intensive science") and a speculated 4th paradigm for doing science (the first three being experimentation, theory and more recently simulation).

While there is debate about precise terminologies, there is definitely a new approach to science that's being used in various fields. It looks something like this:

1 - design a big, powerful experiment that will make measurements over a whole region of scientific parameter space
2 - run the experiment, reduce/process the data and make it available to people
3 - mine the data for interesting science
4 - (optionally) follow up these discoveries with new experiments

This is already the norm in big astrophysics and particle physics projects and works very well. There's also a number of medical/biological projects going in the same direction.

So why is this happening? Short answer: because it's a good way of making new discoveries.

Longer answer: it's the result of two key drivers. One is that (in the area of concern), someone has invented one or more measurement technique that's capable of generating huge volumes of useful data. The second is that we have computers and machine learning/statistics algorithms that are capable of extracting useful information from such large data sets.

This approach has several advantages.

1 - it can be systematic (astrophysicists have discovered whole new classes of celestial object simply by surveying the whole sky to a given sensitivity)
2 - it can be statistically very powerful (sheer volume of data can give small error bars and good signal-to-noise)

3 - there's a wisdom-of-the-crowds aspect to having many scientists working on the same data set (and if the data set is rich enough, it's worth having many people working on it)


Some interesting links on this sort of thing:

Data science Venn diagram

A taxonomy of data science

Skills of data geeks

Rise of the data scientist

What is data science?

Data-intensive science

Why start a research blog?

As I've just started a research blog, I thought "Why do this?" was a sensible question to contemplate. (actually, I thought about it before I started, which seemed even more sensible...)

Sharing some thoughts...
Sometimes the best way to learn or come up with good new ideas is to talk to a fellow scientist. This is why chats over coffee and meeting up a conferences is so important (and why a few glasses of wine at a conference dinner can lead to valuable conversations). This works because of the ideas that are being shared.

There's no reason why talking should be the only medium through which to communicate in this way. And while a blog is less two-way (although please feel free to leave a comment!), it has the great advantage of being able to potentially reach a huge audience. It's difficult to have a conversation with more than a few people, and even a lecture/conference talk is unlikely to have an audience of more than a couple of hundred people (web-casts excepted). But there's nothing to stop thousands upon thousands of people reading a blog post!

Knowing your own mind
There's also a second benefit to writing one's ideas down in a blog post. It helps to clarify them. Writing something for an audience forces you to thing about what you're writing, to work it into a form that will be understandable by the reader, even to challenge your own assumptions. This can develop your own ideas and trains-of-thought in ways that simply thinking about them won't manage.

Returning the favour
Another very important reason is that I've found other people's research blogs very helpful in learning more about how to be a scientist (see myblogroll for some excellent examples). Not every blog will be useful for every reader, but it strikes me that if every researcher kept a blog where they posted their thoughts, observations and experiences, collectively that would form a great body of knowledge for other people to explore.

In conclusion...
There are several great reasons why a scientific researcher should consider keeping a research blog. And there may well be others that I've not thought of yet, but that will occur as I post on more topics. Which is itself an illustration of why research blogging is a good idea :-)