What is Science?

I think a lot about science and the scientific method.  These are very complex topics, but the more I think about it the more I'm convinced their fundamental basis is very simple.

Science is evidence-based reasoning.

Everything else is implementation detail.

This is a very powerful thought because it's hugely general.  We have a provably unique set of mathematical rules that underpin it (probability theory), the human brain has an excellent track record at generating new scientific ideas, and we're getting better and better at generating new empirical evidence (data).

It also begs an interesting question:  If this is the essence of science, how can we improve our ability to do it effectively?

Food for thought...

Cognitive Science of Rationality

Here's an interesting post on the cognitive science of rationality.  I've only just read it so need some time to digest it, but it seems to me anyone wanting to improve themselves as a scientist should be paying attention to how they think rationally.  As with many other things, it's a skill that you can learn and improve upon.  

A mulling layer

After my previous post on my approach to research, I realised I might have missed out a layer in the process (or at least undervalued it).

The idea of a pool-of-memes approach is to collect in your mind a set of useful/relevant memes, then let them mull.  However, what I actually do is a bit more structured than that.  The pool actually has a second layer, containing ideas that have occurred to me as particularly promising.  I'll tend to actually write down these ideas and I'll come back to them periodically and consciously work on improving them.

This has the advantage of letting me refine my most promising ideas, as well as sometimes combining them into 'research arcs' (single, coherent research projects that will lead to multiple, related outcomes).  It also makes it easier to develop ideas that might have some merit but be undercooked, currently.

Having this intermediate layer is also quite useful in terms of time management.  It allows me to focus a bit more time on the ideas I think are promising, but without committing the significant chunk of full-on work time that a "small bet" requires.

Predictive power and explanatory power

Recently, I've been thinking about the goals of science. Science isn't about discovering Truth. This is for the very simple, practical reason that Truth is very hard to positively identify - how would you establish that a prospective Truth wasn't just a very good approximation? Not very helpful...

There are all sorts of philosophical discussions to be had here, but we can sidestep these and take a more practical approach. Specifically, we can choose to care about Predictive Power and Explanatory Power.

Predictive Power is the ability of a given theory to allow us to make predictions about the natural world. We know that Newtonian gravity is an approximation (to General Relativity, at the very least), but it's very good at predicting where the planets in our solar system will be. This is really a flat-out practical consideration - if a theory can't make predictions, it's not very useful (and some would argue it's not even science).

Explanatory Power is the quality of a theory that gives us some deeper understanding of what's going on in a physical system. For example, knowing about atomic electron orbitals allows us to make sense of the periodic table and chemical interactions. It gives us ways to develop other theories.

So, what we're looking for from a scientific theory is the ability to make predictions and for some explanatory insight as to why something happens, so that we can use that insight to develop further theories.

This is also relevant for statistical modeling (and hence data-intensive science), because we can build our models to address either or both of these. Predictive algorithms such as neural networks can perform very well, but the structure of the model is often hard to interpret in any kind of explanatory way. Conversely, a linear regression model might tell us a lot about which variables are important, but may not make good predictions. Ideally, it would be nice to build models that are useful for both prediction and explanation.

Data-intensive Science

I've recently been noticing a number of articles and blog posts on "data science" (or "data-intensive science") and a speculated 4th paradigm for doing science (the first three being experimentation, theory and more recently simulation).

While there is debate about precise terminologies, there is definitely a new approach to science that's being used in various fields. It looks something like this:

1 - design a big, powerful experiment that will make measurements over a whole region of scientific parameter space
2 - run the experiment, reduce/process the data and make it available to people
3 - mine the data for interesting science
4 - (optionally) follow up these discoveries with new experiments

This is already the norm in big astrophysics and particle physics projects and works very well. There's also a number of medical/biological projects going in the same direction.

So why is this happening? Short answer: because it's a good way of making new discoveries.

Longer answer: it's the result of two key drivers. One is that (in the area of concern), someone has invented one or more measurement technique that's capable of generating huge volumes of useful data. The second is that we have computers and machine learning/statistics algorithms that are capable of extracting useful information from such large data sets.

This approach has several advantages.

1 - it can be systematic (astrophysicists have discovered whole new classes of celestial object simply by surveying the whole sky to a given sensitivity)
2 - it can be statistically very powerful (sheer volume of data can give small error bars and good signal-to-noise)

3 - there's a wisdom-of-the-crowds aspect to having many scientists working on the same data set (and if the data set is rich enough, it's worth having many people working on it)


Some interesting links on this sort of thing:

Data science Venn diagram

A taxonomy of data science

Skills of data geeks

Rise of the data scientist

What is data science?

Data-intensive science

Links (philosophy of science)

I’ve recently read a few interesting posts over on the Science 2.0 blog site concerning probability theory. Well worth a read!

Can science be justified?

You, a Bayesian

Probability and induction: The very foundations of science

Science and Sherlock

t’s not often that I write a post based on a TV show. Bear with me on this.

The BBC have just started showing ‘Sherlock’, a contemporary (and very good, so far) update of the Sherlock Holmes stories of Arthur Conan Doyle. And it got me thinking about the inspirations that originally made me want to be a scientist.

Like many people who end up being scientists, I was inspired by stories of the great scientists and their discoveries. I admired Einstein and Feynmann, I used to have undergraduate lectures near where Crick and Watson figured out the structure of DNA - the list goes on.

But my primary inspiration in how to think like a scientist wasn’t a scientist. And he wasn’t even a real person. Holmes’ deductive reasoning has always struck a chord with me and it’s the best written description I know of concerning how to think like a scientist. The focus and precision of it, the attention to detail and the fact Holmes treats it as a craft to be honed.

There are many very good science texts that the aspiring scientist should read. I’d suggest that it’s also worth spending some time reading the Sherlock Holmes stories, for the simple reason that in order to be a scientist you need to think like a scientist!

The memetics of science

I've become fascinated recently by the concept of memes and the dynamics of how they evolve (memetics). I'm particularly curious about what memetics might have to tell us about how science works.

Full disclosure here is that I'm very much an interested amateur at this point - there are people who have spent many years thinking deeply about memes and memetics, and I'm not one of them. But the basic concept is kind of beautiful and not very hard to grasp, and it does give us some insights into how science works.

We should start with some definitions. A meme is a unit of information (such as an idea or concept) that's copied from person to person. The idea of memetics is that memes are subject to an evolutionary process because they are copied, a range of variants exist, and they are subject to selection pressures (some memes spread more effectively than others). So what we then have is a way of thinking about the dynamics of how (scientific) ideas evolve and develop.

It strikes me that science in particular is a memetic process where we have one additional concept: we subject our memes to the selection pressure that they must be confirmed empirically. This is an important difference. Memetics per se does not require any given meme to be true; it just has to be good at propagating. This explains why rumours that are false but appealing can spread so readily. By adding the additional constraint of empirical confirmation, we are adapting memetics in order to learn about the universe.

Thinking in this way, we can define a list of general scientific processes in which we can be involved.

• Validating an existing meme or memeplex (empirically or via theoretical proof or computational analysis)
• Improving an existing meme or memeplex (someone had a good idea, then you're able to refine it)
• Making a new memeplex (a new combination of memes)
• Creating a new meme

There may well also be others that this interested amateur hasn't yet thought of :-)

One additional thought is that this also gives us some insight as to how important is it to fill your brain full of relevant memes, so that you've got more to work with.

Rational inference in science

One of the side-effects of working in science is that you end up thinking a lot about how we reason and learn. What rules do we choose in order to do this? Do we have a range of options? This is pretty important, as it underpins everything we do as scientists (or thinkers of any kind, come to that).

The term 'Rational Inference' is a good way to describe how we do this. It covers both logical deduction (for when we have definite facts: if A and B are true, then C must also be true) and induction (for when there is uncertainty: D and E are true probably, which implies that F is also likely to be true). Rational inference is how we reason in science, whether using experimental results, observations of the universe or even (the most obvious example) mathematics, and how we learn more about laws that describe how the world works.

The fascination for me is that we have a mathematical theory that tells us how rational inference must work - probability theory. If you're comfortable with a Bayesian perspective on the subject (and are therefore happy to allow probabilities to represent states of knowledge), then probability theory contains logical deduction as a special case when all probabilities are either zero (false) or one (true). It also extends this to allow for different degrees of plausibility (0 < probability < 1), giving us the capability to represent any state of knowledge from false, through 'maybe', to true.

This also gives us an answer to the question "am I using the right set of rules to make my inferences?". The physicist Richard Cox provided a proof (using only very general starting assumptions) that any consistent mathematical rules for handling degrees of plausibility must be those of probability theory (or equivalent to them). Meaning the answer to our question is "yes!" - and we realise we are using the only set of mathematical rules for rational inference that make any sense.

So we find ourselves with a uniquely correct approach to reasoning in science (and anywhere else), along with a set of mathematical rules to use. There are philosophical considerations along the way (such as whether to adopt a Bayesian or Frequentist viewpoint) but if you're comfortable with the idea that your degrees of plausibility can include states of knowledge, you've got a unique set of mathematical rules that tell you how to make rational inferences about the world.

The Information Hierarchy

Rands In Repose posted an interesting article which included a concept called the Information Hierarchy (also known as Wisdom or Knowledge Hierarchy) which I'd not previously encountered.

The idea is this: information can be classified in a 4-level hierarchy.

• Data - the raw material of knowledge
• Information - data that have been organised/presented
• Knowledge - information that has been acquired and understood
• Wisdom - distilled and integrated knowledge and understanding

It strikes me that this gives us some insights into the process of science, especially nowadays. At the most basic scientific level, we're simply trying to gather data about the world and progress through the hierarchy to build up information, knowledge and ultimately wisdom about the world/universe/multiverse/whatever in which we live.

In the original version of this process, every stage was carried out by people. This no longer has to be the case, however. Much data gathering is now automated to at least some degree. Even if scientists are ultimately responsible for building and running the experiments/instruments, a lot of the heavy lifting is now carried out by automated or semi-automated systems, with data reduction carried out by software pipelines.

I would argue that we are also able to automate aspects of the second level of the hierarchy, the production of information. Specifically, I think one can regard statistical modeling and machine learning as doing just that. We live in an era of phenomenal scientific data production, so we now routinely use (and create) statistical methods for extracting the useful information from these giant data-sets.

So I think this begs an interesting question: I wonder how much of this process we might ultimately be able to automate, and in what ways? (and what would the implications be of automated systems capable of the Knowledge and Wisdom levels?)

Turning data into knowledge

At some level, science is about turning data into useful knowledge. When the number of data is small (and especially when the signal is strong), just looking at the data can be enough to gain new understanding. The essence of modern science however, is making this transformation with very large amounts of data. And this requires a particular set of approaches.

The blindingly obvious...
Sometimes you'll get lucky and the knowledge will be obvious from the available data. For example, you might have a scientific image with 108 pixels, but the object you're looking at is imaged to high resolution and to huge signal-to-noise ratio. From this, you can probably learn a lot without doing anything more than looking at the image (and perhaps making a few basic measurements). But it's rare that you can't learn more by more fully exploring all those pixels, and to do that you'll need some better tools.

Vanilla methods
If you do need to do something to your data in order to extract some useful knowledge, your first port of call might be "vanilla" methods. These are the bulk-standard, well-understood tools of the data analysis trade. Taking an average, finding a p-value, fitting a regression line, clustering using k-means etc. You are now into the regime where your data cannot all fit into your brain at once, so you have to start using tools to help you extract useful scientific knowledge. Vanilla methods are by definition widely used and tend to be well-understood and easy to interpret. Your aim here is to use these tools to spot the patterns in large amounts of data. Do your genes group into distinct clusters? Are there significant sources in your astronomical image? What's the most likely curve for your measurements, given the noisy measurements you've made? If you can reduce a billion data-points to a hundred clusters, a thousand point sources or a curve defined by ten parameters, you have already made a lot of progress in understanding what your data are telling you.

Clever methods
Of course, you can also try to be more clever than that. If you have a good idea of the sort of general structure you expect in your data then you can build a method that can target that type of structure. Perhaps you have a good physical model of what's going on? The power spectrum of the Cosmic Microwave Background (CMB) radiation is a good example - the major structures in this curve are well-defined by the underlying physics.

You can try to build clever methods that do more of the donkey work for you. Are you clustering your data? Into how many clusters should you be dividing the data? The right clever method can apply a robust principle to determine this optimally, leaving you free to consider the results in greater detail.

You can also build very general clever methods that are capable of spotting a large range of different types of structure (for example, Bayesian non-parametric techniques, and splines for curve fitting). Care must be taken to not simply identify every noise spike as structure, but this can in prinicpal be a great way to spot the unexpected.

Statistical inference
All of this can be viewed as statistical inference. Inference is the extension of logical deduction to include uncertainty (because probability theory extends mathematical logic to include degrees of uncertainty). Indeed, there's a view that the scientific method is all statistical inference. With very certain observations (the sun rises every morning), we are just left with logical deduction. With uncertain/noisy observations, we are left with statistics, maximum likelihood techniques, Bayesian methods, the need for repeated experiments and the like. And consider Occam's razor (often held up as an important part of the scientific method): probability theory actually provides a mathematical derivation of Occam's razor, via Bayesian model selection (if two models fit equally well, the simpler model will have a higher Evidence value, meaning it's more likely given the data).


Science as data compression...?
While we've drifted into philosophy-of-science territory, there's another interesting idea that's highlighted by the advent of data-sets too large for a human brain:

One could consider science as a series of attempts at data compression.

Think about this for a moment.

What are we looking for, as scientists? We're looking for generalisations about the area in which we're working. We want to know how metals behave as we heat them. We want to know how the universe expands over time. We want to know how our favourite set of genes interact with one another in different conditions. We can make a vast number of observations about any one of these, but what we're after is a set of rules that tells us how these things behave and we want those rules to be as general as possible.

Once we find (and test) such a rule, we've encoded the essence of all those observations into one (often simple) rule. Think about Newton's law of gravity; it goes a long way to describing the the motion of a hundred billion stars in our galaxy, but it's just an inverse-square law with a couple of masses and a gravitational constant. In terms of bits of information, that's a pretty awesome compression factor.

So what? Well, we're talking about the need for methods for converting large data-sets into something more interpretable by a human brain. These are compressions in themselves. So we're using algorithms/statistical methods to partially automate the scientific method. Which leads me to wonder how much more of it we could automate, if we really put our minds to it....