Rapid Research Prototyping

Does research have to be this slow?

I find the pace of a lot of research projects frustrating.  Sure, some things just take time, but I have become increasingly suspicious that a lot of bottlenecks in research are problematic simply because we haven't taken the time to find a faster way of doing something.

This can be very challenging.  Experiments and data, for example, can be a lengthy and painstaking process.  Data analysis, simulations/modeling, discussions with collaborators and even writing the paper can all take many months to complete.  And of course we try to be efficient, but should we really care so much if things take a while to get done.


Yes.


And here's why.


Consider what science is.  Science is the generation of new and improved memes for describing the natural world, whose fitness is judged using empirical evidence.  It's a memetic process.  This means that we should be thinking in terms of evolution of ideas.  And one of the key ways in which we can accelerate any evolutionary process is the shorten the generation time.  Because the sooner you get new research into the public domain, the sooner other people can benefit from it and the sooner you can get feedback.


There is also a second key point here.  Scientific ideas gain most of their value from being tested by other people.  And this can't happen until it's been released into the public domain.  We should be thinking of our newly-minted science meme as no more a than prototype, that needs to be poked and prodded by as many other people as possible before it can even start to be thought of as being robust.
The idea of spending years or decades on a scientific magnum opus is the wrong plan; getting your work into the public domain is everything.


This absolutely does not mean lowering our standards with regards to quality; there is so much research being produced nowadays that we need to avoid drowning each other in mediocre research.  But a single researcher or group can only make a science meme so good.  Beyond a certain point, your idea needs to be tested by other people.  At that point, faster is better.  Much, much better.


What form, then, should science take in the 21st century.  It should be about rapid research prototyping -  the production and dissemination of new high-quality prototype science memes, as rapidly as possible.

Publish early, publish often.  And optimise your bottlenecks.

What's the point of a scientific paper?

Academics write research papers.  It's a major way we disseminate our ideas and, increasingly, our continued career progression and funding depends upon it.  We live in an era of metrics and impact factors.

Because of this, I've been thinking recently about what a scientific paper is, what is its purpose and how we might improve upon this.

I have also been thinking recently about science as a memetic process, that is to say in terms of evolving populations of ideas.  I think this is a useful train of thought, which can give us some interesting insights into how science works and how to make it work better.

Thinking about things in this way, I have come to the conclusion that a scientific paper should contain a small number of high-quality, useful, interesting ideas.  Plus whatever evidence is required to back those ideas up.  And that's it.  They should be as compact as possible and as simple as possible.   I'd even go so far as to say that  one should be able to communicate the point of a paper in the abstract.

Nowadays, the scientific literature is very large and keeping up with the new papers even in a small area is challenging.  I skim read at least a couple of hundred abstracts a day (RSS feeds are awesome for this), and it isn't going to get any better.  But if the paper contains a small number of well-supported ideas that are well-communicated in the abstract and title, I can grasp them more easily and pick out the papers I want to read in greater detail.

I think that the point of a scientific paper is to be a communication channel for well-supported, clearly-stated scientific ideas.  And the more succinct and high signal-to-noise the better.

Refine and Simplify

Science is a complicated business nowadays.  The data sets are large, the measurement technologies are complex, the statistical methods are specialisms in their own right, as also in many cases are the code bases.

It's pretty easy to feel overwhelmed by all this.  And I also find that a lot of effort has to be put into guarding against errors and misunderstandings in the work you're doing.  This vexes me at some level, so I spend time thinking about how to improve the situation.  This is tricky because at some level, the complexity is somewhat innate.  But I do think one can adopt a strategy that helps to some degree.

Refine and simplify everything.

What I mean by this is actively (even obsessively) try to refine and simplify every aspect of the work you do.  Make your next paper more succinct.  Try to come up with a small number of very clear, easy-to-express conclusions.  Make your code more compact and tidier.  Refine your statistical algorithms so they're not so arcane.  Make your data processing procedures as simple and clear as possible.

I think this could be hugely beneficial.  Papers would be easier to read and digest.  Conclusions would be easier to communicate and develop.  Code would be less buggy and easier to work with.  Statistical algorithms and data processing pipelines would be faster.

And I think clear, concise scientific ideas are vital.  Think about the great ideas in science.  There's a purity and beauty to them.  So perhaps we should be actively trying to refine and improve the clarity of the ideas we work on, just in case they turn out to be really important.

Scaling your research productivity

Productivity is important in research.  Ultimately, you'll be judged on the importance, quality (and number) of papers that you publish.  This got me thinking about whether there was a simple way of encapsulating this and I came up with the following:

Aim to do more and more important research per unit-time.

It's sort of obvious, but the key quantity is the rate at which you produce important, high-quality research. If you hardly ever publish anything, that's a bad thing.  If you publish plenty, but all your papers are low-grade rubbish, that's a bad thing too.  And even if you produce large numbers of high (technical) quality papers, but they're all studying unimportant problems, that's not great.

Cognitive Science of Rationality

Here's an interesting post on the cognitive science of rationality.  I've only just read it so need some time to digest it, but it seems to me anyone wanting to improve themselves as a scientist should be paying attention to how they think rationally.  As with many other things, it's a skill that you can learn and improve upon.  

A mulling layer

After my previous post on my approach to research, I realised I might have missed out a layer in the process (or at least undervalued it).

The idea of a pool-of-memes approach is to collect in your mind a set of useful/relevant memes, then let them mull.  However, what I actually do is a bit more structured than that.  The pool actually has a second layer, containing ideas that have occurred to me as particularly promising.  I'll tend to actually write down these ideas and I'll come back to them periodically and consciously work on improving them.

This has the advantage of letting me refine my most promising ideas, as well as sometimes combining them into 'research arcs' (single, coherent research projects that will lead to multiple, related outcomes).  It also makes it easier to develop ideas that might have some merit but be undercooked, currently.

Having this intermediate layer is also quite useful in terms of time management.  It allows me to focus a bit more time on the ideas I think are promising, but without committing the significant chunk of full-on work time that a "small bet" requires.

a 'Pool-of-Memes' approach to research

I tend to spend a fair amount of time thinking about how best to go about research. My rationale is that simply working harder isn't scalable - you can't work more than 24 hours a day (and indeed only a lot less than that in a sustainable way). Therefore to be a better scientific researcher, I need to find ways to improve my approach to research.

Study Hack's excellent article on his research system got me thinking about how I'd describe my own system of research. I think the phrase "pool of memes" fits pretty well.

I try to fill my mind with as many interesting/relevant ideas and concepts as possible. This means both being well read in my own subjects, and also hunting out other subject areas that might add something. For example, for the last year or two I've been becoming increasingly interested in computer science. I've found my most productive phases correspond to learning a new set of relevant ideas.

To this pool, I also try to add clear ideas about what questions are important in various areas of research.

Then I just sort of let all these ideas mull. I might think about something in an idle moment at the gym, or I might head to a coffee shop with my log book and tinker with some thoughts.

What I get from this is a list of possible projects on which to work. This list tends to be pretty organic and it evolves over time. I rank the list in terms of how good/important I think they are. And then I try out the top ones.

What I've not had previously, but what I'm just starting to add, is a stage like Study Hack's "small bets". The idea here is to try out the possible good projects for a month or so, with the aim of producing some concrete evidence as to whether or not to take the any further. I'm not very comfortable on a personal level with the idea of discarding projects like this (I don't like the waste), but objectively it makes a great deal of sense, so I may just need to get over myself

:-)

As much as anything, this approach works well for me because I really enjoy learning new things, so giving myself the justification for doing that during work time is nice :-)

Science and Sherlock

t’s not often that I write a post based on a TV show. Bear with me on this.

The BBC have just started showing ‘Sherlock’, a contemporary (and very good, so far) update of the Sherlock Holmes stories of Arthur Conan Doyle. And it got me thinking about the inspirations that originally made me want to be a scientist.

Like many people who end up being scientists, I was inspired by stories of the great scientists and their discoveries. I admired Einstein and Feynmann, I used to have undergraduate lectures near where Crick and Watson figured out the structure of DNA - the list goes on.

But my primary inspiration in how to think like a scientist wasn’t a scientist. And he wasn’t even a real person. Holmes’ deductive reasoning has always struck a chord with me and it’s the best written description I know of concerning how to think like a scientist. The focus and precision of it, the attention to detail and the fact Holmes treats it as a craft to be honed.

There are many very good science texts that the aspiring scientist should read. I’d suggest that it’s also worth spending some time reading the Sherlock Holmes stories, for the simple reason that in order to be a scientist you need to think like a scientist!

Research flexibility

(image by FeatheredTar)

How flexible should you be in your research?

This is one of those questions that I suspect many researchers never (or rarely) ask themselves; often, one can simply progress incrementally through a research career, going where the interesting work is. And that's fine, but I for one think tthat there are benefits to be gained from some strategic thinking in this area.

What I mean by flexible is how willing should you be to move on to new projects, areas of research and entirely new subjects.

By being flexible, there are a number of advantages.

• You can follow the latest hot topics
• Your interests may develop over time
• The interdisciplinary effect (wider skill-set, acting as a vector to transfer ideas from one subject to another)

Of course there are also some downsides.

• You'll need to build up new domain-specific knowledge
• You'll need to build new collaboration networks
• You'll need to build a reputation in th new subject area

I imagine there isn't a unique answer to this question, but I'm also sure that it's valuable to spend some time thinking about thius: Is your research optimally flexible?

How to pick the projects you work on

(image by Jan Tik)
I recently read a very thought-provoking article about the process of picking research projects to work on. This is a topic that's very important, yet is easily overlooked. So I thought I would post my thoughts.

What should our aim be in picking research projects?

It's important (perhaps even vital) to pick projects that are interesting to you. Not only is this sensible on a personal level (why would you want to work on things you don't find interesting?), but it's also hugely important on an academic level; if you can't enthuse, immerse yourself, even obssess about a particular project, you'll struggle to gain the very deepest levels of insight and your results will be less good because of it.

Research that addresses important questions should be our second aim. Imagine that you have a miracle year of research where everything you work on turns to metaphorical gold. Wouldn't you rather this effort went into curing cancer, producing a working theory of quantum gravity or solving climate change, rather than some minutae of an obscure branch of your subject? I've put interesting and important in this order, but I think the key point here is that you want to work on projects that are both.

These two considerations ought to be enough. Sadly, there are also practical considerations because of the realities of building a research career. It's probably prudent to work on at least some projects that will help you secure future funding and/or jobs. This is a tricky topic, especially if you're on fixed term funding (such as a postdoc) and you have a very limited amount of time before you need to find more funding from somewhere (and you might be employed to work on a specific project). Of course, if you're working on important areas of research then it should be a lot easier to sell yourself. But you need to make sure that the projects you're working on will produce some publishable results and material on which you can talk at conferences. Even one great piece of interesting work can have a huge impact here.

At the risk of a sweeping generalisation, many researchers end up working on safe-but-slightly-uninspiring projects. These types of projects can produce a steady stream of publications and to be fair they do often have some incremental scientific value, but I think it's a huge mistake to only work in this way. Our profession is one of creativity and knowledge discovery, so we should spend a proportion of our time working on ideas that are speculative, exploring new intellectual territory. Of course, many of these won't come to anything, but the occasional one that does might have a huge impact. There are scientists who have built stellar careers and created whole new disciplines with one (really, really good) idea.

And what do I think? I think that a deep fascination with your research is vital. Within that, pick the projects that are likely to be important (in both your and other people's opinions) - you might as well work on things that might have some impact. And beyond that, try to build a good CV but if you're doing the first two things well, this shouldn't be a problem.

The pressure to publish...

The modern academic faces a lot of pressure to be productive, especially to publish papers.

There are pros and cons to this. In the "good old days" (I'm told), academics gained a faculty position and then were left to their own research devices for the next few decades. This is great for truly creative research (so-called "blue sky" thinking), because you can focus exclusively on the problem, letting it develop and exploring its various facets without spending time/effort producing incremental publications. Or course, it may also have allowed some academics to coast.

I'm a strong believer that there's a lot of benefit to academic creativity. What we do is intrinsically creative and creativity needs a bit of scope to explore new ideas, without having to worry if they'll turn into a paper or a grant proposal. But I think there's also a pretty good case for a balance. After all, if an academic spends their whole career deep in thought and never writes a word of it down, their research hasn't been useful to anyone. So, the question becomes this: what balance should we strike between productivity and creativity? Between writing papers and trying out new ideas.

To some degree, this is a trade-off between quality and quantity. The academic that publishes all the time runs the risk of writing papers that have very little important content. There are lots of academic papers that get churned out that have some limited merit, but that exist mainly because the authors felt the pressure to publish. On the other hand, the academic who hardly ever publishes should (hopefully) write papers with lots of great content. Just not very many of them.

There is also a subtlety to this trade-off. While publishing more frequently will tend to mean less research goes into each paper, it does mean that you'll get more rapid feedback on your work (from referees and readers). This is important because it crowd-sources your research, getting a whole range of suggestions and criticisms that will help improve and inform the next stage of your work. Research is actually very incremental (think about how your projects progress on a day-to-day basis), so this can be really beneficial.

And of course it can be argued that the funder (the UK tax payer, in my case) has the right to expect some kind of return for their investment. I think this is fair enough, but I think a lot of care has to be taken in how one defines this return. Number of papers is almost certainly a terrible measure (who cares if an academic writes 50 papers if none of them have any lasting impact). Maybe there has to be a degree of trust between funder and academic?

My gut feeling is that one awesome lead-author paper per year is what we should be aiming for. If you generate enough research for more, great. But one really great paper per year where you're the lead researcher seems to me to be a good level. This should give you enough time to try ideas out and develop new projects, while also building a good publication record over time. If you're like me, you'll also spend a fair amount of time contributing to projects where someone else will be lead author on the papers; this is valuable and you should end up being a co-author on papers as a result.

So the message of this post is to strike a balance. Whatever the rights and wrongs of productivity versus creativity, you need to publish papers to build an academic career. And I really do mean it about the 'awesome' bit. Would you rather be known as the researcher who's produced half a dozen fantastic lead-author papers, or the one who has written fifteen that are deeply uninteresting?

Sleeping on the problem...

Some problems are difficult to solve. You can spend all day trying to find a solution and only end up with a list of approaches that don't work, plus a strong need for the alcoholic beverage of your choice. One possible way forward is strangely counter-intuitive: stop working on the problem.

Not for ever (obviously). But long enough to take a break and give your brain a rest. If you've been beating your head against a metaphorical brick wall all day (or week. or month...) and still haven't solved your problem, it's unlikely that a bit of a break will slow you down much. And it might just help. It can be as short a break as stopping for a cup of tea/coffee. You could leave the task until tomorrow, so your brain gets the night off. Or you could even leave the project for weeks (or more), if you really want some separation.

This has a number of advantages. Firstly, it gives your poor overworked (and often frustrated) brain a rest. But there's also a more subtle effect. There are psychological studies (and apologies that I don't have the references to hand) that suggest that problems can become more easily solved if you take a break. The suggestion is that your brain continues to work on the problem subconsciously while you are doing something else, so that when you return to it you might have new insights that your (now-rested) brain can work with. Anecdotally, I can relate to this. I hate leaving a problem unsolved, but sometimes when I've forced myself to leave it until tomorrow, I find a fresh approach and a good night's sleep lead to me solving the problem very quickly the next day.

There are lots of examples where this can be useful. Trying to understand the meaning of your latest set of experimental results. Solving a particularly knotty piece of mathematics. Finding that invisible bug in your code. Or how best to write that troublesome paragraph in your latest paper.

Happily, this strategy is easy to try (provided you can exercise a little willpower to let go temporarily of the problem that's been bugging you). So the next time you're wrestling with a problem and find you're not winning, consider giving up for a while. Take a break. Eat lunch. Get a good night's sleep. Then come back and see if you can't solve the problem.

Why do you work on the science you work on?

What was the decision-making process? How did you make the choice?

Early in my research career, I discovered that I loved working with statistical inference and building the software to do it. I had chosen my PhD because I knew I was interested in astrophysics and I was offered a place at a good department to work on observational cosmology (which I found and still find fascinating). Gradually over the years my horizons have broadened, to the point where I now work mainly on medical and biological data. But the key point is that I'm using statistical inference and programming to do science.

I certainly didn't see this path coming - it evolved.

I suspect this is true for a lot (maybe even most) scientists. Maybe the subject was something that interested them during their degree. Or perhaps it's what they're trained for, given the choice of undergraduate degree they made. That's sobering - the choice you made aged 17 can define your entire career. Anyone else want a 17-year-old picking your career for you? Thought not...

My guess is that many people aren't sure what to do next. They enjoyed their undergraduate degree and therefore (quite reasonably) decided to do a masters in the same or a similar subject. That also turned out to be interesting and, still lacking inspiration as to a career direction, a PhD beckoned (perhaps they were even offered a place by their Msc supervisor, making it an easy option). Suddenly, they're in their mid-twenties, have a doctorate and almost a decade of training in an academic subject. Sounds like a good basis for academia, so off they trot.

Some people want to stay at the same university and this affects their choice of subject (I'll admit to this a little bit). Given a choice of several interesting topics, they take the one that also allows them to stay where they want to be.

Perhaps the subject in question was/is an up-and-coming area with the prospect of lots of interesting science to work on and important problems to tackle. This seems like a not unreasonable consideration.

And of course some people have a burning passion for the subject (something which strikes me as a very good reason indeed!).

As you progress through your career, you'll learn more about what your chosen subject is really like. Are the scientific challenges important? Is it well-funded? What do you really enjoy doing on a day-to-day basis? And do you try to change things in response to this knowledge? What if after five years in one field, you realise that another field might suit you better for whatever reason. Would you change?

I once read a suggestion that in life you shouldn't pick a good destination, but rather focus on a good direction in which to go. The point is that as you live and experience life and learn from it, you'll be better able to make future decisions about where to go. If you pick your path through life now and stick to it, you'll have to turn down all those unexpected opportunities that occur tomorrow. And the You of 5 years time is probably a better judge of where you should be going at that point than you are right now; so why not defer to your (future) superior judgement?

And the point of all this? Think about why you work on the things you work on. And be willing to be flexible. Even if your current plan is a good one, you might happen upon an even better one tomorrow!

The point of clever methods

The phrase "clever methods" is my label for statistical methods and/or algorithms that go beyond basic and/or standard approaches (which I think of as "vanilla methods"). Those of us whose research involves methodological work aim to write papers that detail new clever methods. Clever methods aim to go beyond the capabilities of the relevant vanilla methods in some meaningful way, as well as hopefully doing so without becoming intractably complicated or slow to run. In the same way that software engineers might be trying to craft better software for a given task, in researching clever methods we're trying to find better mathematical/statistical/algorithmic ways of doing something.

(By way of full disclosure, I should mention that I'm a fan of clever methods in that I really enjoy working on them and finding cunning and sneaky ways to make a method work better. This is great for motivation, but comes with the health warning to be careful to not make something more complicated just for the sake of it :-) )

Performance vs. complexity
Clever methods tend to be more complex than the equivalent simple methods. So, our goal in researching clever methods is often an attempt to trade off complexity for performance. The trick then becomes to minimise the increase in complexity, while maximising the improvement in performance. This is the benefit that most vanilla method possess; they provide reasonable performance in a very uncomplicated way.

In many case where reasonable performance is all we need, this is a very good solution. For example, if you're trying to detect local stars in an astronomical image, the signal-to-noise ratio of your image might be very high. In which case, even a basic method should be able to detect them all with little problem, meaning that such a choice will do everything you require and do so in a simple and easy-to-understand way (which is often a hidden benefit of vanilla methods).

When creating clever methods, it's very easy to ignore the complexity aspect and simply go all-out for performance (there are many, many papers for which this is true). While this can be okay if performance is so vital (relative to handling the complexity), usually this leads to methods that are so narrow in their application that they're not very useful.

Happily, there are also many cases where a little extra complexity gives you a significantly better method with which to work. And there can be other benefits; for example, if you generalise a vanilla method, the resulting clever method will probably be more complex but it may also be more reliable or allow the automation of some parts of its use. Consider a clustering method that has a well-defined, automated way for choosing the number of clusters into which to partition the data; the user no longer has to worry about doing this, thus saving them time. They probably don't care that the underlying maths is more complicated.

And of course very occasionally, you'll manage to create a clever method that's no more complex (or in extreme cases, less complex) than the simple method/s. Congratulations, you've probably discovered something genuinely importance!

The 10% improvement
Often, clever methods can provide and order 10% improvement (in whatever metric is important) over the simple method. The question then becomes, "is this worth the effort?"

The answer is "it depends". If you're trying to extract a signal from some noise, but the signal-to-noise ratio (SNR) is already 105 then increasing it by 10% may well be irrelevant. If on the other hand you're trying to detect signals right at the detection limit of your data, then it might be vital in uncovering that Nobel-winning new class of whatever. I've worked on astronomical source extraction where the data-set has had an effective cost of tens of millions of pounds (actually quite common when the data come from a space telescope). In this case (and assuming Gaussian noise), a 10% improvement in SNR using the simple extraction methods would require 20% extra data, at a cost of millions of pounds. Or you can just use the clever extraction algorithm.

One very important consideration in all of this is that if you develop a clever method that is reasonably general, then that 10% improvement will be a benefit many, many times.

The undiscovered country
So far I've focused on the mundane benefits of clever methods. There is also another aspect to consider. If your method of analysis is too simple, you might miss something important.

Think about a very rich, complex data-set where it's not obvious how to model the data. Gene expression measurements of whole genomes are a good example. We can certainly use simple methods to analyse this and to get some useful scientific results. But what if there is structure in the data to which our choice of method is insensitive? Imagine what would happen if you only fitted your data with straight lines! You'd miss peaks, troughs, oscillations and all manner of other interesting structure in your data. Your methods need to be able to account for all the interesting structure in given data-set and if that structure is complex, a vanilla method may well miss it.

A related point is that a good way of spotting complex patterns can be to use your eyes to look at the data. There are many examples where the best signal detection method is a person (eg. objects in an image, CAPTCHAs). But this doesn't work if your data-set is too big for a person to meaningfully do this. In this case, you need a clever method.

Clever methods as their own research discipline
There is justification for creating clever methods simply because doing so adds to the sum total of human knowledge. This is especially useful when that clever method extends an existing method and/or when it can be further built upon by you or other people. Whole new areas of methodology can be uncovered in this way, whether through being created or through making some existing ideas more widely known. And often reading a clever idea in one context can spark a thought in someone's mind about their own area of research (this is why it's very important to be well-read as a researcher).

If, like me, your research involves creating new clever methods, a burden of proof falls to you. Because there are infinitely many clever methods one could create, it's important to find the ones that are actually useful (defined as having superior performance to the vanilla methods, at the very least). And this means that you need to test your methods and compare them to other existing ones. This is actually one of the real tricks of methodological research; figuring out as many ways as you can to test a new method, to see if it's worth using. A few things I think are really important for this include:

• Test in many different ways
• Test on many different data-sets
• Test using many different metrics
• Testing on synthetic data can be good because you know the right answer
• Testing on real data is very important; real data will always contain more junk than synthetic data
• Real data where you know the right answer (eg. from some other source of information) are wonderful to have
• Realistically simulated data can be very useful. But it takes a lot of effort to build a software/hardware simulation of most types of data
  
• Use your methods. Do some science with them (or help other people to do so), because in the process you'll learn more about how the methods work and how to improve them

In conclusion...
The creation of clever methods is a craft, a balancing act between performance and complexity. But the right method in the right context can be a powerful solution and even open up whole new areas of research.

Do you work on important problems?

I recently read the transcript of a talk given by Richard Hamming and realised many of us may have been missing a trick. He makes the point that we should aim to work on the important problems in our field. This is one of those ideas that seems obvious once you read it, but is somehow easy to overlook during the bustle of day-to-day research.

Why is it important to work on important problems?
Imagine you have a miracle year of work. You're in the zone 100% of the time, every hunch you have turns out to be right and every project you touch turns to gold. Now consider the difference in the impact of your work depending on whether you had been working on important problems or unimportant ones. In the first case, you might have done truly great, maybe Nobel-worthy work. In the second case, you've still done good work but it's not going to change the world. Which would you rather have happen?

What if you work on unimportant problems?
I'm not suggesting that you should exclusively identify important problems and only work on them. After all, you can't be 100% sure your list of "important" is complete and/or completely accurate. However, if all you work on are problems that are unimportant, by definition you limit the impact and value that your work will ever have. Unimportant problems are just that. By all means spend a bit of time tinkering with such problems if they really interest you, but don't waste your career on them.

What are the important problems in your field?
This really boils down to being able to identify what the important problems are in your field/s. This is more difficult than one might imagine, but with some time and thought you can make some headway. Take time to think about what you consider the important problems to be. Read some articles for inspiration. See if any other academics in your area have posted on this topic on the Web. And ask people! A great question over coffee or at a conference dinner is, "What do you think the important problems are in our field?".

Some of these problems will be intractable. Finding an exact O(n) solution to the travelling salesman problem would be awesome, but seems unlikely. And formulating a Grand Unified Theory of physics would be nice, but many people have tried and no-one has just succeeded. It can be tricky to distinguish between problems that are difficult and ones that are (or are likely to be) impossible to solve. There are judgement calls to be made here as to how much time to spend on each problem.

And then there are the problems about which we're uncertain. Is it important or not? Again, this is a judgement call. If the problem is interesting to you, plus you think you can make good progress on it quite quickly, then it's probably worth working on just in case it's important.

Nowhere so far have I mentioned a couple of other considerations that I think are also important. You should probably work on problems that inspire/enthuse you. And you should work on problems to which you're suited, in terms of abilities, skills andtemperament. If you're no mathematician, you shouldn't be trying to work on the Reimann hypothesis. And if you thrive on a sense of rapid progress and individuality, perhaps it's not such a great idea to work on that huge physics project that won't begin generating data for 5 more years.

In conclusion...
It's not the only consideration, but deciding on the important problems in your field and working on them is a pretty good starting point for scientific research.