How to Find and Understand Research Papers

In the current climate, the phrase “follow the science” has become an increasingly common soundbite. But the reality is that science is a process, not a consensus, and (particularly at the cutting edge), it’s a process that initially may only be able to offer incomplete or partial answers, answers which will be updated as more work is done, more data comes in, and a clearer picture starts to emerge. In attempting to follow the science, it’s important to understand this process, to be able to grasp how it’s done and what its limitations are. This in turn will help you understand which conclusions are firmly supported by evidence, and which are speculative and likely to be challenged and changed as time passes and further studies are done. The act of publicly changing ones mind is something that has become synonymous with weakness and indecision in the current public discourse, but in science it’s important – necessary, even – to be able to discard outdated ideas which no longer fit the available data. But how do we go about judging which new results are solid and reliable enough to base big decisions on, and how do we weed out low-quality studies which make big claims but can’t back them up with scientifically sound reasoning?

The following is a guideline for how I approach reading scientific papers, both in my field and in other fields that I don’t know much about. While I’m a theoretical physicist by training, and so the majority of papers I read come from this field, I keep loosely up to date with a few non-physics fields which interest me, and I was even offered an editorial job at a prestigious scientific journal based on my ability to rapidly assess papers outside of my field1. So, while the following is my entirely subjective method, it’s one with which I have seen some success. As always, everyone is different and your mileage may vary!

This is aimed at non-specialist readers: for those who are already a specialist in your field, you’ll likely have your own approach to reading papers. (But in that case, you might be interested in my upcoming guide to writing peer review reports, which will appear over on my academic website sometime soon…!)

Where to look for information?

There are two broad classes of primary sources for scientific information, namely pre-prints and peer-reviewed articles. Pre-prints are a practice that has been common in some fields (particularly physics) for a very long time, but are gradually becoming adopted by many other fields. They are available on website such as arXiv (pronounced ‘archive’: the X represents the Greek letter chi), biorXiv, and many others. They are typically publication-ready papers that are made publically available at the same time as they are submitted to a journal for peer review, which can be a lengthy process. The practice of releasing pre-prints means that you can disseminate your work quickly to others in the field, without having to wait a long time for the peer review to be completed before your work can be seen by the community. Pre-prints are a great way to keep up with the latest developments, but keep in mind there’s no review process for preprints: take any conclusions with a grain of salt, read the paper carefully with a critical eye and remember that extraordinary claims require extraordinary evidence. As a crude measure as to whether a preprint is likely to be reliable, it may be worth checking out the publication history of some of the authors to see if they have a strong track record in their field, e.g. on Google Scholar or ORCID, but remember that it’s entirely possible for innovative works to come from young researchers with no prior publication history. Pre-prints are a wonderful resource, but they should be used with an understanding of their place in the scientific landscape.

Peer-reviewed articles have been published in an academic journal and often require either a journal subscription or a one-time payment to acess. At least in theory, they have gone through a rigorous process through which the journal’s editor has first assessed the suitability of the work for publication, then sent the manuscript to several experts in the field for feedback and to check whether the science has been done to a high level and the data shown in the paper support the authors' conclusions. This process typically takes a few rounds of interation: the referees will offer feedback, the editor will decide whether the manuscript merits further consideration, and if so then the authors will be given the chance to modify the manuscript to fix any errors, clarify anything that was unclear, or even do some further work if required. This process will repeat several times until all parties are satisfied with the final result. (Note that some publishers, such as SciPost conduct the review process publicly, so you can read the reviews that a paper received and see which parts reviewers particularly liked or did not like.) If the manuscript makes it through this process without being rejected from the journal, in principle any errors in the manuscript will have been corrected and the result will be considered correct. At least, that’s the idea: as websites like Retraction Watch will be quick to point out, errors and biases can slip through, and so in practice most of the caveats listed for pre-prints still apply. In all cases, papers should be read with a critical eye!

How to read a scientific paper?

There are as many strategies for reading a scientific paper as there are people who read them, and many excellent guides which cover different ways to approach a paper depending on what you want to get out of it. If you’re a researcher, this may be to simply skim-read it to stay abreast of the latest developments in your field, or it may be to understand in detail parts that are vital to your own research. If you’re a journalist, your goal may be to understand the significance of the science and break it down into more digestible terms that can be more easily understood by non-specialists. When approaching a scientific paper, before you read a word of it, it’s worth asking yourself just what it is that you want to get out of it, and which parts of the paper you need to read in order to get it. (For example, unless you want to reproduce a study yourself, parts like Experimental Methods or Supplemental Materials are probably not useful to you, and you may get more out of focusing on the results and discussion sections.)

Many scientists – myself included – favour the process of reading a paper in multiple passes, first skim-reading the main details (introduction, figures, conclusion) to get a feel for the ‘big picture’ of the work, then going back and reading the details, then (provided you already know enough about the field in question!) making a third pass over it with a more critical mindset, interrogating every claim and trying to poke holes in the methodology. This might sound a little harsh, or even disrespectful to the authors, but it’s quite the opposite - it’s very important not to take anything for granted, to question any claims or assumptions until you can verify them.

  1. First pass: Here I’ll focus on the introduction to put the study in context and try to understand what the authors are trying to achieve, followed by a brief read over the main sections of the paper, paying particular attention to the figures. A good paper will use the figures to summarize – and provide evidence for – the paper’s key findings. I’ll also read the conclusion to get an idea of what the authors claim the results say, and see whether this agrees with what I understood the figures to show.

  2. Second pass: This time around, I’ll go through the details. What sort of model/experiment/calculation is being done? Does the method seem appropriate? Are there any sentences that jump out at me, like “If we ignore X, then Y…” without explaining why X can be ignored? This is also usually the point where I’ll start checking others papers cited by the one I’m reading, particularly if the current work relies heavily on details which can only be found in earlier works, or if there is some important-looking terminology that I don’t understand. This is usually the point where I stop if it’s a paper far outside of my own field, e.g. in fields like medicine or economics where I’m unlikely to get much more from repeated reads.

  3. Third pass: At this point, which I’ll only really do for papers on topics which I already know well, I’ll go through the specific steps that are most important for what I want to get from it. Am I interested in using some theoretical method? In that case, I’ll work through the equations or maybe even try to run a few small simulations to see if I can understand what they’re doing. If it’s an experiment and I want to understand how I could model it theoretically, I’ll try to pick out the most important bits of physics and see if I can put together a minimal model that reproduces the behaviour seen in the experiment. If it’s a paper in a field like astronomy, which I’m interested in but don’t work in, I might go through the plots and raw data (if available) to get some understanding of how the results are obtained and what sort of analysis and/or post-processing has been done.

A Rose By Any Other Name…

One thing to be aware of when approaching a scientific field as a non-specialist is that in addition to the jargon and acronyms that may be hard to understand, researchers often use words in slightly non-standard ways when writing for an expert audience. For example, ‘theory’ in colloquiual English is usually taken to mean little more than an unproven guess, whereas in science ‘theory’ is usually taken to mean a rigorous theoretical framework with explanatory/predictive power which has withstood numerous attempts to challenge it. (Your precise definition may differ depending on what field you come from…!)

A similar problem can arise with words like ‘trick’ which is sometimes used in physics or maths to refer to an elegant shortcut, but can easily be misconstrued as an attempt to mislead. This most famously occurred during the Climate Research Unit e-mail leak, where the word ‘trick’ was used to refer to a data anlysis technique but was misunderstood and misrepresented by climate change sceptics. When reading scientific papers, or indeed any research works from any field, it’s important to keep in mind that you may not be the target audience; these sorts of things are written mainly for others in the same or similar research field, and as such it’s worth keeping an eye out for non-standard uses of language that the you may understand in a different way than the authors intended. This problem is most acute for papers published in specialized journals aimed at a niche audience, and is less of a problem for works which appear in very broad journals with a wide target audience (e.g. Science or Nature).

Let It Sink In

Honestly, I rarely understand everything in a paper the first time I read it, and one of the best things you can do it simply give yourself time to process what you’ve read. Put the paper away and go get a cup of coffee, take a walk, or do something else for a little while, then come back to it with fresh eyes and you may find that you suddenly understand some details which at first glance seemed utterly meaningless. (Equally, you may not - some very gifted scientists are terrible at explaining themselves…!)

If all else fails and even after a concerted effort you still can’t make head nor tail of a paper, you can always drop the corresponding author of the paper an e-mail to ask about it. This works best when you’ve got specific queries, e.g. “How did you process the data to generate this figure?” or “Why does this term in your equation vanish?”, rather than just asking someone to explain their entire paper to you. Not all authors will respond to e-mails of this type, but it’s always worth a try if you want to know more, and many authors will be flattered to know that you’re interested in their work!

One last final thought before we finish up here: while it’s important to think critically when reading a scientific paper, remember that the authors are only human and it’s worth giving them the benefit of the doubt when it comes to subjective matters like phrasing, word choice, colour schemes…you get the idea! No one sets out to write a bad paper, and while plenty of mediocre papers exist, mostly they are due to inexperience or external pressures (e.g. funding running out soon, or need to get a paper published soon in order to secure a grant) that lead to the work failing to live up to even the authors' expectations. Of course, misconduct and fraud do exist so it’s worth keeping an eye out for those, but by and large this type of deliberately misleading behaviour is mercifully rare, at least in my field2.

Conclusion

Hopefully this guide has given you a few pointers about how to approach scientific papers as a non-specialist, where to find work on topics that you’re interested in, how to gauge the quality of a piece of research and above all, equipped you with a few of the skills to set out in search of answers yourself. The next time you see a sensationalist headline in a newspaper, now you can track down the scientific study underlying it and judge for yourself whether the newspaper represented the findings accurately, or is simply trying to create fear and outrage to sell copies and drive traffic to their website.

Are there any steps in here that I missed, tips and tricks you use that I should have included, or even things I’ve suggested that are wrong and unhelpful that you can think you can help me improve on? Drop me a message on Twitter and let me know!

This post was written as part of public engagement component of the Ergodicity Breaking in Quantum Matter project (EBQM). This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No.101031489.

  1. After investing a lot of time in the competitive interview process, I eventually turned down this offer and chose instead to remain in research; this was at the time a very difficult decision, but one that I do not regret. ↩︎

  2. Far more common are well-meaning but misleading data analysis techniques, post-selection of only the ‘good’ data and other such self-deceptive practices that can be hard to spot, but which usually come out in the end - science is ultimately self-correcting, and while it may take a while, in the end erroneous results will eventually be found and corrected by others in the research community. ↩︎