Reading Papers Intelligently

Most papers aren’t worth reading

A paper is worth reading if:

Your advisor or a senior student you’re working with recommends it.
It appeared in a reputable venue (Your advisor or more senior students can help here.) There are a lot of second-rate venues; papers in those venues may be methodologically valid, but irrelevant (unlikely to have impact on the field, or incremental compared to previous work) or non-novel (barely different from something already done, but 2nd-rate committee members may not know the field well enough to be aware of that during reviewing).

How to find papers beyond those specifically being recommended to you?

Use Google Scholar (not general Google search; if your topic contains anything remotely like a buzzword, you’ll just get a ton of SEO-driven crap) to search by keyword. Being overly broad is OK at first, and sometimes as you narrow results, you’ll discover other keywords (in titles, e.g.) that help your search.
If there are specific conferences that feature the type/topics you’re interested in, the proceedings of most conferences are online and searchable. You can browse the whole conference program, which is sometimes useful, since things like posters and work-in-progress papers do not always get published in the proceedings and might only be found on the conference program pages.
As you skim/read the papers (see advice below), make note of specific references in each paper that might be worth following up on. This is especially useful as you track down related work.
Try to make sure the paper has appeared in a reputable venue. There are a few “write-only” venues that were set up because someone needed to add “Started a new conference series” to their CV, and their content tends to be low quality. You can often tell by the affiliations of the people on the program committee. If in doubt, ask your advisor.

Pre-read: What, How, How Much, So What

Take notes as you read. You won’t remember everything important. Really.

Prepare for focus. Not all papers are worth reading thoroughly, but those that are can be pretty dense. If at all possible, read the paper in one sitting since it’s hard to page it all back in later. For some of you, long-form reading may be a return to an activity you don’t do very often. Fortunately, it builds character.

Start by reading the abstract and conclusions.

What is the goal of the authors (what do they want to demonstrate or show)?
How do they propose to demonstrate it? What specific things will be built, observed, measured, etc., such that a positive result of those measurements or observations would support the goal?
How much progress do the authors claim to make on this goal? Did they fully show it? Partially? Failed to show it but learned something else useful?
So what: assuming you understand the goal, and assuming the authors’ claims are true, what lessons, ideas, etc. can you take from it to apply to your own work? Can you compare it to other papers you’ve read that are also relevant to your work?

Digest While Reading: Focus on How and How Much

While reading, keep in mind the what while you concentrate on the how and how much. What are the most important parts of the paper in terms of understanding how the problem is being set up and solved?

One way to do this is to question everything as you read it—not with the goal of shooting down the authors’ claims, but with the goal of making sure you yourself understand the answer.

For example, consider a paper about an educational software tool such as a new type of exercise. If the claim is “Using our educational tool, students learn to do X better”, the first question I would ask as how “learning” is measured and what “better” means. Are students evaluated on how well they perform a task? On how much practice they need before performing the task at a certain level of proficiency? On how quickly they complete the task? (In a well-written paper, the connection between what and how will be made very early, possibly even in the abstract. For example, rather than saying “students learned X better,” the authors might say “When students attempted the task before and then again after using our new tool, the likelihood of getting the task right on the first try increased in a statistically significant way,” or “…the number of tries required to get the task right decreased in a statistically significant way.”

Does the experiment design consider and control for possible confounding factors? (That is, if students are “learning better,” are there other plausible explanations besides the effect of the authors’ new tool, such as that repeatedly solving a particular type of problem results in a proficiency increase even without the tool? Did the authors have a control group that did the same set of exercises without access to the tool, and did they confirm that the control and treatment groups differed in their learning gains in statistically significant ways?

At this point, you are evaluating the specific methods and statistical tests used to support the claims. Don’t feel bad about asking for help with this, since there is a lot to know here. But the key thing is to get to the point where you understand the specific relationship between the overall result being claimed (“better learning”) and the specific types of measurements that allegedly demonstrate that result.

Finally, as you’re reading the paper, once in a while you might read a statement to which you may react “Wow, I’d like to learn more about that” or “Hm, I’m not sure I understood what that meant.” If such statements are accompanied by a reference to another paper, consider looking up the other paper and at least pre-reading it. (This is particularly important when you’re reading the paper’s Related Work section, which is the authors’ attempt to position their work relative to other work in the area.) Strategically pre-reading a few of the references can help you establish the same context the authors had when they wrote it.

Preparing to report on a paper

If you’re leading a reading-group discussion on the paper, what should you do to present it?

The best advice is to first take a big step back and ask how the findings of the paper (if they hold up) affect your group’s longer term research agenda and maybe even your own work.

Then, go back to explaining what, how, how much, and so what. (So if you decide to use slides for your presentation, 3 or 4 should be plenty, possibly including diagrams/graphs/etc. from the paper.) Don’t present the paper by going through a linear summary; the goal is to communicate its main ideas, not to rehash the sequence of events. The group discussion can then focus on whether we believe the methods that support the results, and if so, what are the implications for our work.