“A negative result is still a result!” is a common, if slightly ironic, utterance that can frequently be heard around the ivory towers. I must say that I agree with this sentiment, and I agree with it to the extent that I think everyone who reviews a scholarly paper should be asking about what doesn’t work almost as much as what does. It’s the things that don’t work that set the boundaries for a particular method or technique. However, more importantly than that, the unanswered questions as to why something doesn’t work lay the foundations for future research, and I’d argue that this is precisely why peer reviewers should always ask for a few “negative results”* to be included in every paper they see.

Source: iStockphoto

Posing the question

So if you’re a referee, how exactly should you ask about what doesn’t work? As soon as you’ve read the paper there will no doubt be ideas popping into your head, such as “I wonder what would have happened if they had tried X or Y?”. The key here is not to be afraid to ask obvious questions or to assume that the authors did try X or Y and it didn’t work so they simply didn’t include it. If the authors have already thought about or even tested your suggestion, then it won’t be too much of a problem to answer your question. If they haven’t already thought about it, then your question will be helpful in opening up other ideas for them. Your simple question at the refereeing stage might also save someone else trying something that the authors of the current study already know doesn’t work, but they just haven’t addressed this in their paper.

Prioritizing the issues

When asking about the limitations of a study, the skill is to keep your requirements as a reviewer in perspective. You may be able to think of several things that the authors could try, but on the flip side you can’t be too demanding or expect authors to stray too far from the main scope of their study. Prioritize what you think are the most important issues; for example, if you were the editor, could you accept the paper without the authors trying X and Y? If not, why not? Make sure to clearly state this in your referee report. If you have other ideas but you wouldn’t necessarily reject the paper if they weren’t included, you can put these forward as “nice to haves”. These suggestions may be informative for the authors in thinking about future work even if they aren’t studied in depth in the current paper.

Limitations vs. inconsistencies

Another important thing to bear in mind is that limitations are very different from inconsistencies. Where data and conclusions don’t match or one set of results seems to contradict another, as a referee you have a right to closely examine and pose questions until the inconsistency is resolved. In cases where inconsistencies are very severe, you are justified in asking for more work to be done, but the natural limits of a technique shouldn’t detract too much from the results and the advances that have been made.

As an editor, there’s nothing I like to see more in a referee report than a reviewer who pushes to know just what the boundaries of a piece of work are. Therefore, I encourage everyone who reviews papers to ask about limitations and request more data where it’s necessary, but at the same time keep expectations in perspective and remember to focus on what has been achieved in a paper, as well as what hasn’t. Not every method works for every situation, so it’s important to know where the borders lie. That’s why a “negative result” is indeed still very much a result.

*As an aside, I don’t like the term “negative results” and think that it’s distinctly unhelpful to science. Getting published in a “top” journal, whatever your definition of one happens to be, is pretty competitive these days, which leads me to think that scientists are getting more reluctant to include so-called “negative results” in papers for fear of missing out on a spot in that coveted journal. However, “negative results” are an integral part of the development of research, and hiding them away is to the detriment of us all.