For the conference issue of Journal of Accounting Research published in April, all articles were published through a registration-based editorial process.  We discussed Registered Reports in an interview with David Mellor from the Center for Open Science late last year.  We recently spoke with Christian Leuz, one of JAR’s Editors, about the journal’s experience putting this process into practice for their conference issue.

Q. Thanks very much for speaking with us, Professor Leuz. So tell me, how did this special registration-based issue of JAR come about?

A. The origin was a conversation between Rob Bloomfield and me a few years ago. We were discussing trends in research practices and concerns about the publication process. During this conversation, we discovered that the JAR Editors and he had all been thinking about how registered reports could address some of these concerns. However, as registered reports had been used mostly in the natural sciences, we thought it would be good to first experiment, and that is why we chose it as a theme and process for the 2017 JAR conference.

Q. What was JAR’s goal in using the Registered Reports process?

A. There were several goals and reasons for doing it. In recent years, JAR has been trying to innovate on a number of fronts related to the publication process. We introduced new and, relative to other journals, quite stringent policies on data and code sharing. We also updated policies on conflicts of interest and coercive citations, to name a few. In our conversations about our policies, we have thought extensively about the publication process and its subtle and not so subtle biases, e.g., the incentives to produce significant t-stats. Moreover, over the years, we kept hearing stories about papers with results that do not replicate or are fragile.

So, the idea of registered reports was very intriguing to us. We viewed the conference as an experiment to learn about this process, to see how it would work and whether it might provide an alternative way of producing and publishing papers. We were also curious as to whether the papers and the results would look different when produced with a registered report process, e.g., would we see more papers with insignificant findings? Finally, the process was a way to encourage researchers to engage in new research with potentially higher outcome risk that involves gathering new data.

Q. Could you describe the process followed by the authors and editors for this issue?

A. We used a two‐stage editorial process. We followed the core idea that researchers pre-register the analysis and the journal provides in-principal acceptances to the authors for their studies, irrespective of the results of the analysis. The authors submitted proposals to study certain questions and to gather and analyze data to test their predictions. We sent the proposal for review through our normal referee process. After at least two rounds of revision and re-submission, eight proposals received an “in-principle” acceptance, guaranteeing publication as long as authors gathered and analyzed their data as promised, whether or not results supported their predictions.

Q. Were authors receptive to the idea and did you face any challenges getting submissions that met the guidelines set out by the journal?

A. The response by authors was fantastic. We received a huge number of proposals from many different areas in accounting using many different methods, including field experiments, lab experiments, and archival analyses. Of course, there were also challenges as the authors and reviewers were less familiar and had little experience with the process. This meant that there was a lot of learning along the way and that we probably had to provide more guidance than with the regular process.

Q. What do you see as the benefits to the journal and to your subject community of evaluating articles based on the study design rather than the results?

A. For us, one major benefit was that the experiment opened a discussion about the traditional publication process. It also showed that the journal is open to new ideas and changes. But speaking more generally about publishing registered reports, the biggest benefit is that the process provides additional credibility to the studies and their results. It likely also increases the reproducibility of the findings, which is especially important in light of recent evidence that many fields have very low reproducibility rates. Having said all that, it is important to see that registered reports also change how much authors invest in their projects, and when they do so. A registered report requires a lot of upfront investment in the proposal. But once accepted, there is limited room for adjustments. Thus, authors have fewer incentives to invest in learning about their data and to tease out results that were less predictable upfront. Of course, researchers could simply pursue these opportunities in follow-up research. But it is important to recognize that accounting publishes far fewer papers than the natural sciences. As a result, getting it right with the first paper is more important, hence the concern that authors might invest less once they actually have and analyze the data, is to be taken seriously.

Q. What did you learn from your experience and do you have tips for other journals that may be considering Registered Reports?

A. The experience very clearly showed that there are two sides to the discretion that the traditional process gives to researchers. Registered reports basically tie the hands of researchers once they have gathered the data. This increases the credibility and reproducibility of the findings. At the same time, authors learn a lot from seeing and exploring the data. Thus, as mentioned earlier, there is a tradeoff. Interestingly, both effects can explain why we have far fewer papers with significant results in the current JAR issue.

We very clearly learned that proposals need exposure, just as working papers need workshops. So I would encourage the authors to present proposals prior to submission. If we were to do it as part of a conference again, I would recommend holding the conference for Stage 1, when authors present their research plans and pilots, rather than Stage 2 when people present their results. We learned that proposals have to articulate very clearly what we learn from the study, irrespective of whether the predictions are supported or not, and that the authors should be encouraged to conduct power analyses so that there are fewer underpowered studies.

I would also recommend to other social science journals and editors to be very clear about the kind of papers for which the registered report process is better suited. My sense is that it should be offered for papers that require very big investments in data collection (for which the in-principle acceptance reduces uncertainty); it works well for experiments in the lab or the field; and it provides credibility to studies that later could be unfairly criticized as doing something simply because they found a sensational or unexpected result. It would show that the researchers thought about this outcome ex ante and not after analyzing the data.

Q. Do you have plans to use this process again in the future, either for another special issue or as a new publication type for authors interested in using this process?

A. Overall, the experience was positive but it also showed that the process needs tweaking for social science research such as studies in accounting and finance. We are currently discussing what we have learned and how we could design the process so that we could continue to publish registered reports at JAR. So, stay tuned.

Q. Any final thoughts to share on JAR’s experience with Registered Reports?

A. Yes, let me provide a recommendation. Rob Bloomfield who kindly helped us as a guest editor for the registered report conference wrote a very interesting paper (published in the same issue) that discusses the pros and cons of registered reports and, more specifically, lessons from the process. They conducted a survey among authors, reviewers and conference participants as well as a survey among researchers on the role of discretion in research. His paper provides many interesting insights on the benefits, but also tradeoffs that I discussed. For further information on the Registered Reports process at JAR, see our website.

Thanks, Christian. We appreciate you sharing these valuable insights for editors and authors who are interested in following this process for their own research.

And for Wiley journal editors, here’s our closing message. We have a Registered Reports toolkit to help launch Registered Reports quickly. Please speak with your publisher about making this option available to researchers in your subject community.

Christian Leuz (pictured above), along with Philip G. Berger and Douglas J. Skinner, is Editor of the Journal of Accounting Research, one of the world’s top Business/Finance journals, that publishes original research in accounting and related areas.

Photo credit: Christian Leuz