We had questions about Registered Reports. So I recently spoke with David Mellor from the Center for Open Science for some answers. Here they are. And we have a special message for Wiley journal editors (which we repeat at the end): Please, speak with your publisher about our Registered Reports toolkit so together we can make this option available for researchers in your communities.

Q. Thanks for agreeing to speak with us, David. So what’s all the fuss about preregistration?

A. A preregistration is a time-stamped research plan that you can point to after conducting a study to prove to yourself and others that you really are testing a predicted relationship.

Preregistration has been around for a while. Statisticians have been warning us for decades that commonly used tests could misrepresent scientific evidence if we only report some of the answers, let the data inform the hypotheses we’re testing, or switch outcomes. These problems were affecting the credibility of drug trials, and so specifying in advance has been required by law for clinical studies in many countries for about 20 years. However, similar tools can be applied to basic research that informs policy or that affects our understanding of the way the world works, and so preregistration can increase credibility in these situations too.

Q. Seems like yet another hoop for researchers to jump through… What’s the point? Why bother?

A.It is certainly a new step in the workflow for some. However, everyone wants to produce the most rigorous and credible findings. Preregistration makes the distinction between confirmatory research and exploratory research clearer. Confirmatory research tests a very specific prediction. Exploratory research looks for something unanticipated. Both are vital to the advancement of knowledge. However, using the tools we have for confirmatory hypothesis testing when reporting exploratory, preliminary results makes the research more publishable, but at the expense of credibility. Addressing this conflict motivates our work.

Q. Oh, I see. There are likely benefits to research as a whole. But are there benefits for researchers who preregister?

A.There are several benefits for the individual researcher. The first comes from better planning. Every step that goes into a preregistration: writing the hypotheses, defining the variables, and creating statistical tests, are steps that we all have to take at some point. Making them before data collection can improve the researcher’s study design.

We built a structured, easy to use system that guides the researcher through this process on the Open Science Framework. Because this step is so new and so capable of improving science, the Laura and John Arnold Foundation is supporting $1000 awards for one thousand researchers who publish preregistered research through the Preregistration Challenge. This education campaign is our way of introducing this concept to researchers who are not familiar with the practice.

The other major benefit for the researcher comes from working with a journal that offers Registered Reports. These journals (there are currently 80) will consider accepting the study before results are known. Submitted research plans are evaluated through peer review based on 1) the importance of the research question and 2) the ability of the proposed methods to answers those questions. If both of those criteria are positively evaluated, the paper can be given an in-principle acceptance, which is a promise to publish regardless of outcome.

This is a huge benefit to the researcher. The researcher gets earlier feedback on their proposed study, when it has a chance to actually improve the work. The decision to publish is based on the quality of the work and the importance of the research question instead of the outcome, which is very attractive for anyone looking to publish.

Q. Even so, I’m still not convinced. What indication do we have that researchers are even remotely interested in preregistration?

A. Clinical researchers have been using this process for a long time. But it is new to researchers in basic science or pre-clinical fields. We’ve provided tools to any researcher since 2013 and have seen explosive growth in the number of preregistrations on the OSF.

The OSF is designed for researchers in any field, but there is a lot of demand for preregistration among discipline-specific registries, such as those for international impact evaluations, and education. There are even tools made by proponents of preregistration to help researchers make concise preregistrations.

Q. Now we’re getting there. So what role might a journal play in preregistration?

A. There are four specific steps that journals can take, and they’re summarized in the policies outlined in the Transparency and Openness Promotion (TOP) Guidelines.

1. The first is to simply have researchers disclose whether or not any empirical work was preregistered.
2. The next step would be to include comparisons between the preregistered protocols and reported findings to evaluate whether or not the research plan was followed. Linking to a preregistration at least allows this to happen after publication, but we’d love it for these comparisons to occur during peer review.
3. The third step is to require preregistration for any relevant work. Journals that publish clinical research take this step, and others can as well.
4. The final step is to offer Registered Reports as an option for researchers to use.

Q. And how does this fit within the “open research” agenda?

A. Open Science is about connecting each part of the research that went into the final results. Traditionally, we didn’t have the means to easily share data, or to point to time-stamped research plans, or disseminate materials beyond the appendix section of a printed journal. However, now we have the means to realize that ideal workflow, and to make science work in the way it was meant to work.

Q. What does this all mean for early career researchers?

A. In 10 years I think the field will have different expectations. Researchers will be expected to demonstrate reproducible work by pointing to a body of work that contains all of the essential parts: preregistered plans, materials identified with persistent and unique IDs, datasets with clear descriptors, code for running tests on those data, preprints showing preliminary findings, and finally the peer reviewed manuscript that guides the reader through the complete and accurate narrative.

We’re on the cusp of that now. It is possible to cite that complete workflow for those who are on the cutting edge. But it also takes people caring about science and about transparency so that decisions to publish, award grants, and offer jobs are made based on the reproducibility and transparency of the scholar’s opus.

Q. OK… I’m convinced that this is worth trying. So how can a journal help researchers with preregistration?

A. Besides the four specific steps already mentioned, journals can do a lot to make this happen sooner. They can use Open Science Badges to allow researchers to signal when they are taking these steps. They can let their authors know that preregistration is valued by sharing news about the Preregistration Challenge on their website or social media platforms. They can point to example preregistrations to guide researchers.

Q. You can count me in! So… what’s our “take home”?

A. Whether you’re an early career researcher or a seasoned scholar; journal editor or author, there are steps that you can take to make science better. Raise awareness, talk to your colleagues, do one thing in your next project that is a bit more open than your last, and in the near future we will have made a huge difference.

Thanks, David. We’ll add new journals that launch Registered Reports to the 5 Wiley journals listed already on your site here. For researchers who publish in Journal of Neuropsychology (published for the British Psychological Society by Wiley) launch of Registered Reports is imminent. And we have more journals queuing up.

And for Wiley journal editors, here’s our closing message. We have a Registered Reports toolkit to help launch Registered Reports quickly. Please speak with your publisher and we’ll get things organized. Thank you.

David Mellor (pictured) works at the Center for Open Science on incentive programs with researchers, journals, publishers, and funders. He received his PhD in Ecology and Evolution and has a background in behavioral ecology and citizen science. Find him online on Twitter and ORCID.