Repositioning peer review to support open science, reproducibility and research integrity

Peer Review Week is always a good time to step back and take stock of what’s been happening in the fast paced, ever changing industry that is academic publishing. This year’s theme, the importance of peer review in supporting research integrity, is particularly interesting because it raises questions about the role of peer review in the context of open science and reproducibility, which are both inextricably linked to research integrity.

Peer review has had a rocky reputation in recent decades. There was an intense wave of innovation in peer review at the beginning of the century as concerns about the slowness of the process, potential for bias and the lack of accountability were raised. Open peer review was just one innovation aimed at increasing transparency and accountability for editors and peer reviewers. 

In the years that followed, there were attempts to make peer review more efficient and equitable. Peer review innovations were mostly focused on journal peer review because, at the time, peer review was intended to help journal editors make decisions about what to publish in their journals. Despite these innovations, peer review has continued largely in its traditional form, although there are enough variations in the process to need a standard taxonomy for different types of peer review.

But peer review is changing in a fundamental way. With the increasing popularity of preprints in recent years, driven further by the Covid-19 pandemic, peer review has taken on the wider remit. More than ever, peer review is seen to signify the trustworthiness of different types of research publications rather than a tool to inform editorial decisions on what to publish in Journals. 

Peer review reports can now have a heterogeneous readership rather than just journal editors. It is undergoing another wave of innovation, in many cases involving direct communication between author and reviewer. Peer review reports can be written by and for the specialist and layperson alike.

While expectations of peer review have changed, research integrity as a field of specialisation in its own right has also grown. Again, the Covid-19 pandemic along with the phenomenon of paper mills (a form of sophisticated research misconduct) has made research integrity a ‘hot topic’. Concerns about research integrity have driven calls to change a research culture that motivates questionable research practices and misconduct.

Despite this, there has been little joined up thinking about the role of peer review in maintaining research integrity. There is no obvious direct role for peer review as a mechanism for preventing research misconduct or promoting good research practice. It is not designed to do this. 

Can peer review be redesigned to support research integrity? The key is to remember that peer review is the collation of opinions and therefore, subjective. It can never be the last word on validity. Science is objectively validated by the slow process of repeating the research and showing the same findings via reproducibility and replicability. Peer review can certainly play a role in supporting this validation process.

The open science movement promotes behaviours that allow reproducibility and replicability. Many of these behaviours, such as pre-registering intended research and data sharing are encapsulated in journal editorial policies, but they are rarely the focus of peer review.

Traditionally peer reviewers are asked to comment on whether a piece of research is sound, whether the methods and analysis are appropriate for the research question, and whether the conclusions supported by the data. These are good questions, but peer review needs to be positioned to also assess how far the reporting of research allows for its objective validation in the future. 

Such a repositioning might ask peer reviewers the following questions: 

  • Is it clear whether the research was registered before it began? 
  • Are the raw data shared? 
  • Are reporting guidelines followed? 
  • Are details provided on the sources of materials used? 

The above questions should be asked not just of research published in journals that have policies on reproducibility, but of all types of publications as a standard part of peer review regardless of the editorial policies of the platform on which they are published, or the model used for the peer review process. This coupled with open peer review would incentivise transparent and collaborative practices that would ultimately help to maintain research integrity.

Of course, this would all require widespread support of open science practices and the standardisation of peer review across the industry. This may not seem feasible, but consider, if you read a piece of research that was not written in a way that allowed for its objective validation. Would you trust it?

Jigisha Patel

Attribution: This post was written by Jigisha Patel, Founder of Jigisha Patel Research Integrity Limited

About the author: Formerly a medical doctor, clinical researcher, and medical journal editor, Jigisha is an independent research integrity specialist and founder of Jigisha Patel Research Integrity Limited. Previously, she led the first team dedicated to maintaining research integrity at BioMed Central and was Head of Programme Management for the Springer Nature Research Integrity Group. She has extensive experience in a wide range of research integrity issues in publishing, including the investigation and management of complex cases such as paper mills. She uses her experience to help journals and publishers manage cases of research misconduct and develop policies and processes to maintain research integrity. She also provides various training for journal editors and publishing staff, including a CDP-certified course on research integrity strategy. She is an independently elected member of COPE and Senior Associate Affiliate with Maverick Publishing Specialists.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s