For Peer review week 2019 Sense about Science have asked for a series of blogposts, to begin a Research ‘TO DO’ list. What can researchers, universities, funders and governments do to improve the quality of peer review and research? This is just the start, we need your ideas too.
Imagine a world where society has the high-quality evidence it needs to make informed
decisions about crime, health and education. What action can researchers, universities, funders and governments take to get us there?
How can we improve the quality of peer reviewed publications?
By Professor Gary Collins @GSCollins and Patricia Logullo @patlogullo on behalf of the EQUATOR Network @EQUATORNetwork
It has been estimated that more than 80% of articles in biomedical journals lack important information. They are published without the details that would be necessary for their findings to help researchers and health professionals improve people’s lives. Because publications are frequently incomplete or biased, this wastes money and human resources and can lead to patient harm. These are the findings of key systematic reviews on studies published in the last 20 years.
Since the early 1990s, researchers and journal editors have tried to tackle this problem by creating reporting guidelines: these are tools that remind authors to provide a minimum list of information needed to ensure that:
– the article can be understood;
– the study can be replicated by another researcher;
– the results can be used by health professionals or policymakers;
– the study can be included in a systematic review (to inform guidelines for example).
Research has shown that the quality of scientific reporting has improved modestly with the increased use of reporting guidelines. However, there is still a long way to go.
Another source of avoidable harm to patients is when the studies are not reported at all: non publication of research (also called publication bias) is a known cause of research waste. Since 2005, the International Committee of Medical Journal Editors (ICMJE) has mandated prior registration of trial protocols before clinical trials findings are published, Clinical trial registration makes it easier to identify those studies that have completed but not yet published.
The EQUATOR Network (www.equator-network.org) is a global initiative dedicated to improving the quality and transparency of health research. We conduct research on reporting, manage a collection of more than 400 reporting guidelines (which are currently being audited), and train authors and editors how to use reporting guidelines. Our research experience allows us to draft the following “to-do list” about improving reporting of health care research.
How to fix the problem of poor-quality reporting
Based on the EQUATOR Network experience of conducting and evaluating research-on-research, and helping authors, editors and librarians, these are the activities, procedures, approaches or policies thatto improve the completeness of research reporting. Some of these issues are currently being addressed by organisations and researchers, some might not be.
Researchers (research-on-research investigators) TO DO list:
1. Updating reporting guidelines – The first reporting guideline, the CONSORT Statement, designed to guide the reporting of clinical trials, was published in 1996. Since then, numerous reporting guidelines have been developed for other study designs, including observational studies (STROBE), systematic reviews (PRISMA), diagnostic test accuracy (STARD), prognostic model studies (TRIPOD) studies and many more. CONSORT was updated in 2001 and 2010. However, many reporting guidelines have been available for a long time now without having been updated. Periodic updating and revising is important to ensuring their ongoing credibility, reflect advancements made in study methodology and hopefully increase their use by researchers when writing up the findings from their study.
2. Investigating poor adherence – Many reporting guidelines have been published, but evaluations have shown they are often not fully adhered to, with many key details often omitted — even in reports published in high ranking journals. We therefore need to investigate and understand why researchers are not fully following the recommendations contained in reporting guidelines or do it poorly (incompletely), and why journals are allowing that to happen.
3. Testing new interventions — What could help authors use reporting guidelines? What would be alternative approaches to improving reporting quality? Does training help? We should test them as interventions, in controlled and carefully planned trials.
Publishers and journal editors TO DO list:
1. Knowing reporting guidelines well — Editors working for journals that endorse or claim to enforce the use of reporting guidelines are not always sufficiently familiar with the contents of all the main reporting guidelines and the corresponding checklists. This creates the situation where the editors can ask authors to fill in the wrong checklist as a mandatory step of submission. Training editors on the key reporting guidelines is important to understand whether the relevant guideline and checklist has been correctly followed and completed.
2. Standardise the instructions for authors — The journals’ instructions for authors vary substantially, even between journals that share the same publisher: this means they have quite different expectations of how authors should write and format their papers. This variation persists even in journals that have adopted ICMJE (International Committee of Medical Journal Editors) recommendations. In order to achieve greater consistency and quality, journals should consistently endorse the key reporting guidelines (namely those listed on the EQUATOR homepage).
3. Check usage — Many journals endorse or recommend following reporting guidelines, expecting authors to adhere to them and submit a completed checklist (providing page numbers where key information are reported in the submitted article). However, journals seldom check if reporting guidelines were followed, or indeed if the completed checklist is accurate. Effective solutions are required during the submission and peer review process to identify key information, recommended in the reporting guideline, that has not been reported before acceptance of the article.
4. Increase peer reviewers awareness — Ideally, peer reviewers should be aware of the relevant reporting guidelines for the study design of the article they are reviewing and consult them during the peer review. Peer reviewers have the opportunity to help to improve reporting, by requesting that authors provide details on aspects of the study that are missing.
5. Create the citizen box — Journals could start requiring authors to complete a small paragraph of the manuscripts with information directly relevant to patients and citizens in general, about the importance of the study. This could make manuscripts more understandable to the general public.
Systematic reviewers TO DO list:
1. Provide risk of bias feedback — One of the daily tasks of systematic reviewers is to evaluate the risk of bias of studies. For that, reviewers need detailed information about how the study was done and what it found. When there is “risk of bias due to poor reporting”, we don’t know what happened next. We don’t know what information was asked from authors, because little is reported about this communication between reviewers and triallists. Also, there should be a way to publish information that was collected through direct contact with the authors.
2. Improve reporting — While systematic reviews suffer from a lack of information in the papers they review, not all systematic reviews are well reported, understandable by everybody or useful. Therefore, there is room for improvement in the quality of the systematic reviews reports. This goes from reporting a minimum set of information following the PRISMA reporting guideline to help describe the included studies, to improving the quality of language and format. This is critical for the abstracts and plain language summaries that are frequently used by policymakers and patients.
Government, policymakers and funders TO-DO list:
1. Get to know and require the minimum sets — Funders should mandate the use of reporting guidelines, so that policymakers can make informed decisions on whether or not to adopt a new treatment. This means they need information on harms (an item on several reporting guidelines), feasibility of implementation (providing sufficient detail on the interventions, e.g. by following the TiDIER reporting guideline) and cost effectiveness.
2. Examine protocol methods using reporting guidelines — Funders routinely evaluate applications for funding. These do not always provide all the important details on how authors intend to conduct the research (where reporting guidelines such as SPIRIT and PRISMA-P for clinical trials and systematic reviews can help). If funders start to require adherence to reporting guidelines, at least for the methods section (which can then be readily evaluated during peer review), this might help make better informed decisions on whether to fund studies.
Patients TO-DO list
1. Involve! – Patients should be actively involved in research. This is increasingly happening in Europe, but still in clinical areas only. Patients could also be encouraged to participate in research-on-research projects. We need to know their view on what is essential or important to be reported in a manuscript.
2. Review! –Patients and members of the public could and should participate in the revision process of new reporting guidelines. Research is done with them and for them. So why not ask them to verify the draft items of reporting guidelines checklists, even when they did not participate in their building?
We extend our thanks to David Tovey @DavidTovey for his work editing these blogposts.
Please tweet your ideas #ResearchTODOlist #PeerRevWk19 and #QualityInPeerReview or email us at email@example.com.