For Peer review week 2019 Sense about Science have asked for a series of blogposts, to begin a Research ‘TO DO’list. What can researchers, universities, journals, funders and governments do to improve the quality of peer review and research? This is just the start, we need your ideas too.
Imagine a world where society has the high-quality evidence it needs to make informed
decisions about crime, health and education. What action can researchers, universities, funders and governments take to get us there?
Síle Lane, head of international campaigns and policy at Sense about Science which runs the AllTrials campaign for clinical trial transparency @Senseaboutsci #AllTrials
When evidence goes unpublished the evidence base is incomplete – it has gaps. Furthermore, because it is often so-called ‘negative’ results (results that show there was little or no effect from an intervention or even harm, for example) that more often go unpublished, the evidence that exists is biased. There are many reasons why results go unpublished. It happens when researchers and editors believe that research that didn’t find an ‘interesting’ result is of no interest to the community or no use for a researcher’s career. It happens where publication of results in a journal isn’t the normal practice. It happens when research groups run out of time or funding to spend on a particular project and need to move onto something else. It can also happen, of course, when researchers want to keep evidence hidden for commercial or security reasons.
When results from research aren’t published it means the same research can get repeated unnecessarily. This wastes funds and time and in the case of clinical research, it can lead to patient harm. It means opportunities to explore new areas and to progress the evidence base are missed. Failure to publish results from clinical trials puts subsequent research participants at risk unnecessarily. When research is not peer reviewed and is not opened up to scrutiny by the research community, we miss opportunities to improve research. Decisions made on the basis of the evidence base risk being wrong. Systematic reviews and meta-analyses – the gold standard of evidence – are at risk of being incomplete and biased, meaning that guidelines and decisions based on them may be wrong.
The most studied area of publication bias is in clinical trials. Clinical trials are the large tests often involving hundreds or thousands of people, to check whether a medicine works and is safe. The problem of missing evidence in clinical trials has been known about for decades and hundreds of research papers have been published investigating publication bias in different areas of medicine These pieces of research have routinely shown that around half of the body of clinical trials carried out on medicines we use today have never reported results. Modern clinical trial tracker tools which display live information on which clinical trials have reported results and which have not, show the situation has improved recently but still show that today at least 30% of US and EU clinical trials that are due to have reported results have not. Furthermore, research has shown that clinical trials that show a ‘positive’ result (ie that the medicine is effective) are twice as likely to be published than those that show no effect, and are published more rapidly.
Selective publication of results from clinical trials puts patients at risk and means money is wasted.
A clinical trial of the heart drug lorcainide carried out in 1980 identified that more of the people in the trial who were given lorcainide died than people in the trial taking the placebo. These results were not published until 1993. In the more than a decade when the evidence wasn’t available patients continued to be prescribed drugs in the same class as lorcainide. It has been estimated that 100,000 of these patients in the US alone died unnecessarily.
In response to two separate outbreaks of influenza in 2006-7 and 2012-13, the UK’s NHS spent £424 million on the drug Tamiflu. They decided to do this because guidelines based on evidence from clinical trials supplied by Tamiflu’s manufacturer, Roche, suggested that the evidence showed that Tamiflu reduced complications from the flu and helped people recover from the flu quicker. However, researchers at Cochrane found out that 60% of the research these guidelines were based on was unpublished, and some of the published research contradicted reports from the trials submitted to regulators. It turned out that no one had seen the complete evidence. When all the evidence was finally brought together it showed that the evidence was so flawed such that it was impossible to judge whether Tamiflu was any more effective than placebo.
How to fix the problem of missing evidence
Government TO DO list:
- Set up publicly accessible registers for research.
These already exist for clinical research. In the last decade the proportion of clinical trials that are registered before they begin has risen sharply. Now, it is rare that a clinical trial that occurs in the main clinical trial centres does not get publicly registered in at least one place. This means that researchers, patient groups, investors in research and research regulators can scrutinise which clinical trials have reported results and which have not, and chase down results from unpublished trials.
The World Health Organisation, working with the clinical research community, maintains the Trial Registration Data Set, an internationally agreed 24-item list of the pieces of information that must appear in a register in order for a given trial to be considered fully registered.
- Instruct ethics committees, research governance boards and public research funders to make registration of research before it begins and reporting of the results, whatever the result, a condition of approval to get funding for or run research.
The UK’s research regulator, the Health Research Authority, committed in 2104 to making registration of a clinical trial within 6 months of its start date a condition for granting approval to run the trial. The HRA is exploring levers to ensure results from UK clinical trials are published.
- Where appropriate, bring in legislation to mandate registration of research and reporting of results.
In 2014 the European Union has agreed a new Clinical Trials Regulation which mandates that every clinical trial carried out in the EU or EEA must be registered before it begins and report results onto the publicly accessible register within a year of the end of the trial (within 6 months for paediatric trials). There will be sanctions including financial penalties for breaching the law. This will be the law in Europe from 2020.
- Monitor adherence to registration and reporting rules
The US law, the FDA Amendments Act 2007, mandates that certain US clinical trials must be registered om the federal register before they begin and report results there within a year of the end of the trial, and gives the FDA power to fine research sponsors who breach these rules. Despite the fact that hundreds of applicable trials are gong unreported the FDA has never issued any fines and adherence to this law is patchy.
Publisher/journal TO DO list:
- Make the decision about accepting a research paper independently of whether the results are ‘positive’ or ‘negative’.
- Editors can start to encourage researchers to register their research project before it begins by announcing it as a requirement to have a research paper accepted in the journal.
In 2013 the members of the International Committee of Medical Journal Editors committed to only publishing papers from clinical trials that had been proactively registered on a publicly accessible register in their journals. This led to a sharp increase in the proportion of clinical trials that were registered.
Funder TO DO list:
- Make registration of research and reporting of the results, whatever the result, a condition of new grants
- Mandate that researchers must have reported results for all previous research to be considered for new funds
Since 2016 the US federal funder of health research, the National Institutes of Health, has had the power to withhold remaining or future grant funds from a grantee for failure to submit clinical trial registration and results information.
In 2017 twenty-one more of the world’s largest non-commercial clinical research funders signed up to a World Health Organisation initiative, committing them to bring in a new funding policy to mandate registration of all funded clinical trials and reporting of results within a year of their end and that says that a researcher’s past record of registering clinical trials and reporting results would be taken into consideration when assessing an application for new funds.
- Ensure that grantees understand that the costs, if any, of registering research and reporting results can be covered by grant funds
Universities TO DO list:
- Universities, hospitals and research centres should appoint a named person to be responsible for research reporting. This person should put in place internal processes to ensure research carried out by employees of the institute does not go unpublished.
- Ensure researchers are aware that non-publication of research results is increasingly viewed as research misconduct by governments, institutes and professional societies therefore will be viewed poorly by promotion committees/in tenure decisions.
We extend our thanks to David Tovey @DavidTovey for his work editing these blogposts.