Is Outcome Switching Still A Problem In Clinical Trials?

By Ed Miseta, Chief Editor, Clinical Leader

The scientific method is pretty straightforward. Ask a question, conduct background research, construct a hypothesis, perform an experiment to test the hypothesis, and determine if the data supports or rejects the hypothesis. Your experiment should also be fair, meaning only one factor, or variable, is changing while all others are held constant.

Clearly, the method does not stipulate changing your hypothesis midway through the experiment if the results are not as expected. This would involve adding another step to the method which does not exist, and would also violate the fairness test. Unfortunately, many are questioning whether altering the hypothesis, something known in the life sciences as outcome switching, is a problem in clinical trials.

An article I wrote in 2015 addressed this issue. It seems the success rate for clinical trials dropped significantly since ClinicalTrials.gov came into existence. Prior to 2000, companies would perform clinical trials and report the results at the conclusion of the trial. This process left open the possibility for companies to switch outcomes. Post 2000, companies were required to record their trial methods and outcome measures before collecting the data.

A study conducted by PLoS ONE addressed this issue. In a sample of 55 trials testing treatments for heart disease, 57 percent of the studies conducted prior to 2000 reported positive results from the treatments. In studies that were conducted after 2000, the studies reporting a positive result fell to only 8 percent. This obviously had many questioning whether the reporting of outcomes prior to a study was forcing companies to abandon outcome switching.

But is outcome switching a problem for pharma today? The COMPare (CEBM Outcome Monitoring Project), conducted by a team of academics, medical students, and programmers based at the Centre for Evidence-Based Medicine at the University of Oxford, wanted to find out. The COMPare team monitored clinical trials for switched outcomes. The team monitored all trials published in the top five medical journals, which included NEJM (New England Journal of Medicine), JAMA (Journal of the American Medical Association), The Lancet, Annals of Internal Medicine, and BMJ.

As part of the project, each trial was analyzed for outcome switching by comparing the protocol (or registry entry) with the trial report. If a trial is found where the evidence points to a switched outcome, the COMPare team is writing a letter to the journal to correct the record. They also tracked which journals published their letters, and which did not. The research was conducted between October 2015 and January 2016, and examined a total of 67 trials.

The results do not seem to paint a positive picture. Of the 67 trials checked, only nine turned out to be perfect. The other 58 trials showed 301 outcomes that were not reported, and the team found 357 new outcomes that were added. Of the 58 letters sent on those studies, six were published, 16 were rejected by the journal, and 36 remained unpublished and not rejected four weeks after being sent. In an article on Vox.com, Ben Goldacre, a doctor behind the study, notes BMJ issued rapid corrections. NEJM dismissed concerns of outcome switching, JAMA was friendly but ponderous, and Lancet published the correction letters but regarded the issue as a matter for those conducting the trials, and not journal editors. The Annals of Internal Medicine surprised everyone by dismissing concerns, writing error-laden "rebuttals," and telling trialists they needn’t worry about replying to corrections.

All of this may have you wondering what the FDA has to say about outcome switching. In a blog posted to their website, the COMPare team addresses that viewpoint as well. New drugs must comply with the FDA Amendments Act (FDAAA), Section 801. Section 801 specifies the sponsor or primary investigator is responsible for registering the study, and results must be submitted within one year of the trial’s completion. The Protocol Data Element Definitions also specify what elements must be reported to the registry.  The primary outcome measure is listed as a mandatory requirement of the registry for all trials registered on ClinicalTrials.gov on or after December 1, 2012. Secondary measures are also a mandatory requirement.

Of course, we also know that Section 801 specifies stiff daily fines to be applied when results are not reported on time, even though to date no fines have been assessed on the one-third of results missing from the site. As COMPare notes, this only adds to the confusion surrounding the requirements, possibly undermines the value of registry entries, and does not help with the issue of outcome switching.