Challenges In The Reanalysis of Randomized Clinical Trial Data

By Ed Miseta, Chief Editor, Clinical Leader

In September, the Journal of the American Medical Association (JAMA) published a study on the reanalysis of randomized clinical trial (RCT) data. The main objective of the study was to identify published reanalysis of RCT data to characterize methodological and other differences between the original trial and the reanalysis, and to assess whether the reanalysis changed interpretations from the original article.

The study notes that, after careful re-examination, secondary researchers did not always come to the same conclusion as the original researchers. A surprising total of thirteen studies (35 percent of the 37 eligible studies discovered) led to interpretations that were different from that of the original article regarding the types and number of patients who should be treated.

The central issue appears to revolve around the data. In many instances, researchers are not sharing their raw data and are therefore missing important opportunities for additional analysis. But more importantly, the study questions whether researchers always have the right tools to properly analyze the data. These two factors together could create critical holes in clinical trial findings.

Dr. Thomas Nifong, EVP of diagnostic tests at Definiens, has more than 15 years of experience in the clinical arena. Nifong reviewed the JAMA study and notes not all differences were due to errors in the original analysis. “One analysis does not necessarily trump another,” he states. “You may have had an instance where the reanalysis was asking a slightly different question, where the outcome was defined a little differently, or where different statistical tools were used. It is not uncommon for different conclusions to be drawn from the same raw data using different approaches.”

That being said, Nifong notes it is significant that in some of those cases, a fresh look by the same or different authors was able to draw a clinically relevant conclusion that was different from what was initially published. If the original study was published and treated as dogma, it may have led to inappropriate treatment regimens.

Complexity Is Part Of The Problem

Collecting clinical trial data is a complex process, and there are numerous issues that can arise, should there be problems in initial study design or should researchers discover discrepancies in the study. Looking at the JAMA study, most of the trials examined were older studies, ranging from the 1990s through the early 2000s. That means the trials were also conducted prior to the era of big data. As a result, the findings of the reanalysis may not be indicative of what we can expect to see in the future.

“With big data in the picture, we will definitely be adding another layer of complexity with more chances to introduce error,” states Nifong. “There are also several intermediary steps between when the data is generated and when it is analyzed. Problems can result at any point along the way.”

At the beginning of the trial, a decision must be made as to what data will be collected. Therefore, right from the start, there is an opportunity to collect an inadequate amount of data, which could lead to a problem with the final analysis. As the industry moves forward with personalized medicine and big data, there will be concerns, not only around what data to collect, but how to collect and store data. Laboratory data relies on sample integrity, so the raw data itself is subject to pre-analytical variables. That provides ample additional opportunities for errors to occur.

The Right Tools Are Critical

Nifong is quick to note he doesn’t believe there were large errors made in the studies JAMA examined. In some of those cases, he feels the appropriate statistical tools may not have been used in the original analysis. In others, changes in the clinical question or definition of outcome occurred between the original analysis and the subsequent reanalysis. The changes that took place may have been what provided the opportunity for the reanalysis.  

There also does not seem to be absolute agreement in terms of what statistical tools should even be used. In the past each patient either had a measured outcome or he didn’t. Going forward, researchers will have to apply statistical methods to complex data sets, without always having precedents. For example researchers will be confronted with a combination of genomic and/or phenomic information and they will need to model the probability as to whether someone will have a certain outcome or respond to a certain drug. Comparisons are no longer black and white.   

That being said, there are certainly outright errors that occur in clinical trial data analysis as well. While not part of the JAMA report, Nifong notes there have been times, even in recent years, where researchers have not been able to reproduce published results using publically available data. In those instances, if the data had not been publically available, there is a chance no one would have realized there was an error and clinical trials would have continued with patients being treated with inappropriate therapeutic regimens.

How Can We Better Deal With Data?

All of this might make you question what we can do to better deal with data. There are clearly some issues that exist, and big data will only compound the problem, creating more opportunities for error, differences in interpretation, or changes in the tools that are used. “Going forward, as we generate data from trials, in particular methods for stratifying patients for therapies, we have to realize that data will not be conclusive in and of itself,” says Nifong. “Any results we see will no longer be static conclusions. As new therapies are introduced, we need to have methods in place to continue to look at the original data, but also complement it with additional data.”

To accomplish this, there will need to be discussions about ownership of the data itself. For reanalysis to occur there must be public access to it. Some companies and universities will hold their data very dearly and not want to release it. Nifong notes there are indeed privacy issues to deal with, but there are always ways for data to be made anonymous and safely shared without concern. In fact, he believes the privacy issue is oftentimes used as an excuse to not release data. “My personal impression is there has to be a way for data to be reused and reanalyzed,” he adds. “I also believe the original authors should be involved in that process, since freely available data can open up opportunities for bias on the other side as well. The researchers who performed the original research should be able to discuss their experiences with any nuances in the data, which would significantly help with any reevaluation of the data.”

And one final thought: There have been instances of papers being retracted after news surfaced that there were errors in the data. If it can be done in a practical manner, a better review of the statistical analysis could be one way to ensure there is data integrity and that the statistical tools used were in fact appropriate for the analysis.