How Can Non-randomized RWE Studies Complement RCTs?
By Shirley Wang, Ph.D., associate professor, Brigham and Women’s Hospital, Harvard Medical School, and lead epidemiologist, FDA Sentinel Innovation Center
As stakeholders increasingly look to real-world evidence (RWE) to inform clinical and policy decision-making, it has become even more important to assess not only the benefits of RWE but also the limitations. While RWE can provide deep insights into the drug development process, inform regulatory decisions, and supplement clinical research, we must continuously learn about effective use cases and areas needing further development to ensure we reach safe, effective, and meaningful conclusions from RWE studies. Our team of researchers was interested in evaluating the ability of RWE studies to come to similar conclusions as randomized controlled trials (RCTs). Over the course of five years, the RCT-DUPLICATE Initiative has emerged with key learnings to further inform the industry on the capabilities of RWE.
The RCT-DUPLICATE Initiative
The RCT-DUPLICATE Initiative, launched in 2017, is a series of methods and demonstration projects funded by the FDA and NIH and supported by Aetion. The study aimed to understand how and when database studies can generate valid causal inferences to support regulatory decision-making. To achieve this goal, we used RCT results as a reference standard for valid causal inference and set out to emulate the design of 30 completed RCTs and evaluate the similarity of results between the RCT-database study pairs. We are also working on predicting the results for seven trials that were ongoing at the time we designed and implemented a database study to emulate the trial. We developed a transparent and reproducible process for conducting trial emulations with the assistance of an Aetion Evidence Platform that analyzes real-world data (RWD) to produce evidence on the safety, effectiveness, and value of medical treatments.
Over the past few years, the RCT-DUPLICATE team has published several papers discussing the rationale, methodology, and results for this series of projects.1-4 Most recently, we published the results of 32 non-randomized database studies explicitly designed to emulate specific trials and summarize our evaluation of the concordance of the results between RWE-RCT study pairs (emulation of 30 completed trials and two predictions of ongoing trial results)5 in The Journal of the American Medical Association.
Our findings from these studies have illustrated several points and key learnings which will guide our future studies and initiatives.
Database Studies Can Yield Similar Conclusions To RCTs
We found that database studies can come to similar conclusions as RCTs when the design and analysis can be emulated closely. We showed that when our team of researchers were able to closely mimic the designs of the RCT using RWD, the resulting RWE study produced results that were closely concordant with the RCTs. However, it was more challenging to replicate the results of trials designed with multiple constraints to show effects under “ideal” conditions. Therefore, we observed greater concordance in results for trials with more pragmatic design features. In addition to non-randomization, we identified several elements of trial design that may contribute to differences between an RCT and an RWE study, including skewed age and sex distribution, comparator and outcome emulation quality, placebo control, initiation of therapy in-hospital, dose titration during follow-up, run-in window, discontinuation of maintenance therapy at randomization, and delayed effect over a longer follow-up. These design differences can mean that the hypothetical trial investigators can target in the database study may differ from that of an actual completed trial. When this happens, the study questions and the true effects may differ between the trial and the database study pair.
Simple Binary Success Metrics Do Not Tell The Whole Story
In any trial emulation, there will be design differences and potential sources of bias. We often have to dig deep to tease these apart. Residual bias or random error are always potential explanations for divergence in results between a trial and a database study, but if divergence is driven by design emulation differences, it may be that the database study is accurately targeting a different effect than the trial.
Some Trials Are Harder To Emulate Than Others
There are multiple factors that influence our ability to emulate a trial’s design. For example, we saw in our emulation of HORIZON-PIVOTAL6 and several other trials that it can be challenging to replicate trial results for outcomes with long induction windows or time-varying hazards. Given low adherence patterns in clinical practice, patients may not experience the benefit identified in trials that creates “ideal” but unrealistic conditions to maximize their ability to detect an effect. This is known as the efficacy-effectiveness gap. After aligning the follow-up times between these trials and the duration of exposure observed in clinical practice, we observed closely calibrated effect sizes.
Additionally, while we used the primary result of a single trial as a reference standard, we also needed to consider the uncertainty in the result of any single trial and, therefore, the uncertain replicability of a trial’s findings even by other trials. We observed this issue in our emulation of the sister trials EINSTEIN-DVT7 and EINSTEIN-PE.8
Database Studies Are Complementary To — Not A Replacement For — RCTs
We generally observed that when the data and design are fit-for-purpose, non-randomized database studies can come to similar conclusions as randomized trials about drug effects. However, the real benefit of database studies is their ability to generate evidence that is complementary to trials, including tackling relevant clinical questions where, for a variety of reasons, a trial cannot be conducted. Therefore, when we are evaluating how non-randomized database studies can complement RCTs, we must consider: what is the hypothetical target trial that would match the need of the end user? Are they looking for evidence of effect under ideal conditions or clinical practice conditions? Overall, the RCT-DUPLICATE initiative demonstrates the potential of RWE to supplement RCT findings and serves as a guide for examples and cases where RWE and RCTs can be expected to reach comparable conclusions.
Future Considerations
This research, among other ongoing projects, is integral to our understanding of the capabilities and limitations of RWE for generating valid inferences that support regulatory and health technology assessment (HTA) decision-making. Continuing efforts with electronic health records and specialty registry databases will deepen our understanding of the methods and validity of trial emulation to gather RWE.
References:
1. Franklin JM, Glynn RJ, Martin D, Schneeweiss S. Evaluating the Use of Nonrandomized Real-World Data Analyses for Regulatory Decision Making. Clinical Pharmacology & Therapeutics. 2019;105(4):867-877. doi:10.1002/cpt.1351
2. Franklin JM, Glynn RJ, Suissa S, Schneeweiss S. Emulation Differences vs. Biases When Calibrating Real-World Evidence Findings Against Randomized Controlled Trials. Clinical pharmacology and therapeutics. Apr 2020;107(4):735-737. doi:10.1002/cpt.1793
3. Franklin JM, Patorno E, Desai RJ, et al. Emulating Randomized Clinical Trials with Nonrandomized Real-World Evidence Studies: First Results from the RCT DUPLICATE Initiative. Circulation. Dec 17 2020;doi:10.1161/CIRCULATIONAHA.120.051718
4. Franklin JM, Pawar A, Martin D, et al. Nonrandomized Real-World Evidence to Support Regulatory Decision Making: Process for a Randomized Trial Replication Project. Clinical pharmacology and therapeutics. Apr 2020;107(4):817-826. doi:10.1002/cpt.1633
5. Wang SV, Schneeweiss S, Franklin JM, et al. Emulation of Randomized Clinical Trials With Nonrandomized Database Analyses: Results of 32 Clinical Trials. Jama. Apr 25 2023;329(16):1376-1385. doi:10.1001/jama.2023.4221
6. Black DM, Delmas PD, Eastell R, et al. Once-yearly zoledronic acid for treatment of postmenopausal osteoporosis. The New England journal of medicine. May 3 2007;356(18):1809-22. doi:10.1056/NEJMoa067312
7. Oral Rivaroxaban for Symptomatic Venous Thromboembolism. New England Journal of Medicine. 2010;363(26):2499-2510. doi:10.1056/NEJMoa1007903
8. Büller HR, Prins MH, Lensin AW, et al. Oral rivaroxaban for the treatment of symptomatic pulmonary embolism. The New England journal of medicine. Apr 5 2012;366(14):1287-97. doi:10.1056/NEJMoa1113572
About The Author:
Shirley Wang, Ph.D., is an associate professor at Brigham and Women’s Hospital, Harvard Medical School, and lead epidemiologist for the FDA’s Sentinel Innovation Center. She leads the Meta-Research in Pharmacoepidemiology program, with recent projects aimed at improving the transparency, reproducibility, and robustness of evidence from healthcare databases (www.repeatinitiative.org) and informing when and how real-world evidence studies can draw causal conclusions to inform regulatory or other healthcare decision making (www.rctduplicate.org). She is currently PI on multiple NIH R01s and is also funded by the FDA. Her methods work has received three awards from international societies.