What Is Real-World Evidence, Anyway? Industry Experts Weigh In
A conversation with Shawn Li, Bristol-Myers Squibb; Danny Wiederkehr, Pfizer; and Soeren Mattke, University of Southern California.
This is the first article in a two-part roundtable Q&A on real-world evidence (RWE) in the 21st century. In this first installment, we will explore common misconceptions surrounding real-world evidence (RWE), what should be considered to achieve good real-world data (RWD) analyses, and the impact this data can have on clinical treatment and regulatory development. RWD is anonymized health-related information reported and collected in day-to-day clinical settings. This data may provide new insights about a medication beyond what is collected from a clinical trial alone.
Participating in this Q&A are three industry experts:
- Shawn Li, Ph.D., group director, Worldwide Health Economics and Outcomes Research, Bristol-Myers Squibb (BMS)
- Danny Wiederkehr, MPH, senior director and team lead, Global Health Economics and Outcomes Research Team, Pfizer
- Soeren Mattke, M.D., D.Sc., director, Center for Improving Chronic Illness Care, University of Southern California (USC)
What are some of the common misperceptions about real-world evidence (RWE)?
Shawn Li, BMS: In my experience, one of the most common ones is around its value for key healthcare decision makers — people tend to over- or underestimate what it can bring to the table. While RWD analyses cannot replace randomized clinical trials (RCTs) and have several limitations, they certainly serve to complement clinical trial data by taking more of the evidence that exists into account, to help paint a fuller picture of a treatment option. From this greater data collection, RWE helps to inform decisions by giving healthcare professionals new insights about treatment options in clinical practice.
Over the past 20 years, methodologies for analyzing RWD have evolved to better extract relevant information and reduce the impact of confounding bias. Additionally, it is important to note that the generalizability of RWD findings may be limited to the population studied, which can be more varied than those previously studied in the traditional RCT setting. While electronic health records (EHRs) provide clinically comprehensive data, we have to be aware that they do not always capture the full range of a patient’s treatment journey.
Danny Wiederkehr, Pfizer: One of the greatest misconceptions I have encountered is that researchers regularly influence the direction of the RWD results in their favor, which ultimately may not be in the interest of improving patient care or ensuring safety. Similar to what Dr. Li shared, RWD sources and methodologies have greatly evolved in recent years, allowing us to approach these analyses much more rigorously and with more disclosure and transparency.
As the BMS-Pfizer Alliance, we ensure that all primary outcomes and methodologies are pre-specified and vetted by clinical and methods experts prior to analyzing any data, to ensure that there is no potential for cherry-picking. Additionally, we register our RWD analyses on clinicaltrials.gov or EU PASS register, which is required for investigators to do for traditional randomized controlled clinical trials.
Soeren Mattke, USC: People discuss RWE as if it were a single construct, but there are many sources of information from which to collect and analyze data. For example, there are claims and billing data that have limited clinical information; raw clinical data that comes from EHRs, which have rich but unstructured information; and curated data that is preprocessed, abstracted, and audited. An entire continuum of RWD exists.
What separates good RWD analyses from other, lower-quality analyses?
Mattke: Form follows function. If you have a question that can be easily answered with billing or claims data, there is no reason to get fancy with complex methodologies or sources. One can get a lot of data for very large populations through these sources. However, it is important to note that some research questions may require more clinical detail than is collected for billing or claims reporting needs. To obtain good RWE, one must think about their question first and then find the right data source.
Li: Transparency is key. If we want to conduct a RWD study, it is paramount to register the study with clinicaltrials.gov, EU PASS register, or other public study registration sites to confirm the aims of the study ahead of the analysis. We ensure that the study protocols include prespecified objectives, study design, methodology, endpoints, and subgroups the same way we would for an RCT prior to starting the study. Strong RWD analyses typically leverage advanced statistical methods to address issues such as adjusting for confounding variables to ensure the finding is sound.
Wiederkehr: To build on Dr. Mattke’s comment, you need to outline your research question and your source before anything else. Then, once data collection has begun and the initial findings are revealed, it is important to consider how and when to appropriately adjust the approach to the analysis. To that end, another best practice we adhere to at the BMS-Pfizer Alliance is putting the prespecified primary analysis in the main body of the manuscript, then testing those findings with a number of sensitivity analyses by tweaking the methods or populations slightly to confirm the findings. We usually include these findings in a supplementary appendix for transparency.
ISPOR, the professional society for health economics and outcomes research (HEOR), is a global organization committed to advancing the science and practice of HEOR around the world. The society regularly provides updated recommendations within its Good Research Practice for Outcomes Research Reports. As the HEOR landscape grows and evolves, ISPOR will likely continue to evolve these guidelines.
Can you share some examples of the impact that RWE can make?
Li: For instance, the FDA has used claims databases included in the Sentinel network to help monitor the safety of medical products after they have reached the market —and to complement its existing Adverse Event Reporting System. We’re able to use these data sources to conduct similar evaluations on safety and effectiveness.
Wiederkehr: Another really important area we haven’t touched on yet is how RWE can help shape the design of healthcare benefits and coverage. Because many value assessments are based on RCTs and not the real use of treatments in everyday medical settings, these findings may allow payers to offer plans that are more accurately value-based, which could improve or focus access to therapies or care options.
In the past, formulary and contracting decisions were based on economic models that were often indirect comparisons of RCTs. With RWD evaluations we have the capability to generate real-world comparative effectiveness data, longer-term outcomes data, and real-world cost data that can enable outcome-based contracting.
Does this mean we don’t need RCTs anymore?
Li: While the data collected in real-world analyses are of importance to us, they cannot replace RCTs. Instead, they are used to complement existing data by offering new insights on important healthcare topics not often observed in previously run trials. This data helps to provide reassurance that the results observed in earlier studies, such as those for initial drug approval, are consistent with day-to-day clinical care of everyday patients. And in cases where there are no direct comparisons to a RCT, these analyses can offer insights otherwise unavailable to healthcare professionals and provide direction for future clinical research.
Wiederkehr: RWD is increasingly becoming an important consideration in healthcare decision making. In fact, we are seeing an increased emphasis on the importance of RWE and its role in the everyday work of regulatory bodies like the FDA. However, echoing what Shawn shared, I believe that while the information gained from RWD analyses provide important insights into medicines that are being prescribed, it is important that we consider these as complementary data for what was gathered in RCTs.
It is important to remember that observational real-world studies can only evaluate association and not causality. Due to these limitations, RWD analyses are not used as stand-alone evidence to validate the efficacy and/or safety of a treatment.
In the second installment, our experts discuss how RWE is utilized, why RWD analyses are getting more attention, and what this recent attention means for the future of medical practice and drug development.