Guest Column | March 22, 2023

What Core Outcome Sets Can Do — If We All Adopt Them

By Stuart G. Nicholls, Senior Clinical Research Associate & Strategy for Patient Oriented Research (SPOR) Facilitator, Clinical Epidemiology Program & Office for Patient Engagement in Research Activities (OPERA), Ottawa Hospital Research Institute

GettyImages-1179336252

Randomized controlled trials (RCTs) in their various guises are a key method in the development of evidence regarding the efficacy or effectiveness of interventions. However, any single trial provides only a snapshot of evidence. In most cases, clinical practice or policy is informed by a portfolio of evidence gathered over multiple trials and which can provide a summative account of how effective an intervention proves to be under various conditions and in various contexts.

A challenge to developing such summative evidence is the variation in the selected outcomes that are used to demonstrate the effectiveness of interventions. For example, a review of trials in cancer indicated that more than 25,000 outcomes are each used only in one or two trials.1 Reviews conducted in other clinical areas have similarly found that many outcomes are used in only a handful of trials.2 This variation precludes comparisons of interventions across trials, making evidence-informed decisions challenging. Yet even when the same outcomes — such as quality of life — are selected, the ways in which they are measured may differ, further complicating comparisons across trials. At the same time, outcomes selected for use in clinical trials may not even be relevant to patients and trial participants.3

In response, there has been an increase in the development of core outcome sets (COS), defined as an agreed minimum set of outcomes that should be measured and reported in all trials of a specific condition.4,5 The consistent collection and reporting of outcomes supports the development of cumulative knowledge regarding the effectiveness of interventions in a given area, which can lead to improved care. Equally, the collection and reporting of outcomes also addresses selective non-reporting of outcomes that may not achieve statistical significance in a trial, thereby helping to address reporting bias in the literature.4 Importantly, the development of a COS should involve the full range of knowledge users implicated by trials in the area of interest in order to ensure saliency and to promote uptake.5 While approaches vary6, the development of COS commonly includes empirical work to examine the range and types of outcomes assessed in the area of interest and a consensus process through which the most important outcomes are identified. As of March 2023, the COMET initiative database includes information on core outcome sets for 481 diseases.

When And Where Are COS Being Adopted By Clinical Researchers?

Despite the goals of COS developers, the uptake of COS within clinical trials varies. A systematic review by Hughes et al. that examined studies evaluating the uptake of outcomes recommended by a range of COS found that uptake varied substantially. Among 337 COS included, studies covered only 17 (5%). Within these studies, the reported uptake rates of full COS ranged from 0% of RCTs in gout to 82% of RCTs in rheumatoid arthritis.7 In a separate study of trials published in five major medical journals, none of the 95 identified trials reported a full COS and only two trials (2%) reported on some outcomes that were included in a COS.8

Indeed, even when identified core outcomes are included in trials, the way in which they are measured varies. For example, in a study of vitiligo trials, Eleftheriadou et al. found that while the core outcome of repigmentation was reported in 95% of trials from 2016 to 2021 following publication of the COS, repigmentation was measured using 49 different outcome measures in 174 eligible trials.9

Why Is There A Low And Slow Uptake Of COS In Clinical Research?

A number of potential drivers of low uptake of COS have been proposed. An analysis by Williamson and colleagues noted multiple barriers to the uptake of COS by trialists. These included a lack of awareness of the existence of a COS, a perception of increased patient burden in using a COS, that the COS development process did not have the relevant knowledge users, as well as their own outcome preferences and lack of standardization regarding the measurement of outcomes.8,10-12 Other identified reasons for a lack of use have included logistical or practical challenges (such as cost) or discrepancies with relevant regulatory and professional bodies.8

A further challenge may be a disconnect between those who potentially benefit from the use of COS in trials: systematic reviewers, decision makers, and those who bear the burden of implementing them: trialists and trial participants.12 Indeed, the availability of multiple COS for a trial potentially increases researcher and participant perceived burden if all items should be measured. Indeed, in their review, Matvienko-Sikar et al. noted that they could identify 56 potentially relevant COS for 31 trials, 34 of these COS being potentially relevant for nine trials (or over three per trial).8 Furthermore, a survey of clinical trialists undertaken by Bellucci et al. found that only 10 participants (16%) thought that COS can enhance the patients’ and public’s voices in research (i.e., the amount of patient input and involvement), while just over one third (37%) thought it could improve the quality of the public’s voice in research.”11

Other potential tensions also exist. For example, patient engagement is increasingly recognized and advocated for with respect to clinical trials. If this engagement indicates that the engaged patients disagree with a COS outcome, or expresses concern over patient burden for data collection of the full COS, which should take precedent: the patients’ preferences or the COS? There are also an increasing number of trials that seek to use routinely collected data to increase the efficiency of RCTs and minimize patient burden for outcome ascertainment. If the outcomes included within a COS are not routinely collected, then this potentially negates the benefits of using routinely collected data if additional outcomes must be collected from patients. The development of the COSMIN checklist (COnsensus-based Standards for the selection of health status Measurement INstruments)13,14 may help, but there remains a need to better integrate research and clinical outcomes.

What Can Be Done To Increase Uptake Of COS?

A noted factor associated with the uptake of COS has been awareness of the COS among trialists.7 Taking an integrated knowledge translation approach, in which key knowledge users are engaged in the development of the COS and kept aware of progress, may improve awareness, as would extending dissemination activities beyond academic publication.10 Indeed, awareness of a COS and a positive perception of the COS itself — potentially augmented through a belief that the COS was developed in an appropriate manner — have been identified as potential facilitators of COS uptake.8 Williamson et al. also suggest potential roles for patient organizations, research funders, regulators, and journal editors in creating awareness and supporting use of a COS.10

The above challenges —varied uptake and proposed solutions — indicate that there is a need to develop implementation research regarding COS as opposed to the ongoing proliferation of new COS. Key aspects of such an implementation science agenda include: better understanding of who the target end users are and their needs; extending research to better understand the barriers and facilitators to use and how these may be affected by contextual factors; and developing solutions that address the identified barriers and that can leverage noted facilitators.

Indeed, this work has already begun. For example, a review examining factors associated with COS uptake found that industry-sponsored RCTs have a higher likelihood of implementing a COS than RCTs not funded by industry.15 Understanding why industry-funded trials appear to have higher uptake is key to extending the reach of COS.

While COS offer potential to improve the cumulative evidence base regarding the effectiveness of interventions, there remain challenges to uptake that require an implementation science approach in order to better understand the barriers and facilitators to the us of COS, but which also can help develop interventions and solutions that can improve uptake.

References

  1. Hirsch BR, Califf RM, Cheng SK, et al. Characteristics of oncology clinical trials: insights from a systematic analysis of ClinicalTrials.gov. JAMA Intern Med. 2013;173(11):972-979.
  2. Pugliese M, Tingley K, Chow A, et al. Core Outcome Sets for Medium-Chain Acyl-CoA Dehydrogenase Deficiency and Phenylketonuria. Pediatrics. 2021.
  3. Gargon E, Williamson PR, Young B. Improving core outcome set development: qualitative interviews with developers provided pointers to inform guidance. J Clin Epidemiol. 2017;86:140-152.
  4. Gorst SL, Gargon E, Clarke M, Blazeby JM, Altman DG, Williamson PR. Choosing Important Health Outcomes for Comparative Effectiveness Research: An Updated Review and User Survey. PLoS One. 2016;11(1):e0146444.
  5. Williamson PR, Altman DG, Blazeby JM, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13.
  6. Williamson PR, Blazeby JM, Brookes ST, Clarke M, Terwee CB, Young B. Controversy and Debate Series on Core Outcome Sets. Paper 4: Debate on Paper 1 from the perspective of COMET [Core Outcome Measures in Effectiveness Trials]. J Clin Epidemiol. 2020;125:222-224.
  7. Hughes KL, Clarke M, Williamson PR. A systematic review finds Core Outcome Set uptake varies widely across different areas of health. J Clin Epidemiol. 2021;129:114-123.
  8. Matvienko-Sikar K, Avery K, Blazeby JM, et al. Use of core outcome sets was low in clinical trials published in major medical journals. J Clin Epidemiol. 2022;142:19-28.
  9. Eleftheriadou V, Bergqvist C, Kechichian E, et al. Has the core outcome (domain) set for vitiligo been implemented? An updated systematic review on outcomes and outcome measures in vitiligo randomized clinical trials. Br J Dermatol. 2023;188(2):247-258.
  10. Williamson PR, Barrington H, Blazeby JM, et al. Review finds core outcome set uptake in new studies and systematic reviews needs improvement. J Clin Epidemiol. 2022;150:154-164.
  11. Bellucci C, Hughes K, Toomey E, Williamson PR, Matvienko-Sikar K. A survey of knowledge, perceptions and use of core outcome sets among clinical trialists. Trials. 2021;22(1):937.
  12. Hughes KL, Williamson PR, Young B. In-depth qualitative interviews identified barriers and facilitators that influenced chief investigators' use of core outcome sets in randomised controlled trials. J Clin Epidemiol. 2022;144:111-120.
  13. Mokkink LB, Terwee CB, Knol DL, et al. The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: A clarification of its content. BMC Med Res Methodol. 2010;2010(10).
  14. Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737-745.
  15. Kirkham JJ, Bracken M, Hind L, Pennington K, Clarke M, Williamson PR. Industry funding was associated with increased use of core outcome sets. J Clin Epidemiol. 2019;115:90-97.

About The Author:

Dr. Stuart Nicholls is the Strategy for Patient-Oriented Research (SPOR) Program Facilitator in the Office for Patient Engagement in Research Activities (OPERA) at the Ottawa Methods Centre. In his capacity as SPOR Program facilitator, Dr. Nicholls consults with researchers to provide methodological guidance regarding all aspects of patient-oriented research. In addition, Dr. Nicholls provides training and education on patient engagement in research and actively contributes to research in this field. Dr. Nicholls has over 20 years of health research experience, publishing over 90 peer-reviewed articles on topics including public attitudes toward population screening, stakeholder perspectives on the ethical challenges in pragmatic clinical trials, and the development of core outcome sets.