A Practical Framework For Assessing Risk In Clinical Trials

By Cristin MacDonald, Ph.D., The Avoca Group

The purpose of a comprehensive assessment system is to support structured, systematic, objective, and rational decision making. This holds true even when assessing something as difficult to pin down as risk. The Avoca Quality Consortium (AQC) takes the position that a risk assessment system should support decision making with respect to three fundamental questions:

1. What factors are likely to put quality/timelines/budget at risk?
2. To which risks should mitigation resources be allocated, and at what “trigger levels”?
3. How can outcomes be optimized through quantitative risk modeling?

However, before we can build an assessment system that will answer these questions, we need to lay a solid taxonomic foundation for risk.

Laying The Foundation For Risk Assessment

The development of a comprehensive and effective system for the assessment of risk has been complicated in our industry by the fact that the term “risk” has not been well understood. For example, “risk” has been used in the industry to include any and all of the following:

• An unwanted event that could possibly happen that could then possibly create a quality problem, e.g., “site activation may be delayed long past training.”
• An unwanted event that has already happened that could possibly create a quality problem, e.g., “site activation has been delayed.”
• A quality problem that is already in the making, e.g., “high rate of protocol violations to date.”
• A quality problem that is a done deal, e.g., “very high protocol violation rate interferes with data interpretation.”
• A condition that makes it “really bad” when there is a quality problem, e.g., “pivotal Phase 3 study.”

You can readily see that “risk” has become, in common parlance, an all-embracing term that includes not only true risks, but also emerging or realized issues (outcomes), as well as modifiers of consequence.

The AQC examined risk in terms of common usage, dictionary definitions, and ISO principles. The result, documented in our AQC Glossary, is a very specific four-part definition of risk:¹

From this foundational definition, we can then go on to define related terms that are essential for the systematic assessment of risk:

• Risk driver: A hypothetical circumstance believed to impact probability of adverse outcome.
• Risk event: The “realization” of a risk driver (often are contributor metrics).
• Emerging issue: An issue that has already begun to actualize and that is likely to adversely impact outcomes if not addressed (predictor metrics).
• Outcome: An adverse outcome represents a “realized” risk, i.e., a problem that’s a “done deal.”
• Consequence driver: Any variable that influences the significance of the adverse outcome.

Building A Comprehensive Risk Assessment System

By correlating these risk terms with the statements made commonly in the industry about risk and with the three questions we initially posed as critical for decision making around risk, Denise Calaprice-Whitty, Ph.D., Avoca Quality Consortium senior consultant, created a framework for a comprehensive metrics and risk assessment system, as shown in Figure 1.

Figure 1: A framework for risk assessment²

This framework can be mapped to associated actions, as shown in Figure 2. This “thinking tool” supports comprehensive consideration of the full pathway to each outcome and the nature of events that could put each at risk. Prior to launching and during execution of a study, outcome areas identified as having high “loading” for risk drivers may be targeted for preventive action and/or for a higher level of oversight and/or vigilance toward related contributor and predictor metrics. Effective management surrounding risk at each level should reduce the incidence or consequences at the next level, thus reducing the probability of an adverse outcome.

Figure 2: Components of risk and mapping to associated actions

For example, assume that a site was trained today on the assumption that it would be activated within two weeks. You are subsequently informed that, due to a delay in drug manufacture, patient enrollment will likely not begin for three months. This is a risk driver. When patient enrollment will actually begin is unknown — if it starts within a few weeks, the site personnel will still be effectively trained. But if it starts three months later, there is a risk that site personnel will have forgotten the protocols, potentially impacting clinical trial accuracy and patient safety. This risk driver can be proactively addressed by planning for a second training session to precede patient enrollment.

Reactive steps are necessary in other cases. For instance, suppose a site is having a high rate of protocol violations. This is an emerging issue. Once identified, appropriate resources must be allocated to correct the issue and remediate any negative impact to the trial or patients.

Putting Risk Assessment Into Practice

The advent of ICH E6 (R2), with its risk-based approaches to clinical trial management, has not been one of the smoother transitions in the industry. But with more precise definitions of risk and risk-related terms, a comprehensive framework for risk assessment, and a method of mapping actions to issues to guide decision making, organizations can become more efficient and effective in their identification and management of risk, moving from a reactive state to a more proactive state.

Endnotes:

1. Regarding risk being the effect of uncertainty on objectives:
   • An effect is a deviation from the expected -- positive and/or negative.
   • Objectives can have different aspects (such as financial, health and safety, and environmental goals) and can apply to different levels (strategic, organizationwide, project, product, and process).
   • Uncertainty is the state, even partial, of deficiency of information related to understanding of or knowledge of an event, its consequence, and likelihood (ISO GUIDE 73:2009 Vocabulary for Risk Management).
2. As a point of reference, the AQC taxonomy considers quality metrics as falling into the following three categories:
   • Contributor Metrics: Contributor metrics measure conditions and activities that are logically expected to contribute to quality outcomes, but that do not have a strong enough statistical relationship to qualify as predictors. These metrics are most commonly process-related, addressing the question “Are we doing what we believe we need to do to achieve adequate quality?”
   • Predictor Metrics: Predictor metrics should be able to answer the question "How we are doing so far in our attempt to achieve an adequate level of quality?"
   • Outcome metrics: Outcomes metrics are used to answer the questions "How did we do with respect to the quality outcome? Did we achieve an adequate level of quality?"

For more information about The Avoca Quality Consortium’s metrics and risk assessment system, or to learn more about The Avoca Group, visit www.theavocagroup.com or contact Avoca at www.theavocagroup.com/contact.

About The Author:

Crissy MacDonald, Ph.D., is executive director of client delivery for The Avoca Group. As leader of Avoca’s Integrated Consulting, she provides consulting to top pharmaceutical, biotech, and contract research organizations, and oversees client deliverables, systems, and processes across Avoca. MacDonald has 12 years of pharmaceutical industry experience with expertise in clinical research, process development, and strategic management. Her previous professional roles have touched every stage of the clinical trials process, from preclinical research through late-stage clinical development.

MacDonald earned a Ph.D. in biomedical engineering from Drexel University and a bachelor’s degree from Lafayette College.