A Simulation-Based Approach To CRO Selection

Protocol or clinical trial simulations have been on the radar screen of the industry for quite some time as a technique for optimizing trial design and decision making. In a prior Clinical Leader article, the author summarizes the process nicely:

“With regards to the drug development process, M&S (modeling and simulation) involves modeling compounds, mechanisms and disease level data based on historical observations. Computer simulations are run on these models to generate information that can be used to predict outcomes, thereby improving the quality, efficiency and cost-effectiveness of decision-making.”¹

Simulation-based training has also been widely used in the airline industry (think flight simulators), the healthcare industry (think patient manikins and clinical care settings that mimic real-life scenarios), and others for years. In fact, medical simulation derived from the aviation industry, which has utilized simulation-based learning practices to train pilots since just after World War I.² “Simulation-based learning can be the way to develop health professionals’ knowledge, skills, and attitudes, whilst protecting patients from unnecessary risks.”³ Simulation-based training can involve a combination of technology and techniques to allow learners to practice their skills in simulated real-life situations without putting actual patients at risk.

So, what does all of this have to do with CRO selection, you ask? I’ll do my best to connect the dots. Let’s start with what it isn’t. I am not using the concept of simulation in terms of a computer modeling and simulation technique to predict which CRO will perform better than the other as a means of facilitating a CRO-selection decision (although I’m quite confident some really smart statistician could probably come up with an algorithm to do that!). Rather, I’m referring to concepts derived from simulation-based training and Lean Six Sigma techniques to simulate operational aspects of the trial to see how the CRO will respond to real-life situations. Doing so allows sponsor teams to get a better sense of how the CRO employs creative problem-solving, manages and mitigates risk, and deals with the complexities and realities of the inevitable challenges that will arise in the conduct of a trial that can’t be truly vetted during a standard bid-defense meeting.

Definitions And Theoretical Concepts

A “Day in the Life of,” or DILO, is a Lean Six Sigma tool used to observe the activities on the job, measuring time spent on productive vs. non-value-added activities. For our purposes, I will use a DILO to represent a mock run-through of certain operational aspects of a clinical trial or study conduct; not so much to identify non-value added tasks, but to uncover potential pitfalls that could occur during study execution. While there are very sophisticated ways to conduct simulations of study visits with actual subjects and site representatives that can provide incredibly valuable insights,⁴ this takes a lot of planning and often falls outside of the timelines that many teams can accommodate. Nonetheless, applying many of the techniques and doing the dry-run simulations with internal study team members and vendor partners can be equally enlightening. For example:

At the end of the day, these mock-run sessions are designed to uncover unanticipated issues so they can be planned for, managed, and mitigated in a timely and efficient manner. These techniques can identify ways to ease the burden on the sites and subjects and make trial conduct more efficient and cost-effective. But most importantly, simulation sessions are designed to protect subject safety, something everyone can agree on is job No. 1 in our industry.

Practical Application For CRO Bid-Defense Meetings

Much has been written about ways to optimize the RFP process and streamline the bid-defense meetings, which I won’t recount here. As someone who spends a lot of time troubleshooting studies that are outsourced, I see a huge missed opportunity to bring DILO simulations to the CRO-selection process. Imagine a different type of bid-defense meeting where the sponsor creates a simulated environment for the top risk areas associated with the study. Rather than running through a basic capabilities presentation or repeating information from the proposal, the CRO team members (preferably the actual team members from key functional areas that will be assigned to the study) are put through the paces to simulate likely “what if” situations that will arise during the trial.

In some cases, the risk areas are known and shared with the CRO as part of the RFP process and in other cases the CROs are asked to come to the table with new areas identified. In some situations, the dry runs are done in a live setting to see how the team members think and interact on their feet and in other cases the CROs are asked to bring the results of their simulations. In the latter case, the CRO team conducts a mock run-through of the situations internally, demonstrating that it has not only thought through the key risk areas but how it will approach the issues, what recommendations it has, and how the risks can be mitigated.

By way of illustration, some of the more effective sessions I’ve been able to participate in involved walking through the patient journey from where the subject will learn about the study opportunity, to how they will get pre-screened, consented, screened, and randomized. The process flow was literally mapped out by each of the CROs, which also provided sample study awareness materials, job aids, screen shots of the interactive web response system (IWRS), and so forth. This allowed everyone on the sponsor team to see, hear, and experience what the subjects and sites would go through to get a subject enrolled in the trial and how the different CROs were thinking about the process. As you can imagine, the results varied widely in terms of the level of depth, detail, and sophistication of the CRO “presentations,” which provided a lot of information to aid the selection process.

In another study, the sponsor — a small biopharma company with limited experience — was struggling to figure out the best way to integrate some electronic data capture (EDC) data with external vendors and data sources. It used the CRO simulation sessions to really learn and understand from the CRO experts about the best ways to handle this. The CRO that was ultimately awarded the contract actually came to the table with a mocked-up EDC system, a video showing how the sites would upload the data, the central reader’s review process, and example outputs of the analysis reports with mock data from the lab and imaging vendors. While it didn’t do the simulation live per se, it had conducted all of the dry run through steps in advance to simulate what was likely to happen during the course of the trial. The mock-up of the EDC system allowed the sponsor team to “touch” the system, and the video allowed the sponsor to “see and feel” the process the sites, subjects, and data would go through in these specific aspects of the trial.

In both cases, the sponsors had developed objective scoring criteria that enabled them to score the creativity, organization, planning, communication, resources, and other related aspects that enabled side-by-side comparisons of the CROs’ performance.

Incorporating some type of trial simulations, DILOs, or dry runs can be a great insurance policy to ensure the CRO truly has the expertise to translate general capabilities into actual conduct given the nuances and complexities of a given clinical trial. The CROs still have to invest in preparing for and sending representatives to the bid-defense meeting. By taking a simulation-based approach, those that invest are already well ahead of the planning curve and can convert the simulation experiences into actual trial conduct upon award of the contract. For the skeptics in the audience who say that this will all take more time and effort to set up, as the saying goes, pay now or pay later. Either way, you will have to pay. Would you rather pay for productive and real-life related insights or spend time on non-value-added presentations that don’t really provide a way to differentiate the true capabilities of the CROs? You decide!

References:

1. https://www.clinicalleader.com/doc/modeling-and-simulation-in-clinical-trials-real-potential-or-hype-0001
2. https://www.healthysimulation.com/medical-simulation/
3. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2966567/
4. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5545635/

About The Author:

Beth Harper is the president of Clinical Performance Partners, Inc., a clinical research consulting firm specializing in enrollment and site performance management. Harper also is the workforce innovation officer for the Association of Clinical Research Professionals (ACRP). She has passionately pursued solutions for optimizing clinical trials and educating clinical research professionals for over three decades. Harper is an adjunct assistant professor at George Washington University and has published and presented extensively in the areas of protocol optimization, study feasibility, site selection, patient recruitment, and sponsor-site relationship management. She serves on the CISCRP Advisory Board and the Clinical Leader Editorial Advisory Board, among other industry volunteer activities.

Harper received her B.S. in occupational therapy from the University of Wisconsin and an M.B.A. from the University of Texas. She can be reached at 817-946-4728, bharper@clinicalperformancepartners.com, or bharper@acrpnet.org.