How do top performing companies audit their CROs? When speaking to clinical executives, I found this to be one of their top concerns. At the inaugural Clinical Leader Forum, we brought together a panel of experts to discuss the topic. The panel featured Anita Treohan, director of program management and clinical operations for ImmusanT; Frances Grote, director of clinical sourcing advisory services for Information services Group; and Kathryn Stiede, head of clinical operations for Nimbus Therapeutics. The session was moderated by Laurie Halloran of Halloran Consulting Group.
All of the panelists agreed that auditing refers to a set of activities sponsors perform to ensure regulatory requirements are being met by their CROs. Before you engage with any vendors, you have to evaluate them to ensure they’re capable, qualified, and understand the pertinent regulations applicable to the work you want them to perform. When it comes to auditing, you’ll need to do additional evaluations to ensure they’re managing your project, data, and services according to those standards.
Although there are clearly risks when audits are not performed, one panelist noted most vendors, be they CROs or labs, all have a set of SOPs and processes in place that meet regulatory guidelines. The main issue is how easy it will be to work with those processes and procedures. The more flexible the vendor is, the easier it is to work with them.
“You cannot overemphasize the importance of performing audits,” adds Grote. “If you want to get your drug to market, you have to be able to demonstrate that you performed audits. If you continually put off that ‘chore’ until the end, you will be in a situation where everyone is in panic mode.”
All panelists agreed audits should be done at the start of the partnership and thereafter on a regular basis. The end of the study is the wrong time to find out that something should have been done differently, and only regularly occurring audits can ensure you’ll avoid that situation. Performing a successful audit means addressing the following five issues with your CRO.
1. “I Did Not Select Them”
A problem for some executives is having to deal with CROs, CMOs, labs, and other vendors that had already been selected before they joined the company. In those instances, the executive basically inherits all of the vendors and then has to perform audits and due diligence to find and address gaps. When you are the entire clinical operations team for your company, that task can be daunting. Additionally, if your company does not have any clinical SOPs, or you do not understand the SOPs of your vendors, one option is to bring in outside help. Stiede stated that is exactly what she does at Nimbus.
“I do employ an outside QA (quality assurance) resource to help with audits,” she says. “We did not do audits on all of our vendors prior to selection. We based our selection on six criteria we developed based on our internal culture and the solutions the vendors provided. Once these vendors were selected, we performed the audits. The diligence process then has to continue throughout the life of the study and your partnership with the vendor.”
“Visiting again early on in the program to identify corrective actions using real project processes will save issues from becoming unchecked larger problems later,” added Halloran.
2. Who Should Conduct Audits?
One discussion that arose during the session was around who should conduct the audit. If audits are supposed to be conducted independently, does that mean they should be done by an outside company or at least performed by someone outside the clinical operations team?
Grote notes when she was on the sponsor side, she would have discussions with personnel from clinical operations, data management, safety, and QA to determine how these areas might work together. In some instances, she found the QA department expected clinical operations to take on the burden of resolving issues that were important to QA.
“These were not necessarily audit findings, but QA felt they were risks to the program, and they expected operations to figure out how to address them and run the CAPAs (Corrective and Preventive Actions). It was a huge learning experience for me.”
But is clinical operations really the right group to address CAPAs? One panelist noted that a clinical group will oftentimes have no idea what to do about them. For example, if your CRO is maintaining your documentation, the CAPA should address the specifics of any gaps that may exist regarding CRO SOPs or regulatory requirements. Additionally, someone will always need to ensure a CAPA has been managed to completion. A good deal of thought should be put into who is truly in the best position to do that.
3. Top Audit Objectives
When performing a CRO audit, what should your top objectives be? Treohan states her top priority tends to be personnel and the experience and expertise they have in specific areas. After that, she will examine SOPs and the flexibility of the processes the CRO has in place. The flexibility of the CRO is always a key attribute.
Stiede agrees experience of teams and individuals is a primary concern. As the entire clinical operations team for her company, she needs that CRO to have the needed expertise. “The cultural fit is really important to me. I am the only person on my team, but I may have to work with a team of 20 or 30 individuals at the CRO. It’s important to me that we all are able to work together. How they communicate with me is also critical.”
Employee stability is always a concern, and it was agreed that performing research or speaking to colleagues is one way to evaluate it. Individuals in your network might know if the CRO is experiencing a lot of turnover because of unhappy personnel opting to leave the company. A company that seems to be putting a lot of effort into recruitment might also be experiencing turnover issues. Knowing if the CRO is rumored to be the subject of a possible merger could also fit into your decision-making plans. Any major turnover at a CRO could mean an increase in employees who do not understand the company’s culture and SOPs. A critical factor during audits is addressing the transition planning process of the CRO.
Learning about the folks you’ll be working with can be a challenge. One way to understand employee culture and experience is to personally meet and interview them. A good time to do this is when performing an audit, and it may be necessary for you to insist on it. Halloran added, “while not a standard practice in QA-type audits, having the ability to perform due diligence on the qualifications of the proposed team, especially the project manager, is critical and should be an integral expectation to set with vendors.”
Training is another area to consider. Stiede has worked with full-service CROs for 15 years, and notes training is no longer conducted as extensively as it was in the past. “That is a large risk,” she says. “You are required to make sure your staff is qualified, but you can’t do that if the stuff is kept behind a curtain. Most CRO training today seems to be Web-based. It can be very cursory and has little to do with project training. Here is a simple rule to keep in mind: If a CRO is doing basic GCP training online, they’re not actually teaching skills. They’re just checking a box.”
One panelist suggested performing the audit when at least 90 percent of sites have been initiated and there has been some patient enrollment. Sponsors can check documentation at that time to ensure patient data is being collected correctly. If a trial is close to being ready for submission when critical errors are discovered in the way data was collected, the trial may have to get trashed at a huge cost to the companies involved. When your sites are initiated, you can still fix that, if necessary, by amending procedures.
Treohan recommends performing some co-monitoring visits with CRAs, especially if they’re supplied via a CRO. This will ensure the CRAs understand the protocol, are establishing the right relationships with investigator sites, and are performing required tasks according to your specifications.
4. Mock Initiation Vists Yield Insights
Mock initiation visits are another way to gain insights into the experience of site staff. While some CROs might object to them, there are insights to be learned from getting the personnel in a room and having them perform a walk-through. “These mock visits provide plenty of opportunities for ‘on-the-fly’ education,” Halloran added. “They are effective, simple to perform, and can be added on to any kickoff meeting.”
“We really need to get back to conducting these with our CROs and CRAs,” says Grote. “Many CRAs are simply going in and performing a task. The CRAs don’t really manage the site as a whole anymore, and they don’t have a complete understanding of the project and the program. With all of the outsourcing going on in pharma, these co-monitoring visits become more important. You need to see how CRAs are interacting with sites.”
5. Identify Your Risks
Halloran also stressed the need to identify risks upfront, at a kickoff meeting or perhaps a workshop. The risk assessment should be based on the program itself and the protocols.
“You should be able to identify those risks up front and produce a plan based around those risks,” states Halloran. “Then a spreadsheet or other tool can be used to track those risks. When something does pop up, it’s much easier to deal with and find a solution.”
“Almost everything is a potential risk,” adds Stiede. “To determine those risks, simply sit down and discuss what might happen. This could take place during one of those workshops. This discussion forces your vendors to think about how they would apply their experience in particular situations so they are better prepared for it. There is no standardized list of possible risks. It could be anything from a global trial having a government shutdown or travel bans (which is rarer) to supplies not arriving in time or a shortfall in patient enrollment (which is more common).”
If a CRO presents a list of identified risks and possible solutions to you as part of the RFP (request for proposal), that is a sure sign they understand the trial. Treohan notes CROs will generally have some understanding of general risk management. Where they generally lack knowledge of possible risks is when they relate to a certain therapeutic area. CROs might be knowledgeable about a disease like Alzheimer’s for instance, but tend to be less informed about rare conditions in dementia.
One of the more common risks a company faces is transition on the study team. The panel felt some CROs handle transitions better than others. Those CROs usually have a transition plan in place ahead of time and are better able to deal with situations when they happen. Obviously, informing you two days out that an individual will be taken off a program is never acceptable. Having a continuity plan in place is, however, one way for the CRO to demonstrate it is prepared to handle transitions.
A bigger issue that might come up is simply being able to schedule an audit with a large CRO because they have so many occurring. The panel felt one way for the industry to overcome this problem is putting a CRO certification program in place. Audit findings could be accessed by anyone requiring them, without having to send in their own people. Audits tend to consume a lot of time and resources for both the sponsor and the vendor, which seems to be a waste of resources. The entire industry could benefit by not having every company conduct audits that produce the same findings.