Consistent And Clear Terminology, Best Practices Needed To Scale Sensor-Based DHTs
By Jessie Bakker, MS PhD; Digital Medicine Society (DiMe) Program Director
Sensor-based digital health technologies (DHTs) have been increasingly adopted in clinical research, clinical care, and everyday life, but a lack of clarity regarding usability best practices means that not all of these tools can be successfully implemented at scale.
Anyone who has used health-focused websites or smartphone apps or digital measurement, diagnostic, or therapeutic products will know that some tools are more comfortable, convenient, easy, and satisfying to engage with than others. In addition to user experience, usability — or lack thereof — has important implications for the safety and effectiveness of medical devices, as emphasized by a 2016 report, which found that medical error, including use error, is the third leading cause of death in the United States behind heart disease and cancer.
Consider the case of a blood glucose monitor recalled by the FDA after it was found that users may inadvertently switch the display units from mg/dL to mmol/L, and vice versa, when inserting new batteries, increasing the probability of a patient making an inappropriate adjustment to their insulin dose. This case illustrates a usability failure carrying a clear risk to patient safety.
Paying inadequate attention to the principles of user-centered design and usability engineering can have adverse consequences beyond product safety and effectiveness, including: the need for costly product redesign; delays in getting a product to market; sub-optimal product uptake and adherence resulting in missing or uninterpretable data; and the potential for increased health disparities resulting from a design process that does not account for diversity, equity, and inclusion.
Given the high stakes, why is usability an ongoing source of confusion for those working in the field of digital medicine? How can we ensure that DHTs meet the needs of all users — including patients, caregivers, clinicians, study participants, clinical research investigators, and sponsors — when scaled up to real-world deployment? We have identified two industry challenges preventing optimization of user-centered design and human factors studies in digital medicine, thereby limiting the benefits that could be realized by deploying digital tools that are scalable for real-world use.
Industry Challenge 1: Inconsistent Terminology, Standards, And Regulatory Requirements
Concepts and processes related to usability have been borrowed from fields such as biomedical engineering, software development, and cognitive/behavioral psychology, resulting in inconsistent terminology and definitions. For example, within software development the word “utility” typically refers to whether a tool provides the features that a user needs; however, in medicine the term “clinical utility” has arisen from the genetic testing literature and refers to the likelihood that a test/tool results in improved health outcomes.
The field of digital medicine sits at the intersection of technology and healthcare and must therefore reconcile the differences in terminology that have arisen within these communities.
In addition, regulatory requirements for usability are housed in multiple locations and vary by geographical region and by the regulatory status of the product. For example, human factors and usability engineering are described in depth within FDA guidance for medical devices and combination products, while less detail is devoted to usability testing within the draft guidance on using DHTs for remote data acquisition in clinical investigations. In Europe’s Medical Device Regulation (EUMDR), aspects of usability are addressed in relation to various topics, such as the description of intended purpose, risk management system requirements, measures to eliminate or reduce risks related to use errors, interactions with other devices, and post-market surveillance.
International standards are similarly disparate; for example, ISO 20282 focuses on usability of consumer products, IEC 62366 focuses on usability of medical devices, and ISO 9241 contains recommendations for human-centered design applied to consumer digital products such as software and banking systems, much of which can be adapted and applied to DHTs.
Industry Need 1: Technology developers need support to help them determine exactly which regulatory guidance and industry standards apply to the usability of their DHT.
Industry Challenge 2: Unclear Best Practices For User-Centered Design And Human Factors Testing
At one end of the spectrum, usability testing is a checkbox exercise undertaken when product design is nearly final, allowing limited opportunity to incorporate what is learned during the research process into the final product design. In other cases, we see teams developing an in-depth usability engineering plan including a use-related risk analysis, and engaging in complementary methodologies including heuristic evaluations, cognitive walk-throughs, look-and-feel assessments, mixed methods formative testing, and quantitative summative/validation testing under simulated or actual use conditions with pass/fail thresholds established a priori.
There is no one-size-fits-all approach to follow during DHT design and usability testing, and it can be difficult for product developers with limited training on these topics to know what steps to take. For clinicians, investigators, and sponsors, it can be difficult to know whether the usability work undertaken pre-market by a developer is sufficient to meet their specific post-market use case or whether there is a need to proceed with further human factors studies and/or comparative usability studies.
Industry Need 2: Technology developers, clinicians, investigators, and sponsors need resources to support their product development and scientific strategies specific to DHT usability. The knowledge gaps described above bring to mind the lack of clarity that prompted the Digital Medicine Society (DiMe) to develop the Verification, Analytical Validation, and Clinical Validation (V3) Framework for biometric monitoring technologies:
“…terminology, approaches, and evidentiary standards are not aligned across these [engineering, technology, data science, regulatory, and clinical] communities, slowing the advancement of digital medicine for improved health, healthcare, and health economics.”
The V3 Framework outlines a modular approach to ensuring that sensor-based DHTs are fit for purpose, meaning that the technical, scientific, and clinical performance of the product is suitable for a defined purpose and context of use. At each step of the V3 approach, but particularly during analytical and clinical validation, usability must be addressed. With broad industry adoption, the V3 Framework has become the de facto methodological framework for evaluating DHTs and is the logical “home” for describing usability best practices during the development and evaluation of sensor-based DHTs.
DiMe has assembled a multi-stakeholder team of experts to address these industry challenges and extend the V3 Framework to include user-centered design and human factors studies of sensor-based DHTs. The extended framework will include clear, actionable recommendations along with case studies, supporting those working in the field to meet applicable regulatory requirements while ensuring that their products meet the needs of all users they are designed to help.
About The Author:
Jessie Bakker, MS PhD, is a program director at the Digital Medicine Society (DiMe). Dr. Bakker has extensive experience in the development, validation, and deployment of digital tools in clinical trials within academia as well as several device manufacturers. She is a founding member of DiMe, served as co-chair of the DiMe research committee from 2019-2021, and contributed to DiMe’s first project, which described the V3 framework as a means of evaluating sensor-based digital health technologies. A graduate of the University of Otago, New Zealand, Dr. Bakker completed her postdoctoral fellowship before joining the faculty of Harvard Medical School in 2013, specializing in epidemiological studies and clinical trials in sleep medicine.