Why Human Factors Validation Studies Do Not have Pass/Fail Criteria
Why Human Factors Validation Studies Do Not have Pass/Fail Criteria
Some of the most frequently asked questions to Human Factors Engineers and Human Factors Consultants are:
“How do we know our Human Factors study is successful?” or
“How do we know if we’ve passed validation?”
It is a reasonable question that in other realms of engineering would render a direct answer. While many engineering tests and studies adhere to protocols with clear Pass/Fail criteria that deem an effort a success or failure, Human Factors doesn’t produce a black and white answer.
There are many types of Human Factors exercises that are performed over the course of product development and a product’s lifecycle. The most widely known and used methods are simulated use formative and summative studies. Naturally, people might expect summative studies, also known as Validation studies, to have Pass/Fail criteria considering the Pass/Fail nature of other portions of the Design Validation effort.
Some researchers have attempted to apply success rates to HF data in an effort to set a Pass/Fail threshold to confidently render an effort a success or a failure. Unfortunately, those success rates aren’t as informative as one would expect.
Now you’re thinking, “So, if there is no threshold for success, how do we know our validation study was sufficient to support Design Validation?”
Good question.
First, you must understand that Human Factors is intended to inform the risk and likelihood of harm associated with your product. Ultimately, you must assess the details of the human factors data to understand the risk associated with the product and whether you have defensible results to present your product as safe and effective to the Food and Drug Administration (FDA). There is no magical number that designates a product as safe to use as possible. If you have a 90% task success rate, meaning 90% of people performed the task(s) successfully with no use errors or difficulty, you must examine the remaining 10% to consider what harms may occur to the user if they were to not perform the task successfully. If there is a chance within those 10% of use errors that a serious harm could occur in actual use, additional work is required.
Let’s take a look at the process to put together the best possible application to increase your likelihood of a successful submission to a regulatory agency:
Here are some of post validation analysis questions your team should ask during each of the aforementioned Steps:
Step 1: Validation Study
Is my study methodology sound?
Does my study methodology include representative use scenarios developed from a solid understanding of the context of use, user interface, user groups, task analysis, and Use-Related Risk Analysis (URRA)?
Do I have a plan to collect both objective and subjective data to provide ample opportunities for user input?
Proper preparation, set-up, and data collection are vital to presenting your case to the FDA or other regulatory authority. Without the key elements of a sound, representative validation study, you face an uphill battle.
To ensure the industry is preparing human factors studies with sound methodology, major industry guidance and standards should be followed, such as:
International standard IEC 62366-1:2015+AMD1:2020
US FDA’s Applying Human Factors and Usability Engineering to Medical Devices
Additional guidelines and standards may apply based on your specific product type and submission.
Step 2: Assessing Safety Implications of Use Errors
Are there any patterns or trends that will suggest an issue with product use when introduced to a larger population?
Do the errors seen in the study have safety implications for users?
Do I need to implement Design Changes or additional Risk Mitigations to prevent a use error from occurring to eliminate or reduce the likelihood of harm?
Are these changes significant?
Do these changes necessitate a formative study, repeated validation study, or supplemental validation study?
As you can see, there are a number of questions that must be asked at this stage. These questions are accompanied by thorough updates of existing risk management documentation like the Use-Related Risk Analysis (URRA) to aid in determining answers to these questions.
It is important to note that choosing a feasible update to your device user interface late in the process is a potentially costly endeavor. Regardless, consideration must be given to the true cost of either incorporating changes before FDA submission and delaying product approval and launch versus a failed attempt at FDA approval, or worse, adverse events and harm to users after product launch.
Step 3: Can risks be eliminated or reduced to the furthest extent possible?
Do these changes reduce or eliminate the risk?
Do the change introduce new risk?
What risks remain for my final device design (residual risk)?
As Human Factors professionals we recognize that some use errors and risks cannot be feasibly eliminated such as users needing to clean injection sites or wash hands, so we have to ask ourselves, “Is the remaining risk reduced to the lowest extent possible?” Once you’re confident that is the case, proceed to the next set of questions.
Step 4: Are the benefits of the product worth the remaining risk presented?
How does the residual risk compare to the benefits of using the product?
Do the risks outweigh the benefits? Do the benefits prevail?
Be prepared to use data and solid evidence for this argument. You will reasonably want to rely on Steps 2 and 3. Proceed through those exercises dutifully as those are the most rigorous, thorough, and safety focused exercises that will put forth the safest product possible. Once you feel no more can be done to reduce risk holistically, not just risk related to use errors in the validation study, you then can rely on the benefit-risk analysis.
Step 5: Submit to FDA
If the data and argument support that benefits outweigh the risks, you can submit to the FDA, otherwise, you may need to consider changes to the User Interface that could further inform the iterative Steps 1-4.
Final Thoughts
The answer to “How do I know if my validation was successful?” is a complex one. Reach out to see how Agilis can help you through your Human Factors activities and answer the many questions required throughout your Human Factors process.
About the Author:
Ray Talbott, MS
Ray brings over a decade of experience in executing Human Factors activities for medical devices and combination products. Ray has extensive Human Subjects research experience across industry and academia from executing clinical trials to usability testing for clinical and commercial use. Ray is experienced in human factors and regulatory strategy, integrating human factors activities into design controls, and risk management.