Use Related Risk Analysis (URRA) – A Critical Living Tool within the Human Factors Process
Use Related Risk Analysis (URRA) – A Critical Living Tool within the Human Factors Process
In October 2020, Hanniebey Wiyor, PhD, FDA CDRH presented a webinar discussing risk assessment of medical devices and shared a remarkable finding that use-related risk analyses (URRAs) were the leading cause for deficiencies in 542 human factors (HF) submissions reviewed by CDRH in 2019 [1]. Issues with URRAs were more prominent then training issues, test environment, testing issues, test participants, and HF test results. In this article, we present common issues FDA CDRH identified with URRAs, as well as the Agilis approach to URRA and task categorization.
URRA Definition and Common Issues
URRA is the process by which a manufacturer documents potential hazards and harms end users may encounter via interaction with their product’s user interface. A URRA relies on several different inputs. FDA CDRH described that Sections 2-7 (Figure 1) of the FDA-recommended HFE/UE Report structure encapsulates a manufacturer’s overall URRA process [2].
Figure 1. Outline of HFE/UE report
To produce an effective URRA, it must be understood that it is a living process that must be revised throughout the product development cycle and during post-market surveillance. A manufacturer’s URRA should be updated a minimum of four times as the product design evolves, and data is collected throughout the design control process [3]. How to document a URRA is up to the manufacturer, although guidance documents and industry standards mention that Task Analysis, PCA Analysis, Failure Modes and Effects Analysis, and Fault Tree Analysis are all acceptable. FDA has acknowledged that estimating probability of use errors is difficult and can even be impractical, therefore the URRA process should focus more on potential harm of use errors [2, 4]. One of the most important outcomes of a URRA is the categorization of tasks and identification of critical tasks which will be the focus of HF validation testing [2].
The following have been identified as common issues involving the URRA during HF reviews by regulators [1]:
No URRA provided
Unknown or unclear determination of critical tasks
Submitting entire risk management report (forcing the HF reviewer to find and review use-related risk portions within)
Elimination or selection of critical tasks based on risk priority number (specific to Failure Modes and Effects Analysis)
Inappropriate determination for the levels of severity of harm
Non-specificity of risk mitigation control information
The issues above illustrate that a disconnect between the URRA process and HF activities is not uncommon. However, a clear connection between URRA and HF where the URRA is utilized to identify relevant tasks which are critical and need to undergo HF testing by end users is necessary for manufacturers to achieve the end goal, which is to, “Maximize the likelihood that new medical devices will be safe and effective for the intended users, uses, and use environments,” [2]. One theme within the common issues with URRAs listed above is that manufacturers are not always clear and/or reasonable when defining critical tasks. With 20 years of experience helping medical device and combination product manufacturers achieve successful human factors submissions, Agilis has developed practices to ensure the proper back-and-forth flow of information between URRA and HF.
Agilis Approach to URRA Development and Critical Task Identification
Agilis employs the human factors engineering process shown in Figure 2 when supporting development of medical devices and combination products. Within Phase 1, Concept & Design and Phase 2, Testing, Re-design & Instructional Labeling, lives the URRA development process. New information regarding intended use, users or use environments, as well as data from preliminary HF activities can trigger iterations of the URRA during Phases 1 and 2. In Phase 3, Clearance and Marketing, the URRA identifies all critical tasks for HF validation testing, feeds into the residual risk evaluation and potentially a risk-benefit analysis (RBA), and is maintained based on post market surveillance. It is worth noting that unanticipated use errors may occur during HF validation testing which may necessitate further revisions of a URRA and subsequent HF testing.
Figure 2. Agilis human factors engineering process.
During Phases 1 and 2, the manufacturer should document all tasks, use-related hazards, foreseeable use errors, potential harms, and severity of potential harms. Additionally, the URRA should identify specific risk control measures meant to eliminate or mitigate occurrence of foreseeable use errors. Prior to entering Phase 3, a manufacturer should have high confidence that they have identified and can test all critical tasks during HF validation testing.
Case Study – Iteration of a URRA
To illustrate a URRA iteration during Phases 1 and 2, we will use an example formative evaluation with a single-dose auto-injector. Most single-dose auto-injectors have a similar operational sequence including removing a cap, activating the device to inject a dose of medication, waiting for the full dose to be administered, and then removing the auto-injector. See Table 1 for one example row of a URRA for the formative evaluation. Of most interest, the task, potential use error, potential hazard/hazardous situation, potential harm, severity of potential harm, and risk control measures are included. At the point of a formative evaluation, it helpful to have risk control measures associated with task performance clearly identified. This allows formative evaluation data to demonstrate if current risk control measures are effective.
The blue highlighted data presented in Table 2 show that risk control measures documented in Table 1 were not effective in mitigating the use error of twisting the cap.
Risk control measures added and highlighted in yellow in Table 3 demonstrate how a URRA could be iterated based on formative evaluation data. Additionally, the risk control measures added in Table 3 are inherent safety by design which is best practice and most likely to be effective in mitigating use-related risk [5].
Critical Task Identification
Identifying critical tasks can be challenging. However, for medical devices to be reviewed by FDA CDRH, insight has been provided regarding their expectation when labeling a task as critical. A critical task is defined as, “A task which, if performed incorrectly or not performed at all, would or could cause serious harm to the patient or user, where harm is defined to include compromised medical care,” [2]. In our experience, the words “serious” and “compromised medical care” trigger debates when going to define critical and non-critical tasks. However, FDA CDRH pointed to 21 CFR 803.3 (w) shown in Figure 3 to provide insight on this matter [1]. When you compare 21 CFR 803.3 (w) to the five-level qualitative severity levels in ISO 14971 shown in Figure 4, it becomes apparent that tasks with foreseeable use errors and associated potential harms that are serious, critical, or catastrophic would be labeled as critical.
Figure 3. Content of 21 CFR 803.3 (w).
Figure 4. Example of five-level qualitative severity rankings.
In contrast, for combination products FDA has defined critical tasks as, “user tasks that, if performed incorrectly or not performed at all, would or could cause harm to the patient or user, where harm is defined to include compromised medical care [6].” It is important to note that the term “serious” is not included in this definition. The Division of Medication Error Prevention and Analysis (DMEPA) typically reviews human factors data for combination products where the drug or biologic is the primary mode of action. DMEPA’s vision is to, “Eliminate medication errors in the U.S. healthcare system [7].” Based on this, any task with a combination product where a foreseeable use error could lead to incorrect delivery of the labeled dose of medication should be categorized as critical.
To ensure a seamless connection between your URRA and HF evaluations, it is prudent to ensure risk control measures are specific, especially those mitigating risks associated with critical tasks. This is because once all critical tasks are tested in an HF validation study, the data produced will provide evidence demonstrating if implemented risk control measures are effective or ineffective. If risk control measures show signs of ineffectiveness, the manufacturer may have to revisit the user interface design and associated risk control measures which will delay regulatory approvals and have financial consequences!
Summary
We hope this article has help provide some clarity on definition of critical tasks and the importance of treating a URRA as a process. The Agilis team has many years of experience consulting on URRAs and with providing HF data that seamlessly feeds into a manufacturer’s risk management process. We look forward to helping you with your next project that may include URRA creation, defining critical tasks, and collecting HF data to ensure end users can complete critical tasks safely and effectively!
Stay tuned for next month’s newsletter where we will discuss how to ensure cybersecurity risks are incorporated into your next HF evaluation!
References
[1] Wiyor, H.D. (October 22, 2020). Risk Assessment of Medical Products in Human Factors Submissions with a Focus on EU countries. MHFN Webinar.
[2] FDA guidance entitled, Applying Human Factors and Usability Engineering to Medical Devices that was published by CDRH, FDA on February 3, 2016.
[3] AAMI TIR59:2017 – Integrating Human Factors into Design Controls.
[4] FDA draft guidance entitled, Contents of a Submission for Threshold Analyses and Human Factors Submissions to Drug and Biologic Applications that was released September 2018.
[5] ANSI/AAMI/ISO 14971:2019, Medical Devices – Application of risk management to medical devices.
[6] FDA draft guidance entitled, Human Factors Studies and Related Clinical Study Considerations in Combination Product Design and Development that was published in February 2016.
[7] Office of Surveillance and Epidemiology (OSE) – Divisions. Content last updated March 23, 2020. Retrieved from https://www.fda.gov/about-fda/center-drug-evaluation-and-research-cder/office-surveillance-and-epidemiology-ose-divisions
AAMI Book
Applied Human Factors in Medical Device Design