Thinking Outside the Box—How a Threshold Analysis Can Be a Useful Human Factors Strategy

Thinking Outside the Box—How a Threshold Analysis Can Be a Useful Human Factors Strategy

This newsletter is based on an Agilis presentation for the HFES Medical Conference on March 27, 2019. Contact Agilis Consulting Group to receive a copy of this presentation.

Leverage Existing Use-Related Data to Enhance Your Human Factors Process

Sponsors are always interested in finding the most cost effective strategies to achieve a successful FDA human factors submission the first time. Human factors best practices and processes, such as use-related risk documentation and formative evaluations, should be applied to every new device user interface (UI) so that risks to patients and end users are eliminated or minimized. The extent of data necessary for a human factors submission to show safe and effective use can fluctuate based on many variables, such as type of device, use-related risks, and specific agency requests, among other things. The degree of human factors use-related data required can be reduced in some cases by applying a threshold analysis strategy, where sponsors leverage existing use-related data to comprehensively evaluate known problems with similar device UIs and identify use-related risks associated with the new device that are also found in existing similar devices on the market (i.e., comparator products). In effect, sponsors who choose to implement a threshold analysis are still utilizing extensive human factors methods to identify use-related risk with their device, but the approach involves the added strategy of using a comparator product currently on the market as a baseline for acceptable use-related risk.

What is a threshold analysis?

At a very basic level, a threshold analysis is a comparison of use-related risk for a new product UI versus a currently marketed product UI (i.e., comparator) with the same or very similar UIs and intended users.

FDA draft guidance documents for threshold analyses outline the threshold analysis human factors process for determining whether use-related risk for a new product/device warrants a human factors validation study when compared to existing similar devices (i.e., comparators):

Although the FDA CDER draft guidance documents for threshold analyses are specific to drug products, Agilis has successfully applied the spirit of the approach across different types of submissions reviewed by both CDER and CDRH for several years.  

While CDRH does not specifically refer to a “threshold analysis” in their human factors guidance (2016), the same general approach is described for testing modified devices (see Section 8.2, Human Factors Validation Testing of Modified Devices). According to CDRH human factors guidance, when sponsors make changes to a device already on the market, they must complete a gap analysis to compare use-related risk of the original unmodified product/device to the use-related risk associated with the implemented modifications. Then, the sponsor should conduct a residual risk analysis to determine if risk levels associated with the modifications are acceptable or if a supplemental validation study is required to demonstrate that the modifications are safe and effective for end users and patients. In essence, this is the same type of process that forms the framework for CDER’s guidance on threshold analysis.

When should sponsors consider using a threshold analysis?

A threshold analysis can be a viable tool for helping a sponsor determine use-related risks (including potential gaps in risk profiles) and may provide an alternative to validation testing on a case-by-case basis. Keep in mind that a threshold analysis may not be the best strategy in all cases. For example, it may not be appropriate to perform a threshold analysis for very high risk devices such as robotic surgery systems or ventilators because FDA has identified these devices as having clear potential for serious harm and requires human factors validation data to support that these devices are safe and effective in the hands of end users (See FDA's 2016 List of Highest Priority Devices for Human Factors Review).

What are the high-level steps for a threshold analysis?

Using human factors processes, a threshold analysis (or gap analysis) culminates in a thorough analysis of gaps in use-related risks and can bridge studies by leveraging existing human factors data (e.g., post-market surveillance, recalls, known adverse events, and problem reports for similar comparator products). A threshold analysis compares differences in use-related risks between the new device/product and existing comparator(s) to identify if the new product presents less risk, no more risk, or the potential for additional use-related risks over the risks identified for the comparators. According to CDER (2017, 2018), a threshold analysis includes the following human factors analyses:


What are possible outcomes of a threshold analysis?

Sponsors must interpret the results of a threshold analysis and perform a residual risk analysis to determine if the risk to the user or patient is acceptable for the risks associated with parts of the UI that present greater risk than a comparator. This process generally can lead to three different sponsor conclusions (although FDA may or may not agree with the sponsor):


Even if the conclusion is not what the sponsor hoped for, a threshold analysis can provide a sponsor with valuable use-related risk information that can help them to optimize the product before going to market, which can reduce the risk of human factors validation “surprises” and potential post-market issues.

Communication with FDA is key!

We encourage sponsors to have early conversations with FDA prior to submissions to obtain agency feedback and initial responses to the regulatory strategy pursued by the sponsor. It is important to remember that FDA may or may not agree with the sponsor’s threshold analysis/gap analysis conclusion, so there is still the potential for sponsors to receive a request for human factors validation data.

A successful threshold analysis can save time and resources in the long run on a case-by-case basis. When determining overall strategy for a submission and whether or not a threshold analysis may be an appropriate approach to implement, sponsors should consider the following:

Chart 3.png

Agilis Case Study—Pre-filled Syringe

Agilis successfully applied a threshold analysis strategy to a drug/device combination product that ultimately led to FDA approval of the product. Note that details of this case study have been generalized due to client confidentiality.  The sponsor developed a pre-filled syringe product used by adult lay users in non-clinical use environments, which incorporated many of the same user interface components, the same intended users, and the same intended use environment as products that already exist on the market. The sponsor met with FDA several times to align expectations on the human factors processes and the regulatory strategy the sponsor was pursuing. Agilis worked with the sponsor to develop the use-related risk analysis (URRA) for the new product, identified appropriate comparators, and submitted a threshold analysis for FDA review that included a task analysis comparison, labeling comparison, and physical device UI comparison. The original threshold analysis identified several elements of the UI that may present additional risk to users and patients over the comparators.

The sponsor met with FDA and decided to re-design a specific element of the product as well as to revise the IFU to minimize risk to users and to better align with the risk profiles of the comparator products. Agilis redesigned the new product’s IFU and subsequently conducted a formative evaluation to collect performance and subjective data from representative users. Data demonstrated that the IFU revisions were effective at mitigating risk on critical tasks.

Based on IFU formative findings and the sponsor’s re-design of a specific component of the new product, Agilis iterated the threshold analysis. The sponsor analyzed the results of the threshold analysis, determined that the new combination product posed no more risk than comparator products already on the market, concluded that no human factors validation data were needed, and submitted the conclusion (along with rationale) to the agency. In this case, FDA accepted the sponsor’s conclusions.


Remember—Threshold analysis does not mean a sponsor can ignore human factors!

To complete a threshold analysis/gap analysis, extensive human factors analysis and use-related risk documentation are necessary. As part of the larger program, sponsors should ideally be incorporating human factors processes into all phases of their design life cycle in order to achieve an optimized UI that presents minimal use-related risks to end users and patients (e.g., contextual analysis, formative evaluations, etc.).

Home based Human Factors evaluations

Home based Human Factors evaluations