Should your formative evaluation be testing the full user interface at once?
Should your formative evaluation be testing the full user interface in one study?
Agilis considers different strategies based on where you are in the product development lifecycle
Agilis works with clients who have products at all stages of development, and when a formative is being planned, it is very common for a sponsor to intend to test multiple elements of the user interface (UI) during a study. The user interface includes all device components the user interacts with to prepare the device for use, use the device, or perform maintenance on the device. The user interface includes:
Instructions for Use, User Manual, Quick Reference Guides or other printed instructional materials
Packaging including the outer carton and secondary packaging of individual components
On-device labeling, prompts, on-screen instructions, mobile apps
Training
However, the focus of a formative evaluation can be motivated by a variety of factors which need to be considered when the study is designed as it may not be advantageous to test all aspects of the UI at once.
Questions to consider when designing a formative study
An early-stage formative can help develop individual components of the user interface (UI) in a variety of ways. At the earliest stage, it is often helpful to perform interviews with the intended user group to evaluate terminology to be used in the instructions for use (IFU). For example, what is the common terminology a person with Type 1 diabetes would use for a specific device component? Getting targeted feedback from a group of users in a focus group could be very helpful in using the correct, understandable terminology in an IFU, so that in a later study, users don’t express confusion about the component names and misuse device or put themselves at risk due to this confusion.
After the IFU has been drafted, a formative focused on “think aloud” could help evaluate whether the flow of information, as presented, is most appropriate. Next, a formative with directed IFU use of the device would pull together all parts of the UI. At this stage during the directed IFU use of the device, the IFU should ideally be in the general format of the final product (e.g., accordion style booklet, large foldout, etc.). Finally, a formative with no directed IFU use could help prepare for validation. Depending on the stage of product development, it may not be reasonable to start with a focus group which evaluates naming conventions, but it should not be a baseline expectation that all formative evaluations start with the full UI.
Often, sponsors have user information that they can leverage ahead of conducting a formal formative evaluation. Is there any previous formative evaluation or user feedback that could be leveraged ahead of a later stage formative? For example, if the device had been used in clinical trials by persons with Type 1 diabetes, did they provide any feedback on device component names? Should that information be used to update the IFU ahead of a commercial formative? Did they provide any feedback on actual device use that could be leveraged to make design updates? Could formative evaluation results from similar products or UIs that were conducted by other working groups within the company be investigated? Sponsors may be able to leverage feedback gained in a variety of ways to make decisions on UI design and to leverage when planning future formative evaluations.
How to properly test the uI in a formative
Looking at too many UI elements at once can create a real mess with the data
Often sponsors want to conduct a formative by evaluating the device, the IFU, and maybe a quick reference guide (QRG) or an instructional video, all in the same study. If there have not been previous formative evaluations to evaluate individual parts of the UI, looking at them all at once can yield data that may or may not be very helpful. Formative evaluations generally have smaller sample sizes for each intended user group, so it’s difficult to have a reasonable number of participants evaluate multiple components of a UI.
For example, imagine a formative study with 2 user groups, n=5 user per group, which intends to evaluate the product, the IFU, and a QRG. Having all users interact with the product while using the IFU and then while using the QRG biases the users to whichever instructional material they use first. You could have user Group A use the IFU and user Group B use the QRG, but then you have no data for Group A with QRG or for Group B with the IFU. You could split the groups and have half the participants use the IFU and the other half the QRG, but then you have only 2-3 users per group for each instructional design element. A potentially cleaner approach would be to have a small formative with only the IFU, then a small formative with only the QRG and for the sponsor to then decide which instructional material they are going to use, not both. Then, conduct the larger formative with the instructional material chosen. Alternatively, if the intent is to use both IFU and QRG, develop the QRG once the IFU is evaluated with a formative, to ensure the content is common to both.
Anyone who has observed a participant in a usability study has probably thought to themselves at some point, “why don’t they just look at the instructions?” If that has ever been you, then you also know that many participants don’t pick up the instructions in a usability study unless the moderator tells them to. Imagine a late-stage formative evaluation in which the sponsor wants to see if (1) any final updates are needed to the IFU and (2) to evaluate readiness for validation. In a non-IFU directed use situation, the moderator should let the participant interact with the UI however they see fit, just like would happen in a validation study. In this kind of scenario, the participant may never even pick up the instructions while using the product. If that is the case and the sponsor’s goal is to use the information from the formative evaluation to make IFU design updates, then this is a problem. At the kickoff of any formative work on a product, the goals of the formative should be discussed by the entire project team. During protocol development, these goals should drive study design and simulated scenario development. It’s possible to have a non-IFU directed use scenario followed by an IFU directed use scenario, however, the participant may already be biased based on their experience in the first scenario. It may be more advantageous to have a formative that is specifically focused on IFU-directed use with users truly naïve to the UI, focusing probing on specific design elements of the IFU, etc. and then a separate study to evaluate validation readiness with the full UI and only non-directed use.
Conclusion:
Of course, there are many constraints to performing consecutive formative evaluations, including budget, timeline, etc. Though, it is known that involving human factors early in the process actually helps to save money by reducing redesign costs. Regardless, there needs to be alignment on expectations of how the data from a formative will be used, depending on what components of the UI are being evaluated. Identifying a holistic human factors strategy early in the product development cycle can help sponsors to leverage data across formatives to optimize the user interface. Agilis assists clients at all stages of product development, working to map out and implement the human factors strategy which is most advantageous to your submission goals.
About the Author:
Jessie Huisinga, PhD
Dr. Jessie Huisinga is a Senior Human Factors Consultant with Agilis Consulting Group, LLC and an expert in assessing human performance with extensive experience working with individuals with neurological impairments. She has a background in Biomedical Engineering and Biomechanics, with specialized training in Neurology in order to evaluate movement patterns and task performance in persons with performance limitations. She has experience assessing a diverse spectrum of home and professional use medical devices and products as well as conducting in-home and actual use usability studies.