4 Tips for Moderating Usability Studies
Moderating usability studies is a skill that is developed with practice and by observing experienced moderators. There are little, if any, training courses or in-depth guides on how to become an effective moderator. Moderating itself is multi-pronged; you need to be unbiased and observant, yet a leader who is neutral and approachable. The following are 4 tips and practices for moderating usability studies.
Understanding the type of usability study that you are conducting will determine the techniques that are used to conduct your study and the way you interact with participants. For the purposes of this article, we will focus on two types of usability studies: human factors formative and summative validation studies. For both types of studies, you will want to develop a well-organized script, or moderator guide, that can be repeated for every participant to ensure consistent moderation and robust data collection.
Formative evaluations can be conducted with varying degrees of formality depending on the phase of product development, the maturity of the user interface to be studied, and goal of the study. Although, determining the root cause of use errors and if the User Interface (UI) is safe and effective, since the product is still in the development phase, there is opportunity to dive deeper into the design.
Best practices for formative evaluations:
The moderator may frequently interrupt the participant’s workflow to probe on use errors and the design of the UI.
Participants are encouraged to think-aloud.
Participants are asked about preferences and suggestions on how to improve usability of the User Interface.
Unlike formative evaluations, HF summative validation studies are conducted at the end of product development with production equivalent versions of the UI.
Best practices for HF summative validations:
There will be little to no interruptions of the participant’s workflow by the moderator.
Participants are not asked about preferences or UI improvements.
Primary focus of probing is to determine the root cause and if the UI is safe and effective.
Are we evaluating the participant’s performance or their abilities? No. We are evaluating the usability of the UI. At the beginning of the session, we want to clearly explain that the purpose of the study is to evaluate the UI, not the participant. Listen to what the participant says as well as what they don’t say, including their body language, indirect comments, and micro expressions. If the participant starts to blame themselves or express self-doubt, remind them that we are not evaluating them. Be careful not to defend the design of the product and the design team’s intent. Participants will stop being honest if they feel you are defending the existing design rather than being open to feedback and suggestions.
We want to make sure that probing or follow-up questions do not lead the participant to their response. Allowing the participant time to think through your questions can be particularly difficult, especially if the silence is uncomfortable for you. It can be tempting to want to help the participant find the words they are looking for. The way you ask follow-up questions can lead the participant to an answer.
Instead of asking the participant “How difficult was that?”, ask them, “How did that go for you?”.
Here are some recommendations for remaining neutral:
Watch your non-verbal cues, facial expressions, body language, and tone.
Don’t rescue participants when they struggle. Encourage them to verbalize their thoughts and feelings.
Give similar feedback and react the same for both positive and negative comments about the product.
Since the moderator guides the participants through the study session, you will want to ensure the participant feels comfortable in this new, unfamiliar environment. You will want to understand the disease state of the participants recruited for the study. In some cases, comorbidities can add to patient participant frustrations. Some participants may struggle with limitations, such as arthritis, cognitive, sight, or dexterity issues. Look for cues that participant’s emotions are running high, listen for tension, sighs, huffs, and obviously anger. If you notice a participant is visibly becoming agitated or upset, ask them if they need a break. If you moderate in a monotone voice or with a ‘poker face’, you may want to soften your approach to be more inviting and relaxing to participants.
Here are some recommendations for moderator intervention:
If a participant comes into the session room noticeably uncomfortable or nervous, take some extra time to get them acquainted with the room set up and let them know they can take a break any time.
If a participant struggles with a task, reassure them they are providing valuable information, but don’t commiserate with how much difficulty they are experiencing.
If a participant experiences a medical situation, there are some safety measures you can take. Always have a first aid kit nearby. Some participants with diabetes can have their blood sugar drop quickly, have sugary snacks and juice nearby.
Moderators have a lot to juggle. Understand that even experienced moderators have challenges. It is impossible to prepare for every scenario, but you can build yourself a tool kit of go-to tips and tricks. In general, the best practice is to be as prepared as possible.
Pro-tips:
Watch old study videos of yourself moderating and take notes on areas for improvement and to keep yourself sharp. Most people hate to watch themselves, but this will give you insight into your moderating habits and language.
Practice moderating with a study guide via a dry run – first by yourself then with a colleague.
Record yourself practicing and take notes on areas for improvement.
Practice, practice, practice. Practice in the mirror, with a friend, or with your pet.
About the Author:
Jennifer Mead, MS
Jennifer Mead is a human factors psychologist and human factors consultant with Agilis Consulting Group, LLC. Jennifer joins Agilis with experience in consulting and the Biotech space with all phases of combination product development, ensuring insights gained from usability research are transferred to labeling and device requirements to ensure products are safe and effective for use.
