Go through the discussion guide with the participant, asking permission and recording the session if possible. Take notes, especially if not recording. Note when and how participants first attempt the tasks, if their first attempt is right/wrong, note how long the participant spends on each task (each task should take no more than a 1-2 minutes to complete), if any tasks are left incomplete or skipped, etc. Prompt with follow up questions if necessary and remember to prompt the user to think out loud as they work through the tasks.
Introduction to participant: Thank you for agreeing to participate in our research. I will ask you to complete a series of tasks with a visualization of an online store's order tracking pipeline. When completing these tasks, please think aloud and talk through how you're thinking about the task. We are interested in how you use the tool, what expectations you may have, and particularly where the tool falls short. Your honest feedback will help us to improve this tool for future use.
Interviewer Instructions: [Bring up the summary stage of the tool]
Ask the following questions during each view. Specific subsamples may have additional questions and tasks. Encourage the user to speak outloud and ask follow up questions, if something is unclear.
-
Tell me about this view.
- What do you think the view is about?
- What is the data you're looking at?
- What is an insight you gain from this data?
-
Filter the data and answer a question
-
Filter/Use another interaction to answer a question
-
Filter/Use another interaction to answer a question
-
Navigate to each of the different stages.
- How is the navigation?
- Does it make sense?
- Does it meet your expectations?
For View 1,
-
Tell me about this view.
- What do you think the view is about?
- What is the data you're looking at?
- What is an insight you gain from this data?
How challenging was this task?
1 - Not challenging at all 2 3 4 5 - Very challenging
-
What are the most popular categories in the following cities?
- Are the categories stable across cities?
- Sao Paulo
- Belo Horizonte
- Salvador
- Vitoria
How challenging was this task?
1 - Not challenging at all 2 3 4 5 - Very challenging
-
Use the tool to explore the spiked sales.
- What were the total sales ($) on that day?
- What was the most popular item category?
How challenging was this task?
1 - Not challenging at all 2 3 4 5 - Very challenging
For View 2...
- Tell me about this view.
- What do you think the view is about?
- What is the data you're looking at?
- What is an insight you gain from this data?
For View 3...
- Tell me about this view.
- What do you think the view is about?
- What is the data you're looking at?
- What is an insight you gain from this data?
Likert Questions
Scale: (1 Strongly disagree - 5 Strongly agree)
- I found the system unnecessarily complex.
- I thought the system was easy to use.
- I think that I would need the support of a technical person to be able to use this system.
- I found the various functions in this system were well integrated.
- I thought there was too much inconsistency in this system.
- I would imagine that most people would learn to use this system very quickly.
- I found the system very cumbersome to use.
- I felt very confident using the system.
- I needed to learn a lot of things before I could get going with this system.
Source: https://www.trymyui.com/sus-system-usability-scale
-
What are your overall thoughts on the tool?
-
Were there any tasks you found particularly challenging or non-intuitive? Please tell me more about those and any ideas for improvement you may have.
-
Do you have any other suggestions or comments?
-
Overall, how would you rate this tool?