Usability Testing

Research and Usability: Design Evaluation

Introduction

Conducted usability testing to understand how users interacted with Motorola’s Smart Actions automation platform. Evaluated onboarding, rule creation, terminology, discoverability, and overall user comprehension. Identified key usability barriers and translated findings into design improvements that made automation more intuitive and approachable.

Dates: August 1st and August 2nd
Duration: 60-minute sessions
Participants: 6 participants (4 male, 2 female)
Qualifications: All participants were owners of touch screen smartphones

Methods

Participants were asked to perform various tasks such as adopting pre-scripted sample rules, creating a custom rule, deleting an action block, editing rule name, turning off all rules, and looking for more information.

Feedback

Participants provided feedback on the welcome screen, first impressions, actions and triggers, sample rules, language, and iconography.

Objectives

Evaluate users' ability to complete tasks (creating custom rule, adopting sample rule) independently.
Observe user interactions with the fully functioning application.
Gather feedback on application understanding, including language used.

First Impressions

After exploring the app briefly, users had a general understanding of the application.
Users described the app as a tool for setting up or automating tasks on their phones.
However, users struggled to comprehend how the app actually worked.
Some users initially believed they needed to open the app and select a "mode" to activate it.
Users found the app interface confusing, with many menus within menus.
Most users selected "Get Started" instead of "Learn More" on the initial screen.

Teach Screen

Users had difficulty connecting the information presented on the "teach" screen to the rule list.
Some users found the information confusing and struggled to make sense of it.
Users expected further assistance beyond the teach screen, such as examples or a guided walkthrough.
Some users mentioned that the term "custom rule" was not included on this page, causing confusion.
Several users skipped this screen initially, later realizing they needed it but couldn't find it.

Other Observations

Users didn't understand why they were asked to save changes when exploring a rule.
All users opened the pre-scripted rule to understand the app's functionality.
Users expected the back button to take them to the beginning or the teach screen.

Adding a Sample Rule

Users had significant difficulty adopting their first sample rule due to a lack of understanding that rules needed to be "added."
The save and cancel buttons were not immediately noticed, making it harder for users to grasp the concept of adding rules.
Most users understood the need to set up Bluetooth but didn't understand its connection to the rule automatically turning on.
Users questioned the reliability of the rule if they forgot to manually turn it on.

Auto Reply Text

The label for "Auto reply Text" caused confusion as users interpreted it as automatically replying to incoming text messages.
Users were puzzled when they encountered information about callers instead of texting in the action setup.
Uncertainty arose about whether the rule played a message for calls and sent a text for texts.

Launch Application

Users expressed confusion about the purpose of the Launch Application action.
Users mistakenly believed it would launch the pre-scripted feature instead of the Vehicle mode application.
Some users thought they could specify which apps they wanted to work while driving but found the process tedious.
Users questioned the need for an app to launch while driving and were unsure of its utility.
The list of applications displayed after touching the action box further confused users.

Issue: Misleading Label for "Auto Reply Text"

The label for "Auto Reply Text" is misleading and causes confusion.
Users interpret "auto text reply" as automatically replying to incoming text messages.
Example user feedback: "I like this thing about texting - driving and texting is dangerous."

Issue: Confusion about Action Setup

Users are confused when they see information about callers instead of texting after entering the action setup.
Example user feedback: "What do callers have to do with this? Oh wait, does it only reply to callers and not people who text me? I don't think I get this."
Users question whether the action plays a message for calls and sends a text for texts.

Issue: Lack of Clarity about Launch Application Action

All users expressed confusion about the purpose of the Launch Application action.
Users were not familiar with the Vehicle mode application and mistakenly thought the action would turn on.
Example user feedback: "Oh, this will launch the script?”
Some users suggested having checkboxes instead of adding apps one at a time.

Triggers and Actions

Users did not spontaneously comment on what triggers and actions were until asked by the researcher.
Upon explanation, users understood that triggers turn the rule on, and actions occur when the rule is on.

Confusion about Save and Cancel

Users often answered "yes" when asked if the setup was complete without saving the rule.
Users realized they needed to save the rule only after being asked what they would do next.
Example user behavior: Finding and saving the rule after being prompted.

First-Time Landing Screen

Users who landed on the screen for the first time could recall accurately what actions were included in their sample rule.
Most users were unsure of the meaning of "ready" but assumed it indicated whether the rule was on or would be on shortly.

Issue: Bluetooth-Related Confusion Post Rule Set Up

Some users experienced confusion due to having set up Bluetooth and assuming that "ready" meant the rule was on.
Users associated "ready" with Bluetooth being on and paired with their car, leading to confusion.

Difficulty Finding the "+" Button

Most users struggled to discover the plus button to add a new rule and found it through trial and error.
Users usually touched everything at the top of the screen to locate the plus button.

Creating a Custom Rule: Sleep Mode

Users were asked to create a rule to prevent interruptions during sleep.
Triggers: phone not moving and phone charging.
Actions: silence ringer, dim screen brightness, and allow incoming calls from VIPs (e.g., your mom).
Most users easily set up the rule, attributing it to clear instructions for triggers and actions.
All users named the rule immediately (e.g., Sleep, Zzzzz, Night).

Creating a Custom Rule: Adding Actions and Triggers

Most users had no difficulty finding or setting up actions.
Some users found the terminology "When my charging status changes" to be awkward and confusing.
Example user feedback: "It would be better to just call this charging. 'When my charging status changes' seems like the rule only turns on the moment my charging changes, like I unplug it from the outlet."

Creating a Custom Rule: Trigger Details

Only one user discovered that clicking on the text allowed specifying the charging source.
Other users weren't interested in this detail and didn't see a need for it.

Editing, Deleting, and Turning Rules On/Off

Editing the Name: All users successfully changed the name of the rule.
Deleting an Action: Users attempted to delete the block in the rule builder or clicked the + next to actions to re-select the grayed-out action.
Accidental Discovery: Some users accidentally disconnected blocks while scrolling through a rule.
Turning Rules On/Off: Most users found the ON/OFF switch on the rule builder. Some users were confused by the menu button that disabled all rules, mistaking it for three buttons representing the rules.

Actions and Triggers List

Launching a website or application was considered fun by several users, despite their confusion during the rule task.
Ring Loud for Important People was considered exciting and innovative.
"When my display status changes" trigger was confusing to most users. They were unsure of its meaning or how a display status could change.
"When my charging status changes" trigger was also confusing, with users interpreting it as triggering only during the change from charging to not charging or vice versa.

Summary

Language-related issues were prevalent throughout the session, leading to understanding difficulties.
Confusion with Triggers: "When my display status changes" and "When my charging status changes" triggers caused confusion among users.
Confusing Descriptions: Clearer language is needed for actions and triggers.
Discoverability Issues: Icons, plus sign to add new rules, and the menu key were difficult for users to discover.
Saving Rules: Users didn't immediately understand that rules needed to be saved to be adopted.
Updated User Interface: Users found the updated screenshots and click-through prototype of the Drive Rule to be a major improvement, describing it as more user-friendly, intuitive, and visually appealing.
Helpful Explanation Text: Users found the explanation text helpful in understanding how the applications worked.
Awareness of Rule Status: Users noticed the connection between color and rule state, with on as blue and off as gray.
Visual Cues: The flashing "thinking" and flashing gray to blue when adding an action drew users' attention and informed them about the rule's progress.

< Back

Google Sites

Report abuse