UX Summary

Research and Usability + Design Evaluation

Version 1.0

Overview:

• Dates: August 1st and August 2nd

• Duration: 60-minute sessions

• Participants: 6 participants (4 male, 2 female)

• All participants were owners of touch screen smartphones

Methods:

• Participants were asked to perform various tasks such as adopting pre-scripted sample rules, creating a custom rule, deleting an action block, editing rule name, turning off all rules, and looking for more information.

Feedback:

• Participants provided feedback on the welcome screen, first impressions, actions and triggers, sample rules, language, and iconography.

Objectives:

• Evaluate users' ability to complete tasks (creating custom rule, adopting sample rule) independently.

• Observe user interactions with the fully functioning application.

• Gather feedback on application understanding, including language used.

First Impressions:

• After exploring the app briefly, users had a general understanding of the application.

• Users described the app as a tool for setting up or automating tasks on their phones.

• However, users struggled to comprehend how the app actually worked.

• Some users initially believed they needed to open the app and select a "mode" to activate it.

• Users found the app interface confusing, with many menus within menus.

• Most users selected "Get Started" instead of "Learn More" on the initial screen.

First Impressions: Teach Screen

• Users had difficulty connecting the information presented on the "teach" screen to the rule list.

• Some users found the information confusing and struggled to make sense of it.

• Users expected further assistance beyond the teach screen, such as examples or a guided walkthrough.

• Some users mentioned that the term "custom rule" was not included on this page, causing confusion.

• Several users skipped this screen initially, later realizing they needed it but couldn't find it.

First Impressions: Other Observations:

• Users didn't understand why they were asked to save changes when exploring a rule.

• All users opened the pre-scripted rule to understand the app's functionality.

• Users expected the back button to take them to the beginning or the teach screen.

Adding a Sample Rule:

• Users had significant difficulty adopting their first sample rule due to a lack of understanding that rules needed to be "added."

• The save and cancel buttons were not immediately noticed, making it harder for users to grasp the concept of adding rules.

• Most users understood the need to set up Bluetooth but didn't understand its connection to the rule automatically turning on.

• Users questioned the reliability of the rule if they forgot to manually turn it on.

Adding a Sample Rule: Auto Reply Text

• The label for "Auto reply Text" caused confusion as users interpreted it as automatically replying to incoming text messages.

• Users were puzzled when they encountered information about callers instead of texting in the action setup.

• Uncertainty arose about whether the rule played a message for calls and sent a text for texts.

Adding a Sample Rule: Launch Application

• Users expressed confusion about the purpose of the Launch Application action.

• Users mistakenly believed it would launch the pre-scripted feature instead of the Vehicle mode application.

• Some users thought they could specify which apps they wanted to work while driving but found the process tedious.

• Users questioned the need for an app to launch while driving and were unsure of its utility.

• The list of applications displayed after touching the action box further confused users.

Issue: Misleading Label for "Auto Reply Text"

• The label for "Auto Reply Text" is misleading and causes confusion.

• Users interpret "auto text reply" as automatically replying to incoming text messages.

• Example user feedback: "I like this thing about texting - driving and texting is dangerous." 

Issue: Confusion about Action Setup

• Users are confused when they see information about callers instead of texting after entering the action setup.

• Example user feedback: "What do callers have to do with this? Oh wait, does it only reply to callers and not people who text me? I don't think I get this." 

• Users question whether the action plays a message for calls and sends a text for texts.

Issue: Lack of Clarity about Launch Application Action

• All users expressed confusion about the purpose of the Launch Application action.

• Users were not familiar with the Vehicle mode application and mistakenly thought the action would turn on.

• Example user feedback: "Oh, this will launch the script?” 

• Some users suggested having checkboxes instead of adding apps one at a time.

Issue: Lack of Understanding about Triggers and Actions

• Users did not spontaneously comment on what triggers and actions were until asked by the researcher.

• Upon explanation, users understood that triggers turn the rule on, and actions occur when the rule is on.

Issue: Confusion about Save and Cancel

• Users often answered "yes" when asked if the setup was complete without saving the rule.

• Users realized they needed to save the rule only after being asked what they would do next.

• Example user behavior: Finding and saving the rule after being prompted.

Issue: First-Time Landing Screen

• Users who landed on the screen for the first time could recall accurately what actions were included in their sample rule.

• Most users were unsure of the meaning of "ready" but assumed it indicated whether the rule was on or would be on shortly.

Issue: Bluetooth-Related Confusion Post Rule Set Up

• Some users experienced confusion due to having set up Bluetooth and assuming that "ready" meant the rule was on.

• Users associated "ready" with Bluetooth being on and paired with their car, leading to confusion.

Issue: Difficulty Finding the "+" Button

• Most users struggled to discover the plus button to add a new rule and found it through trial and error.

• Users usually touched everything at the top of the screen to locate the plus button.

Creating a Custom Rule: Sleep Mode

• Users were asked to create a rule to prevent interruptions during sleep.

• Triggers: phone not moving and phone charging.

• Actions: silence ringer, dim screen brightness, and allow incoming calls from VIPs (e.g., your mom).

• Most users easily set up the rule, attributing it to clear instructions for triggers and actions.

• All users named the rule immediately (e.g., Sleep, Zzzzz, Night).

Creating a Custom Rule: Adding Actions and Triggers

• Most users had no difficulty finding or setting up actions.

• Some users found the terminology "When my charging status changes" to be awkward and confusing.

• Example user feedback: "It would be better to just call this charging. 'When my charging status changes' seems like the rule only turns on the moment my charging changes, like I unplug it from the outlet."

Creating a Custom Rule: Trigger Details

• Only one user discovered that clicking on the text allowed specifying the charging source.

• Other users weren't interested in this detail and didn't see a need for it.

Editing, Deleting, and Turning Rules On/Off:

• Editing the Name: All users successfully changed the name of the rule.

• Deleting an Action: Users attempted to delete the block in the rule builder or clicked the + next to actions to re-select the grayed-out action.

• Accidental Discovery: Some users accidentally disconnected blocks while scrolling through a rule.

• Turning Rules On/Off: Most users found the ON/OFF switch on the rule builder. Some users were confused by the menu button that disabled all rules, mistaking it for three buttons representing the rules.

Actions and Triggers List:

• Launching a website or application was considered fun by several users, despite their confusion during the rule task.

• Ring Loud for Important People was considered exciting and innovative.

• "When my display status changes" trigger was confusing to most users. They were unsure of its meaning or how a display status could change.

• "When my charging status changes" trigger was also confusing, with users interpreting it as triggering only during the change from charging to not charging or vice versa.

Summary:

• Language-related issues were prevalent throughout the session, leading to understanding difficulties.

• Confusion with Triggers: "When my display status changes" and "When my charging status changes" triggers caused confusion among users.

• Confusing Descriptions: Clearer language is needed for actions and triggers.

• Discoverability Issues: Icons, plus sign to add new rules, and the menu key were difficult for users to discover.

• Saving Rules: Users didn't immediately understand that rules needed to be saved to be adopted.

• Updated User Interface: Users found the updated screenshots and click-through prototype of the Drive Rule to be a major improvement, describing it as more user-friendly, intuitive, and visually appealing.

• Helpful Explanation Text: Users found the explanation text helpful in understanding how the applications worked.

• Awareness of Rule Status: Users noticed the connection between color and rule state, with on as blue and off as gray.

• Visual Cues: The flashing "thinking" and flashing gray to blue when adding an action drew users' attention and informed them about the rule's progress.