Capturing User Intent
Problem to be Solved
In situations with multiple similar elements, mabl can struggle to choose the correct one, resulting in inaccurate and unreliable tests.
Let’s say a user trains mabl to click a button in their application, but there are 6 similar buttons on the page. Maybe they clicked that button because it was first in the list, or maybe because it was in a certain state— but mabl doesn’t know that.
This means that when mabl runs the test later, it will have to guess which of those 7 buttons the user wanted it to click, resulting in a test failure that shouldn’t have happened.
Solving this problem created a unique and effective way to handle repeated elements scenarios, and drastically improve test accuracy and reliability.
Discovery
Understand why repeated elements pose such a challenge for the mabl app
Interview users to get a sense for what problems they’re struggling with, with respect to test reliability
Iterate and test
User flows to understand how this slice of work fits in the overall mabl user journey
Work with the lead engineer and PM to establish a short- and long-term strategy
Medium-fidelity wireframes
Multiple rounds of usability testing, rapidly iterating between rounds
Collecting feedback from the broader team
Evaluation
Does this solve the problem?
Design Process
Discovery
Team Structure
Engineering — 4 engineers; 1 tech lead
Product — 1 product manager
Design — Lead designer (myself); UX researcher
Understand Problem
Engineering noticed a trend where users were reporting much higher number of tests as false failures.
Our team’s TL, PM, and I reviewed the complaints together and determined that the uptick in false failures was largely due to mabl not knowing how to handle scenarios where it encounters similar elements on a page.
If you trained mabl to click one of the “delete” buttons, it would be unsure which of the six delete buttons to click on when the test runs in the cloud later.
User Interviews
The mabl trainer had a number of workarounds for handling repeated element cases, but none of the workarounds were user friendly, reliable, or scalable.
These methods involved writing JavaScript, and relying on static attributes like X-Paths.
Users told us that they don’t want to write JavaScript to solve this problem.
“Using JavaScript is a real pain. I'm not really a pro, so it can take me anywhere from five minutes to over an hour just to specify which element I'm targeting.”
— External user during user interviews
Iterate and Test
User Flows
This new feature would not exist in a vacuum. We needed to make sure that what we did here did not have a negative impact elsewhere in the mabl experience.
To do this, the team worked through a UX-led user flow session.
We broke down every step in the trainer process and identified where this feature sat within that process.
“If we think about this journey as having to physically walk where we want to go, look how far a user has to walk to go from the test results back to editing something in the middle of the flow.”
— Engineer during collaborative user flow exercise
Click to view larger version
Design and Testing
Our initial iteration went down a very technical path, consisting of a series of forms that mabl would prompt the users with as a means to gather intent.
With each iteration, the design became less technical and more visual. After testing and iterating over the course of a week, the design evolved and became simpler and more conversational, and users loved it.
Version 1
The first version was a form with three sections for the user to configure. Visual design was not a focus at this point.
Target — the attributes mabl recognized for the element that was clicked
Context — the attributes for elements around the one that was clicked
Wait: — how long mabl will wait for the target element to appear, and what to do if that element never appears
Version 1 was too technical, and heavy-handed. This version was only tested internally, and never made it in front of users.
Version 1 was too complicated and difficult to use, and was never tested with external users.
Version 2
Version 2 attempted to organize the information on the page more effectively.
However, the problem was that each category felt optional — the segmented control wasn't the right mechanism for sorting the information.
Version 2 did do well with explaining the difference between these three categories without relying on verbose copy.
Ultimately, version 2 was changed because it was still too difficult to use, and concepts like “one of many” simply weren’t clear to users.
Version 3
This was the designs breakthrough moment. It became clear that this could become more of a conversational experience. If mabl needed to understand user’s intent, why couldn’t it simply ask the user?
With that in mind, version 3 went in an entirely different direction. This is also when visual design became a focus.
By checking off answers to the question “Why did you click this element?”, users reduce the number of candidate elements and increase mabl’s confidence.
The closer to “one” the number of candidate elements is, the higher confidence mabl has of being able to find that element again later.
The confidence bar proved to be effective in driving users forward.
During testing, users told us that they would want to get that bar as close to “high” as possible, and likened it to a password strength indicator.
Users suggested that it was fun to use, as well.
During usability testing, users told us they wanted a degree of customization with these intent selections in order to make their tests more flexible.
Customization options are displayed on a bottom sheet, and follow the same conversational design as the intent selections themselves.
Usability Testing
Participants were given a prototype with a simulated app, and were asked to train mabl to extend the trials of users in that simulated app:
Click the “Extend trial button” for the following users:
• whichever user is first in the list
• for the user “Monika Jackson”
• for a user whose trial has expired
When completing these tasks, participants would eventually run into situations where mabl needed to prompt them for their intent.
Notes from each testing session were recorded in Dovetail.
Dovetail enabled me to “zoom out” and look at the notes as a complete body of work.
This allowed me to consider edge cases appropriately, and evaluate the design’s performance in aggregate.
Evaluation
Final Design
Testing confirmed that the conversational design was superior to the original form-based design .
Users saw the value in providing intent, and no one felt that the prompt was intrusive or annoying.
Users expressed their excitement to use the new intent prompt and were confident this approach would make their tests more reliable.
Problem Solved
Every user we tested with expressed their excitement to try the new feature out for themselves after it launched.
The conversational design achieves the goal of gathering intent to make tests more reliable, and does so without being obtrusive or invasive to users.
It’s fun to use and doesn’t require much additional time or effort from users.
This project and all associated designs and process artifacts are owned by mabl. The parts of the project that I can share publicly are limited — shown on this page are select pieces of the overall project.