Quantitative Usability Testing in IBM Watson Health



We love to measure things. It is our way to understand how good or bad something is. Numbers help to evaluate the situation in a quick and easy manner. In this post, I summarized the responses to my six questions related to quantitative usability testing focusing on the real world methods and cases during an interview with Oliver S.


About Oliver S.

Oliver S, Design Lead at IBM Watson Health. During years of experience in creating digital products on different platforms for leaders in finance and Healthcare industry, he has helped clients envision their new user-centric enterprise product experiences and established solid foundations for a consistent and scalable design framework across their entire portfolios.


1. Which usability testing method do you or your company use more? Quantitative or qualitative? And why?

We use both in equal amount, they obviously serve different purposes. It is always good to have quantitative data( like A/B testing or A/B/C testing or accessibility testing). There is no distinction between those two, often time it’s like the combination of both.   


2. What specific quantitative usability testing methods do you usually use and in which phase of the design?

A/B testing is the most common method we use. We do that sometimes before the development cycle starts. We create prototypes to validate our ideas, interactions, UI, layout component, etc.

We also do Free-form Usability Testing. We give the user the latest version and test users for 10 to 15 mins. We usually start with a few tasks and record the accomplishment rate. This is more of a later phase of the development cycle. Those feedback can inform the next iteration of the future decision-making process, and prioritization.

Quantitative is almost a must and qualitative is always a part of it, so you should capture both of them.


3. What do you think are the pros and cons of the methods you use? (Quantitative)


  1. Very persuasive, in terms to justify your design decisions and align stakeholders internally. This happens all the time when we have a group of people like business, development, and sales.  If you have strong data on behave of the users, you can have a very strong statement to support your suggestion.


  1. It needs a lot of data for the conclusion. And it is not easy to recruit users. The process is very expensive. Time-consuming and cost heavy.
  2. Design is not really driven by the data we get. So this is the limitation of the data bais. If you only have 5 users all from one state you can’t get different perspectives from different states. So we need to get a big volume of the data to perform unbiased analysis.


4. What do you usually measure during a quantitative usability testing?

You can measure how much time to complete the interaction. Completion rate. You can’t give them unlimited time, at some point, you have to cut off and see it as a fail. Eye tracking. You can see which portion of the screen caught people’s attention.


5. Can you describe a case when quantitative usability results influence design decision?

During a navigation system validation that we did before for a web app. We were designing a search engine, so it was very information heavy and took a lot of horizontal space. A designer’s instinct would hide the navigation menu. But business stakeholders want to give users immediate access to all different products.

So we did  A/B Testing. But the result is: People don’t care whether it’s hidden or not. Because they use two big monitors all the time, so horizontal space really doesn’t matter to them. So we came up with a hybrid design.  Hamburger button and Icons of different applications are shown on the sidebar and after clicking the hamburger button, all the labels will show.

Then we tested another 5 users and 7 out of 10 liked the third option.  We let the data tell the story.


6. Can you describe a case when “number” convinced folks like executives?

If you want to enhance experiences, we use a behavioral analysis tool called “Amplitude” to track users’ interactions with the application. After improving a certain step, you can run a test to see if the result is better than before. For example, if the result is 20% better. That can give you enough confidence to change the step.

So besides the visual part, quantitative usability testing can also help improve the overall experience.


Quantitative User Testing Methods Retrieved from http://usabilitytesting.sg/user-experience-course/lesson-6-quantitative-user-testing-methods/

Jakob Nielsen (January 21, 2001) Usability Metrics Retrieved from https://www.nngroup.com/articles/usability-metrics/

Jakob Nielsen (November 21, 2011) Accuracy vs. Insights in Quantitative UsabilityRetrieved from https://www.nngroup.com/articles/accuracy-vs-insights-quantitative-ux/

Kate Moran (April 22, 2018) Quantitative User-Research Methodologies: An Overview Retrieved from https://www.nngroup.com/articles/quantitative-user-research-methods/

Measuring UX and ROI Retrieved from https://www.nngroup.com/courses/measuring-ux/

ARMEN GHAZARYAN (JANUARY 21, 2015) 5 UX KPIs You Need To Track Retrieved from https://designmodo.com/ux-kpi/

Design Critique: Winc(iPhone App)


Winc is an iOS app I use to order wine online. Users take their simple 6-question Palate Profile Quiz and Discover their personalized wine recommendations. The selected wine will be delivered per month. Users can use the app to:

  • Sign up/Sign in
  • Manage membership
  • Discover personalized wine recommendations
  • Customize orders
  • Ship box immediately
  • Rate wines
  • Invite friends

In this design critique, I am going to analyze the experiences of Manage membership, Discover personalized wine recommendations, and Customize orders.


Manage Membership

Overall Rating: ★★★☆☆

Well Designed:

To get access to membership, there are only two tappable icons provided. One clearly indicates the number of wine user added(top right). Proper using of semantic constraints suggests that users should tap the “hamburger button” to find their membership information even though it is the knowledge of head that “hamburger button” means “menu”.

Could Be Improved:

Invite Friends is highlighted while not being used by users often. The highlighted number causes problems because it dominates over other more useful features too much, which is not designed with the needs and interests of the user in mind. 

The SMS center and email center has poor mapping and system image, the user flow is incomplete, users can’t predict the outcome after checking or unchecking all the checkboxes.


The action of “Invite Friends” can be designed when users finish their order. So it won’t appear every time in the menu with the red notification. After failing to clear the red notification multiple times, users would decide that they cannot do the task and ignore it. Relocating the“Invite Friends” action to the finishing order stage will provide a better user-centered design.  

Clear word instructions (what type of text message and email services they will get) can be provided with the checkbox, which will improve the visibility of the design and complete the system image in order to make the users know what to do and understand the outcomes.


Discover personalized wine recommendations

Overall Rating: ★★★★☆

Well Designed:

The experience of discovering personalized wine recommendations is designed with clear and separated cards with the name and the location of the wine, the price, and an “ADD TO BOX” button, which provides great visibility and natural mapping. Users know exactly what they will expect after tapping the button. After tapping “ADD TO BOX”  button, the number of wine added will show on the top right which provides solid feedback.

Could Be Improved:

There are no clear instructions on how to get access to the detailed description of the wine, users have to rely on their knowledge of the head to tap the card to get to the destination, which also doesn’t help users complete a smooth seven stages of the gulfs of execution and evaluation.


By adding a light grey background color and a simple Learn More button will improve the affordances of the card and provide a better conceptual model. Users will know the physical properties of the “Learn More” button affords to tap and the outcome from that would be the detailed description of the wine.


Customize orders

Overall Rating: ★★☆☆☆

Well Designed:

The experience to choose the preference wine has a good balance between usability and aesthetics which makes it easier to use. Two clear selection boxes are provided to customize the amount of red wine and white wine with no other useless features. The total number of wines per month changes after the selection which provides good feedback.

Could Be Improved:

Users can’t cancel the order or the monthly subscription without contacting the support, which also leads to a low rating result in Apple Store (two stars). With the current design of the ordering system, wines are sent monthly even without personalized selection, which creates Loss-of-Activation Errors and makes users feel stupid by forgetting to choose their wines.  


To prevent human error, it’s better to make it possible using the app to cancel the order by adding a “Cancel Order” section under the menu. Notifications should be sent before the order date to remind the users to pick their desired wines, which will create a better system with less human error.