Quantitive Usability

By / / Tools & Techniques

In the field of user experience, there is a wide range of research available, researchers mapped 20 methods across 3 dimensions: “Qualitative vs. Quantitative”, “Attitudinal vs Behavioral” and “Context of Use”. This article will focus on introducing the “Quantitative research” method, and in order to make it more clear, it will also compare it with another direction in this dimension—“Qualitative”.

(figure 1)

What is Quantitive usability?

“Quantitative usability testing (or benchmarking) is based on a large number of participants (often more than 30); (quant) data, in form of one or more metrics (such as task completion rates or task times) that reflect whether the tasks were easy to perform.” (When to Use Which User-Experience Research Methods, 2014) In most cases, quantitive data provide a more indirect assessment of the usability.

It can reflect results about task-completion times, success rates, the perception of users (figure 2) or a number of errors occurred in participants’ user journeys.

(figure 2)

Its characteristics can be summarized as below:

What?—Focus on “how many?” and how much?”. The outcomes are statistically meaningful results that are likely to be replicated in a different study.

Why?—its goals are to evaluate the usability of an existing site; to track usability overtime; to compare the site with competitors or to compute ROI;

When?— You can use it when you have a working product, either at the beginning or end of a design cycle.

The methodology: Involve many participants, the test should be well-defined, strictly controlled study conditions, and don’t need a think-aloud protocol.

Pros and Cons of Quantitative Usability

Pros:

It is statistical significance. Quant data can be very supportive when delivering design, In addition, quantitative methods do a much better job answering “how many and how much” types of questions. The numbers can help prioritize works, for example, to focus on issues with the biggest impact. It’s also regarded as an essential instrument in calculating return on investment. (14 Usability Testing Tools Matrix and Comprehensive Reviews, 2014)

Cons:

-Time-consuming

Since a large amount of data is required, it’s often very time-consuming. In emergency circumstances, such as when an action is required urgently, this method is not recommended.

-Can be hard to interpret in the absence of a reference point. For example, in a test, if 60% of participants were successful, it is hard to say it’s successful or not. Experts said it is the reason why this method is rather to describe the usability of a site, it is often used to compare the new design with a previous one.

-It doesn’t point out “What problem”users encountered.

According to a quant usability test, you can know data such as “only 20% of participants completed the tasks”, but you won’t know what difficulties users met and how to make it easier.

What needs to be careful when doing “Quantitative Usability”

  • For quant studies, tasks need to have a single well-defined answer.

It means the answer cannot be open, for example, a task like “Find the address and phone number of MET”, if the user only fount its address but didn’t find it’s phone number, it’s hard to define if the task is a success or a failure.

  • need to be stringently controlled from session to session.

That is, you need to make sure that your tests should be run under a similar environment as possible, which means: you cannot run one session in person but get the rest done remotely.

Conclusion:

When trying to get general insights of usability of one site, Quantitive usability can be an appropriate choice, such as “how many people are satisfied with their experience here?” Or “How many clicks went to the wrong direction?” It can also help you compare one design version with another. It is a statistical type of method which needs relatively strict condition control. It often is used when there’s an existing system already.

 

References: