Quantitative Usability: Check the usability by A/B testing

A/B testing


Quantitative usability methods, relying on computer data collection and statistical analysis, are relatively objective because the presence of the research is always the number and quantitative data. The quantitative data is one of the most important elements of usability testing. Quantitative data provides information that can be counted to answer such questions as “How many?”, “What were the outcomes?”, and “How much did it cost?”. So how can we analyze a series of quantitative data? For example, a web page has 3 thousand visitors traffic every minute. That means nothing. How can the usability experts evaluate their hypothesis and recommendations? The comparison, in this case, is the key issue.

There are some design methods based on the quantitative data in the real world’s design process, such as the Eye-tracking technology, Satisfaction survey, and A/B testing. The most convenient and low-budget method is A/B testing. This method is the most frequent method implied during my previous job because it is easy to observe the testing result and evaluate the design work’s efforts. The principle of this method is to define the variable element and compare with the pre and post edition. The researchers just need to observe the traffic of the pages or interfaces. For example, You compare two web pages by showing the two variants (let’s call them A and B) to similar visitors at the same time. The one that gives a better conversion rate, wins. By testing many times, the evaluators can figure out which elements of the interface influence the traffic and which iterating directions perform better and prove or refute their hypothesis.

Case Study

Here are the case studies in the real world. The first case is about a German company that offers various software training courses, MedienReich ComputerTrainings. The UX experts’ hypothesis is that replacing ‘course categories’ with ‘best selling courses’ on the homepage will boost engagement.

Case 1


The “Control” one (A) is the original page, offering visitors the traditional information in the home page. On the other hand, the “Variation” page (B) shows the most popular courses on the homepage.

The result proves the experts’ hypothesis. The engagement on the homepage increased by a robust 40.87% at a confidence level of 99.9%. The year-on-year value of the homepage also increased by 106.42%. The variation won because it served visitors with the right information at the right place. showing the popular courses on the homepage can give some visitors, who are not sure what courses they want to learn, chances to directly access the best-selling courses. So this information will enhance the engagement of these kinds of visitors.

The second case is about Price information page of Paperstone, which is a UK based eCommerce website that deals in office supplies. The Usability experts thought that displaying competitors’ higher prices on product pages will increase clicks on ‘Add To Basket’ and overall website conversion rate.

This test is special because it has 2 variation versions.



Variation 1


In this case, Variation 1 couldn’t perform better than the original page. This happened probably because the competitors-prices module didn’t have a differentiating presence on the page. The module was a plain text version and might have become a victim of banner blindness (Banner blindness is a phenomenon in web usability where visitors to a website consciously or unconsciously ignore banner-like information, which can also be called ad blindness or banner noise).

Variation 2


The second variation worked much better and won over the control. The competitors-prices module was more prominent — Paperstone’s price was also highlighted against the competitors’ prices, and a header text was added. The module was also repositioned below the ‘Add to Cart’ button in a bid to clear up the CTA’s surrounding space. This case provides some necessary questions in the similar design process:

  1. Would displaying competitors’ prices drive potential customers to your competitors, if the saving is only small? Or if visitors were not previously aware that the competitors sold the product?
  2. Would you appear more expensive on product pages that do not have the competitor price module?
  3. Would adding the competitor price module to your product detail pages create poor UX which distracts and puts off your users?

In the nutshell, A/B testing will effectively help the UX designer to evaluate their features and evaluate the usability in a cheap and objective way. However, A/B testing still has its limitation, such as the testing target or element of the A/B testing is single and the test result must require statistical data; A/B testing can only test the immediate user’s operation; A/B testing cannot provide the details of user’s behavior. There is no most perfect way for the UX field but there is the most suitable one. Usability evaluation method should be used in the right contexts and the right scenarios.



Nitin Deshdeep, “10 Kickass A/B Testing Case Studies From Our Archive”, https://vwo.com/blog/10-kickass-ab-testing-case-studies/

Jan Panero Benway David M, “Banner Blindness: Web Searchers Often Miss ‘Obvious” Links’ ”, Lane Rice University Houston, Texas

Design Critique: The Operation System of my Desktop Display Screen (electronic device)

The operation systems are very common and deeply influence our daily life. As we known, computer operation systems like Microsoft PC, Apple Mac; the mobile phone like iPhone and Google phone; even recorders, radios, watches have their own operation system. Good operation system will largely enhance our working efficiency. But bad operation systems will make us crazy.

My desktop display

I encounter many problems when I am using my desktop display to adjusting my desktop screen’s attributes (Brightness, Contract, etc.). Before I read <The Design of Everyday Things>, I thought that was my problem for some wrong operations. However, after this book taught me the systematic methods to evaluate design or user experience, I think the problem is the Usability of the screen’s operation system itself.

There are 3 mainly usability problems of this screen, I use the methodology from Don Norman to correct the design problems after I make some mistakes.

First of all, the problem is about visibility. When I wanted to turn on my screen first time, I could not find the power button to turn on the screen (P1). That very frustrating experience, because I couldn’t see any obvious signal to notice me where is a trigger to turn on, I could just touch everywhere of the screen to find the power switch. Finally, I found that the button, it was at the bottom of the screen (P2), And there are also few unknown buttons paralleled with the turn-on button on the left side. The problem of this product is people cannot recognize where the buttons are, which deviates the visibility principle.

P1, no signs can be seen when the it is off

P2, the button is at the bottom the screen.

The second problem is the mapping of the operations. When I wanted to adjust the brightness of the display, I had to randomly press the bottom of screen to activate the adjusting interface(P3). Only one specific button was available(P2), but I don’ know which button is the right one. The bottom of interface on the screen shows some icons(P4), but I cannot understand well the meaning of that (I even thought that is touchable). I could only try to press the buttons and guess that that icon’s meaning. I try for about 5 times to figure out which button matched to which operation. Actually, the meaning of each button is the alignment relationship between icons and buttons. The alignment is inaccurate so I often do wrong operation. The whole operation process was full of my mistakes and complains.

P3, bad mapping system in the screen system

P4, ambiguous icons and bad alignment

The last problem is constraints. I often press a wrong button because the bad alignment between icons and buttons below. Many buttons are still unavailable when it doesn’t align to a icon. In P4, you may see my finger is going to press the position of “down” command, but the actual button is “off”. So if I mistakingly press the “off” button, the screen will suddenly turn off and the operation interrupts. This experience is very bad because I have to reopen the screen and operate again.

The first recommendation for the screen design is to add some more obvious signal to show where is the power switch. The fluorescence material button can be used in the front of the screen. If there is a luminescent button in the front, user may more easily know how to turn on the screen. The second recommendation for the screen is to move the other button to another place where is relatively far away the power button. This change may decrease the possibility of interruption from suddenly turning off. The third recommendation is to move the buttons to visible place that can help user to recognize the meaning of each button and align the icons. Last but not least, I think some design detail like the alignment problem and the distance between each button, should be reconsidered.



<The Design of Everyday Things> Don Norman