The Unsung Hero in UX Design Process: Quantitative Usability Research


Introduction: As a user experience design student, quantitative research is mentioned far less than qualitative research. My brief encounter with quantitative usability data was during a class assignment where we were supposed to perform tree test through Optimal Workshop, but I was not fully aware why and when we need to quantitative data in usability research. Quantitative usability research is the unsung hero that points out the problem with solid statistic backup if done well, while qualitative research adds context to the problem and informs researchers how the problem can be solved.

With qualitative research, such as interview, survey, and remote usability testing, we get very interesting user feedback that brings close-up perspective to look at a problem. However, how do we identify a problem first? That is where quantitative research comes into play. With statistics, quantitative research gives a “Yes” or “No” answer to questions like, “is the event section on the website homepage can draw enough users to click on it?”

Quantitative usability research measures interface through 3 aspects: effectiveness, efficiency, and satisfaction. Effectiveness is indicated by how users correctly performed a task. Efficiency is measured by whether a task is completed and how long it takes to complete the task. Satisfaction is measured through questionnaires to indicate user’s experience with the interface.

The greatest benefit of quantitative research is that the results it delivers is very unlikely to be a fluke, which could be a problem in qualitative research when the test participants change. This benefit is a result of strictly controlled consistency of quantitative research task questions, test settings, and participants’ familiarity with the interface.

Despite of its obvious benefit, quantitative research is usually very expensive, takes a long time to complete, and needs many participants (ie. over 30 participants). Due to its expensive nature, many company are hesitate to use such research method. When companies consider whether a decision is expensive, it considers the actual cost, the cost–benefit ratio, and the available budget. The main concern for investing in quantitative research is that it may not generate result that worth the investment, or its actual cost may exceed the existing research budget. This is a common phenomenon among companies where the company is not aware of the value of quantitative research. Without quantitative research, the company’s UX team could risk the time and money working on a less significant problem which is tangible to the change of test participants’ population. More importantly, when combining with qualitative research, quantitative research is a very valuable tool to test the hypothesis problem qualitative research brings up.

When company doesn’t give quantitative research credit, such research becomes difficult to be carried out. However, a even more severe challenge lies in performing quantitative research. A recent survey conducted by Nielson Norman Group among 429 UX professionals indicates UX team lack the knowledge on how to conduct, run, analyze and report quantitative research. It seems quantitative research is an intimidating subject among UX designers.

Quantitative research should go hand in hand with qualitative research to better identify the problem, create informed design solutions, and verify the design solution. As companies’ UX culture grows, the budget on quantitative research is increasing. The future for implementing quantitative research without too much resistance within companies seems not too far, but the how and why quantitative research should be used needs to be thoroughly addressed in future UX professionals’ training.


Design Critique: Kindle Notebook Interface (PC app)

Figure 1.          

This critique evaluates the interface of Notebook function on Kindle PC app. The Notebook stores the following type of notes: bookmarked page, user’s written notes, colored highlights, starred highlights, and popular highlights from other users. Users can select from a drop-down menu to review their preferred type of notes. In addition, users can star or delete a specific highlight or written note.

Icon Paired with Text to Initiate Notebook Stress-free

Icon use helps the interface look clean (Figure 1.), avoiding too much text cramming on the already text-rich interface. The imagery nature of icon distinguishes itself from the text body, thus icon is visible enough to catch user’s attention when user navigates through the interface. However, since users could interpret the icon that looks like a piece of paper differently, this icon has ambiguous affordances on its own. Therefore, when the mouse touches the icon, the text label, “Notebook”, as signifier shows up, sending clear message of the icon’s purpose. Overall, combined use of icon and text increases the discoverability to initiate the Notebook, and therefore, reduces user’s stress when navigating across the content-rich interface. It is a success on the reflective level.

Intuitive Notes Viewing by Scrolling Down

Figure 2.

The notes are displayed in the order from the first to the last page of the book, rather than the time when the note is added. In other words, if a note on page 291 is added a few days after a note on page 292 is added, the note on page 291 is still going to show up first when the user scrolling down the Notebook column (Figure 2.). This is good natural mapping through culture. Experience on reading paper books teaches readers that the notes in the beginning of the book shows up before notes that are near the end of the page. Kindle Notebook’s system model matches user’s mental model that are learned through their past interaction with paper books. Furthermore, natural mapping also happens when the user scrolls up to reach notes in previous pages and scrolls down to reach later pages. In U.S, text content progresses in the direction from top to bottom on a single page. Therefore, it makes perfect sense to move up when searching for content in previous pages and move down when searching for content in later pages.

Weak Relationship between a Highlight and Associated Written Note

Figure 3.

When the user highlighted a passage and generated a written note for the same passage, Kindle Notebook does not have strong signifier to address those are very different types of notes, or the association of the written note to the highlight passage. Without close attention, the written note is easily mistaken as a highlight (Figure 3.).

Solution: Instead of showing the written note on the vertical level (ie. above or below) to the highlight note, the written note could be placed in a conversing shaped box that can be extended from the highlight note. All highlight notes that are associated with written notes should look clickable. When the highlight note is clicked on, the conversing box containing written notes expands. This design signifies the relationship that the written note is an intellectual product generated from the highlight note. The shape of the conversing box carries the interactive nature of the reader and the text, and therefore semantically maps the relationship between the reader and the text.

Colored Highlight Has Ambiguous Affordances

Kindle Notebook users are given four color options when they highlight, but the fact that users can view notes selectively based on highlight’s color is not signified in the design. It requires certain effort for users to assign different meanings associated with the color and to remember what they mean, such as “pink is absolutely important”, “yellow is fun reading”, “blue is scientific/technical concept”, “orange is worth to read again. The user would not actively use different colored highlights if he/she is not able to see its greater value in viewing notes in an organized way.

Solution: Whenever a novice Kindle user highlights a passage, an interactive prompt pops up as he/she is going to pick the highlight color, “Choose a color and view notes in color of your choice in Notebook later”. This prompt shows up in the first 5 times when the new user wants to highlight. This prompt indicates what actions the user can take with the highlight and what they could expect, and therefore, it improves the color highlight’s affordances.