System Usability Scale: A Quick and Efficient User Study Methodology

 

Summary

Fig 1. SUS Score diagram, (Caglarca, Sercan, fig. 4. http://www.sercancaglarca.com/portfolio/destinationuppsala.html )

This article briefly described the differences between qualitative and quantitative usability study methods, indicated that high cost is one of the disadvantages of quantitative methodologies, and introduced the System Usability Scale (SUS), which is a quick, efficient and reliable usability study method, and explained SUS’s working process.

Quantitative Usability vs. Qualitative Usability

The quantitative usability study is usually been considered as an expensive method that can cost “four times as much as” qualitative studies, and qualitative studies “often generate better insight” (Neilson, 2001). However, quantitative studies is still a good supplement to the qualitative usability study methodology (NNGroup) and can offer an assessment of tasks’ difficulty (Budiu). As Jenna Erickson has stated, both qualitative and quantitative data are both necessary if you want to fully understand an average user’s potential behavior (Erickson). Qualitative research might help researchers understand the specific flaws of a product (Budiu), and can tell you why and how your test participants conduct a certain task intuitively, it also can be biased. While qualitative testing determines “users’ experiences, perceptions, opinions, and feelings”  (Usability Testing) and answers questions such as why and how to fix a problem, quantitative methods answers questions such as how many and how much (see fig.2), in order to obtain specific numbers such as “success or time on task”, with with “rigorous well-controlled conditions” (Meyer). In other words, while qualitative studies usually “tell stories”, quantitative “gives contexts” that its feedback “allows you to understand your site’s usability in the context of a much bigger picture” (Trymyui).

Fig. 2. Questions Answered by Research Methods Across the Landscape. (Rohrer, fig.2.)

Qualitative and quantitative studies are supplements to each other. When quantitative studies have a great strength of providing descriptive data (Madrigal, et. al), qualitative research fulfill its difficulties of interpretation. However, in reality, with a small budget, companies usually conduct only qualitative studies for its lower cost and direct fruit of users’ insight that provide better investment returns, and the quantitative study is often been neglected (Nielsen, 2001).

Introduction of System Usability Scale

When most quantitative study methods are considerably expensive, the System Usability Scale (SUS) was developed as a quick and efficient way of testing the usability of a user interface. As a fairly short quiz, it doesn’t require a lot of resources to administer (Thomas). In fact, according to John Brooke, the developer of SUS, intended to be a “quick and dirty” methodology. It is “quick” for the reason that it “allows us to do so in the very short time available to us during an evaluation session”; And it is “dirty” because this method “provides us with a measure of people’s subjective perceptions of the usability of a system” (Brooke, 2013, pp. 33).

Moreover, as a tool that measures only users “subjective view” on a system’s usability, SUS has been proved “valid and reliable” (Brooke, 2013, pp. 33), and that it is comparable to other standardized usability questionnaires (Sauro, 2011, pp. 72). SUS has become an industry standard with references in over 600 publications (Sauro, Feb 2011) and has become a part of ISO 9241-11 (usabilityscale.com) in the year 1998.

How to Conduct a System Usability Scale

To conduct a System Usability Scale, you will need participants to answer 10 questions on a Likert Scale (Thomas). Participants will rank each question from 1 to 5 based on how much they agree with the statement they are reading. 5 means they agree completely, 1 means they strongly disagree (similar to Pratt’s course evaluation form).

The 10 questions are:

  • I think that I would like to use this system frequently.
  • I found the system unnecessarily complex.
  • I thought the system was easy to use.
  • I think that I would need the support of a technical person to be able to use this system.
  • I found the various functions in this system were well integrated.
  • I thought there was too much inconsistency in this system.
  • I would imagine that most people would learn to use this system very quickly.
  • I found the system very cumbersome to use.
  • I felt very confident using the system.
  • I needed to learn a lot of things before I could get going with this system.

and each question should be answered in the format as below:

Fig. 3. A sample statement from SUS, (Sauro, Feb 2011, Fig. 2)

With a simple strategy like this, SUS is cheap to start using, particularly online, and it’s still one of the “most efficient ways of gathering statistically valid data and … a clear and reasonably precise score.” (Thomas)

Thomas has concluded the calculation of three simple rules from John Brooke’s 1996 revised version of SUS (Thomas; Brooke, 1996, pp. 6).

“Your users will have ranked each of the 10 templates questions above from 1 to 5, based on their level of agreement.

      • For each of the odd-numbered questions, subtract 1 from the score.
      • For each of the even numbered questions, subtract their value from 5.
      • Take these new values which you have found, and add up the total score. Then multiply this by 2.5.

The result of all these tricky calculations is that you now have your score out of 100. This is NOT a percentage, but it is a clear way of seeing your score.”

The calculation became even easier when an Excel spreadsheet has been created to help us calculate the SUS result automatically, which can be easily found on the web.

After we generate a system’s score, the number itself has no meaning without interpretation. After a 500 of evaluation with over 5000 users, Jeff Sauro provided a simple and quick method for us to interpret the SUS score that has been generated (Sauro, Feb, 2011). If the system gets an 80.3 or higher is an A, which means the system is good and will be recommended to others; 68 is the average score among all 500 tests that Sauro has run; and 51 is about the F, which means usability is the priority problem (see fig. 4). 

Conclusion

In summation, on the one hand, as Sauro has indicated, “data from over 5000 users and almost 500 different studies suggest that SUS is far from dirty” (Sauro, Feb 2011). On the other hand, to gain a valid and fairly accurate result with only a 10-question scale, SUS is surely a “quick” method of the usability study. What’s more, with the tools developed in the 30 years after SUS’s release, such as the SUS calculator and Sauro’s interpretation diagram, this method became even quicker. Therefore, I would conclude that compare with the other quantitative method (which might have other advantages), SUS is a quick and “quick” usability scale.

References

Brooke, John, SUS: A Retrospective, Journal of Usability Studies, Vol. 8, Issue 2, February 2013, pp. 29-40. Retrieved on April 10th, 2018, http://uxpajournal.org/wp-content/uploads/pdf/JUS_Brooke_February_2013.pdf

Brooke, John, SUS – A quick and dirty usability scale, Usability evaluation in industry, Vol. 189, Issue 194, June 11, 1996, pp. 4-7. Retrieved April 10, 2018. https://hell.meiert.org/core/pdf/sus.pdf

Budiu, Raluca, Quantitative vs. Qualitative Usability Testing, www.nngroup.com, October 1, 2017. https://www.nngroup.com/articles/quant-vs-qual/

Erickson, Jenna, UX Designers: Do Not Be Scared Of Quantitative Data, usabilitygeek.com, April 10, 2017. https://usabilitygeek.com/ux-designers-quantitative-data/

Madrigal, Demetrius, and Bryan McClain, Strengths and Weaknesses of Quantitative and Qualitative Research, www.uxmatters.com, https://www.uxmatters.com/mt/archives/2012/09/strengths-and-weaknesses-of-quantitative-and-qualitative-research.php

Meyer, Kate, Writing Tasks for Quantitative and Qualitative Usability Studies, www.nngroup.com, January 21, 2018. https://www.nngroup.com/articles/test-tasks-quant-qualitative/

Mortensen, Ditte, Best Practices for Qualitative User Research, www.interaction-design.org, March, 2018. https://www.interaction-design.org/literature/article/best-practices-for-qualitative-user-research

Nielsen, Jakob, Quantitative Studies: How Many Users to Test?, www.nngroup.com, June 26, 2006. https://www.nngroup.com/articles/usability-metrics/

Nielsen, Jakob, Usability Metrics, www.nngroup.com, January 21, 2001. https://www.nngroup.com/articles/usability-metrics/

NNgroup, Analytics vs. Quantitative Usability Testing, Youtube.com, July 14, 2017. https://www.youtube.com/watch?v=YWCfjy1BeA8

Qualitativeataanalysis, Qualitative and Quantitative Data Analysis: 7 Differences and the Common Sense, www.qualitativedataanalysis.net. http://www.qualitativedataanalysis.net/qualitative-and-quantitative-data-analysis/

Rohrer, Christian, When to Use Which User-Experience Research Methods, www.nngroup.com, October 12, 2014. https://www.nngroup.com/articles/which-ux-research-methods/

Sauro, Jeff, Measuring Usability with The System Usability Scale (SUS), MeasuringUsability.com. February 2, 2011. https://measuringu.com/sus/

Sauro, Jeff, A Practical Guide to the System Usability Scale (SUS), MeasuringUsability.com.2011. Retrieved on April 17, 2018, https://lms.pratt.edu/pluginfile.php/784763/mod_resource/content/0/practicalGuidetoSUS.pdf

Thomas, Nathan, How To Use The System Usability Scale (SUS) To Evaluate The Usability Of Your Website, usabilitygeek.com, July 13, 2015. https://usabilitygeek.com/how-to-use-the-system-usability-scale-sus-to-evaluate-the-usability-of-your-website/\

Trymyui, Quantitative and Qualitative testing, www.trymyui.com. https://www.trymyui.com/quantitative-and-qualitative-testing

Usabilityscale, Methodology, usabilityscale.com. https://usabilityscale.com/methodology/

The iPhone’s Keyboard: Pros and Cons

iOS’s keyboard, as one of the most commonly used keyboard, is a very well, fully developed keyboard system. It supports numerous of languages (including more than 2,500 emojis) and variety of layouts, and can smartly auto-correct the text in most cases.

iPhone Keyboard in different settings.

( Left to right: English, QWERTY; Chinese (Simplified), 10 keys; Chinese (Traditional), QWERTY; Emoji )

Note: This critique focuses on iOS’s keyboard, and the research is based on iOS 11.2.5 that runs on an iPhone 6s. Some functions can be different if you have a different version of iOS or running on a different device such as an iPad.

Intro

In an earlier report, Sarah Perez displayed a diagram of research launched by Kantar Worldpanel in August 2017. According to that, iOS shared 35% of the smartphone market in the US. World widely, the number is slightly lower but still considerable large. The statistic shows that almost one-third of American adults are using an iOS system, which means here in the States, there are approximately 0.1 billion people are texting with an iPhones keyboard.

I myself am a long-term fan of Apple’s product. Since the year 2007, I started texting mainly with iOS keyboard (on an iTouch). Therefore, many “hidden” functions of the keyboard become a knowledge in the head to me. If you knew how to use these functions, your experience with iPhone can be much better. Such hidden functions often lack discoverability or understandability, which cause this considerately designed keyboard was usually underestimated.

Android or iOS, new versions or old, the basic arrangement of their keyboards are based on the American standard QWERTY keyboard. This allows new users can easily find either the shift key or any other keys such as space or return, as long as they have used a physical keyboard before. The usage of a physical keyboard has been repeated for so many times so that where the shift key and the space key should locate are become a behavioral level cognition. If you want to delete something, you go to the right side, and to type a space, you look down. Where these keys are arranged on the keyboard become somehow become a natural mapping. Therefore, with this inherited arrangement, most users can text with iPhone’s keyboard. And as long as people can text their friends, they won’t really complain about their phone, or more specifically, their phone’s keyboard. Even though, people still expecting a better experience.

Auto-Correct

The iPhone keyboard succeeds in providing such better experiences, in most cases. First of all, it allows you to mistype. For the reason that on an iPhone, the QWERTY keyboard has been shrunk down to a considerable small size, mis-typing occurs all the time, for the similarity of the actions to tap “T” and “Y”, there is only two millimeter’s difference. In this case, slips occurred on the performing stage of action. Instead of making the user to backspace and retype, the system would auto-correct the word by estimate what word the user actually wanted to input. For example, I always mistype “T” and “Y”. If I were trying to input the word “typical”, sometimes “ytpical” would be typed. The system would immediately auto-correct it and input the correct “typical” instead. On the other hand, as a second language user, mis-spelling also happens a lot. In fact, since auto-correct exists, many native speakers make lots of spelling mistakes. Mis-spelling, instead, is a knowledge-based mistake. The iPhone keyboard allows this, too. Even better, sometimes when I don’t know how to spell a word, I would simply type an approximately correct word and let the system do the rest. The auto-correct function makes my life so much easier in this perspective.

On the other hand, however, sometimes you will hate the auto-correct so much that you want to turn it off once and for all. For instance, when you were typing an uncommon, long and weird Scandinavian name, and you finally finished typing and click space, then “bang”,  “Sigurjon” magically turns into “Sign on.” Fairly saying, I can tell that Apple’s designers have considered this mistake thoroughly. The key is you, the user, should not click the space button as you usually do. Instead, you can manually choose the “Sigurjon” displayed with a quotation mark above, listed with “sign on” and other guesses that iOS provided. Moreover, after you retype this word for several times, the system will learn that you intended to input this word and will stop auto-correcting it. (That’s why I can’t provide an image of this auto-correction, my iPhone has already learned this word.) Finally, you can always choose to turn this function off from the Settings. Even though, this process is still frustrating for many users. I have to say that there’s no alternative solution for the problem has come out since there’s a paradox exists within. Maybe we should just wait for a bigger database for spellings and a smarter AI been invented. For now, Apple designers have already done a great job by providing three alternative solutions.

Hidden Function, Shift & Caps Lock

Except for obvious function like typing, iPhone keyboard has a lot of hidden functions that can optimize user experiences in many ways. Some of them, such as the punctuation and number keyboard that signified with a key with “123” on it, or the microphone icon affords the function of the audio input, have a good discoverability and most users can discover them, since they turn the information into the knowledge in the world. After not too many times of using, actions such as tap the “123” key to call out a punctuation keyboard will become a behavioral action and will be done really quick since it’s sub-conscious. Admittedly, “123” doesn’t signifies punctuation well. But on the one hand, without too much thinking, you will try to hit the “123” key to call out a punctuation keyboard. It is because, firstly, most people share a common understanding that a keyboard would always include a punctuation keyboard. And secondly, rare people would try the other keys have clear signifiers to other functions. I think Apple’s designers actually did a good job of taking advantages of user’s behavioral cognition. On the other hand, the way of changing the keyboard modes varied a lot of times. (I still miss that in earlier versions the keyboard has the “@” key on the first page, a feature that you can still find on Chinese keyboards.) Therefore, for many users who have experience on the iOS system, where to find this function (the left side by the earth icon) became a knowledge in the head, and a key with “123” can give a fairly enough affordance to them. 

However, Apple relies on their user’s knowledge in the head too much. Take my girlfriend as an example, who become an iPhone user since last August. Before that, she had been using cellphones for twelve years and the Android system for seven years. Although she is experienced with using smartphones, she still couldn’t find some functions. She once asked me how to keep typing capital letters without repeatedly tap the “shift” button (the one with an arrow point upward). To me, it is a function that I use all the time: if you tap the key once, it will be highlighted, indicating one time of capital input; if you quickly tap the button twice, it will display a slightly different icon with a line lies below the arrow, and you can keep typing capital letters without hit this button once and once again. This feature provides a good feedback to users who know this function beforehand – whether my action effected and which mode was activated are both indicated immediately. However, for those users who don’t know this function before, this function is almost impossible to discover, for the reason that it doesn’t have a signifier. Although there’s a wide upward arrow as those drawn on the physical keyboard’s shift keys, which here affords the function “shift” to the users. However, this signifier doesn’t hint the function of “caps lock”. For a long-term iOS user, it might be very easy to figure out, even he doesn’t know this specific function. The gulf of execution might be quite easy to be bridged with several times of sub-conscious attempt. Tap and hold, doesn’t work? Double click, bingo. However, a new user such as my girlfriend might have a totally different process, for the reason that the Androids’ keyboard has a differently functioned “shift” key.

Hidden Function, Possibilities of Tap and Hold

Similarly, recently another feature was introduced with iOS 10. This function might not be so hard for an experienced apple device user to discover. If you tap and hold the earth key, a menu would pop-up, which includes the selection of different type method and one hand keyboard options, as well as a link to the keyboard setting menu, which is deeply hidden in the Settings. This function, again,  is hard to be found unless you are so bored and keeping trying possible functions of each key. Honestly, this is how I found this menu. Furthermore, if I haven’t built up a conceptual model that base on my understanding of Apple products – sometimes a key has multiple functions, and in many cases the way to call it out is tap and hold, I won’t have such a behavior that tapping and holding different keys to see whether there are any secrets. If you actually keep trying, you will find some keys for letters, numbers and punctuations have some hidden input options. All these convenient functions are not fully functioning because lacking discoverability and too much reliance on the user’s knowledge in the head.

In my perspective, the reason for that iPhone keyboard lacks signifiers for hidden functions has two main reasons. First and foremost, Apple cares the simplicity and aesthetic of the interface’s visual layout a lot. In fact, Don Norman thought this is the main reason. “Apple has done the world a great disservice by emphasizing the appearance and actually making their products more difficult to use,” He said in an interview, “Apple has completely forgotten that an important part of design is being able to understand and use your stuff.” (Salim) Admittedly, Apple’s designers must have an aesthetic concern. However, I think they also considered elsewise. On an iPad’s keyboard, you can find more signifiers than on iPhones’. There is smaller grey icon/symbol on the side. On an iPhone, instead, because of the screen is much smaller, the size of the interface is limited. Putting too many icons/symbols on will surely damage the not only the visual simplicity but also the understandability. Too much information on a single key won’t be recognizable. Imagine a button smaller than a quarter inch and has two different icons of the same size overlapped, no one would be able to recognize anyone of them. The iPad’s solution won’t work as well, the secondary icon would be way too small to be recognizable.

Another reason is that, personally saying, Apple focuses more on its loyal customers, who usually already have built up a conceptual model of how to use their products that help them bridge up the gulf of execution. Likewise, if I were facing the same problem my girlfriend had, I would directly go to the shift key and try different actions with that key – tap and hold, double click, etc. don’t’t really have the same problem – after a while.

Nevertheless, the keyboard still needs to provide a better user experience to the new users, or users who haven’t dig into these hidden functions – just as Don Norman has stated that designers should not expect all their users would explore the instructions. In order to do that, I would suggest two different solutions. Firstly, providing an instruction in front would be helpful, just like many apps do. In fact, when some new functions were published, Apple usually provides a brief introduction beforehand. On the contrast, some super basic functions such as the shift key, or a well thought through new function like a single hand keyboard for both left handy and right handy people, don’t have such an introduction – neither when an iPhone was newly bought, nor when the system was updated. Another solution is specifically for the “shift” key. On an Android phone, there’s actually no such a function equals to the shift on a physical keyboard. That actually works. Instead of having a key with two different modes, it provides only the “caps lock” function. The users only have to tap it on or off, and the system would do the rest – a function that iPhone’s keyboard also has. Auto-caps the first letter of a sentence, auto-correct some familiar names or terms, etc. The two modes of iPhone’s keyboard are useful, but without discoverability, it will be useless.

Hidden Function, Shake to undo

Compare with the shift key and the type method menu, another hidden function is much easier to discover. What would you do if you mistyped something and feels anxious? Many would shake the device that they made the mistake with. If you did that with an iPhone when typing, a dialog box would pop-up as a feedback, telling the user who didn’t know this function before that with the former gesture operation you can actually undo your last action. Instead of learned skills, it relies on people’s visceral level of cognition. Moreover, if you do this action for multiple times, the system will provide you a different dialogue box includes a redo function. With this gesture operation, iPhone keyboard provides a function to make up mistakes, as Don Norman has advised, “make it possible to correct problems directly from help and guidance messages”.

Conclusion

There are so many more aspects can be discussed about iPhone’s keyboard, such as the backspace operation, the transition between different typing methods, the emoji selection and the different skin color selection, AZERTY and QWERTZ keyboard support, T9 typing method, and etc. Although this post has pointed out a lot of problems of discoverability, these hidden functions actually enhanced the iPhone keyboard’s usability. The more functions a single product can provide, the more complexity there is. There will be nothing to complain about iPhone’s keyboard, as long as we don’t have to Google “hidden functions of iOS.” Just as Don Norman said, “complexity is good; It is confusion that is bad.”Reference:

Perez, Sarah. iOS and Samsung market share now tied in the US
https://techcrunch.com/2017/10/13/ios-and-samsung-market-share-now-tied-in-the-u-s/

Salim, Ayesha. Design guru Don Norman slams Apple’s ease-of-use “disservice”. 2015.
http://www.idgconnect.com/abstract/10261/design-guru-don-norman-slams-apple-ease-disservice

Norman, Donald. The Design of Everyday Things. Basic Books, 2013.