Pure Methodology : A Usability Evaluation Method


Source: http://oxzin.com


What is Pure Methodology and why is it important?

While interning with Techsmith this summer, I got the opportunity to participate in a usability evaluation method called Pure (Pragmatic Usability Rating by Experts) Methodology. This was my first experience participating in PURE usability evaluation method and found the method to be very productive and easy to accomplish.

According to Nielsen Norman Group, “PURE is a usability-evaluation method in which usability experts assign one or more quantitative ratings to a design based on a set of criteria and then combine all these ratings into a final score and easy-to-understand visual representation.”


Quantitative and Qualitative Data: As a UX professional, it is important to communicate findings from your research/testing in a way that can be interpreted by your team and other stakeholders involved. There is a constant need of quantitative data to represent your results/findings. As the results from PURE methodology are in the form of ratings/scores/numbers, it becomes relatively easy to inform the results to different stakeholders who may not understand the qualitative data easily, especially business.

Requires less Time and Resources: You don’t need to involve real users in this process, as its an analytical usability evaluation method. Definitely, its not a replacement for usability testing, but it is a great method to evaluate your application in different phases of design/testing or evaluate your competitors.


What do you need for PURE?

Usability Experts: To perform this method, the panel should consist of at least three usability experts.

While conducting PURE method, our panel consisted of designers as well as researchers. Also, we had more than 3 experts that evaluated the intended applications. So, you can have designers included in this process as well, but they require prior understanding of user centered design.

A Facilitator:  The facilitator is responsible to keep track of time, organize meetings, create and manage required documents (ScoreCard/ScoreSheet) and ensure that everyone is adhering to the predefined target audience.

Target User/Persona: It is necessary to “know your audience” well before you proceed with PURE method.In this method, focusing on a Persona or an Empathy Map would be truly beneficial as you could revisit your target user as per requirement, in turn keeping evaluators, the facilitator and all associated stakeholders on the same page.


How to do it?

Identify Target User: The target user type can be decided by the product manager, lead designer or the  expert panel. I think its highly important to consider a specific persona/target audience so that it would assist expert panel during the evaluation process. 

Identify the Main Tasks: You need to identify the key/critical task for the persona/empathy map selected. When you are selecting the Tasks, its important to consider the business context for the product as well. Example-‘Buy Tickets’. Nielsen Norman Group recommends “to keep the number of tasks close to 10, and no more than 20, at least when you first start using this method”. We had in all 7 tasks defined across two applications(our competitors), while running PURE for the first time. For example: Installing an application, Purchase Items.

Determine Happy Paths: There could be many ways a task could be accomplished. The Happy Path is the most desired path the target audience would take to accomplish a task. Evaluating and improving the happy path would be an effective way to make the task simpler for users. In our PURE process, a senior researcher and designer collaborated to determine all the fundamental tasks and their happy paths.

Step boundaries for each task Break the happy path into small steps. A step begins when the system presents the user with a set of options and is waiting for user input to proceed. Micro interactions need not be considered as a separate step.


Individual Rating by Expert Panel

The facilitator needs to create a document for evaluators to enter ratings for all steps of each task. This document is referred as the Score Sheet. In this sheet, the facilitator lists all predefined Tasks and Steps involved in each task.


Then the evaluators would rate each step individually (1, 2 or 3) depending upon the following criteria:


Calculate inter rater reliability score

The facilitator would now collect the individual scores. To understand the friction among the evaluators ratings, the facilitator uses the scores to calculate inter rater reliability score.

Final Scores

Now, the expert panel would walkthrough all the steps of each task and discuss about it. The discussion would help in understanding the perspective and reasoning behind the scores allocated by other evaluators in the panel and help to decide a single score for each task. The decided score is then considered as the PURE Score for the respective step. However, we twisted this process a little bit while performing PURE. Instead of deciding a single score, the evaluators had the freedom to change their scores after understanding the perspective of other expert raters. The facilitator then calculated the average of scores given by all the evaluators for each step  and rounded of to the nearest whole number, which acted as the ‘Decided Score’.

ScoreCard: All the decided scores are incorporated onto a single sheet called Scorecard. Scores are summed for each task and depicted on the right of the respective tasks which can be referred as Task Score. All Task Scores are summed to calculate the final score for the application.

Any Improvements?

Multiple Happy Paths: Instead of sticking to just one happy path for each task, it would be great if multiple happy paths for a single task could be evaluated. When we were reflecting on our PURE process, it was also found that the evaluators were interested to discuss together and determine different happy paths for each task.

More than 3 point Scale: When we were assigning ratings to all steps of each task, it was really difficult to determine a score when we had only a 3 pointer scale to use. Additionally, we were constrained to provide ratings on the basis of ‘Cognitive Load’, leading us to disregard few other usability issues that we encountered during our evaluation process. It would be interesting to have at least a 5 pointer scale to assign ratings that considers other parameters  to make the evaluation more inclusive and comprehensive.



Finally, I would say that Pure methodology is an impressive evaluation method, providing results in the form of qualitative as well as quantitative data. But, there are definitely few areas where the PURE methodology could be improved as discussed in the article, but its still in its nascent phase. It would be interesting to see how PURE would grow in next few years. Overall, It is a good answer to the question ‘Give me the numbers’.






Can UX change the world? 

UX is an empathetic and user centric approach of designing products and services for users. It is highly used in resolving complex problems associated with the usability of products and services in digital spaces. However, there are many other conflicting problems faced by the society outside this ‘Digital World’, which requires our immediate attention. Can we possibly use UX to address these complex social challenges ? Can we use UX to bring social change? Can we use ‘UX for Good’?

The answer to these questions is ‘YES’.

Design agencies like IDEO, Daylight and Smallspaces are ideating services and products for social impact using combination of experience design, design thinking and other design principles.

This article highlights the use of UX for social good through products/services that incorporated user experience design for development.

Why UX for Social Impact?

Design as a discipline is being used to counter social challenges by developing products.However, it just improves the life of people who used these product. But to initiate a social change/impact, this isn’t enough.

To create a social impact, it is important that we motivate people to perform certain ‘Actions’ which, in turn would lead to social change.

The user centric approach of designing products would help to identify how users could be motivated to perform certain actions. Thus, I believe that the process of user experience design is a perfect fit.

Experiences for Social Good


UNICEF Kid Power Band 


Mission- Giving Kids the power to save lives
Company – Daylight and UNICEF


The UNICEF Kid Application tracks the physical activity of a kid through the UNICEF Kid Power Band. After completion of a mission (physical activity), the kids earn points for their mission and this unlocks therapeutic food packets for severely malnourished children around the world.

This experience is an initiative to motivate physical activity of kids in United States and teach them the importance of being a global citizen.

It was interesting to see that the impact itself acted as a motivator for the kids to perform an ‘Action’ (physical activity).

“Badges are cool, but they don’t really help anyone. I would rather earn more food for other      kids.” –

-Anna, age 10

It seemed that they conducted user research to understand their target audience (kids) and capitalized on the analysis that kids inherit desire to help their peers. Also the application was designed to give a feel of an adventure and travel app which additionally helped the entire experience.


Genocide Museums in Rwanda, Germany


Mission- Improve the experience of visitors at Genocide Museum
Organization – UX for Good


The museum basically educates the visitors about the horrors of the 1994 Rwandan genocide against the Tutsi in the hopes of inspiring change. Nearly one million people were killed due to the unfortunate incident. However, the visitors were emotionally broken after learning the facts, artifacts and stories  about the mass atrocity .

‘UX for Good’ incorporated user experience design process where they conducted user research and determined that the experience lacked moments to reflect, and to share stories of hope. Maintaining the balance between empathy and compassion during the entire experience was the key. Incorporating the newly designed plan of the entire experience lead to visitors being motivated to do ‘act for humanity’.

Inzovu Curve


The museum underwent a digital transformation as well which helped them to stay connected with their visitors and spread words of hope for humanity. During the process, they also produced the ‘Inzovu Curve’ that helped modify the experience design. It also helped designer map the emotional impact of institutions around the world.



UX design encompasses before, during and after designing process of the product/service. This aspect of the UX process makes it very flexible to be incorporated in different forms and capacities. I think UX research specifically played a major role in designing products and services for social impact. Also, in both the projects, it was cited that the impact of the product/service on the target audience was recorded to understand whether the design served the purpose. So how can we address social challenges? I think, UX is a good place to start from.





Designing for Action, Not Reaction: UX for Good and The Kigali Genocide Memorial




Capturing social value in UX projects, Andrew Mara, Miriam Mara

Design Critique : Citymapper – Transit Navigation


Citymapper is a transit application that helps users to map their journey’s efficiently by evaluating different city transport options and suggesting the best routes available with Real-Time updates. It is helping travelers across the major cities in the world to make their transit carefree and effortless.

This mobile application is critiqued on the basis of principles and design theories put forward by Don Norman in the book “Design of Everyday Things”.

Citymapper – Main Page



Main page provides good discoverability of varied transport modes through visual illustrations acting as signifiers.

‘Get me Somewhere’ tab with the ‘Search’ symbol acts as a signifier comprehending that the destination location is to be entered.

Within ‘Places’ section, ’Home’ and ‘Work’ tabs are available for users to set their work and home locations respectively thereby reducing chances for a user to make a mistake or slip.

According to Don Norman, “Well designed systems are resilient against failure”. Citymapper offers offline maps of different transport modes that helps travelers to be anxiety free even without an internet connection.

Citymapper – Suggestion Page




This page provided different route suggestions to reach desired destination.


The ‘Now’ tab helps to set an arrival or departure time for the journey. But due to poor discoverability of the tab, functionality of the button is not conceptualized by the user. Discoverability of the ‘Now’ Tab can be improved if Natural Mapping principle by Don Norman is followed. The ‘Now” tab can be placed after the destination location is specified for better conceptualization.

The time duration for journeys are displayed in the format of minutes only. Time is standardized to be evaluated in Hours and Minutes format in most of the scenarios. Thus conversion from ‘Minutes’ format to ‘Hours and Minutes’ format could lead to a slip or mistake by user.

If only knowledge in the world (App) is used, it would be difficult for the user to comprehend that ‘A’ refers to a Subway Train. This may lead to similarity error, if not addressed appropriately. (Similarity between Platform ‘A’ of a bus and Subway Train ‘A’). It is advisable that a correct signifier same as ‘Symbol of the Subway Train on the Main Page’ is incorporated for better discoverability.

Citymapper – Selected Route Summary



Overlapping of the data on the map is cited at many occasion while transiting.

The activities of the journey namely Walking, Waiting and Riding varied city transports are displayed in different colors. This helps the user to build a better conceptual model to access the information. Even though the information is complex in nature, distinct colors helps to get rid of any confusion that may exist while accessing the information.

Citymapper – Journey 

Flash cards (Separate Modules) are used to map the journey which constraints users to concentrate on one task at a time.

No directions are provided while walking thereby forcing the user to continuously look on the screen. This could lead to a slip, if he/she is an avid user of the app or a mistake, if he/she is a new user. This issue can be resolved if proper Real – Time visual (e.g Turn Left after 20m) and sound signifiers are incorporated within the application. As said by Don Norman, “Sound is essential as visual information because sound tells us about things we can’t see , and it does while our eyes are occupied elsewhere.”



It suggests best sections of subway trains to the users by excellent use of signifiers. This also helps to build a relationship between the application and the users.

Feedback is provided once the journey is completed by the user. It also gives an account of calories burnt, trees saved and money saved after completing the journey thereby generating a visceral response within users.


Overall, Citymapper is an excellent application for commuters and explorers. It provides authentic information and estimations with little or no error. The mobile application has few minor design problems which could be resolved easily. At the end of the journey, Citymapper brings a smile to its users face with its amazing feedback feature thereby encouraging people to use city transports.