Quantitative Usability in the Corporate World: Data-Driven Design


This review briefly explores how quantitative and qualitative usability methods are leveraged by product managers as data-driven design to develop and deliver memorable user experiences.

Quantitative versus Qualitative

What is quantitative usability? According to the Nielsen Norman Group, it’s important to note the difference between quantitative and qualitative usability and what is required to carry out each. Specifically, the two methods are complementary to each other and serve different purposes (Budiu, NNG, 2017). Budiu states that quantitative or “qual” consists of “one or more metrics (such as task completion rates or task times) that reflect whether the tasks were easy to perform” (Budiu, NNG, 2017). Qualitative or “qual” consists of “observational findings that identify design features easy or hard to use” (Budiu, NNG, 2017).

While Budiu distinguishes the two with clear inputs, outputs, and use cases, as mentioned, there are there opportunities to conduct a study which leverages both. In doing so provides robust insights not just for UX/UI specialists but also for other data-driven specialists whose career is centered on leveraging data for how users interact with their product and reiterating based on those insights. One notable professional is the product manager.

Data Analysis Infographic

Behold, The Product Manager!

According to O’Reilly Design, as a product manager, “much of your job comes down to translating between the needs, perspectives, and skill sets of your stakeholders and customers” (O’Reilly Design, 2017). This role unifies all parts of the organization to conceptualize products according to business needs, competitive analysis, and user data; they must engage in cross-function collaboration to unify the work of UX/UI researchers and designers, data analysts and scientists, developers, and company executives. This exciting role allows individuals to wear many hats and apply many skills, however, with that comes a price of requiring a strong emotional state to not only manage the work but colleagues’ varying skillsets, expectations, and feedback (O’Reilly, 2017). According to the Harvard Business Review, some key characteristics of a successful product manager are:

  • *Team player*
  • Emotional intelligence
  • Agile project management
  • UI/UX concepts
  • Design abilities
  • Technical knowledge
  • Analytic abilities
  • Strong writing skills
  • Cross-functional business acumen

Despite the level of intensity, the product manager is truly an admirable individual. A key characteristic of this role is that they unify data and design–a rather difficult yet fruitful marriage of often conflicting ideologies. Product managers collaborate with the intent to design products and experiences with data in mind. This practice acknowledges that “data capture, management, and analysis is the best way to bridge between design, user experience, and business relevance” (King, Churchill, & Tan, O’Reilly, 2017). King, Churchill, & Tan acknowledge that “that design intent and evaluation are often poorly matched to the data capture and analysis because designers with a desire to understand user experience have not been in effective dialogue with data scientists and machine-learning experts” (Designing with Data, 2017).

Product Design

The product manager’s role and responsibilities will differ across industries and organization size. Product managers at smaller organizations may find themselves leveraging more out-of-the-box technologies for less complex workflows, but with less team support. Wearing of many hats will go beyond the general expectations of the role and expand to satisfying the requirements of other roles due to the lack of team support. However, corporate product managers, depending on the industry, may use an array of tools for a more complex workflow consisting of many team members. Either way, the product manager’s toolbox will generally consist of the following:

Design/Code Version Control

  • GitHub
  • Sketch
  • Visio


  • Heap/Segment/Google Analytics
  • Optimizely/VWO/Hotjar
  • Typeform/SurveyMonkey

Project/Product/Feature Management

  • Invision
  • JIRA/Confluence/Aha!
  • Mural/Trello/Asana/Monday


  • SQL
  • Tableau/Looker/Airtable
  • Heroku/Rails/Frontend/Backend

Many of the above tools will support the product manager’s workflow in addition to the analysis they conduct to ensure new products, redesigns, or iterations are successful. The analysis a product manager conducts could range depending on the features or products they support. Sample metrics or key performance indicators (KPIs) to guide a product manager’s strategy are:

  • Testing the success of marketing messages, email/web/platform features.
  • Assessing the performance of a marketing campaign and which high-performing campaign should receive funding.
  • Constructing synthetic versus treated groups to predict the interest and identify core impact.
  • Measuring unengaged users or ‘drop-off’ usage of specific user types over time (predictive/time windows).


Design Critique: Sephora Skincare & Beauty App


Sephora is a multinational online and in-store makeup, skincare, fragrance, and beauty services, retailer. Upon opening the Sephora mobile application, a user is presented with options to either view latest skincare and beauty deals, to begin shopping or exploring products, to receive product recommendations, to search for specific products, or to navigate for additional features.


This evaluation specifically reviews Sephora’s navigational design, or user experience, as the user interface design is simple and sufficient.

Good Design: Good Signifiers and Cues

Sephora’s mobile app contains just the right number of signifiers for how an action should be taken without overwhelming users. There are additional signifiers for in-app notifications and confirmations such as items added to the shopping cart:

Much of the user’s interaction is as simple as a “tap” or “swipe”, with excellent spatial cues for controls directly mounted on the items they are designated to manipulate. However, “tap” and “swipe” do not possess explicit affordances other than the mobile phone’s touch screen interface. In contrast, additional signifiers, but with explicit affordances, include the “drop down”, “roll-up”, or “menu” icons.

Within the main “Shop” view, a user relies on perceivable cues to scroll through the below list layout and tap to select a product category—all in the absence of a scroll bar and font suggesting the presence of a hyperlink.

Bad Design: Poor Discoverability

Due to poor mapping within the Sephora’s mobile app, a user has difficulty deciding between which version of “Home” or “Shop” will take you to the preferred view or results. Similarly, “New for You” items, which are recommended new product arrivals, are also available via “Inbox”. These numerous or misguided pathways lead to plenty of user slips.

The user’s intention or goal is simple: to begin shopping or to see the main view. The plan to get there is fraught with poor and duplicate pathways—an onus on the designer and not the user.


Recommendations: Build the User’s Experience from the User’s Activities

Sephora UX/UI researchers and designers should consider applying the human-centered design (HCD) process to acknowledge and articulate the problem. Additionally, they should employ design thinking tools, such as the double-diamond diverge-converge model, to prototype and reiterate designs based on that initial problem and user feedback.

More specifically, and following Norman’s concept of design thinking, I would suggest Sephora UX/UI researchers and designers incorporate the following in their next design iteration/mobile app update:

  • Observe actions: Sephora design researchers should observe the user’s actions in real-time to identify which user actions are associated with specific goals. They’ll gain a better sense of how to simplify mapping so that a specific action is enabled through a clear pathway to an expected end goal.
  • Ideate redesigns: Sephora designers should ideate on possible designs to improve mapping. Specifically, I would suggest they start with working to consolidate the main and menu views by:
    • Remove “Home” and “Inbox” categories from the main menu and consolidate into main view equivalent “Home” and “New for You”.
    • Remove “drop-down” icon from the menu next to sign-in details as subcategories, “My Community”, and the “sign-out” option are also available from the main menu.
    • However, keep the welcome bar at the top of the main menu displaying a user’s profile picture, name, Sephora points, and status.
  • Prototype versions: Sephora designers should work alongside the design researchers to attribute a design experience that matches the user actions and scenarios. Additionally, the removal of features and signifiers should either be replaced or redirected accordingly. It doesn’t hurt to also add a redesign of those areas to accommodate new space.
  • Test prototypes: Sephora designers should work alongside the design researchers to test the prototype as the right problem might have been identified but the wrong solution attributed. This might include running tests before and after prototype to compare key performance indicators such as the length of time it took for a user to complete a specific or set of actions within the app.
  • Iterate on design and performance: Sephora designers should work alongside the design researchers to continuously explore potential impediments or design improvements.