Qualitative or Quantitative: When do you ask your users and when your data?

Designing information experiences takes into consideration perception (engaging with the senses), cognition (engaging with the mind), emotion (engaging with the heart), and action (engaging with the body). Other factors such as capabilities, constraints, and context also influence experience, and the process of examining these details involves a great deal of research and evaluation both initially and continually as products are tested and improved.

Design is ultimately problem solving, which means decision making coupled with creative thinking. Information architects work with interaction designers to create meaningful relationships between people and the products and services they use, and both disciplines are often referred to as both an art and a science.

As such, design research involves both quantitative and qualitative methods. When you start evaluating certain factors like the emotional elements of design, research methods become much more qualitative as you rely on things like diary studies and user interviews. There are also times when you need numbers to answer a question and quantitative methods come into play. 

So how do designer researchers balance the two? It has to do with context and the type of questions that need to be answered.

Matthew Pearson is a behavioral economist at Airbnb who brings methods and insights from economics and​ psychology t​o user experience. Matthew outlines when to listen to your data versus your user, pulling from articles by data scientists, product designers, and UX researchers.

As the outside world often has a much larger effect on metrics than product changes do, controlled experiments isolate the impact of the product change while controlling for external factors. When you test a single change, the methodology is often called A/B testing and it  provides a clean and simple way to make causal inference. At the same time there are pitfalls to relying too heavily on experimentation in the place of innovation, imagination, or decisive thinking in the face of uncertainty.

“Data and A/B test are valuable allies, and they help us understand and grow and optimize, but they’re not a replacement for clear-headed, strong decision-making.”

And even in the age of big data, there’s still no substitute for good qualitative research.

But ultimately, there are certain things your user can’t tell you. At World IA Day this year there was a speaker Pamela Pavliscak, a researcher and data scientist who collects stories about how people engage with technology. Based on data from a large-scale study, combining online studies, interviews, and social experiments, Pamela found that people don’t remember the design or appearance of a website or app, they remember the feeling they take away from the experience.

In his article, Pearson points to the work of Psychologist and Economics Nobel Laureate Daniel Kahneman, who distinguishes between the “experiencing self” and “remembering self” and whose theories about human irrationality and bias have major implications for qualitative product research. What makes the “experiencing self” happy is not the same as what makes the “remembering self” happy. The “remembering self” retrospectively rates an experience by the peak level of pain or pleasure in the course of the experience, and by the way the experience ends. It is this self that makes the decisions. For example a college student will decide whether or not to repeat a spring break vacation based on the peak-end rule applied to the previous vacation as opposed to evaluating the break by how it was throughout, moment to moment.

In the end, any design effort will be judged by how successfully it meets user needs and the objectives of the organization commissioning it. This means taking into consideration the complicated psychology behind how individuals interact with products and tailoring your research method to the context of your question.

Slippy UX

Like many people, I spend a lot of time interacting with screens and being interrupted by various devices. I can’t remember the last time I went an hour without checking my phone, and I spend more time staring at pixels than looking at people. In this part of the world, we are never not connected, and this only becomes increasingly true. The wearable technology industry—already a $3-5 billion market—is set to grow to over $40 billion within the next five years according to research from Credit Suisse, radically altering the way we interact with technology, our environment, and each other.

Social media has become so embedded in personal and business communication that it is one of the leading ways in which we engage with each other and feel connected. Everywhere you look people are communicating and “socializing” but more and more these interactions take place through new and multiplying mediations.


In a recent article UX Magazine listed their top predictions for UX trends in 2015 which included the rise of what they called “slippy” UX. Compared to the old model of “sticky” UX—experiences that are meant to pull you in, keep you interested, and bring you back for more—slippy UX is about creating experiences that operate effectively, yet invisibly. Coined by Jake Zukowski, Assistant Creative Director at Frog Design, he perfectly illustrates the need for this kind of design within the automobile industry.

“Here, UX literally finds itself in a life-or-death situation: can a digital experience in a car operate effectively, yet invisibly enough to not affect the safety of a driver and her passengers? A successful design in this concept is ‘slippy’ because it is designed for glance-ability, minimal copy, and use in potentially high-stress situations.”

This extends into the concept of the connected home—people don’t want to live in a house that is constantly interrupting their lives. The idea is designing seamless experiences to enhance not distract from analog activities—laundry, sleeping, comfort, and (gasp!) talking to each other? With the addition of wearable devices, technology that it is literally attached to you body, these things should really be enhancing our world and personal experiences, both analog and digital, rather than interfering with them.

In another recent relevant article ACM Interactions Magazine talks about designing for the “connected everyday”.

[It] is a question of how we can facilitate meaningful opportunities for communication and interaction—not just a question of how digital technologies can make our lives more effective. It is a fundamental design question about the kind of life we want to live: one where we often feel overloaded, distracted, and fractured in our social engagements, with notifications that pull us away from our lives and sometimes from the things we really want to be doing. Or the kind of life where the ability to connect across networks is commensurate with our current practices in the physical world.

The article goes on to discuss this idea of designing for commensurability. In order for digital objects to integrate into our lives, we need to make our ability to connect to people and things across networks commensurate with our practices in the physical world. We need to move beyond the artificial dichotomy of online and offline. For designers this means enabling people to connect to data and information in a more tactile way that supports connections and interactions grounded and organized around practices in the physical world, not just layered on top of them.