Designing information experiences takes into consideration perception (engaging with the senses), cognition (engaging with the mind), emotion (engaging with the heart), and action (engaging with the body). Other factors such as capabilities, constraints, and context also influence experience, and the process of examining these details involves a great deal of research and evaluation both initially and continually as products are tested and improved.
Design is ultimately problem solving, which means decision making coupled with creative thinking. Information architects work with interaction designers to create meaningful relationships between people and the products and services they use, and both disciplines are often referred to as both an art and a science.
As such, design research involves both quantitative and qualitative methods. When you start evaluating certain factors like the emotional elements of design, research methods become much more qualitative as you rely on things like diary studies and user interviews. There are also times when you need numbers to answer a question and quantitative methods come into play.
So how do designer researchers balance the two? It has to do with context and the type of questions that need to be answered.
Matthew Pearson is a behavioral economist at Airbnb who brings methods and insights from economics and psychology to user experience. Matthew outlines when to listen to your data versus your user, pulling from articles by data scientists, product designers, and UX researchers.
As the outside world often has a much larger effect on metrics than product changes do, controlled experiments isolate the impact of the product change while controlling for external factors. When you test a single change, the methodology is often called A/B testing and it provides a clean and simple way to make causal inference. At the same time there are pitfalls to relying too heavily on experimentation in the place of innovation, imagination, or decisive thinking in the face of uncertainty.
“Data and A/B test are valuable allies, and they help us understand and grow and optimize, but they’re not a replacement for clear-headed, strong decision-making.”
And even in the age of big data, there’s still no substitute for good qualitative research.
But ultimately, there are certain things your user can’t tell you. At World IA Day this year there was a speaker Pamela Pavliscak, a researcher and data scientist who collects stories about how people engage with technology. Based on data from a large-scale study, combining online studies, interviews, and social experiments, Pamela found that people don’t remember the design or appearance of a website or app, they remember the feeling they take away from the experience.
In his article, Pearson points to the work of Psychologist and Economics Nobel Laureate Daniel Kahneman, who distinguishes between the “experiencing self” and “remembering self” and whose theories about human irrationality and bias have major implications for qualitative product research. What makes the “experiencing self” happy is not the same as what makes the “remembering self” happy. The “remembering self” retrospectively rates an experience by the peak level of pain or pleasure in the course of the experience, and by the way the experience ends. It is this self that makes the decisions. For example a college student will decide whether or not to repeat a spring break vacation based on the peak-end rule applied to the previous vacation as opposed to evaluating the break by how it was throughout, moment to moment.
In the end, any design effort will be judged by how successfully it meets user needs and the objectives of the organization commissioning it. This means taking into consideration the complicated psychology behind how individuals interact with products and tailoring your research method to the context of your question.