In 2001, Jakob Nielsen said the first rule of usability is not to listen to users. He wrote this in the context of user preferences and desires—users aren’t always the best judges of what a design should be—but there is a lot of value in analyzing user feedback in the context of usability testing. Observing user behavior and their responses to testing provides insight into design problems and potential solutions and passing on those insights to clients can be an impactful way of reinforcing research findings using qualitative data. However, as researchers, we should be careful to present qualitative data in an accurate and ethical way.
Usability and UX research is not usually subject to review by an ethics committee or institutional review board, since the research conducted isn’t usually considered official human subject “research” as defined by the U.S. Department of Health and Human Services. Data is also usually anonymized, stripping identifying markers down to the basics. When researchers begin to analyze and present findings based on the gathered data, there can be a lot of room for interpretation. This is especially true for qualitative data, where the verbal feedback a participant provides often needs to be refined from an in-the-moment reaction to an articulated response to a design or product. However, as Amaya Becvar Weddle points out in an article on UXMatters, “there is a fuzzy line beyond which a crafted quotation goes beyond being a cleaned up version of a participant’s spoken statement and becomes something that is artificial.” The journey from the user test transcript to the final product can often be like playing a game of telephone: the message can get muddled as it gets passed along.
Becvar Weddle points researchers to two reasons for choosing to include a quotation in a report or presentation: representativeness and inspiration. Does the opinion provided represent a larger trend in the data? Can the quotation be used to inspire a potential solution? Once appropriate quotations are chosen, place them in the appropriate context. Even if each participant is anonymous, basic demographic information can still be attributed to provide context for the user’s point of view and experience using the product.
Researchers can also draw inspiration from other fields were qualitative data and quotations are used in research, such as journalism or social science research. If the user were to see their words used, would they recognize them and what would their response be? Were they fully informed of how their responses would be used? A study on how participants in a social science research study reacted to their quotations in drafts of the final report found that participants were surprised at the form the report took and raised “issues about people’s levels of understanding of the research process…and whether researchers should show participants in advance the kind of report they expect to write” (Corden & Sainsbury 2006).
Finally, qualitative data shouldn’t be presented in isolation. Whenever possible, researchers should try to triangulate data. Qualitative data and quotations can be used to support other forms of data and other methods of usability testing. A project could also make use of investigator triangulation: different researchers will interpret data in different ways, leaving less room for subjectivity and more rigorous analysis of the data available.
References:
Corden, A., & Sainsbury, R. (2006). Exploring ‘Quality’: Research Participants’ Perspectives on Verbatim Quotations. International Journal of Social Research Methodology, 9(2), 97–110. https://doi-org.ezproxy.pratt.edu/10.1080/13645570600595264