Users interest are relentlessly changing all the time and their attention span are as short as they have ever been. Given this companies are constantly thinking of new ways to maintain engagement with their users. Sometimes that comes in the form of change with their product and how the user interacts with them. It’s understood that with every potential change, whether major or minor, testing should always be involved. Not only to ensure it works works from a technical perspective but also ensure it achieves the desired goal or effect. One effective way a lot of companies are using to ensure proper testing is done is with a method called A/B testing.
According to Optipedia, A/B testing is an experiment where two or more variants of a user interface are shown to users at random, and statistical analysis is used to determine which variation performs better for a given conversion goal. [Optipedia]
Figure 1 – With A/B testing there are two (or more) variants. Variants A (control) and a variant B (variation) are nearly identical user interfaces with the exception of one variation. This variation can be as small as the labeling of a checkout button from “shop now” to “buy now” or as big as the manipulation of a feature functionality.
It’s easy to see how effective A/B testing can be with its controlled test and comparative analytical data however the general practice used by companies raise some ethical questions. I honestly was surprised to find out how often companies I use on a regular basis, Google, Amazon, Instagram, perform A/B testing on its products. They are performed all the time and most of the time the users are unaware of it. User agreements aside, this ongoing testing is essentially done without the users explicit agreement to participate and the lack understanding on the goals and objectives of companies/test and what potential impact the testing could have on the user is troubling.
For instance, in March 2014, Facebook conducted an emotional response test that included close to a million of its users without their knowledge. During this emotional contagion experiment, the social platform gave some the test users a positive newsfeed while others received a negative newsfeed. The study, published in collaboration with researchers from Cornell University, found that users with negative newsfeeds posted more negative words, and those with positive newsfeeds posted more positive words. While the results are not surprising, the duplicitous nature in which they performed their experiment and the possible emotional impact it could have on its user is alarming and unethical. You can find more details on 10 other user experiments conducted by Facebook during 2014 on at Forbes.
In 2012, due to the growing toxic behavior in their community, Riot games tested a new chat feature in their immensely popular multiplayer online battle area (MOBA) game, League of Legends. Their objective wasn’t to eliminate toxic behavior from the games, as that would be near impossible, but to protect players from the harm of the disruptive behaviors from some members. In many online multiplayer games, you are able to chat (text or voice) as you play the game: with only people on your team, party chat or with everyone in the game, cross team chat. This testing, without the players knowledge, consisted of a variant with the cross team chat functionality disabled by default. The feature was still available the game but the players had to manually turn it on via the menu settings. After analyzing the results from the test, they found a 30 percent swing from negative messages to positive ones even while actual chat activity remained the same. You can find more details on this test can be found in an archived video on Game Developer Conference’s website.
Even though the goals and hypothesis presented for these experiemnts were possibly met, they both suffer from own ethical issues such as confidentiality, transparency, or user consent. While not all A/B testing is conducted this way it’s not hard to question the ethics in conducting a test that intentionally presents a frustrating, stressful, and emotionally draining experience is a good practice or ultimately helpful.
Some might argue that testing both “good” and “bad” experiences are essential to completely understand a user’s overall experience and usability research methods, such as A/B testing, tracks along the same ethical lines as a normal product launch. But I’d assert that companies should be more transparent and upfront to their users about their testing and the intentional exposure to a negative and potential harmful experience does little to advance the product or industry and instead halts any progress to the detriment of the users.
References
Optipedia, A/B Testing. Available online: https://www.optimizely.com/optimization-glossary/ab-testing/ [Accessed on March 18, 2019]
American Marketing Association (AMA), Are A/B Tests Ethical?. Available online: https://www.ama.org/marketing-news/are-a-b-tests-ethical/ [Accessed on March 18, 2019]
Forbes, 10 Other Facebook Experiments On Users, Rated On A Highly-Scientific WTF Scale. Available online: https://www.forbes.com/sites/kashmirhill/2014/07/10/facebook-experiments-on-users/#13f0685a1c3d [Accessed on March 15, 2019]
Game Developer Conference (GDC), The Science Behind Shaping Player Behavior in Online Games. Available online: https://gdcvault.com/play/1017940/The-Science-Behind-Shaping-Player [Accessed on March 15, 2019]