Honesty in Ethical User Testing

Learning how to conduct respectable, ethical research is crucial skill for any aspiring usability expert. Ethics, defined by D. Resnik at the National Institute of Environmental Health Science, is “norms for conduct that distinguish between acceptable and unacceptable behavior. The foundation for clean ethical research are the principles of honesty and integrity”[1]. The UXPA Code of Conduct for User Experience Professionals states, “be honest with everyone, act with Integrity, do no harm and if possible provide benefits,.”[2] These principles should not only be applied to usability participants, but should also extend to the clients as well.

Why is ethical research so important?

Adhering to certain ethical guidelines promotes truthful, and unmanipulated results. In research teams, ethics promotes fairness and accountability amongst researchers, and counters against plagiarism. Ethical standards also hold researchers liable to the public, and keeps the best interest of the public in mind. Studies that maintain strong ethics are likely to be more favored by the public, and be held to a higher esteem.

Now armed with basic industry principles, its easy to identify unethical research. Lets take a look at the following example:

Facebook’s Experiment

In 2012, Facebook secretly ran a psychological experiment on over 600,000 of their users without their informed consent. Facebook modified users’ newsfeeds, showing either more positive news stories, or more negative stories. According to the published study by Kramer et al [3], the aim was to examine how emotional states can spread to others through social media. Results found that when more negative stories were shown, users were less likely to positive statuses, and vice versa. This result supported the researchers’ hypothesis that the emotions of others can influence one’s own emotions over social media.

In their newsfeeds, people are used to commercial manipulation, in which they are being shown advertisements for products based on their search history, and interests. However, in this case, Facebook intended to intentionally manipulate users’ emotions without their knowledge, how scary is that! Even scarier, this unethical study seemed to receive almost no backlash from the community. The stock of Facebook didn’t fall after this issue became public, and Facebook is still arguably the most widely used social media platform.

Looking at this case, ethically-speaking, what did Facebook do wrong?

  • Didn’t get informed consent from participants
  • Didn’t act in the best interest of everyone- they may have actually harmed users by increasing negative emotions, such as depression and anxiety!
  • Dishonest, didn’t let users know they were part of an experiment until their study was published.

Ethical Research

Image cred: http://www.sec-ed.co.uk/best-practice/how-honest-are-you-and-your-school/

Now that we know what NOT to do, lets identify how key aspects of ethical research. As a user researcher, it is your duty to protect your participant’s physical and mental wellbeing, anonymity, and to represent their opinions honestly.

Be Honest with your participants… 

Trust is easy to lose, and very difficult to gain back. Therefore, it is not worth the risk to put yourself in a situation where you might be caught in a lie. It is better to be transparent with both participants and the client throughout the testing process.

From the beginning of your user test, it is important to be open and honest with the participants. It is a good idea to begin the user test with some background information on why you are running this research, who your client is, how you are going to use the result from this test, and what tasks will be asked of the participant. Disclosing this information will allow for participants to give their informed consent to be a part of this research. Also, if you will be videotaping or voice recording the session, it is ethically important to obtain their consent. Another benefit of disclosing all relevant information to the participant is that the more well-informed the participant is, the more comfortable and confident they will be, yielding in more authentic results.

However, at times, it might be important to withhold certain info

rmation if it would create a bias within the participant. If it is known that a participant has had a previous negative experience with your client’s company, not disclosing the name of the company immediately would be beneficial so that the participant doesn’t draw upon that negative experience when testing. As soon as you obtain your results, it is an ethical decision to then inform your participants about the undisclosed information.

Another practice of honesty in user testing is to introduce each person that will be involved in the test truthfully. If someone in the room works from the company of the product they’re testing, do not disguise them as a member of your usability team, be truthful about their background. If you fear that the identification of the company member may cause bias for test, get permission from participant to wait until test is completed to introduce everyone’s role.

Truthfully representing your participants…

After you have gathered all of your user test results, it is now your duty as a researcher to represent your participants as honestly as possible. At this stage it is important to be educated about certain biases in order to reduce their effects that may tamper with your data. The Confirmation Bias is one such bias that can affect the truthful representation of your participant data. According the confirmation bias, researchers may subconsciously cherry-pick certain test results to fit their hypothesis. In doing so, the majority of participants’ opinions may be tossed aside to make way for data that does support the researcher’s hypothesis.

Wait! What about being honest with your client?

Be truthful to your clients, never try to mislead or lie to your employer! Don’t try to exaggerate the results because you think thats what a client wants. For example, there should be an appropriate relationship between the the sample size and the resources that was provided to you. Usability experts aren’t miracle workers, if there is a low budget, the likelihood of a vast sample size, and detailed user testing results is low. Be truthful to your clients, if they want more data, they will need to provide further time and resources.

In order for your user research to be respected and yield accurate results, it is crucial to follow ethical guidelines at all stages of your user testing. Honesty is the best policy. It is better to be honest with all individuals, including participants and clients, and fully disclose all relevant information to them, or else there is a possibility for trust to be broken. If the trust is broken, the validity of your research will be put into question.

 

References

[1]  What is Ethics in Research & Why is it Important? (n.d.). Retrieved March 07, 2018, from https://www.niehs.nih.gov/research/resources/bioethics/whatis/index.cfm?links=false

[2]  UXPA Code of Professional Conduct. (n.d.). Retrieved March 07, 2018, from http://uxpa.org/resources/uxpa-code-professional-conduct

[3] Schekman, R. (2010). Editorial expression of concern and correction. PNAS, 107(15), 7113.

[3] Facebooks Newsfeed Study: Was It Ethical Or A Violation Of Privacy? (2014, July 01). Retrieved March 07, 2018, from https://www.npr.org/2014/07/01/327248369/facebooks-newsfeed-study-was-it-ethical-or-a-violation-of-privacy

[4] Suri, J. F. (2015). The little book of design research ethics. Place of publication not identified: IDEO.

[5] Mortensen, D. (n.d.). Conducting Ethical User Research. Retrieved April 03, 2018, from https://www.interaction-design.org/literature/article/conducting-ethical-user-research

Design Critique: MTA eTix (Android app)

Launched in the summer of 2016, MTA eTix is a free mobile app that allows users to purchase tickets for the Long Island Rail Road and Metro-North directly through their own mobile device. This app aims at eliminating unfortunate events that commuters struggle with on a normal basis, such as losing tickets or missing trains due to long lines at the ticket machines.

In this critique, the Android version of MTA eTix will be analyzed using Don Norman’s design principles discussed in his book The Design Of Everyday Things. 

Finding Train Times

MTA eTix was designed solely for the purpose of purchasing tickets.This is signified by the poor mapping between the Train Time button, located on the homepage, and information on train schedules.

Once clicked, users expect to be taken to a page of app that displays train time information. However, that is not the case as the user is immediately redirected to the Google Play Store and directed to download other apps created for the MTA, such as LIRR Train-Time, or Metro-North Train Time. Only users that have downloaded the specific train app beforehand are afforded to view upcoming train times through the secondary app.

Clicking on ‘Train Time’ button sends user to Google Play Store

Recommendation:

This poor design can be frustrating to users in a hurry who expect to be able to immediately view upcoming train schedules. The need to download, or be redirected, to a secondary app when clicking that particular button may be seen as deceitful mapping from a user’s perspective.

Forcing the user to open and use a secondary app to view crucial information can provide for a negative experience. Therefore, it is recommended, if possible, to merge all apps into one, so trains times can be viewed by all users immediately, and allow for better mapping between the Train Time button and the information.

Streamlining the Purchasing Process

The process of  purchasing tickets is made quick and easy by providing the user with a restrictive interface that guides the user with consistent signifiers, such as CTA buttons and arrows. Actions of the user are limited, and they are directed to follow a predetermined path. For example, the user is unable to select a destination without selecting their origin station first. Similarly, the user is constrained to picking a ticket type before being allowed to chose Off peak or Peak.

Users can star/favorite stations they travel to often, thus skipping the majority of the ticket purchasing process, which reduces the chance to make any mistakes/slips.

On the Payment page, the Accept & Pay button is grayed out with a lock icon, which signifies that it can not be pressed until the user enters the necessary information.

The ticket purchasing process

 

Transaction & Ticket Activation Confirmation 

The MTA eTix app provides multiple positive examples of feedback.

After the user enters their credit card information and clicks the Accept & Pay button, not only is the user redirected to a confirmation page, but also a transaction receipt is immediately sent to their email. Thanks to these to forms of feedback, the user is not left doubting if the payment went through.

Confirmation page on MTA eTIX app & transaction receipt sent to email

 

Now that the train is approaching, it is time for the user to activate their newly purchased ticket located in their Ticket Wallet. In the Ticket Wallet, once activated, the online ticket now has a bright green accent color to signify that it is active. On the ticket page itself, it now has a colorful barcode with the current time and date at the top, as well as the time the ticket was activated. The color alterations supply the user with immediate visual feedback that their ticket was indeed activated successfully.

Before and After activation in Ticket Wallet & on Ticket page

 

Supplemental Feedback Recommendation:

The app already provides exceptional feedback, however I would recommend adding one additional feature that would not only give feedback, but it would also provide the user with clear visibility of their ticket and travel information.

Although the discoverability and mapping of the process of opening an active ticket is clear, it is a time consuming process.

I would recommend that once a ticket is activated, the app pushes a notification that contains your travel information to your phone’s notification bar, and if clicked on takes your directly to the activated ticket. This feature would provide the user with immediate feedback that the ticket was activated, and also provides the user with a shortcut to a tedious process.

Mockup of proposed notification feature

Conclusion:

The MTA eTix app is widely used in the NYC area due to its convenience. The app’s restrictive interface, high discoverability, and consistency makes it easy to use. Still being relatively new, available for slightly less than 2 years, there are still few details that can be improved upon to increase the overall experience for the user.