Measuring and Detecting Emotions: Do We Really Know How You Feel?

Image Credit: MIT News

Think about the apps, websites, and other digital products that we encounter everyday. How do we feel when using them? Have we been pleased by some of them while annoyed by others? Are we aware of our reactions and behaviors when such feelings occur? Think of the times when we have the urge to abandon a website or uninstall an app out of frustration, and we will understand how much emotion weighs in those interactions. Centering on users affective responses to digital products, this article will provide some insights into the technologies used to detect emotions, as well as the pros and cons of such approach.


Emotional Technologies

The user experience field has been striving to find valid measurement of users’ feelings to a product, because a successful usability is necessary, yet not sufficient, to sustain continuing use if it fails to elicit positive emotions from users. Compared to a self-reported technique which could be subjective to bias and errors, using technologies that analyze physical responses is a more reliable way to objectively reflect how users really feel (Reynolds, 2017).

Understanding emotions requires understanding of human physiological metrics, including facial expressions, body language, tone of voice, etc., which could be measured and analyzed by a variety of sensing technologies. The main methods used are: motion capture systems or accelerometer sensors that tracks one’s body movements; biosensor placed on palms or fingers to measure increase in sweat; and cameras for measuring facial expressions (Preece, Sharp, & Rogers, 2015).


Facial Coding Example: Affdex

Image Credit: Affectiva

In recent years, advanced softwares and algorithms have been developed to detect facial expressions and analyze different emotions. An leading example of facial recognition or facial coding tools would be the Affdex emotion analytics and insights software from Affectiva. Captured through a webcam, users reactions to digital contents is analyzed using Affdex’s machine-learning algorithms, and classified into six categories (Preece et al., 2015). Below is a list of Affdex’s fundamental emotions and a simplified procedure of how Affdex works.

Affdex 6 Fundamental Emotions:
1) Happiness
2) Sadness
3) Disgust
4) Fear
5) Surprise
6) Anger
(Preece et al., 2015)
Affdex cloud-based face video process procedures:
1) Detect & Extract Features: 

Face Detection.
Image Credit: Affectiva

- Locate a face, and extract the key feature points on the face
- Identify 3 Regions of interest: mouth, nose and upper half of face

2) Classify Emotional States:

Analyze key regions
Image Credit: Affectiva

- Assess movements, shape and texture of the entire face at pixel level
- Take extracted facial features and classify them into emotional states
- Affdex data is received at 14 frames per second
- Once the facial features have been categorized, the resulting emotions
  are assigned numeric values for each frame

3)Assess & Report Emotion Response:

Affdex dashboard.
Image Credit: Affectiva

- Infer emotional and mental states to feed analytical models 
- The Affdex dashboard visualizes a time series curve for each emotion
  metrics that aggregates user's emotional experiences 

(Affdex, n.d. p.4-5)


Pros & Cons

Emotional Technologies provide honest feedback to designers, as it measures people’s physical responses on a non-conscious level. In addition to informing designs, further applications could also help people to self assess their emotional states. For example, an app called “Moon Phrases” was developed to analyze users’ expressions on social media (E.g. Tweets, Facebook posts) through the choice and frequency of words, phrases, hashtags, emoticons, etc. This measurement of positive or negative emotions, in turn, is visualized using a series of moon icons representing different phases (Preece et al., 2015). Reflecting on the patterns arise from the history of moons, users can be conscious of the fluctuations in their mood and perhaps be more motivated to improve their emotional well-being.

Image Credit: Semantic Scholar

On the down side, how information as such and within what context it should be used might raise considerable concerns. Question is: when technologies are capable of comprehending people’s emotion – when knowledge of one’s emotion is not only tailored towards design of an app or a website, but could also be exploited for massive digital contents such as ads, movies, news, merchants, etc, that properly match your mood (Preece et al., 2015) – will you think it is creepy or intrusive?



While basic emotions such as the ones classified by Affdex might be explicit enough to identified and measured, there are many other emotions, or perhaps a mixture of different states, that are more nuanced and thus not always recognizable. Also it could be tricky for researchers or designers to tell exactly where that affective response is coming from. For example, if the facial coding produced a result of confusion, is it due to unclear signifier as of what to do next? Or a lack of feedback that informs their past action? Another concern raised by researchers is such measurements’ capability to be generalized across different countries or ethnic groups, because the expression and perception of emotions might vary depending on different cultures. (Ko, 2017)



In addition to the technologies mentioned above, eye-tracking, speech, words/phrases (E.g. Tweets, Facebook) analysis, EG wearables (E.g. headsets that picks up electric signals from the brain) are widely applied to reveal users mental state. It is recommended to adapt a combination of methodologies for more useful insights. There is not yet a scientific way to precisely measure emotion (and I don’t think there will ever be, for they could as complex, nuanced and ephemeral as you could imagine), but these knowledge are relatively reliable to inform the design of user experience.



Affdex (n.d.). Exploring the emotion classifiers behind Affdex facial coding. Retrieved October 18, from…#

Ko, D. (2017). Measuring emotions to improve UX. Retrieved October 18, from

Preece, J., Sharp, H., & Rogers, Y. (2015). Interaction design (4th ed.). United Kingdom: John Wiley & Sons Ltd.

Reynolds, A. (2017). How to measure emotion in design. Retrieved October 18, from


Design Critique: Roomi Mobile App

Roomi is a free mobile application that offers a safer and easier way for people to find co-living solutions: apartments/rooms and roommates. Users who have an empty room to rent out or look for a suitable candidate to take over the lease publish their listing information on the market. Home seekers search, connect and apply to move into an ideal place that’s compatible with their budget, lifestyle, or other needs and exceptions.

Roomi tackles the three levels of processing in different aspects.  First of all, its adaption of a rounded, bubbly design style, less saturated color scheme and lighthearted graphics capture users’ at the visceral level, creating a positive emotion associated with the app by invoking a sense of friendliness and coziness.

As users click into the listing detail, their gaze falls up the number of views during the past week. Sometimes this number could be as high as over 1,000. Giving out this data trigger a desired reaction from users primarily on the behavioral level. See this piece of information might not be as enjoyable, because a higher number means there are more potential competitors out there who’s eyeing on the same place, This creates an unsettling feeling that have less control over something that they would expect, but in the meanwhile, also motivates them to quickly take actions and move forward with the request.

In order to connect the right person to the right place and to individuals who share similar lifestyle/interests, Roomi places an emphasis on one’s profile building. It encourages users to establish an in-app presence by adding profile picture, age, self-description, etc., and even provides a list of adjectives and questionnaires that reveals some insights about one’s personality traits and living habits. Users are also prompt to connect with social media such as Facebook and LinkedIn. In addition, the built-in messaging option is a more convenient and casual means of communication compared to other channels such as emails. These instruments essentially boil downs to the social aspect resides in the reflective level where everyone is conscious of. Roomi’s popularity does not simply come from the tangible elements seen on the surface but really heighten the value of co-living and potential for community building, especially when people just move to a new place.


Due to a large amount of inventories within certain areas, markers placed on the map often cluttered and overlap on top of each other; but the selected one will be highlighted and brought forward to acknowledge the interaction is registered. This provides immediate visual feedback to users to indicate the current button state and their position on map. However, when I returned from listing detail back to map view, I always kept forgetting which ones I already looked at, which ones I favorite, and which ones I decided not to consider, since each marker simply displayed the price. Therefore I had to perform the action of tapping in and backing out repeatedly. According to Norman, such occurrence of errors could be considered a slip, specifically a memory-lapse slip, as they are caused by memory failure with the right intention. Norman’s suggestion for combating memory-lapse slip is to reduce the number of steps and to provide distinct indicators that remind users of the next step. In my case, my recommendation would be to gray out the listings that have already been viewed, add a symbol next to price or use a different color to highlight the marker as an indication for favorite listing. Also, giving users the option to hide the options that are passed on could effectively clear up the map and cause less confusion.

After users send out their request, the “Request to Chat” button is darkened and disabled. This not only provides a feedback to users that they have reached out to this apartment, but also imposes a nice constraint that prevent users from repeatedly sending messages before the other person accepts and replies back.