How to automate customers’ feelings.
Marketers want to read consumers’ minds, but they are not psychics. They do the following
They combine psychology with technology to read consumers’ thoughts by analyzing their facial expressions.
Emotion recognition is a relatively new technology that is at the intersection of AI and machine learning, the camera on your mobile device or desktop computer and the software needed to understand whether you are smiling or frowning. While the software encodes what the camera sees, facial expressions must be matched against a database of millions of examples. This will allow the user’s facial expression to be correlated with known patterns of emotion expressed by the face and then evaluated for measurement purposes.
Marketers can use this information to figure out how to make their product more interesting and appealing – at least, that’s how it should work.
According to Max Kalehoff, vice president of marketing for attention and emotion measurement company Realeyes, facial recognition and facial coding are sometimes mixed in the mainstream press. “Facial recognition is software for identification or identity verification, while facial coding is the detection of emotion from facial signals. Facial coding has nothing to do with identity identification.”
This tool has a lot of potential and many pitfalls.
FCERtain?
Perhaps a more accurate name would be “face coding for emotion recognition (FCER),” said Seth Grimes, founder of Alta Plana, an information technology analytics and consulting firm and an expert in sentiment analysis. The technique attempts to classify expressions and changes in expressions and to derive emotional recognition from facial recognition, he explained. However, Grimes also noted an important caveat: emotion recognition also depends on context.
Grimes gave an example: a picture of the late basketball star Kobe Bryant smiling. “Someone looking at it might be sad,” he said. However, a smile is associated with happiness. This example shows that some associations can be problematic, he noted. Looking at the association between facial expression and emotion may not be enough if text and speech are not also considered, he said.
“Recognition of emotion by face is now controversial,” Grimes continued. The technology is based on the work of psychologist Paul Ekman. Ekman once postulated that there are six universal emotions: fear, anger, joy, sadness, disgust and surprise. (Ekman himself is skeptical of its commercial application).
Critics point out that this premise may not be universal for all cultures. “It is perfectly feasible to try to use machine (learning) to model emotional expression,” Grimes says. The model can account for differences in emotional meaning across cultures. Not all smiles are alike. “You have to make sure the model is as free of bias as possible,” Grimes said. For example, “if you’re selling to the elderly, you shouldn’t include children in the model.” Who builds the model is also a factor. “White male engineers may not be aware of diversity issues.”
Clashing with reality
Still, one way to overcome the limitations of the model is to expand it by collecting more data. In the case of Realeyes, that meant bringing in psychologists and annotators to train the software to recognize different human emotions, Kalehoff explained. “Our investment in emotion and attention AI now includes nearly 700 million AI tags, nearly six million video measurement sessions, and 17 patents. Our historical archive now includes 30,000 video ads.”
The company’s approach takes into account that smiling is not the same in every culture, so bringing in psychologists and annotators in other countries helps deepen and expand the database while mitigating bias. “For example, ‘Emotions in (some) cultures are more subtle, so we have to make sure the annotations reflect those nuances.'” said Kalehoff.
The whole purpose of Realeyes is to measure human attention and emotion as factors in improving digital video content. If a promotional video isn’t working, marketers will know where to “tweak” the visuals to improve viewer attention.
Realeyes assesses users by emotional states (happiness, surprise, confusion, disdain, disgust, sympathy, fear) as well as engagement, negativity and valence. Facial recognition is not thwarted by beards or glasses.
“In our ad measurement products, reading and interpreting emotional states began as a better diagnosis of individual video creatives by trained creative and media analysts.” recalls Kalehoff. “Over time, we learned how to synthesize raw measurement data that can be automatically processed to predict real-world outcomes such as market attention, video completion and even brand favorability.”
It’s not a smile, it’s a “facial action block.”
Noldus Information Technology is one step further from the customer. “We give them (UX designers) the tools to understand the user’s mental state and experience,” says company founder and certified biologist Lucas Noldus. UX designers can then apply that data to create more effective online commerce sites.” Noldus’ two related products are FaceReader, which can be used in a lab setting, and FaceReader Online, which can be used to test subjects anywhere.
A typical approach to analyzing facial expressions is to “follow Eckman.” Researchers test smiles and frowns on faces to match the “big six” emotions (fear, anger, joy, sadness, disgust and surprise), Noldus explained. But these six basic expressions are “too crude” and insufficient to describe human expression, he explained. To express their emotional state, humans use a specific combination of 43 facial muscles. Each individual action is a “unit of facial action” that offers a more detailed method of measuring emotional state, Noldus explained. It allows for measure factors such as confusion, distraction, attention and boredom — emotional states that can affect the online shopping experience.
“We humans have evolved into facial expressions to communicate with each other,” Noldus said. When two people are together, smiles and frowns matter, whereas people make different faces when interacting with inanimate objects. Here, the AI must distinguish between the frown a person makes when surfing a Web site, which means focus, and the same frown when talking to another person, which may express anger or disgust.
The units of facial action can provide more insight here. “Confusion is important when designing interactive systems. We want the person to find what they’re looking for,” Noldus says. Of course, in the customer journey, “confusion is a negative factor.”
Or take an online game. “You want people to be surprised or shocked or angry or show fear,” Noldus says. “That’s what makes a game interesting. But that wouldn’t be a good design feature for an online banking site aimed at seniors. “You want to convey trust, control and ease of use,” he said, so in this case, boredom is a good thing.
A frown is suspect
As with any new technology, abuse and misuse cases are possible, and ethical boundaries are still being defined. Data on emotional states can be misused. Grimes proposed a hypothetical situation: What if an on-board camera mounted on the dashboard of a car reads the “anger” on the face of a driver stuck in traffic? Would his insurance rate go up?
Noldus draws a different line – he will not allow his company’s products to be used in public places where users have not given their consent to be monitored. This is different from laboratory use, where subjects knowingly consent to being observed as part of a project. Grimes quoted Google‘s old slogan as a guide: “Don’t be mean.”
