3
Take a look at the woman in figure 3-1, who is screaming in terror. Most people who were born and raised in a Western culture can effortlessly see this emotion in her face, even with no other context in the photograph.
Except . . . she isn’t feeling terror. This photograph actually shows Serena Williams immediately after she beat her sister Venus in the 2008 U.S. Open tennis finals. Turn to page 310 (appendix C) to see the full photograph. In context, the facial configuration takes on new meaning.1
If Williams’s face subtly transformed before your eyes once you knew the context, you are not alone. This is a common experience. How did your brain accomplish this shift? The first emotion word I used, “terror,” caused your brain to simulate past facial configurations that you have seen of people feeling fear. You were almost certainly not aware of these simulations, but they shaped your perception of Williams’s face. When I explained the photo’s context—winning a crucial tennis match—your brain applied its conceptual knowledge of tennis and winning to simulate facial configurations that you’ve seen of people experiencing exultation. These simulations again influenced how you perceived Williams’s face. In each case, your emotion concepts helped you make meaning from the image.2
In real life, we usually encounter faces in context, attached to bodies and associated with voices, smells, and other surrounding details. These details cue your brain to use particular concepts to simulate and construct your perception of emotion. That’s why, in the full photo of Serena Williams, you perceive triumph, not terror. In fact, you depend on emotion concepts each time you experience another person as emotional. Knowledge of the concept “Sadness” is required to see a pout as sadness, knowledge of “Fear” to see widened eyes as fearful, and so on.3
According to the classical view, you shouldn’t need concepts to perceive emotion, because emotions are supposed to have universal fingerprints that everyone around the world can recognize from birth. You’re about to learn otherwise. By applying the theory of constructed emotion, combined with a little reverse engineering, you’ll see that concepts are a key ingredient for perceiving emotions. We’ll begin with the best experimental technique for demonstrating that certain emotions are universal: the basic emotion method used by Silvan Tomkins, Carroll Izard, and Paul Ekman (chapter 1). Then we’ll systematically reduce the amount of emotion concept knowledge available to our test subjects. If their emotion perception becomes more and more impaired, then we’ve revealed that concepts are a critical ingredient to constructing emotion perceptions. We’ll also learn how emotions can appear to be universally recognized under certain conditions, opening the door to a new, better understanding of how emotions are made.4
The basic emotion method, you may recall, was designed to study “emotion recognition.” On each trial of an experiment, a test subject views the photograph of a face, carefully posed by a trained actor, to represent the so-called expressions of certain emotions: smiling for happiness, scowling for anger, pouting for sadness, and so on. Accompanying the photo is a small set of English emotion words, depicted in figure 3-2, and the subject chooses the word that best matches the face. The same words appear trial after trial. In another version of the basic emotion method, a test subject selects the best of two or three photos to match a brief story or descriptive phrase, such as “Her mother died, and she feels very sad.”
Test subjects from all around the world (Germany, France, Italy, United Kingdom, Scotland, Switzerland, Sweden, Greece, Estonia, Argentina, Brazil, and Chile) choose the expected word or face about 85 percent of the time on average. In cultures that are less like the United States, such as Japan, Malaysia, Ethiopia, China, Sumatra, and Turkey, subjects match faces and words slightly less well, responding as expected about 72 percent of the time. Hundreds of scientific studies have used these findings to conclude that facial expressions are universally recognized and therefore universally produced, even by people in faraway cultures that had little contact with Western civilization. Ultimately, these emotion “recognition” findings have been so well replicated over the last several decades that universal emotions seem to qualify as one of those rare bulletproof scientific facts, like the law of gravity.5
The thing is, universal laws have this annoying habit of losing their universality. Newton’s law of universal gravitation was only universal until the theory of relativity showed that it wasn’t.
Watch what happens when we change the basic emotion method very slightly. Simply remove the list of emotion words. Test subjects must now freely label the same posed photographs from the dozens (or even hundreds) of emotion words that they know, as depicted in figure 3-3, instead of choosing a response from a short list of possibilities, as depicted in figure 3-2. When we do this, the subjects’ success rate plummets. In one of the first free labeling studies ever conducted, subjects named the faces with the expected emotion words (or synonyms) only 58 percent of the time, and in subsequent studies the results were even lower. In fact, if you ask a more neutral question without referring to emotion at all—“What word best describes what’s going on inside this person?”—the performance is even worse.6
Why does such a small change make such a large difference? Because the short list of emotion words in the basic emotion method—a technique called a forced choice—is an unintentional cheat sheet for the test subjects. The words not only limit the available choices but also prompt the subjects to simulate facial configurations for the corresponding emotion concepts, preparing them to see certain emotions and not others. This process is called priming. When you first looked at Serena Williams’s face, I primed you in a similar way by telling you the woman was “screaming in terror.” Your simulation influenced how you categorized the sensory input from her face to see a meaningful expression. Likewise, test subjects who see a list of emotion words are primed with (i.e., they simulate) the corresponding emotion concepts to categorize the posed faces they see. Your knowledge of concepts is a key ingredient for experiencing other people as emotional, and emotion words invoke this ingredient. And they could be largely responsible for producing what looks like universal emotion perception in the hundreds of studies that use the basic emotion method.7
Free labeling reduced the ingredient of concept knowledge, but only somewhat. In my own lab, we went a step further and removed all emotion words, printed or spoken. If the theory of constructed emotion is correct, then this small change should impair emotion perception even more. On each trial of an experiment, we presented subjects with two wordless photographs side by side (figure 3-4) and asked, “Do these people feel the same emotion?” The expected answer was merely yes or no. The results of this face-matching task were telling: subjects identified the expected matches only 42 percent of the time.8
Next, our team reduced the ingredients even further. We actively interfered with our test subjects’ access to their own emotion concepts, using a simple experimental technique. We had them repeat an emotion word like “anger” over and over. Eventually, the word becomes just a sound to the subject (“ang-gurr”) that’s mentally disconnected from its meaning. This technique has the same effect as creating a temporary brain lesion, but it’s completely safe and lasts less than one second. Then we immediately showed subjects two wordless faces side by side as before. Their performance dropped to a dismal 36 percent: nearly two-thirds of their yes/no decisions were incorrect!9
We also tested subjects with permanent brain lesions who suffer from a neurodegenerative illness called semantic dementia. These patients have trouble remembering words and concepts, including those for emotion. We gave them thirty-six photographs: six actors each posing six different basic emotion facial configurations (smiles depicting happiness, pouts depicting sadness, scowls depicting anger, wide-eyed gasping depicting fear, nose-wrinkling depicting disgust, and neutral). The patients then sorted the photos into piles in any way that was meaningful to them. They were unable to group all scowling faces into an anger pile, all pouting faces into a sadness pile, and so on. Instead, the patients produced only positive, negative, and neutral piles, an arrangement that merely reflects pleasant versus unpleasant feeling. We now had solid evidence that emotion concepts are necessary for seeing emotion in faces.10
Our findings are reinforced by research on young children and infants, whose emotion concepts aren’t fully developed yet. A series of experiments by psychologists James A. Russell and Sherri C. Widen showed that two- and three-year-old children, when shown basic emotion facial configurations, are not able to freely label them until they possess clearly differentiated concepts for “Anger,” “Sadness,” “Fear,” and so on. Such young children use words like “sad,” “mad,” and “scared” interchangeably, like adults who exhibit low emotional granularity. It’s not an issue of understanding the emotion words; even when these kids learn the meanings, they struggle to match up two pouting faces, whereas they find it easy to match a pouting face to the word “sad.” Results for infants are similarly telling. Infants who are four to eight months old, for example, can distinguish smiling faces from scowling faces. This ability, however, turned out not to be related to emotion per se. In those experiments, the posed faces for happiness showed teeth while those for anger did not, and that’s the cue that infants picked up on.11
From this sequence of experiments—removing the list of emotion words, then using wordless photographs, then temporarily disabling emotion concepts, then testing lesion patients who can no longer process emotion concepts, and finally testing infants who don’t yet possess clearly defined emotion concepts—a theme emerges. As emotion concepts become more remote, people do worse and worse at recognizing the emotions that the posed stereotypes are supposedly displaying. This progression is strong evidence that people see an emotion in a face only if they possess the corresponding emotion concept, because they require that knowledge to construct perceptions in the moment.12
To really see the power of emotion concepts, my lab visited a remote culture in Africa with little or no knowledge of Western practices and norms. With the fast pace of globalization, very few such isolated cultures exist anymore. My doctoral student Maria Gendron traveled to Namibia, Africa, to study emotion perception in a tribe known as the Himba, along with the cognitive psychologist Debi Roberson. Visiting the Himba was no simple task. Maria and Debi flew to South Africa and then drove for about twelve hours to their base camp in Opuwo, northern Namibia. From there, Debi, Maria, and their translator traveled many hours to reach individual villages near the Angola border, following tracks through the bush in an all-terrain vehicle, using the mountains and sun as landmarks. At night, they slept in a tent mounted on top of the car to avoid snakes and scorpions, which were numerous. I unfortunately could not join them, so they were equipped with a satellite phone and a generator so we could speak whenever a signal was available.13
Life among the Himba is decidedly non-Western. The people live mainly outdoors and in communal compounds made from saplings, mud, and dung. The men tend cattle day and night, while the women prepare food and care for the children. The children tend goats near the compound. The Himba speak a dialect of Otji-Herero, and they use no written language.
The Himba’s reaction to the research team was fairly low-key. The children were curious and would hang around in the early morning before their chores. Some of the women were initially unsure if Maria was female since she was wearing (from their perspective) boyish clothing, which led to some finger pointing and laughter. The men must have figured it out, however, because at one point, one proposed marriage. Maria’s Namibian translator took the simple approach by explaining politely, in Otji-Herero, that Maria was “already married to another man with a very big gun.”
Maria used the face-sorting experiment with the thirty-six posed faces. It doesn’t depend on words at all, let alone emotion words, so it worked nicely across the language and culture barriers. We’d created a set of photos using dark-skinned actors, because our originals featured Western faces that didn’t look like Himba tribespeople. Our Himba subjects understood the task immediately, as we had hoped, and were able to sort the faces spontaneously by actor. When asked to sort the faces by emotion, the Himba clearly diverged from Westerners. They placed all the smiling faces into a single pile, and most of the wide-eyed faces into a second pile, but then made many different piles with mixtures of the remaining faces. If emotion perception is universal, then the Himba subjects should have sorted the photographs into six piles. When we asked our Himba subjects to freely label their piles, smiling faces were not “happy” (ohange) but “laughing” (ondjora). Wide-eyed faces were not “fearful” (okutira) but “looking” (tarera). In other words, the Himba participants categorized facial movements as behaviors rather than inferring mental states or feelings. Overall, our Himba subjects showed no evidence of universal emotion perception. And since we omitted all reference to English emotion concepts in our experiments, those concepts are a prime suspect for why the basic emotion method appears to give evidence of universality.14
Figure 3-5: Maria Gendron (right) working with a Himba subject in Namibia, beneath a tent attached to Maria’s truck
There was still one mystery remaining, however: another group of researchers, led by psychologist Disa A. Sauter, had visited the Himba a few years earlier and reported evidence of universal emotion “recognition.” Sauter and her colleagues brought the basic emotion method to the Himba using vocal sounds (laughs, grunts, snorts, sighs, etc.) instead of photos of posed faces. In their experiment, they offered brief emotion stories (translated into Otji-Herero) and asked their Himba participants to select which of two vocalizations matched each story. The Himba did this well enough that Sauter and her colleagues concluded that emotion perception was universal. We were unable to replicate these results with a different group of Himba participants, even using the published method and the same translator as Sauter did. Maria also asked another group of Himba subjects to freely label the vocal sounds, without accompanying stories, and again, only the laughing sounds were categorized as expected (although they labeled the sounds as “laughing” rather than “happy”). So why did Sauter and her team observe universality when we did not?15
In late 2014, Sauter and her colleagues inadvertently solved the mystery. They revealed that their experiment included an extra step not reported in their original publication: a step that’s rich in conceptual knowledge. After the Himba participants heard an emotion story but before they listened to any sound pairs, they were asked to describe how the target person in the story was feeling. To help them in this task, Sauter and colleagues “allowed participants to listen several times to a given recorded story (if needed), until they could explain the intended emotion in their own words.” Whenever Himba participants described something other than the English emotion concept, they received negative feedback and were told to try again. Test subjects who were unable to provide the expected description were disqualified from the experiment. In effect, Himba participants were not permitted to listen to any sounds, let alone pick the ones that matched the story, until they had learned the corresponding English emotion concepts. When we attempted to replicate Sauter and colleagues’ experiment, we used only the methods in their published paper, without the extra, unreported step, so our Himba test subjects did not have the opportunity to learn English emotion concepts before listening to the vocalizations.16
There was one other difference between our experimental method and the one used by Sauter and her colleagues. Once a Himba participant had explained the emotion concept satisfactorily—let’s say it was sadness—Sauter’s team played a pair of sounds, such as a cry and a laugh, and the subject chose the better match for sadness. The participant then heard more pairs of sounds, each one containing a cry: perhaps a cry and a sigh, then a cry and a scream, and so on. From each pair, the participant selected one sound as the better match for sadness. If the Himba participants were not confident of the link between cries and sadness at the beginning of these trials, they certainly were by the end. Our experiments avoided this problem. In each trial, Maria would read a story (through the translator), then present a pair of sounds, and then have the participant choose the best match. Trials were in random order (e.g., a sadness trial, followed by an anger trial, followed by a happiness trial, and so on), which is a standard way to avoid learning within this type of experiment. We saw no evidence of universality.17
There is one emotion category that people seem able to perceive without the influence of emotion concepts: happiness. Regardless of the experimental method used, people in numerous cultures agree that smiling faces and laughing voices express happiness. So “Happy” might be the closest thing we have to a universal emotion category with a universal expression. Or it might not. For one thing, “Happiness” is usually the only pleasant emotion category that is tested using the basic emotion method, so it’s trivial for subjects to distinguish it from the negative categories. And consider this fun fact: the historical record implies that ancient Greeks and Romans did not smile spontaneously when they were happy. The word “smile” doesn’t even exist in Latin or Ancient Greek. Smiling was an invention of the Middle Ages, and broad, toothy-mouthed smiles (with crinkling at the eyes, named the Duchenne smile by Ekman) became popular only in the eighteenth century as dentistry became more accessible and affordable. The classics scholar Mary Beard summarizes the nuances of the point:
This is not to say that Romans never curled up the edges of their mouths in a formation that would look to us much like a smile; of course they did. But such curling did not mean very much in the range of significant social and cultural gestures in Rome. Conversely, other gestures, which would mean little to us, were much more heavily freighted with significance.
Perhaps sometime in the last few hundred years, smiling became a universal, stereotyped gesture symbolizing happiness.* Or . . . perhaps smiling in happiness is simply not universal.18
Emotion concepts are the secret ingredient behind the success of the basic emotion method. These concepts make certain facial configurations appear universally recognizable as emotional expressions when, in fact, they’re not. Instead, we all construct perceptions of each other’s emotions. We perceive others as happy, sad, or angry by applying our own emotion concepts to their moving faces and bodies. We likewise apply emotion concepts to voices and construct the experience of hearing emotional sounds. We simulate with such speed that emotion concepts work in stealth, and it seems to us as if emotions are broadcast from the face, voice, or any other body part, and we merely detect them.
A perfectly reasonable question for you to ask at this point is: how can my colleagues and I have the audacity to claim that our handful of experiments disconfirm hundreds of others that found evidence that emotions are universally recognized in expressions? The psychologist Dacher Keltner, for example, estimates that “there are a zillion data points on a perspective that conforms to Ekman.”19
The answer is that most of these zillion experiments use the basic emotion method, which you have just seen contains a secret stash of concept knowledge about emotion. If humans actually had an inborn ability to recognize emotional expressions, then removing the emotion words from the method should not matter . . . but it did, every single time. There is very little doubt that emotion words have a powerful influence in experiments, instantly casting into doubt the conclusions of every study ever performed that used the basic emotion method.20
To date, my lab has made two expeditions to Namibia and one to Tanzania (visiting a hunter-gatherer group called the Hadza) with consistent results. The social psychologist José-Miguel Fernández-Dols has also replicated our results in an isolated culture on the Trobriand Islands in New Guinea. So, science now has a reasonable, alternative explanation for those “zillions of data points.” The basic emotion method guides people to construct perceptions of Western-style emotions. That is, emotion perception is not innate but constructed.21
If you look closely at the original cross-cultural experiments from the 1960s, you can see clues that the conceptual elements within the basic emotion method pushed the results toward the appearance of universality. Of the seven samples using test subjects from remote cultures, the four that used the basic emotion method provided strong evidence for universality, but the remaining three used free labeling and did not show evidence of universality. These three contrary samples were not published in peer-reviewed journals but only as book chapters—a lesser form of publishing in the world of academia—and are rarely cited. As a result, the four samples supporting universality were lauded as a major breakthrough in research on our underlying human nature and set the stage for the research avalanche to come. Hundreds of subsequent studies employed the basic emotion method with forced choice, largely in cultures that had exposure to Western cultural practices and norms, taking a key condition for universality out of the experimental design but still claiming it as fact. This explains why today, many scientists and the public fundamentally misunderstand what is known about “emotional expressions” and “emotion recognition” from a scientific point of view.22
What might the science of emotion look like today had someone drawn different conclusions from those original studies? Consider Ekman’s account of his first visit to the Fore tribe in New Guinea:
I asked them to make up a story about each facial expression [photograph]. “Tell me what is happening now, what happened before to make the person show this expression, and what is going to happen next.” It was like pulling teeth. I am not certain whether it was the translation process, or the fact that they have no idea what it was I wanted to hear or why I wanted them to do this. Perhaps making up stories about strangers was just something the Fore didn’t do.
Ekman might be right, but it is also possible that the Fore did not understand or accept the concept of a facial “expression,” which implies an internal feeling that seeks release in a set of facial movements. Not all cultures understand emotions as internal mental states. Himba and Hadza emotion concepts, for example, appear to be more focused on actions. This is also true of certain Japanese emotion concepts. The Ifaluk of Micronesia consider emotions as transactions between people. To them, anger is not a feeling of rage, a scowl, a pounding fist, or a loud yelling voice, all within the skin of one person, but a situation in which two people are engaged in a script—a dance, if you will—around a common goal. In the Ifaluk view, anger does not “live” inside either participant.23
When you look at the development and history of the basic emotion method, there’s a surprising amount to criticize from a scientific standpoint. Over twenty years ago, the psychologist James A. Russell catalogued many of the concerns. And remember that the “six basic facial expressions” were not a scientific discovery; the Western architects of the basic emotion method stipulated them, actors posed them, and a science was built around them. There is no known validity to these particular facial poses, and studies that use more objective methods like facial EMG and facial coding do not find evidence that people routinely make these movements in real life during episodes of emotion. Yet scientists continue to use the basic emotion method regardless. After all, it produces very consistent results.24
Each time a scientific “fact” is overturned it leads to new avenues for discovery. The physicist Albert Michelson won a Nobel Prize in 1907 for disproving a conjecture made by Aristotle, that light travels through empty space via a hypothetical substance called luminiferous ether. His detective work set the stage for Albert Einstein’s theory of relativity. In our case, we’ve cast substantial doubt on the evidence for universal emotions. They only appear to be universal under certain conditions—when you give people a tiny bit of information about Western emotion concepts, intentionally or not. These observations, and others like them, set the stage for the new theory of emotion that you are about to learn. So Tomkins, Ekman, and their colleagues did contribute to a remarkable discovery. It just wasn’t the discovery that they expected.25
The many cross-cultural studies employing the basic emotion method suggest something else exciting: it may be easy to teach emotion concepts across cultural boundaries, even unintentionally. Such a worldwide understanding would be hugely beneficial. If Saddam Hussein’s half-brother had only understood the American emotion concept of anger, he might have perceived anger in Secretary of State James Baker, which might have averted the first Gulf War with the United States, saving thousands of lives.
Given how easy it is to teach emotion concepts by accident, there is also a danger in using Western stereotypes of emotion in cultural research. For instance, an ongoing series of studies called the Universal Expressions Project is attempting to document what is universal about emotional expressions in the face, body, and voice. So far, they’ve identified “about 30 facial expressions and 20 vocal expressions that are very similar around the world.” The catch is that the project uses only the basic emotion method, so it’s investigating universality with a tool that cannot provide such evidence. (Also, they’re asking people to pose what they believe are their cultural expressions, which is not the same thing as observing actual body movements during emotion.) More importantly, if the project reaches its goal, everyone in the world might learn the Western stereotypes for emotions.26
In the long run, scientists who still subscribe to the basic emotion method are very likely helping to create the universality that they believe they are discovering.
Closer to home, if people believe that a face alone displays emotion, it can lead to serious mistakes with damaging repercussions. In one case, this belief changed the course of a U.S. presidential election. In 2003–2004, Governor Howard Dean of Vermont was seeking the Democratic nomination for president of the United States, an honor that ultimately went to Senator John Kerry of Massachusetts. Voters saw a lot of negative campaigning that season, and one of the most misleading examples was a video of Dean taken during a speech. In a snippet of video that went viral, Dean’s face was shown alone, without context, and he looked furious. But if you watched the entire video in context, it becomes obvious that Dean was not enraged but excited, firing up the crowd with his enthusiasm. The snippet circulated on the news, spread widely, and, ultimately, Dean dropped out of the race. We can only wonder what might have happened if viewers had understood how emotions are made when they saw those misleading images.
Guided by a constructionist approach, scientists continue to replicate my lab’s findings in other cultures (data from China, East Africa, Melanesia, and other regions are looking promising at press time). As they do, we are speeding the paradigm shift to a new understanding of emotion that goes beyond Western stereotypes. We can cast aside questions like “How accurately can you recognize fear?” and instead study the variety of facial movements that people actually make in fear. We can also try to understand why people hold stereotypes about facial configurations in the first place, and what their value might be.
The basic emotion method has shaped the scientific landscape and influenced public understanding of emotion. Thousands of scientific studies claim that emotions are universal. Popular books, magazine articles, radio broadcasts, and TV shows casually assume that everyone makes and recognizes the same facial configurations as expressions of emotion. Games and books teach preschool children these allegedly universal expressions. International political and business negotiation strategies are likewise based on this assumption. Psychologists assess and treat emotion deficits in people suffering from mental illness using similar methods. The growing economy of emotion-reading gadgets and apps also assumes universality, as if emotions can be read in the face or in patterns of bodily changes in the absence of context, as easily as reading words on a page. The sheer amount of time, effort, and money going into these efforts is mind-boggling. But what if the fact of universal emotions isn’t a fact at all?
What if it’s evidence for something else entirely . . . namely, our ability to use concepts to shape perception? This is the crux of the theory of constructed emotion: a full-fledged, alternative explanation for the mystery of human emotion that does not rely on universal emotion fingerprints. The next four chapters dive into the details of this theory and the scientific evidence that supports it.
