Affective Cognition

with Jamil Zaki and Noah Goodman
September 2012 -- July 2014
Stanford University

Humans reason about others' emotions all the time, effortlessly and accurately. However, scientists have few formal computational models of how humans perform these feats of inference. Having computational models (i) allows testing of precise theories of emotion perception and reasoning, and (ii) opens the doors to many potential applications, such as building emotionally-intelligent robots and computer agents (Affective Computing) and building models to help diagnose and treat mental illness (Computational Psychiatry). This page details the start of my research program on computational models of affective cognition, and is based on our 2015 Cognition paper (linked at the bottom of the page).

Reasoning about emotion as logical, principled reasoning

We propose that reasoning about emotion occurs just like reasoning about anything else: using (i) domain-specific knowledge of how emotions work, and (ii) domain-general reasoning principles. For example, humans have a very intuitive grasp of physics: Most people can visualize how a thrown baseball would fly, even if they could not write down the equations that describe that trajectory. Similarly, people have an intuitive understanding of personality. We know that some individuals are just more irritable, while others are angelic: this knowledge allows people to predict how different individuals would react to different events. This knowledge constitutes our lay theories of the world (Other scholars may refer to this type of knowledge as concepts, schemas, or scripts). We have lay theories of physics and lay theories of personality that comprise the domain-specific knowledge about how thrown objects travel through the air and how personality affects behavior.

Now, given this knowledge, we can use logical principles of reasoning to arrive at optimal inferences. For example, if you see person A (but not persons B, C) react in an annoyed fashion after events X, Y, and Z, you might come to the conclusion that person A is an easily annoyed person. If you see, by contrast, that persons A, B, and C all react in an annoyed manner after event X (but not events Y and Z), then perhaps you might infer that it is, in fact, event X that is incredibly annoying. This type of social reasoning was formalized by Harold Kelley in his Covariation model (1973). In his proposed view, social cognition -- reasoning about others -- relies on reasoning using covariational evidence and knowledge about personality traits. Such principles of reasoning are domain-general: in other words, they are the same whether one is reasoning about physics or personality. Or, as we propose, emotions.

Despite how enigmatic emotions may be to define, people do have a intuitive theory of emotions. While most of us will have difficulty describing what happiness is, we would have no difficulty identifying a happy person††, inferring what type of events happened to them to make them happy, or predicting what they would do next. This knowledge is part of our intuitive theory of emotions. And this knowledge makes reasoning about emotions possible. In fact, if reasoning about emotions does follows logical principles, then we can write down a formal, mathematical model to describe these inferences. In Experiment 1 and 2 of our 2015 Cognition paper, we show that a mathemtical model based on Bayes' rule can accurately predict how people reason "forwards" and "backwards" about emotion (i.e. from events to emotions, and back from emotions to events).

Affective Cognition using Lay Theories of emotions

Social Cognition, or reasoning about others, relies on domain-specific knowledge (e.g. what we think about race, personality...) and on domain-general reasoning over that domain-specific knowledge.
We propose that Affective Cognition, reasoning about emotions, similarly relies on domain-general reasoning principles over domain-specific emotion knowledge.

Cue Integration

So far I've laid the case for a Bayesian model of emotion for reasoning from cause to effect and vice versa. Let's take our model a step further to examine a relatively common problem: cue integration. Cue integration refers to situations where there are multiple "cues" over which one has to make inferences. It is an extremely well studied problem because it appears all over science. Investigators trying to locate a plane wreckage would try to piece together bits of information (the plane's last known location and heading; weather patterns at the time; information about ocean currents if the crash was at sea) to "integrate" those different cues and infer the most likely location of the wreckage. Medical diagnosis relies on inference from multiple observed symptoms and known patient medical history. Within psychology, organisms often have to combine information from different senses (a well studied example is how the Barn Owl localizes prey using the signal from both of its ears).

Experiment 1

A proposed lay theory of emotion. Arrows indicate direction of causal influence.
Outcomes cause Emotions (which are unobservable), which in turn causes the person to act (Action), make an utternace (Speech), display a facial expression (Face), and so forth.

We cannot observe emotion directly, and the best we can do is to infer emotion from observing different cues (such as the person's facial expression, etc). Thus, there are many situations where we might observe different, possibly conflicting cues to someone's emotions. For example, you might see a person who has just won a prize or an award but has a facial expression that looks sad (such as Olympic medalists crying, or a glum Messi winning the 2014 World Cup Golden Ball Award).

Among emotion scientists, there has been a long debate on how we resolve situations with conflicting emotional cues. A long tradition steeped in the Basic Emotion theories of Paul Ekman and colleagues suggest that facial expressions are the only cues that matter in such conflict situations. A second side argue that no, facial cues aren't everything, and contextual cues matter as well (e.g. the situation that occured). Both sides' arguments do have some merit, and there are numerous studies supporting both sides (see our paper for more details on this debate).

We propose that the solution is that in fact, all types of emotional cues matter, to some extent -- and the real question is then, how much do they matter? A Bayesian model of emotional cue integration suggests that observers weigh different emotional cues based on how reliable the cues are. If one observers a very reliable cue that signals happiness, and a far less reliable cue that signifies sadness, then one should weigh the "happy" cue more than the "sad" cue when making one's inferences. Importantly, reliability gives a quantifiable way of measuring how much each cue should be weighted with respect to other cues. In our 2015 Cognition paper, we test the predictions of this Bayesian model with combinations of context and facial expression (Experiment 3), and with context and utterances (Experiment 4). Thus, we show that an ideal observer Bayesian model accurately predicts the types of inferences that people make given multiple, sometimes conflicting cues. This suggests that people do reason about emotional cues and resolve conflicts in a pretty optimal manner!

For more information, check out our 2015 publication in Cognition!

The principle of covariation was known by experimental scientists who performed the earliest experiments in physics, chemistry, and biology. Its entry into psychology was far more recent.
††A recent study suggests that continued use of digital devices might make kids poorer at detecting emotions. Sigh. Well at least our digital devices themselves may be getting better at emotion perception.

Relevant Publications

Ong, D. C., Zaki, J., & Goodman, N. D. (2015). Affective Cognition: Exploring lay theories of emotion. Cognition, 143, 141-162.
[ pdf ] [ pre-journal-formatted version ] [ github (data, code) ]

Zaki, J. (2013). Cue integration: A common framework for physical perception and social cognition. Perspectives on Psychological Science, 8(3), 296-312.
[ pdf from SSNL lab website ]