Understanding the internal states of others is among the most vital and ubiquitous tasks humans face. In meeting this challenge, perceivers rely on multiple cognitive mechanisms including (i) vicariously sharing the sensorimotor and affective states of others and (ii) using contextual information to draw explicit inferences about others’ experiences. Research demonstrates that these cognitive processes are engaged by qualitatively different types of social cues, and rely on anatomically distinct neural systems. Although these processes have been studied largely in isolation, this approach may not closely model information processing involved in more naturalistic, “everyday” social cognition. Instead, like the sensory system, social cognition likely incorporates multiple “senses” that are each optimized for processing different types of social information, and that often work together to support interpersonal understanding. I will present two neuroimaging and two behavioral studies focusing on emotion perception and empathic accuracy, which together bolster this case. Together, these studies highlight interactions between multiple cognitive processes—as well as the contexts in which they are deployed—as critical to determining how individuals fare in their attempts to understand others. This approach provides a novel method by which to connect information processing to social function, and to understand the social symptoms that characterize psychiatric conditions such as autism spectrum disorders.