Pop summary of Tamir*, Thornton*, Contreras & Mitchell (2016) PNAS

How do we navigate other people's mental states? We use fMRI to discover the 'longitude and latitude' of our maps of others' minds.

*equal contributions

December 1, 2015 by Mark

What does it mean to us when we perceive someone to be happy, angry, or sad? How do we compare the mental states of planning and exhaustion, or pity and fury? Fundamentally, what are the principles we use to make sense of other people's minds? These are questions that I, together with my former labmates (now gone on to bigger and better things) Diana Tamir and Juan Manuel Contreras, and my advisor Jason Mitchell, have worked towards answering in our research, recently published in the Proceedings of the National Academy of Science (PNAS).

We view these as questions of great importance because our ability, as humans, to understand other humans, resides near the heart of what makes our species special. Humanity's unique achievements - spreading across the globe, building massive structures, cities and civilizations, developing intricate and diverse cultures, and studying the nature of the universe - rely on our ability to cooperate with each other and socialize in highly complex ways. Dealing with the complexity of the social world requires a keen understanding of other people's minds.

Navigating other people's mental states poses a particular challenge. Unlike the things we encounter in our physical environment, we cannot directly perceive people's mental states. There is no scent of hope, nor sound of curiosity. We must therefore infer what other people are thinking and feeling from indirect clues: the way they act, their facial expressions and body posture, and the overall context. While usually quite effective, these clues are inherently fuzzy and imperfect - it is sometimes hard to be sure whether a new acquaintance's bonhomie is sincere or feigned, or whether a partner's silence reflects fatigue or resentment.

Despite these challenges, people are generally quite good at navigating this fuzzy, hidden world of others' minds. The question, for us, became: what principles do people use to do so? In particular, we investigated what dimensions the brain uses to organize its understanding of other people's mental states. A reasonable analogy here might be GPS. A GPS receiver picks up on satellite signals to determine its location with respect to two dimensions: latitude and longitude. Knowing your coordinates on these dimensions tell you exactly where you are on the surface of Earth. If you know the coordinates of two places, you (or your phone) can calculate how far apart they are. Positions on individual dimensions also carry specific information: for example, a high very latitude indicates a cold climate, while longitude can tell you which time zone you're in (approximately).

We hoped to learn what dimensions the brain's mental state GPS might be attuned to. Naturally latitude and longitude would make no sense on a map of others' minds, but what would? Fortunately, we were not the first to think along these lines, and a number of other scholars have proposed (sets of) dimensions that might carve up the social landscape. We generated a list of 166 different mental states and asked a large group of online participants to rate them on 16 dimensions from existing theories. The scatterplot below visualizes the positions of the states on these dimensions.

These original dimensions came from diverse sources and were never intended to be fully complementary to one another. As a result, they embody some overlapping (i.e. correlated) intuitions. We thus reduced them down to a set of four non-redundant dimensions: rationality (how rational or emotional a mental state is), social impact (how intense and intrinsically social a state is), human mind (how uniquely human and purely mental a state is), and valence (how positive or negative a state is). These four dimensions captured most of what the original 16 reflected about mental states, while remaining uncorrelated with each other. At this point we also carefully chose a subset of 60 mental states for use in a neuroimaging experiment. The positions of these 60 states on our four uncorrelated hypothetical dimensions are shown in the graph below.

We hoped to learn whether people spontaneously organize mental states along some or all of these four dimensions when they think about other people's minds. To test this, we turned to the brain, in the form of a functional magnetic resonance imaging (fMRI) experiment. FMRI is a costly and difficult technique, but it carries significant advantages in this context. In addition to being a rich, multidimensional source of data, one particular advantage for our work is that fMRI is what is known as an implicit measure. The scanner acquires data directly from participants' brains without any intervention on their part, and thus participants have little ability to consciously control what is measured. The implicit nature of fMRI helped us to keep our participants completely blind to our hypotheses until after the experiment was over. This minimized any chance that, in their cooperativeness, participants might place greater emphasis on the hypothetical dimensions than they might have done spontaneously.

Participants in the fMRI experiment saw the mental state terms above paired with two scenarios known to evoke the presented state. For example, on one trial, participants would might see the word "embarressment" paired with the scenarios "forgetting someone's name" and "laughing and accidentally snorting." The participant's task was to judge which of the two scenarios would elicit more of the mental state in question in another person, and indicate their judgment by pressing a corresponding button. This process repeated itself many times over the course of the experiment with all 60 of the mental states terms shown above appearing 16 times each for each participant.

We analyzed this data using a technique called representational similarity analysis, something I've written about previously. The basic procedure was as follows: we generated patterns of neural activity associated with each of the 60 mental states, using the general linear model (basically just a regression). We then correlated these patterns with each other to determine how similarly the brain represented each pair of mental states. Naturally we only considered the portion of the pattern within parts of the brain sensitive to others' mental states (i.e. the "social brain network"). We then tried to predict the neural similarity between mental states using the states' positions on the four hypothetical dimensions shown above. If, for example, two mental states were very similar in terms of their valence (e.g. peacefulness and affection, both positive) then valence would predict that the corresponding neural patterns associated with peacefulness and affection should be quite similar. The group-level results of this analysis can be seen below.

The points in the plot above indicate the correlations between neural similarity and the predictions made by each of the four hypothetical dimensions. The error bars around each point indicate the 95% confidence intervals (determined via a process called bootstrapping) around each correlation. The three correlations for which the respective confidence intervals do not include zero are statistically significant - i.e., there is a low probability that values so large would have been observed by chance if the real correlations were 0. Of the four dimensions tested, only human mind was not statistically significant. The other three dimensions together account for approximately 47% of the variability in the true dimensions underlying the neural representation of others' mental states. In other words, rationality, social impact, and valence explain nearly half of the neural basis of our understanding of what others are thinking and feeling. (N.B.: arriving at this estimate requires correcting for reliability based attenuation and an underestimation inherent in representational similarity analysis.).

There's clearly a lot more work to be done in this vein. At best, we're almost half-way to a basic understanding of the way we think about other people's mental states. However, there are doubtless more important dimensions yet to discover. We're also interested in how the importance of the three dimensions we identified here might change depending on who you imagine having them. For instance, might rationality matter more when considering a friend's mind and valence matter more when trying to understand a stranger? More broadly, we hope to learn how these dimensions allow us to perform everyday 'mindreading' - that is, how they might tie into our ability to predict and explain others' thoughts and feelings.

For anyone interested in digging into this further, de-identified data from this study are publicly available online. See here for our Open Science Framework project page, which contains the rating data used to generate the graphs above, behavioral data from participants in the fMRI experiment, and code (R, Matlab, and Python). Raw imaging data are available on the Harvard Dataverse here. The interactive graphs on this page were built with d3.js. If you have any questions about this research, you can get in touch with me on twitter or via email.

2020 update: We have recently published a new paper on this topic in Cortex (preprint). In this article we build on the study described above to provide an even more thorough test of what we now call the '3d Mind Model'. These results can be summarized in three points:

1. The 3d Mind model is robust: We combined four fMRI datasets, composed of data from more than 113 participants, to conduct a more powerful test of the influence of rationality, social impact and valence on neural pattern similarity. We found that these three dimensions are significant predictors in this mega-analysis, despite heterogeneity across studies.
2. The 3d Mind model is comprehensive: In this follow-up study, we examined a total of 58 possible dimensions of mental state representation, including the 16 we examined in the initial study. Despite examining this much broader set of candidates, we found that a 3d Model based on dimensions reflecting rationality, social impact, and valence was the best explanation for neural pattern similarity. Moreover, these three dimensions explain 80% of reliable variance across studies, suggesting that the 3d Mind Model explains a clear majority of our shared concepts of mental states.
3. The 3d Mind model is generalizable: In addition to testing the more dimensions on more fMRI data, we also examined the ability of the 3d Mind Model to generalize to new measures of mental state representation. Specifically, we found that rationality, social impact, and valence are each significant predictors of both behavioral judgements of mental state similarity made by participants on MySocialBrain.org and estimates of semantic similarity between mental state terms in text derived from the fastText word embedding. This suggests that the 3d Mind Model can be applied to a broad range of different psychological phenomenon involving thinking about other people's thoughts and feelings.

You can read more about what we think the brain uses this 3d map of mental states for in blog posts on other recent papers here and here.