Sensory encoding of emotion conveyed by the face and visual context

• Type of Material: Article
• Authors: Katherine Soderberg, Emory University; Grace Jang, Emory University; Philip Augustus Kragel, Emory University
• Language: English
• Date: 2024-07-11
• Publisher: Cold Spring Harbor Laboratory
• Publication Version: Preprint (Prior to Peer Review)
• Copyright Statement: The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
• License: Creative Commons Attribution-NonCommercial 4.0 International
• Final Published Version (URL): https://doi.org/10.1101/2023.11.20.567556
• Title of Journal or Parent Work: bioRxiv
• Keywords: amygdala; facial features; artificial neural networks (ANNs); interpretation; sensory signals; posterior temporal sulcus (pSTS)
• Research Categories: Biology, Neuroscience

Abstract

Humans rapidly detect and interpret sensory signals that have emotional meaning. The posterior temporal sulcus (pSTS) and amygdala are known to be critical for this ability, but their precise contributions—whether specialized for facial features or sensory information more generally—remain contentious. Here we investigate how these structures process visual emotional cues using artificial neural networks (ANNs) to model fMRI signal acquired as participants view complex, naturalistic stimuli. Characterizing data from two archival studies (Ns = 20, 45), we evaluated whether representations from ANNs optimized to recognize emotion from either facial expressions alone or the broader visual context differ in their ability to predict responses in human pSTS and amygdala. Across studies, we found that representations of facial expressions were more robustly encoded in pSTS compared to the amygdala, whereas representations related to visual context were encoded in both regions. These findings demonstrate how the pSTS operates on abstract representations of facial expressions such as ‘fear’ and ‘joy’ to a greater extent than the amygdala, which more strongly encodes the emotional significance of visual information more broadly, depending on the context.