Acoustic and Visual Stimulus Parameters Underlying Sound Symbolic Crossmodal Correspondences

• Type of Material: Article
• Authors: Sara M. List, Emory University; Kelly McCormick, Emory University; Simon Lacey, Emory University; Krishnankutty Sathian, Emory University; Lynne Nygaard, Emory University
• Language: English
• Date: 2019-01-11
• Publisher: Cold Spring Harbor Laboratory
• Publication Version: Preprint (Prior to Peer Review)
• Copyright Statement: The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
• License: Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International
• Final Published Version (URL): https://doi.org/10.1101/517581
• Title of Journal or Parent Work: bioRxiv
• Grant/Funding Information: This work was supported by grants to KS and LN from the National Eye Institute at the NIH (R01EY025978) and the Emory University Research Council. Support to KS from the Veterans Administration and to SML from the Laney Graduate School is also acknowledged.
• Keywords: pseudowords; language; multisensory; represenational similarity analysis
• Research Categories: Language, Linguistics; Biology, Neuroscience

Abstract

It is often assumed that a fundamental property of language is the arbitrariness of the relationship between sound and meaning. Sound symbolism, which refers to non-arbitrary mapping between the sound of a word and its meaning, contradicts this assumption. Sensitivity to sound symbolism has been studied through crossmodal correspondences (CCs) between auditory pseudowords (e.g. ‘loh-moh’) and visual shapes (e.g. a blob). We used representational similarity analysis to examine the relationships between physical stimulus parameters and perceptual ratings that varied on dimensions of roundedness and pointedness, for a range of auditory pseudowords and visual shapes. We found that perceptual ratings of these stimuli relate to certain physical features of both the visual and auditory domains. Representational dissimilarity matrices (RDMs) of parameters that capture the spatial profile of the visual shapes, such as the simple matching coefficient and Jaccard distance, were significantly correlated with those of the visual ratings. RDMs of certain acoustic parameters of the pseudowords, such as the temporal fast Fourier transform (FFT) and spectral tilt, that reflect spectral composition, as well as shimmer and speech envelope that reflect aspects of amplitude variation over time, were significantly correlated with those of the auditory perceptual ratings. RDMs of the temporal FFT (acoustic) and the simple matching coefficient (visual) were significantly correlated. These findings suggest that sound-symbolic CCs are related to basic properties of auditory and visual stimuli, and thus provide insights into the fundamental nature of sound symbolism and how this might evoke specific impressions of physical meaning in natural language.