Stimulus Parameters Underlying Sound-Symbolic Mapping of Auditory Pseudowords to Visual Shapes

• Type of Material: Article
• Authors: Simon Lacey, Emory University; Yaseen Jamal, Emory University; Sara M. List, Emory University; Kelly McCormick, Emory University; Krishnankutty Sathian, Emory University; Lynne C. Nygaard, Emory University
• Language: English
• Date: 2020-09-01
• Publisher: Wiley
• Publication Version: Author Accepted Manuscript (After Peer Review)
• Copyright Statement: © 2020 Cognitive Science Society, Inc. All rights reserved.
• Final Published Version (URL): http://dx.doi.org/10.1111/cogs.12883
• Title of Journal or Parent Work: Cognitive Science
• Volume: 44
• Issue: 9
• Start Page: e12883
• End Page: e12883
• Grant/Funding Information: This work was supported by grants to KS and LCN from the National Eye Institute at the NIH (R01EY025978) and the Emory University Research Council.; Support to KS from the Veterans Administration and to SML from the Laney Graduate School is also acknowledged.
• Keywords: Language; Speeded classification; Sound symbolism; Similarity; Speech; Correspondences; Psychology; Prosody; Words; Social Sciences; Vibratory patterns; Multisensory; Representational similarity analysis; Semantics; Bouba; Sensitivity; Psychology, Experimental
• Research Categories: Psychology, Cognitive; Psychology, Behavioral; Health Sciences, Speech Pathology; Biology, Neuroscience

Abstract

Sound symbolism refers to non-arbitrary mappings between the sounds of words and their meanings and is often studied by pairing auditory pseudowords such as “maluma” and “takete” with rounded and pointed visual shapes, respectively. However, it is unclear what auditory properties of pseudowords contribute to their perception as rounded or pointed. Here, we compared perceptual ratings of the roundedness/pointedness of large sets of pseudowords and shapes to their acoustic and visual properties using a novel application of representational similarity analysis (RSA). Representational dissimilarity matrices (RDMs) of the auditory and visual ratings of roundedness/pointedness were significantly correlated crossmodally. The auditory perceptual RDM correlated significantly with RDMs of spectral tilt, the temporal fast Fourier transform (FFT), and the speech envelope. Conventional correlational analyses showed that ratings of pseudowords transitioned from rounded to pointed as vocal roughness (as measured by the harmonics-to-noise ratio, pulse number, fraction of unvoiced frames, mean autocorrelation, shimmer, and jitter) increased. The visual perceptual RDM correlated significantly with RDMs of global indices of visual shape (the simple matching coefficient, image silhouette, image outlines, and Jaccard distance). Crossmodally, the RDMs of the auditory spectral parameters correlated weakly but significantly with those of the global indices of visual shape. Our work establishes the utility of RSA for analysis of large stimulus sets and offers novel insights into the stimulus parameters underlying sound symbolism, showing that sound-to-shape mapping is driven by acoustic properties of pseudowords and suggesting audiovisual cross-modal correspondence as a basis for language users' sensitivity to this type of sound symbolism.