Stimulus Parameters Underlying Sound-Symbolic Mapping of Auditory Pseudowords to Visual Shapes

• Authors: Simon Lacey, Emory University; Yaseen Jamal, Emory University; Sara M. List, Emory University; Kelly McCormick, Emory University; Krishnankutty Sathian, Emory University; Lynne C. Nygaard, Emory University
• Journal Title: Cognitive Science
• Volume: Volume 44, Number 9
• Publisher: Wiley | 2020-09-01, Pages e12883-e12883
• Type of Work: Article | Post-print: After Peer Review
• Publisher DOI: http://doi.org/10.1111/cogs.12883
• Subjects: Psychology, Cognitive; Health Sciences, Speech Pathology; Biology, Neuroscience; Psychology, Behavioral
• Keywords: Social Sciences; Psychology, Experimental; Psychology; Language; Multisensory; Representational similarity analysis; Sound symbolism; Speeded classification; Vibratory patterns; Correspondences; Sensitivity; Similarity; Speech; Semantics; Prosody; Words; Bouba

Abstract

Sound symbolism refers to non-arbitrary mappings between the sounds of words and their meanings and is often studied by pairing auditory pseudowords such as “maluma” and “takete” with rounded and pointed visual shapes, respectively. However, it is unclear what auditory properties of pseudowords contribute to their perception as rounded or pointed. Here, we compared perceptual ratings of the roundedness/pointedness of large sets of pseudowords and shapes to their acoustic and visual properties using a novel application of representational similarity analysis (RSA). Representational dissimilarity matrices (RDMs) of the auditory and visual ratings of roundedness/pointedness were significantly correlated crossmodally. The auditory perceptual RDM correlated significantly with RDMs of spectral tilt, the temporal fast Fourier transform (FFT), and the speech envelope. Conventional correlational analyses showed that ratings of pseudowords transitioned from rounded to pointed as vocal roughness (as measured by the harmonics-to-noise ratio, pulse number, fraction of unvoiced frames, mean autocorrelation, shimmer, and jitter) increased. The visual perceptual RDM correlated significantly with RDMs of global indices of visual shape (the simple matching coefficient, image silhouette, image outlines, and Jaccard distance). Crossmodally, the RDMs of the auditory spectral parameters correlated weakly but significantly with those of the global indices of visual shape. Our work establishes the utility of RSA for analysis of large stimulus sets and offers novel insights into the stimulus parameters underlying sound symbolism, showing that sound-to-shape mapping is driven by acoustic properties of pseudowords and suggesting audiovisual cross-modal correspondence as a basis for language users' sensitivity to this type of sound symbolism.