Through a Dog's Eyes: fMRI Decoding of Naturalistic Videos from the Dog Cortex

• Authors: Erin M. Phillips, Emory University; Kirsten D. Gillette, Emory University; Daniel D Dilks, Emory University; Gregory S Berns, Emory University
• Journal Title: Neuroscience
• Publisher: JoVE | 2022-09-13
• Type of Work: Article | Final Publisher PDF
• Publisher DOI: http://doi.org/10.3791/64442
• Subjects: Biology, Neuroscience; Health Sciences, Opthamology
• Keywords: visual stream; pathways; what/where dichotomy; cortex; perception; AI based fMRI decoding; dog cortex; naturalistic video classification; object based vs action based classification; cross species comparison of visual perception

Abstract

Recent advancements using machine learning and functional magnetic resonance imaging (fMRI) to decode visual stimuli from the human and nonhuman cortex have resulted in new insights into the nature of perception. However, this approach has yet to be applied substantially to animals other than primates, raising questions about the nature of such representations across the animal kingdom. Here, we used awake fMRI in two domestic dogs and two humans, obtained while each watched specially created dog-appropriate naturalistic videos. We then trained a neural net (Ivis) to classify the video content from a total of 90 min of recorded brain activity from each. We tested both an object-based classifier, attempting to discriminate categories such as dog, human, and car, and an action-based classifier, attempting to discriminate categories such as eating, sniffing, and talking. Compared to the two human subjects, for whom both types of classifier performed well above chance, only action-based classifiers were successful in decoding video content from the dogs. These results demonstrate the first known application of machine learning to decode naturalistic videos from the brain of a carnivore and suggest that the dog's-eye view of the world may be quite different from our own.