Automated annotation of birdsong with a neural network that segments spectrograms

• Type of Material: Article
• Authors: Yarden Cohen, Boston University; David Aaron Nicholson, Emory University; Alexa Sanchioni, Boston University; Emily K Mallaber, Boston University; Viktoriya Skidanova, Boston University; Timothy J Gardner, University of Oregon Eugene
• Language: English
• Date: 2022-01-20
• Publisher: eLife Sciences Publications Ltd
• Publication Version: Final Published Version
• Copyright Statement: © 2022, Cohen et al
• License: Creative Commons Attribution 4.0 International
• Final Published Version (URL): http://dx.doi.org/10.7554/eLife.63853
• Title of Journal or Parent Work: eLife
• Volume: 11
• Grant/Funding Information: The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.
• Keywords: neural network
• Research Categories: Biology, General

Abstract

Songbirds provide a powerful model system for studying sensory-motor learning. However, many analyses of birdsong require time-consuming, manual annotation of its elements, called syllables. Automated methods for annotation have been proposed, but these methods assume that audio can be cleanly segmented into syllables, or they require carefully tuning multiple statistical models. Here, we present TweetyNet: a single neural network model that learns how to segment spectrograms of birdsong into annotated syllables. We show that TweetyNet mitigates limitations of methods that rely on segmented audio. We also show that TweetyNet performs well across multiple individuals from two species of songbirds, Bengalese finches and canaries. Lastly, we demonstrate that using TweetyNet we can accurately annotate very large datasets containing multiple days of song, and that these predicted annotations replicate key findings from behavioral studies. In addition, we provide open-source software to assist other researchers, and a large dataset of annotated canary song that can serve as a benchmark. We conclude that TweetyNet makes it possible to address a wide range of new questions about birdsong.