Automatic Recognition of Element Classes and Boundaries in the Birdsong with Variable Sequences.

Takuya Koumura,Kazuo Okanoya,David S Vicario

doi:10.1371/journal.pone.0159188

Abstract

Researches on sequential vocalization often require analysis of vocalizations in long continuous sounds. In such studies as developmental ones or studies across generations in which days or months of vocalizations must be analyzed, methods for automatic recognition would be strongly desired. Although methods for automatic speech recognition for application purposes have been intensively studied, blindly applying them for biological purposes may not be an optimal solution. This is because, unlike human speech recognition, analysis of sequential vocalizations often requires accurate extraction of timing information. In the present study we propose automated systems suitable for recognizing birdsong, one of the most intensively investigated sequential vocalizations, focusing on the three properties of the birdsong. First, a song is a sequence of vocal elements, called notes, which can be grouped into categories. Second, temporal structure of birdsong is precisely controlled, meaning that temporal information is important in song analysis. Finally, notes are produced according to certain probabilistic rules, which may facilitate the accurate song recognition. We divided the procedure of song recognition into three sub-steps: local classification, boundary detection, and global sequencing, each of which corresponds to each of the three properties of birdsong. We compared the performances of several different ways to arrange these three steps. As results, we demonstrated a hybrid model of a deep convolutional neural network and a hidden Markov model was effective. We propose suitable arrangements of methods according to whether accurate boundary detection is needed. Also we designed the new measure to jointly evaluate the accuracy of note classification and boundary detection. Our methods should be applicable, with small modification and tuning, to the songs in other species that hold the three properties of the sequential vocalization.

Highlights

Sequential vocalizations, in which voices are produced sequentially, have been a target of wide variety of researches. This is because they include human spoken language, and because they serve as excellent models for precise motor control, learning, and auditory perception
Songs in two birds which had more than 1% of manually unrecognizable notes were discarded
Songs in each bird were individually processed because songs were largely different among birds

Summary

Introduction

Sequential vocalizations, in which voices are produced sequentially, have been a target of wide variety of researches This is because they include human spoken language, and because they serve as excellent models for precise motor control, learning, and auditory perception. Birdsong is one of the most complex and precisely controlled sequential vocalizations, and has been widely and intensively studied [1,2,3]. Rules for note sequencing are unique to individuals and acquired by learning [17,18,19,20,21] This rule for note sequence production is called song syntax. In analyzing birdsong it is important to group notes into classes, extract timing information, and consider song syntax

Methods

Results

Conclusion