Semantic Labeling of Nonspeech Audio Clips

Xiaojuan Ma,Perry Cook,Christiane Fellbaum

doi:10.1155/2010/404860

Abstract

Human communication about entities and events is primarily linguistic in nature. While visual representations of information are shown to be highly effective as well, relatively little is known about the communicative power of auditory nonlinguistic representations. We created a collection of short nonlinguistic auditory clips encoding familiar human activities, objects, animals, natural phenomena, machinery, and social scenes. We presented these sounds to a broad spectrum of anonymous human workers using Amazon Mechanical Turk and collected verbal sound labels. We analyzed the human labels in terms of their lexical and semantic properties to ascertain that the audio clips do evoke the information suggested by their pre-defined captions. We then measured the agreement with the semantically compatible labels for each sound clip. Finally, we examined which kinds of entities and events, when captured by nonlinguistic acoustic clips, appear to be well-suited to elicit information for communication, and which ones are less discriminable. Our work is set against the broader goal of creating resources that facilitate communication for people with some types of language loss. Furthermore, our data should prove useful for future research in machine analysis/synthesis of audio, such as computational auditory scene analysis, and annotating/querying large collections of sound effects.

Highlights

Natural language is a highly complex yet efficient means of communication with great expressive power, and it is the primary mode of human communication and information exchange [1]
Very little research has been done on the use of nonspeech audio to convey and express concepts in Augmentative and Alternative Communication (AAC)
This work suggests that for both language-impaired populations and for healthy speakers whose comprehension is compromised for other reasons, nonspeech audio have the potential of conveying concepts and assisting language comprehension

Summary

Introduction

Natural language is a highly complex yet efficient means of communication with great expressive power, and it is the primary mode of human communication and information exchange [1]. Nonverbal representations of concepts that people communicate about have been explored and evaluated as a means to support linguistic representations. These include animations and videos [2] and especially still pictures [3]. Some research indicates that nonspeech audio perception may be impaired together with speech perception for people with specific pathological profiles because the process may share certain channel and brain regions [4]. This work suggests that for both language-impaired populations and for healthy speakers whose comprehension is compromised for other reasons, nonspeech audio (environmental sounds) have the potential of conveying concepts and assisting language comprehension

Objectives

Methods

Results

Discussion

Conclusion