Electroencephalogram-Based Single-Trial Detection of Language Expectation Violations in Listening to Speech.

Hiroki Tanaka,Hayato Maki,Sakti Sakriani,Satoshi Nakamura,Hiroki Watanabe

doi:10.3389/fncom.2019.00015

Hiroki Tanaka, Hayato Maki + Show 3 more

Open Access

https://doi.org/10.3389/fncom.2019.00015

Copy DOI

Abstract

We propose an approach for the detection of language expectation violations that occur in communication. We examined semantic and syntactic violations from electroencephalogram (EEG) when participants listened to spoken sentences. Previous studies have shown that such event-related potential (ERP) components as N400 and the late positivity (P600) are evoked in the auditory where semantic and syntactic anomalies occur. We used this knowledge to detect language expectation violation from single-trial EEGs by machine learning techniques. We recorded the brain activity of 18 participants while they listened to sentences that contained semantic and syntactic anomalies and identified the significant main effects of these anomalies in the ERP components. We also found that a multilayer perceptron achieved 59.5% (semantic) and 57.7% (syntactic) accuracies.

Highlights

In speech communication, we often face several types of language expectation violations, such as prosodic, semantic, and syntactic errors, especially in conversation through machine output
To achieve single-trial detection of such errors, we focus on other event-related potential (ERP) components, e.g., N400 and P600
Post hoc analysis revealed that the left posterior and the right anterior were significantly different between two conditions

Summary

Introduction

We often face several types of language expectation violations, such as prosodic, semantic, and syntactic errors, especially in conversation through machine output (e.g., human–computer interaction; Koponen, 2010). Regarding errors in the responses of spoken dialogue systems and machine translation, human examiners in previous research judged each sentence on an error scale from 1 to 5, unlike automatic evaluation metrics, e.g., word error rate (Lippmann, 1997; Och et al, 1999; Papineni et al, 2002). Even though this approach is quick and practical, it suffers from several problems. We assume that this system can be used for assessing people who exhibit the anomalies of semantic context sensitivity (e.g., autism spectrum, dementia, Olichney et al, 2008; Pijnacker et al, 2010; O’Connor, 2012; Tanaka et al, 2012, 2015, 2017a,b, 2018a; Ujiro et al, 2018)

Objectives

Methods

Results

Conclusion