Multi-channel fusion convolutional neural network to classify syntactic anomaly from language-related ERP components

Abstract

Event-Related Potential (ERP) analyses have revealed several language-related components in sentence processing literature. More recently, researchers attempted to apply machine-learning techniques to classify the language-structure dependent ERP signals in a more reliable and efficient way. The purpose of the current paper is to propose a classification technique based on data-driven approach to detect syntactic anomaly from language-related ERP components. We specifically examined whether sentences with syntactic violations elicited differential patterns of ERP signals and the abnormal patterns can be reliably classified by machine-learning techniques. The specific aim of the study is to develop a multi-channel fusion convolutional neural network (MCF-CNN) including two branches of CNNs and a trunk merged by an intermediate fusion layer to obtain trained linguistic features from the raw data and perform the classification. We extracted different linguistic ERP components from syntactic violations and put them in the fusion. As a next procedure we combined the features in the fusion layer of the proposed neural network architecture. Experimental results demonstrate that the proposed method provides more than 92% classification accuracy.