Modality encoded latent dataset for emotion recognition

Abstract

Variational autoencoder (VAE) is an unsupervised learning that represents high dimensional input data into normally distributed latent space. Multi-channel physiological signals, namely EEG and peripherals are mostly preferred for affective computing. The DEAP dataset is converted into multimodal latent dataset for emotion recognition in this study. 40-ch recordings of 32 participants are encoded to different modalities of peripherals and 32-ch EEG. First, short-time Fourier transform (STFT) is used to extract time–frequency (TF) distribution for training VAE. Thus, the localized components in the each channel of the modalities is converted to 100-dimensional space using VAE. The proposed method is applied to each participant’s recordings to obtain new latent encoded dataset. Within and between subject classification results using latent dataset are compared to the original data for peripheral, 32ch EEG and peripheral with EEG modalities. Naive Bayes (NB) classifier is used to evaluate the encoding performance of the 100-dimensional modalities, and compared to original results. The error rates of leave-one participant-out cross-validation (LOPO CV) 0.3322 and 0.3327 are yielded for high/low arousal and valence states while the originals are 0.349 and 0.382.