A GAN model encoded by CapsEEGNet for visual EEG encoding and image reproduction

Abstract

In last few decades, reading the human mind is an innovative topic in scientific research. Recent studies in neuroscience indicate that it is possible to decode the signals of the human brain based on the neuroimaging data. The work in this paper explores the possibility of building an end-to-end BCI system to learn and visualize the brain thoughts evoked by the stimulating images. To achieve this goal, it designs an experiment to collect the EEG signals evoked by randomly presented images. Based on these data, this work analyzes and compares the classification abilities by several improved methods, including the Transformer, CapsNet and the ensemble strategies. After obtaining the optimal method to be the encoder, this paper proposes a distribution-to-distribution mapping network to transform an encoded latent feature vector into a prior image feature vector. To visualize the brain thoughts, a pretrained IC-GAN model is used to receive these image feature vectors and generate images. Extensive experiments are carried out and the results show that the proposed method can effectively deal with the small sample data original from the less electrode channels. By examining the generated images coming from the EEG signals, it verifies that the proposed model is capable of reproducing the images seen by human eyes to some extent.