Correspondence between the Video-Learning Deep Neural Networks and EEG Brain Activity during Naturalistic Video Viewing

Abstract

Recent findings have suggested that the responses of brain activity and those of deep neural networks (DNNs) to sensory inputs correspond well. Such correspondence is expected to enable the use of DNNs as brain simulators. However, previous studies have been conducted mainly by measuring brain activity responding to static images using functional magnetic resonance imaging, which has a low temporal resolution. In this study, we examined human brain responses to naturalistic videos using electroencephalography (EEG) with time resolution on the order of milliseconds. Therefore, we used a video processing DNN model pre-trained under a self-supervised learning framework. The power spectral density of EEG responses to the presentation of documentary videos and short clips were both predictable from the responses of the DNN to the same video inputs. The prediction performance depended on the frequency bandwidth and was particularly accurate for high-frequency brain activity (i.e., beta- and gamma-band dynamics).