Visual Image Decoding of Brain Activities Using a Dual Attention Hierarchical Latent Generative Network With Multiscale Feature Fusion

Abstract

Reconstructing visual stimulus from human brain activity measured with functional magnetic resonance imaging (fMRI) is a challenging decoding task for revealing the visual system. Recent deep learning approaches commonly neglect the relationship between hierarchical image features and different regions of the visual cortex, and fail to use global and local image features in reconstructing visual stimulus. To address these issues, in this paper, a novel neural decoding framework is proposed by using a dual attention hierarchical latent generative network with multi-scale feature fusion (DA-HLGN-MSFF) method. Specifically, the fMRI data is firstly encoded to hierarchical features of our image encoder network which employs a multi-kernel convolution block to extract the multi-scale spatial information of images. In order to reconstruct the perceived images and further improve the performance of our generator network, a dual attention block based on channel-spatial attention mechanism is then proposed to exploit the inter-channel relationships and spatial long-range dependencies of features. Moreover, a multi-scale feature fusion block is finally adopted to aggregate the global and local information of features at different scales and synthesize the final reconstructed images in the generator network. Competitive experimental results on two public fMRI datasets demonstrate that our method is able to achieve promising reconstructing performance compared with the state-of-the-art methods. The codes of our proposed DA-HLGN-MSFF method will be open access on https://github.com/ljbuaa/HLDAGN.