Visual Image Reconstruction Research Articles

A novel DRL-guided sparse voxel decoding model for reconstructing perceived images from brain activity

BackgroundDue to the sparse encoding character of the human visual cortex and the scarcity of paired training samples for {images, fMRIs}, voxel selection is an effective means of reconstructing perceived images from fMRI. However, the existing data-driven voxel selection methods have not achieved satisfactory results. New methodHere, a novel deep reinforcement learning-guided sparse voxel (DRL-SV) decoding model is proposed to reconstruct perceived images from fMRI. We innovatively describe voxel selection as a Markov decision process (MDP), training agents to select voxels that are highly involved in specific visual encoding. ResultsExperimental results on two public datasets verify the effectiveness of the proposed DRL-SV, which can accurately select voxels highly involved in neural encoding, thereby improving the quality of visual image reconstruction. Comparison with existing methodsWe qualitatively and quantitatively compared our results with the state-of-the-art (SOTA) methods, getting better reconstruction results. We compared the proposed DRL-SV with traditional data-driven baseline methods, obtaining sparser voxel selection results, but better reconstruction performance. ConclusionsDRL-SV can accurately select voxels involved in visual encoding on few-shot, compared to data-driven voxel selection methods. The proposed decoding model provides a new avenue to improving the image reconstruction quality of the primary visual cortex.

Visual image reconstruction and strategic innovation of Chengdu restaurant time-honored brand

Visual image reconstruction and strategic innovation of Chengdu restaurant time-honored brand

Semantics-Guided Hierarchical Feature Encoding Generative Adversarial Network for Visual Image Reconstruction From Brain Activity.

The utilization of deep learning techniques for decoding visual perception images from brain activity recorded by functional magnetic resonance imaging (fMRI) has garnered considerable attention in recent research. However, reconstructed images from previous studies still suffer from low quality or unreliability. Moreover, the complexity inherent to fMRI data, characterized by high dimensionality and low signal-to-noise ratio, poses significant challenges in extracting meaningful visual information for perceptual reconstruction. In this regard, we proposes a novel neural decoding model, named the hierarchical semantic generative adversarial network (HS-GAN), inspired by the hierarchical encoding of the visual cortex and the homology theory of convolutional neural networks (CNNs), which is capable of reconstructing perceptual images from fMRI data by leveraging the hierarchical and semantic representations. The experimental results demonstrate that HS-GAN achieved the best performance on Horikawa2017 dataset (histogram similarity: 0.447, SSIM-Acc: 78.9%, Peceptual-Acc: 95.38%, AlexNet(2): 96.24% and AlexNet(5): 94.82%) over existing advanced methods, indicating improved naturalness and fidelity of the reconstructed image. The versatility of the HS-GAN was also highlighted, as it demonstrated promising generalization capabilities in reconstructing handwritten digits, achieving the highest SSIM (0.783±0.038), thus extending its application beyond training solely on natural images.

NeuroZoom: Denoising and Super Resolving Neuromorphic Events and Spikes.

Neuromorphic cameras are emerging imaging technology that has advantages over conventional imaging sensors in several aspects including dynamic range, sensing latency, and power consumption. However, the signal-to-noise level and the spatial resolution still fall behind the state of conventional imaging sensors. In this paper, we address the denoising and super-resolution problem for modern neuromorphic cameras. We employ 3D U-Net as the backbone neural architecture for such a task. The networks are trained and tested on two types of neuromorphic cameras: a dynamic vision sensor and a spike camera. Their pixels generate signals asynchronously, the former is based on perceived light changes and the latter is based on accumulated light intensity. To collect the datasets for training such networks, we design a display-camera system to record high frame-rate videos at multiple resolutions, providing supervision for denoising and super-resolution. The networks are trained in a noise-to-noise fashion, where the two ends of the network are unfiltered noisy data. The output of the networks has been tested for downstream applications including event-based visual object tracking and image reconstruction. Experimental results demonstrate the effectiveness of improving the quality of neuromorphic events and spikes, and the corresponding improvement to downstream applications with state-of-the-art performance.

Hierarchical Bayesian Causality Network to Extract High-Level Semantic Information in Visual Cortex.

Functional MRI (fMRI) is a brain signal with high spatial resolution, and visual cognitive processes and semantic information in the brain can be represented and obtained through fMRI. In this paper, we design single-graphic and matched/unmatched double-graphic visual stimulus experiments and collect 12 subjects' fMRI data to explore the brain's visual perception processes. In the double-graphic stimulus experiment, we focus on the high-level semantic information as "matching", and remove tail-to-tail conjunction by designing a model to screen the matching-related voxels. Then, we perform Bayesian causal learning between fMRI voxels based on the transfer entropy, establish a hierarchical Bayesian causal network (HBcausalNet) of the visual cortex, and use the model for visual stimulus image reconstruction. HBcausalNet achieves an average accuracy of 70.57% and 53.70% in single- and double-graphic stimulus image reconstruction tasks, respectively, higher than HcorrNet and HcasaulNet. The results show that the matching-related voxel screening and causality analysis method in this paper can extract the "matching" information in fMRI, obtain a direct causal relationship between matching information and fMRI, and explore the causal inference process in the brain. It suggests that our model can effectively extract high-level semantic information in brain signals and model effective connections and visual perception processes in the visual cortex of the brain.

Mental image reconstruction from human brain activity: Neural decoding of mental imagery via deep neural network-based Bayesian estimation

Visual images observed by humans can be reconstructed from their brain activity. However, the visualization (externalization) of mental imagery is challenging. Only a few studies have reported successful visualization of mental imagery, and their visualizable images have been limited to specific domains such as human faces or alphabetical letters. Therefore, visualizing mental imagery for arbitrary natural images stands as a significant milestone. In this study, we achieved this by enhancing a previous method. Specifically, we demonstrated that the visual image reconstruction method proposed in the seminal study by Shen et al. (2019) heavily relied on low-level visual information decoded from the brain and could not efficiently utilize the semantic information that would be recruited during mental imagery. To address this limitation, we extended the previous method to a Bayesian estimation framework and introduced the assistance of semantic information into it. Our proposed framework successfully reconstructed both seen images (i.e., those observed by the human eye) and imagined images from brain activity. Quantitative evaluation showed that our framework could identify seen and imagined images highly accurately compared to the chance accuracy (seen: 90.7%, imagery: 75.6%, chance accuracy: 50.0%). In contrast, the previous method could only identify seen images (seen: 64.3%, imagery: 50.4%). These results suggest that our framework would provide a unique tool for directly investigating the subjective contents of the brain such as illusions, hallucinations, and dreams.

Visual image reconstruction based on EEG signals using a generative adversarial and deep fuzzy neural network

The decoding of the brain stimulated by visual images has been discussed in this study in order to classify and reconstruct the stimulating images. The dataset utilized was given by Stanford University and included the EEG signal recorded from ten healthy people. In the first stage, the proposed approach combines the ability of the Long-Short-Term Memory (LSTM) network to extract sequence (EEG signals) features with the ability of fuzzy network to extract features in high-uncertainty contexts. The features extracted in the previous stage, which is a noise-free and compressed form of the EEG data, were employed as a conditional vector in the CGAN network in the following step. As a result, this network was used to reconstruct visual stimuli that evoke specific brain responses. When compared to earlier research, the evaluation results produced utilizing the suggested approach demonstrate a greater classification accuracy, precision, recall, and F1 score. Also, the similarity of produced and real images was measured quantitatively using two parameters, FID and SSIM; the SSIM results for 10 participants were on average 0.37, 0.35, 0.35, 0.34, 0.42, and 0.42 for groups (1) to (6), and the FID results were 0.44, 0.45, 0.51, 0.50, 0.39 and 0.39, respectively. Therefore, it can be stated that the method suggested in this study can reconstruct visual stimuli using brain response (EEG signal) with acceptable accuracy, so taking a significant step towards the objective of mind reading.

The brain-inspired decoder for natural visual image reconstruction.

The visual system provides a valuable model for studying the working mechanisms of sensory processing and high-level consciousness. A significant challenge in this field is the reconstruction of images from decoded neural activity, which could not only test the accuracy of our understanding of the visual system but also provide a practical tool for solving real-world problems. Although recent advances in deep learning have improved the decoding of neural spike trains, little attention has been paid to the underlying mechanisms of the visual system. To address this issue, we propose a deep learning neural network architecture that incorporates the biological properties of the visual system, such as receptive fields, to reconstruct visual images from spike trains. Our model outperforms current models and has been evaluated on different datasets from both retinal ganglion cells (RGCs) and the primary visual cortex (V1) neural spikes. Our model demonstrated the great potential of brain-inspired algorithms to solve a challenge that our brain solves.

A New Visual Image Reconstruction Method of Production Equipment for Industrial Intelligent Terminals

A New Visual Image Reconstruction Method of Production Equipment for Industrial Intelligent Terminals

Development of Brain-Computer Interface Technology Using Scalp Electroencephalography

Studies on electroencephalography-based brain computer interface (EEG-based BCI) has been widely studied since around 2000. In the studies, three methods, event-related desynchronization/synchronization (ERD/ERS) during a motor imagery task, a P300 component of event-related potential, and steady state visual evoked potential (SSVEP) has been studied as standard methods. Since around 2013, with the acceleration of research in the United States and other countries, there has been a wide variety of studies that is not limited to the standard methods, such as research and development toward practical application and studies on extracting new brain activity signals. Some of the research results have been developed as commercial products. In this paper, after reviewing the studies on EEG-based BCI, topics in the research and development toward practical application such as new EEG systems and artifact rejection methods are introduced, topics in the studies on extracting new brain activity signals such as applying deep learning, emotion estimation and visual image reconstruction are also introduced, and topics in combination of BCI and information and communications technology and combination of BCI and XR technology are introduced. Finally, prospects of BCI technology are discussed.

視認する単純な画像の形と色を反映する脳波信号特徴量とその判別の研究

In this study, as a basic investigation for EEG-based visual image reconstruction, we investigated whether EEG signal features reflect shapes and colors of simple visual images which subjects viewed and whether the features can be discriminated. First, we investigated how the shapes and the colors are reflected in event-related potentials (ERP), the event-related spectrum perturbations (ERSP), and the inter-trial phase synchronization (ITC). The results showed statistically significant differences in ERP among the colors and ERP, ERSP and ITC among the shapes depending on time periods, frequency bands and electrodes. Second, based on the results, we explored learnable input data sets. Then, learnability for discriminating the shapes were shown in EEG waveforms on 100ms time periods in single trials at all channels and phase of time frequency analysis on the limited time-frequency domain and electrodes. Finally, we investigated whether two discriminators using LSTM and CNN discriminate the shapes from the learnable data for each subject. Then, it was found that accuracies of discrimination of the shapes were over a chance level with all the learnable data sets, subjects, and discriminators. We concluded that the distinct shapes can be discriminated from EEG signals by exploring appropriate features of input signals for discriminators.

Inter-individual deep image reconstruction via hierarchical neural code conversion

The sensory cortex is characterized by general organizational principles such as topography and hierarchy. However, measured brain activity given identical input exhibits substantially different patterns across individuals. Although anatomical and functional alignment methods have been proposed in functional magnetic resonance imaging (fMRI) studies, it remains unclear whether and how hierarchical and fine-grained representations can be converted between individuals while preserving the encoded perceptual content. In this study, we trained a method of functional alignment called neural code converter that predicts a target subject’s brain activity pattern from a source subject given the same stimulus, and analyzed the converted patterns by decoding hierarchical visual features and reconstructing perceived images. The converters were trained on fMRI responses to identical sets of natural images presented to pairs of individuals, using the voxels on the visual cortex that covers from V1 through the ventral object areas without explicit labels of the visual areas. We decoded the converted brain activity patterns into the hierarchical visual features of a deep neural network using decoders pre-trained on the target subject and then reconstructed images via the decoded features. Without explicit information about the visual cortical hierarchy, the converters automatically learned the correspondence between visual areas of the same levels. Deep neural network feature decoding at each layer showed higher decoding accuracies from corresponding levels of visual areas, indicating that hierarchical representations were preserved after conversion. The visual images were reconstructed with recognizable silhouettes of objects even with relatively small numbers of data for converter training. The decoders trained on pooled data from multiple individuals through conversions led to a slight improvement over those trained on a single individual. These results demonstrate that the hierarchical and fine-grained representation can be converted by functional alignment, while preserving sufficient visual information to enable inter-individual visual image reconstruction.

Illumination modelling for reconstructing the machined surface topography

Illumination estimation is crucial for surface detection methods based on visual image reconstruction. The luminance of the machined surface topography with desertification fractal structure is difficult to accurately describe using existing illumination models, which depend on the macrosurface geometric features and reflection characteristics. Therefore, machined surface topography illumination model under coaxial light microscopic vision is proposed in this paper for the online detection of complex surfaces and special functional surface topography in the manufacturing field. Based on light scattering theory, the luminance of the surface topography is studied under the coaxial light microscopic vision; the influencing factors of luminance of surface topography are revealed and the illumination model of machined surface topography is established. The experimental results show that less calculation error occurs when using the proposed illumination model, which can be used to more accurately describe the luminance of machined surface topography. The research results will lay a foundation for online detection and monitoring of machined surfaces.

Functional Alignment-Auxiliary Generative Adversarial Network-Based Visual Stimuli Reconstruction via Multi-Subject fMRI.

Functional Magnetic Resonance Imaging (fMRI) provides more precise spatial and temporal information to reconstruct stimulus images than other technologies that can be used to measure the human brain's neural responses. The fMRI scans, however, generally show heterogeneity among different subjects. The majority of the existing methods aim primarily at mining correlations between stimuli and evoked brain activity, disregarding the heterogeneity among subjects. Therefore, this heterogeneity will impair the reliability and applicability of multi-subject decoding results, leading to sub-optimal results. The present paper proposes the functional alignment-auxiliary generative adversarial network (FAA-GAN) as a novel multi-subject approach for visual image reconstruction that employs functional alignment to alleviate the heterogeneity between subjects. Our proposed FAA-GAN includes three key components: 1) a generative adversarial network (GAN) module for reconstructing visual stimuli, which consists of a visual image encoder as the generator that uses a nonlinear network to convert stimuli images into an implicit representation and a discriminator that generates the images comparable to the original images in detail; 2) a multi-subject functional alignment module, which is used to precisely align the individual fMRI response space of each subject in a common space to reduce the heterogeneity among different subjects; and 3) a cross-modal hashing retrieval module used for similarity retrieval of two modalities of data, i.e., the visual images and the evoked brain responses. Experiments on real-world datasets show that our FAA-GAN method does better than other state-of-the-art deep learning-based reconstruction methods with fMRI.

Images Reconstruction from Functional Magnetic Resonance Imaging Patterns Based on the Improved Deep Generative Multiview Model

Reconstructing visual stimulus images from the brain activity signals is an important research task in the field of brain decoding. Many methods of reconstructing visual stimulus images mainly focus on how to use deep learning to classify the brain activities measured by functional magnetic resonance imaging or identify visual stimulus images. Accurate reconstruction of visual stimulus images by using deep learning still remains challenging. This paper proposes an improved deep generative multiview model to further promote the accuracy of reconstructing visual stimulus images. Firstly, an encoder based on residual-in-residual dense blocks is designed to fit the deep and multiview visual features of human natural state, and extract the features of visual stimulus images. Secondly, the structure of original decoder is extended to a deeper network in the deep generative multiview model, which makes the features obtained by each deconvolution layer more distinguishable. Finally, we configure the parameters of the optimizer and compare the performance of various optimizers under different parameter values, and then the one with the best performance is chosen and adopted to the whole model. The performance evaluations conducted on two publicly available datasets demonstrate that the improved model has more accurate reconstruction effectiveness than the original deep generative multiview model.

Visual Saliency and Image Reconstruction from EEG Signals via an Effective Geometric Deep Network-Based Generative Adversarial Network

Reaching out the function of the brain in perceiving input data from the outside world is one of the great targets of neuroscience. Neural decoding helps us to model the connection between brain activities and the visual stimulation. The reconstruction of images from brain activity can be achieved through this modelling. Recent studies have shown that brain activity is impressed by visual saliency, the important parts of an image stimuli. In this paper, a deep model is proposed to reconstruct the image stimuli from electroencephalogram (EEG) recordings via visual saliency. To this end, the proposed geometric deep network-based generative adversarial network (GDN-GAN) is trained to map the EEG signals to the visual saliency maps corresponding to each image. The first part of the proposed GDN-GAN consists of Chebyshev graph convolutional layers. The input of the GDN part of the proposed network is the functional connectivity-based graph representation of the EEG channels. The output of the GDN is imposed to the GAN part of the proposed network to reconstruct the image saliency. The proposed GDN-GAN is trained using the Google Colaboratory Pro platform. The saliency metrics validate the viability and efficiency of the proposed saliency reconstruction network. The weights of the trained network are used as initial weights to reconstruct the grayscale image stimuli. The proposed network realizes the image reconstruction from EEG signals.

3D Modeling System of Indoor Environment Art Landscape Design under Statistical Equation

Abstract This paper establishes a three-dimensional virtual landscape model based on statistical equations. This paper completes the analysis of interior design requirements by analyzing the structure of the interior design platform and the indoor positioning algorithm from the three-dimensional virtual perspective. We realize the 3D reconstruction of indoor landscape through the 3D visual image reconstruction module and VRLM browser module. The experimental results show that the designed 3D virtual landscape is more realistic. Its artistic effect is better.

Adaptive Event Address Map Denoising for Event Cameras

Event cameras provide high temporal resolution and high dynamic range and have application prospects in areas such as visual navigation, robotics, and image reconstruction. Event cameras output an asynchronous stream of address events that can be highly polluted by noise due to lighting and motion conditions. In order to better analyze and process noise with different spatiotemporal correlation, we introduce a method using an event address map (EAM) representation and classify event noise into background noise, near-edge noise, and recessive noise. In addition, since near-edge noise and recessive noise have a closer spatiotemporal correlation with real events, they are difficult to suppress with current denoising methods. Accordingly, we propose an adaptive EAM denoising method by adjusting the time window to form the EAM by measuring the number of events after pre-denoising and performing median filtering and opening. The method can better adapt to scenes of different motions to process near-edge noise, and can generate events to fill recessive noise. The output EAM can be directly used for subsequent process. Alternatively, to provide high temporal resolution, we restore the stream of address events from the output EAM. To suitably evaluate event denoising performance on real-world data, we introduce the average local event variance and amount of recessive noise, and the proposed method achieves improvements of 24.94% and 72.79%, respectively, compared with conventional denoising.

An ultra-high-speed hardware accelerator for image reconstruction and stereo rectification on event-based camera

Event-based cameras are novel bio-inspired vision sensors, which sense brightness changes rather than the actual intensity level. In contrast to conventional cameras, such cameras capture new information about the scene in the form of sparse events at a very short latency. Recently, some researchers have studied on efficient event-based camera reconstruction approaches to obtain high-quality images. These efforts make performing stereo vision based on event-cameras being possible. However, traditional stereo architecture cannot process such high-speed image streams generated by the event-based camera in real-time (in the order of μs), which calls for new approaches.In this paper, we provide a set of practical solutions on visual image reconstruction and stereo rectification for spike camera, which serves two important procedures of stereo vision. We first provide FPGA-accelerated hardware architecture to achieve ultra-high-speed visual image reconstruction with full texture of natural scenes from spike data. Then, an FPGA-accelerated ultra-high-speed stereo rectification architecture is proposed to rectify the reconstructed images generated for stereo vision. In this architecture, a fully pipelined calculation module is designed to process the complex coordinate transformation operations, which puts in pipelined registers to maximize the clock frequency. To further improve the throughput, we propose a parallel processing architecture that uses multiple processing elements (PEs) to process multiple pixels per cycle. In addition, we design a memory management unit (MMU) to optimize the memory usage of the hardware resource. The whole architecture is effectively implemented on a Xilinx Zynq7100 FPGA chip. We evaluate the proposed architecture with different settings. The experiments show that our architecture can process the spike streams in real-time manner.

Real-time, wide-field and high-quality single snapshot imaging of optical properties with profile correction using deep learning.

The development of real-time, wide-field and quantitative diffuse optical imaging methods to visualize functional and structural biomarkers of living tissues is a pressing need for numerous clinical applications including image-guided surgery. In this context, Spatial Frequency Domain Imaging (SFDI) is an attractive method allowing for the fast estimation of optical properties using the Single Snapshot of Optical Properties (SSOP) approach. Herein, we present a novel implementation of SSOP based on a combination of deep learning network at the filtering stage and Graphics Processing Units (GPU) capable of simultaneous high visual quality image reconstruction, surface profile correction and accurate optical property (OP) extraction in real-time across large fields of view. In the most optimal implementation, the presented methodology demonstrates megapixel profile-corrected OP imaging with results comparable to that of profile-corrected SFDI, with a processing time of 18 ms and errors relative to SFDI method less than 10% in both profilometry and profile-corrected OPs. This novel processing framework lays the foundation for real-time multispectral quantitative diffuse optical imaging for surgical guidance and healthcare applications. All code and data used for this work is publicly available at www.healthphotonics.org under the resources tab.