Human Visual Processing Research Articles

Multi-Sensor Image Fusion: A Survey of the State of the Art

Image fusion has been developing into an important area of research. In remote sensing, the use of the same image sensor in different working modes, or different image sensors, can provide reinforcing or complementary information. Therefore, it is highly valuable to fuse outputs from multiple sensors (or the same sensor in different working modes) to improve the overall performance of the remote images, which are very useful for human visual perception and image processing task. Accordingly, in this paper, we first provide a comprehensive survey of the state of the art of multi-sensor image fusion methods in terms of three aspects: pixel-level fusion, feature-level fusion and decision-level fusion. An overview of existing fusion strategies is then introduced, after which the existing fusion quality measures are summarized. Finally, this review analyzes the development trends in fusion algorithms that may attract researchers to further explore the research in this field.

The effect of flanker category on crowding is modulated by processing time

When the human visual system processes complex scenes, crowding is considered a bottleneck in interpreting visual information. Several studies have reported that visual crowding seems to occur in the early stage of visual possessing. After that, extensive evidence suggested that high-level visual information has a major effect on crowding, which showed substantial interest in the impact of high-level visual information on crowding. However, the temporal property of high-level crowding is not well understood. Here, we performed a series of behavioural experiments to show how flanker category information and exposure time affected crowding's intensity. We found that as the category's effect becomes stronger, the intensity of crowding will be reduced in longer exposure time. The peak of this effect occurs at the time window of 100 ms-150 ms after the stimulus appears. We propose that exposure time may modulate high-level information processing depth, which can adjust multi-level crowding.

Knowledge-guided semantic computing network

The excellent performance of deep neural networks mainly relies on the attributes of the dataset, such as the size, diversity, completeness. However, it is usually difficult to obtain a high-qualified training dataset for some scenarios. Inspired by the human visual cognition process with few sample learning, and strong robustness, we believe the experience knowledge is more powerful than large-scale data. To combine the power of knowledge and data, we propose a knowledge-guided semantic computing network (SCN) in this paper, which is constructed with a primary knowledge-guided semantic tree module and an auxiliary data-driven lightweight neural network module. The semantic tree module can calculate the classification results by a forward computing process rapidly. The lightweight neural network module can aid the semantic tree module for higher classification ability. We also propose a hinge cross-entropy loss function to train the SCN, which enables the SCN to focus on those misclassified training samples and further improve the classification accuracy. The experimental results on MNIST and GTSRB data sets prove that the SCN achieves excellent classification accuracy comparable to the state-of-the-art methods on the original training samples and higher classification accuracy than the state-of-the-art methods on few training samples. What is more, at BIM eps = 0.3 on MNIST and FGSM eps = 0.03 on GTSRB adversarial test samples, the proposed SCN(1/4) and SCN(1/8) obtain over 75% and 14% accuracy improvement than the original CapsNet.

Dense Attention Convolutional Network for Image Classification

Convolutional neural networks (CNN) have made rapid progress for a series of visual tasks, but the bottom-up convolutional feature extraction process is not qualified to mimic the human visual perception process, which takes advantages of discriminative features. Although attention modules extract the top-down discriminative features and have been widely investigated in the past few years, current attention modules interrupt the bottom-up convolutional feature extraction process. To tackle this challenge, in this paper, we introduce dense connection structure to fuse discriminative features from attention modules and convolutional features, which we term as dense attention learning. Also, to alleviate the over-fitting problem caused by rapid feature dimension growth, we propose a channel-wise attention module to compress and refine the convolutional features. Based on these strategies, we build a dense attention convolutional neural network (DA-CNN) for visual recognition. Exhaustive experiments on four challenging datasets including CIFAR-10, CIFAR-100, SVHN and ImageNet demonstrate that our DA-CNN outperforms many state-of-the-art methods. Moreover, the effectiveness of our dense attention learning and channel-wise attention module is also validated.

Optimal templates for signal extraction by noisy ideal detectors and human observers.

The optimal template for signal detection in white additive noise is the signal itself: the ideal observer matches each stimulus against this template and selects the stimulus associated with largest match. In the noisy ideal observer, internal noise is added to the decision variable returned by the template. While the ideal observer represents an unrealistic approximation to the human visual process, the noisy ideal observer may be applicable under certain experimental conditions. For template values constrained to lie within a specified range, theory predicts that the template associated with a noisy ideal observer should be a clipped image of the signal, a result which we demonstrate analytically using variational calculus. It is currently unknown whether the human process conforms to theory. We report a targeted analysis of the theoretical prediction for an experimental protocol that maximizes template-matching on the part of human participants. We find indicative evidence to support the theoretical expectation when internal noise is compared across participants, but not within each participant. Our results indicate that implicit knowledge about internal variability in different individuals is reflected by their detection templates; no implicit knowledge is retained for internal-noise fluctuations experienced by a given participant during data collection. The results also indicate that template encoding is constrained by the dynamic range of weight specification, rather than the range of output values transduced by the template-matching process.

The Spatiotemporal Power Spectrum of Natural Human Vision

When engaging in natural tasks, the human visual system processes a highly dynamic visual data stream. The retina, performing the very first steps in this processing, is thought to be adapted to take advantage of low-level signal regularities, such as the autocorrelation function or power spectrum, to produce a more efficient encoding of the data (Atick & Redlich, 1992). Previous work examined the joint spatio-temporal power spectrum of handheld camera videos and Hollywood movies, showing that power falls as an inverse power-law function of spatial and temporal frequency, with an inseparable relationship (Dong & Attick, 1995). However these data are far from a true characterization of a day in the life of the retina due to body, head and eye motion. In addition, the distribution of natural tasks will influence the statistics of this signal. Here, we aim to characterize these statistics of natural vision using a custom device that consists of a head-mounted eye tracker coupled with high frame-rate world cameras and orientation sensors. Using video data captured from this setup, we analyze the joint spatiotemporal power spectrum for three conditions: 1) a static camera viewing a natural task being performed, 2) a head mounted camera worn by a subject engaged in a natural task, and 3) videos simulating the dynamic retinal image, created by overlaying the subject's eye motions on the head-mounted camera video stream. Results suggest that compared to a static camera, body and head motion have the effect of boosting high temporal frequencies. Eye motion enhances this effect, particularly for mid to high spatial frequencies, causing this portion to deviate from the power law and become nearly flat. These data will be important for developing efficient coding models relevant to natural vision.

Deep Neural Networks as a computational model for early vision: Lateral masking and contour integration

Background: The deep neural network (DNN) models developed for image classification have been recently suggested as biologically inspired models for different brain functions. Here we apply a standard visual DNN model to explore early-vision mechanisms of integration, as reflected in the psychophysical phenomena of lateral masking, contrast summation, and contour integration. Methods: We used the standard ImageNet-trained VGG network model. The model correlate of perceptual distance was the L2 distance between the average response to images corresponding to different stimuli conditions. Higher values of the metric (indicating larger changes in the DNN representation) correspond to better discrimination. Results: For lateral masking, the model produced a close match to the basic behavioral data (Polat & Sagi 1993), with a facilitation of ~0.4 log units at 2.5 wave-length distance, as well as: (1) inhibition for very close flankers, (2) no facilitation for orthogonal flankers and decreased facilitation with deviation from collinearity, (3) more facilitation of a vertical configuration compared to horizontal and oblique, (4) scaling of the effects with wave-length, and more. These results were obtained with a growing facilitation and longer interaction range from the mid layers of the model and up, indicating hierarchical integration that possibly substitutes for the assumed lateral interactions in V1. For contrast summation we replicated the configuration dependent (smooth but not jagged) summation (response vs number of patches along a Gabor contour) power-law (Bonneh and Sagi 1998). For Gabor contour detection in noise (Field et. al. 1993) the model showed the known effects of spacing, contour smoothness, and scaling. For noise detection in natural images (Alam et al. 2014), perceptual thresholds were strongly correlated with model predictions (R=0.78, N=1080 images). Conclusions: These findings demonstrate effortless replication in a DNN of classic findings concerning early human visual processing, suggesting convergent evolution of biological and artificial vision.

Study on Representation Invariances of CNNs and Human Visual Information Processing Based on Data Augmentation.

Representation invariance plays a significant role in the performance of deep convolutional neural networks (CNNs) and human visual information processing in various complicated image-based tasks. However, there has been abounding confusion concerning the representation invariance mechanisms of the two sophisticated systems. To investigate their relationship under common conditions, we proposed a representation invariance analysis approach based on data augmentation technology. Firstly, the original image library was expanded by data augmentation. The representation invariances of CNNs and the ventral visual stream were then studied by comparing the similarities of the corresponding layer features of CNNs and the prediction performance of visual encoding models based on functional magnetic resonance imaging (fMRI) before and after data augmentation. Our experimental results suggest that the architecture of CNNs, combinations of convolutional and fully-connected layers, developed representation invariance of CNNs. Remarkably, we found representation invariance belongs to all successive stages of the ventral visual stream. Hence, the internal correlation between CNNs and the human visual system in representation invariance was revealed. Our study promotes the advancement of invariant representation of computer vision and deeper comprehension of the representation invariance mechanism of human visual information processing.

Depth in convolutional neural networks solves scene segmentation.

Feed-forward deep convolutional neural networks (DCNNs) are, under specific conditions, matching and even surpassing human performance in object recognition in natural scenes. This performance suggests that the analysis of a loose collection of image features could support the recognition of natural object categories, without dedicated systems to solve specific visual subtasks. Research in humans however suggests that while feedforward activity may suffice for sparse scenes with isolated objects, additional visual operations ('routines') that aid the recognition process (e.g. segmentation or grouping) are needed for more complex scenes. Linking human visual processing to performance of DCNNs with increasing depth, we here explored if, how, and when object information is differentiated from the backgrounds they appear on. To this end, we controlled the information in both objects and backgrounds, as well as the relationship between them by adding noise, manipulating background congruence and systematically occluding parts of the image. Results indicate that with an increase in network depth, there is an increase in the distinction between object- and background information. For more shallow networks, results indicated a benefit of training on segmented objects. Overall, these results indicate that, de facto, scene segmentation can be performed by a network of sufficient depth. We conclude that the human brain could perform scene segmentation in the context of object identification without an explicit mechanism, by selecting or "binding" features that belong to the object and ignoring other features, in a manner similar to a very deep convolutional neural network.

Variational Models for Color Image Correction Inspired by Visual Perception and Neuroscience

Reproducing the perception of a real-world scene on a display device is a very challenging task which requires the understanding of the camera processing pipeline, the display process, and the way the human visual system processes the light it captures. Mathematical models based on psychophysical and physiological laws on color vision, named Retinex, provide efficient tools to handle degradations produced during the camera processing pipeline like the reduction of the contrast. In particular, Batard and Bertalmío (in J Math Imaging Vis 60(6):849–881, 2018) described some psychophysical laws on brightness perception as covariant derivatives, included them into a variational model, and observed that the quality of the color image correction is correlated with the accuracy of the vision model it includes. Based on this observation, we postulate that this model can be improved by including more accurate data on vision with a special attention on visual neuroscience here. Then, inspired by the presence of neurons responding to different visual attributes in the area V1 of the visual cortex as orientation, color or movement, to name a few, and horizontal connections modeling the interactions between those neurons, we construct two variational models to process both local (edges, textures) and global (contrast) features. This is an improvement with respect to the model of Batard and Bertalmío as the latter cannot process local and global features independently and simultaneously. Finally, we conduct experiments on color images which corroborate the improvement provided by the new models.

Proposal for human visual pathway in the extrastriate cortex by fiber tracking method using diffusion-weighted MRI

The extrastriate cortex in the human visual cortex is divided into two distinct clusters: the “what-information” processing area and the “where-information” processing area. It is widely accepted that the “what-information” cluster is processed through the ventral stream to the temporal cortex, and the “where-information” cluster through the dorsal stream to the parietal cortex. In human neuroanatomy, fiber bundles for the ventral stream (such as the inferior longitudinal fasciculus) are well defined, whereas fibers for the dorsal stream are poorly understood. In this study, we attempted to trace the dorsal stream fibers using a fiber tracking method using 7.0T diffusion-weighted MRI. We used data from a healthy male subject as well as from an unbiasedly selected nine-subject dataset in the Human Connectome Project. The surface of the visual area, including V1, V2, V3, V4, MT, was determined from the Brainnetome atlas (Fan et al., 2016), which is the connectivity-based parcellation framework of the human brain. The resulting visual pathway indicated that the putative pathway for the classical dorsal stream is unlikely to exist. Instead, we demonstrated that fiber connections exist between the angular gyrus with MT in the visual cortex, and between the angular gyrus and IT in the temporal cortex. Through that, we composed a two-pathway model for where-information processing that passes through the angular gyrus. Finally, we proposed a modified human visual pathway model based on our fiber tracking results in this report. The modified where-information pathway will provide a new aspect for the study of human visual processing.

MLGaze: Machine Learning-Based Analysis of Gaze Error Patterns in Consumer Eye Tracking Systems.

Analyzing the gaze accuracy characteristics of an eye tracker is a critical task as its gaze data is frequently affected by non-ideal operating conditions in various consumer eye tracking applications. In previous research on pattern analysis of gaze data, efforts were made to model human visual behaviors and cognitive processes. What remains relatively unexplored are questions related to identifying gaze error sources as well as quantifying and modeling their impacts on the data quality of eye trackers. In this study, gaze error patterns produced by a commercial eye tracking device were studied with the help of machine learning algorithms, such as classifiers and regression models. Gaze data were collected from a group of participants under multiple conditions that commonly affect eye trackers operating on desktop and handheld platforms. These conditions (referred here as error sources) include user distance, head pose, and eye-tracker pose variations, and the collected gaze data were used to train the classifier and regression models. It was seen that while the impact of the different error sources on gaze data characteristics were nearly impossible to distinguish by visual inspection or from data statistics, machine learning models were successful in identifying the impact of the different error sources and predicting the variability in gaze error levels due to these conditions. The objective of this study was to investigate the efficacy of machine learning methods towards the detection and prediction of gaze error patterns, which would enable an in-depth understanding of the data quality and reliability of eye trackers under unconstrained operating conditions. Coding resources for all the machine learning methods adopted in this study were included in an open repository named MLGaze to allow researchers to replicate the principles presented here using data from their own eye trackers.

An Efficient Method for Infrared and Visual Images Fusion Based on Visual Attention Technique

Infrared and visible image fusion technology provides many benefits for human vision and computer image processing tasks, including enriched useful information and enhanced surveillance capabilities. However, existing fusion algorithms have faced a great challenge to effectively integrate visual features from complex source images. In this paper, we design a novel infrared and visible image fusion algorithm based on visual attention technology, in which a special visual attention system and a feature fusion strategy based on the saliency maps are proposed. Special visual attention system first utilizes the co-occurrence matrix to calculate the image texture complication, which can select a particular modality to compute a saliency map. Moreover, we improved the iterative operator of the original visual attention model (VAM), a fair competition mechanism is designed to ensure that the visual feature in detail regions can be extracted accurately. For the feature fusion strategy, we use the obtained saliency map to combine the visual attention features, and appropriately enhance the tiny features to ensure that the weak targets can be observed. Different from the general fusion algorithm, the proposed algorithm not only preserve the interesting region but also contain rich tiny details, which can improve the visual ability of human and computer. Moreover, experimental results in complicated ambient conditions show that the proposed algorithm in this paper outperforms state-of-the-art algorithms in both qualitative and quantitative evaluations, and this study can extend to the field of other-type image fusion.

Low frequency transcranial magnetic stimulation of right posterior parietal cortex reduces reaction time to perithreshold low spatial frequency visual\xa0stimuli

Research in humans and animal models suggests that visual responses in early visual cortical areas may be modulated by top-down influences from distant cortical areas, particularly in the frontal and parietal regions. The right posterior parietal cortex is part of a broad cortical network involved in aspects of visual search and attention, but its role in modulating activity in early visual cortical areas is less well understood. This study evaluated the influence of right posterior parietal cortex (PPC) on a direct measure of visual processing in humans. Contrast sensitivity (CS) and detection response times were recorded using a visual detection paradigm to two types of centrally-presented stimuli. Participants were tested on the detection task before, after, and 1 hour after low-frequency repetitive transcranial magnetic stimulation (rTMS) to the right PPC or to the scalp vertex. Low-frequency rTMS to the right PPC did not significantly change measures of contrast sensitivity, but increased the speed at which participants responded to visual stimuli of low spatial frequency. Response times returned to baseline 1-hour after rTMS. These data indicate that low frequency rTMS to the right PPC speeds up aspects of early visual processing, likely due to a disinhibition of the homotopic left posterior parietal cortex.

Unveiling the Secrets of Escher's Lithographs.

An impossible structure gives us the impression of looking at a three-dimensional object, even though this object cannot exist, since it possesses parts that are spatially non-connectable, and are characterized by misleading geometrical properties not instantly evident. Therefore, impossible artworks appeal to our intellect and challenge our perceptive capacities. We analyzed lithographs containing impossible structures (e.g., the Necker cube), created by the famous Dutch painter Maurits Cornelis Escher (1898–1972), and used one of them (The Belvedere, 1958) to unveil the artist’s hidden secrets by means of a discrete model of the human retina based on a non-uniform distribution of receptive fields. We demonstrated that the ability of Escher in composing his lithographs by connecting spatial coherent details into an impossible whole lies in drawing these incoherent fragments just outside the zone in which 3D coherence can be perceived during a single fixation pause. The main aspects of our paper from the point of view of image processing and image understanding are the following: (1) the peculiar and original digital filter to process the image, which simulates the human vision process, by producing a space-variant sampling of the image; (2) the software for the filter, which is homemade and created for our purposes. The filtered images resulting from the processing are used to understand impossible figures. As an example, we demonstrate how the impossible figures hidden in Escher’s paintings can be understood.

Spatial attention enhances cortical tracking of quasi-rhythmic visual stimuli

Successfully interpreting and navigating our natural visual environment requires us to track its dynamics constantly. Additionally, we focus our attention on behaviorally relevant stimuli to enhance their neural processing. Little is known, however, about how sustained attention affects the ongoing tracking of stimuli with rich natural temporal dynamics. Here, we used MRI-informed source reconstructions of magnetoencephalography (MEG) data to map to what extent various cortical areas track concurrent continuous quasi-rhythmic visual stimulation. Further, we tested how top-down visuo-spatial attention influences this tracking process. Our bilaterally presented quasi-rhythmic stimuli covered a dynamic range of 4–20 ​Hz, subdivided into three distinct bands. As an experimental control, we also included strictly rhythmic stimulation (10 vs 12 ​Hz). Using a spectral measure of brain-stimulus coupling, we were able to track the neural processing of left vs. right stimuli independently, even while fluctuating within the same frequency range. The fidelity of neural tracking depended on the stimulation frequencies, decreasing for higher frequency bands. Both attended and non-attended stimuli were tracked beyond early visual cortices, in ventral and dorsal streams depending on the stimulus frequency. In general, tracking improved with the deployment of visuo-spatial attention to the stimulus location. Our results provide new insights into how human visual cortices process concurrent dynamic stimuli and provide a potential mechanism – namely increasing the temporal precision of tracking – for boosting the neural representation of attended input.

Time series data interpretation for ‘wheel-flat’ identification including uncertainties

Train wheel flats are formed when wheels slip on rails. Crucial for passenger comfort and the safe operation of train systems, early detection and quantification of wheel-flat severity without interrupting railway operations is a desirable and challenging goal. Our method involves identifying the wheel-flat size by using a model updating strategy based on dynamic measurements. Although measurement and modelling uncertainties influence the identification results, they are rarely taken into account in most wheel-flat detection methods. Another challenge is the interpretation of time series data from multiple sensors. In this article, the size of the wheel flat is identified using a model-falsification approach that explicitly includes uncertainties in both measurement and modelling. A two-step important point selection method is proposed to interpret high-dimensional time series in the context of inverse identification. Perceptually important points, which are consistent with the human visual identification process, are extracted and further selected using joint entropy as an information gain metric. The proposed model-based methodology is applied to a field train track test in Singapore. The results show that the wheel-flat size identified using the proposed methodology is within the range of true observations. In addition, it is also shown that the inclusion of measurement and modelling uncertainties is essential to accurately evaluate the wheel-flat size because identification without uncertainties may lead to an underestimation of the wheel-flat size.

Unraveling nonlinear electrophysiologic processes in the human visual system with full dimension spectral analysis

Natural sensory signals have nonlinear structures dynamically composed of the carrier frequencies and the variation of the amplitude (i.e., envelope). How the human brain processes the envelope information is still poorly understood, largely due to the conventional analysis failing to quantify it directly. Here, we used a recently developed method, Holo-Hilbert spectral analysis, and steady-state visually evoked potential collected using electroencephalography (EEG) recordings to investigate how the human visual system processes the envelope of amplitude-modulated signals, in this case with a 14 Hz carrier and a 2 Hz envelope. The EEG results demonstrated that in addition to the fundamental stimulus frequencies, 4 Hz amplitude modulation residing in 14 Hz carrier and a broad range of carrier frequencies covering from 8 to 32 Hz modulated by 2 Hz amplitude modulation are also found in the two-dimensional frequency spectrum, which have not yet been recognized before. The envelope of the stimulus is also found to dominantly modulate the response to the incoming signal. The findings thus reveal that the electrophysiological response to amplitude-modulated stimuli is more complex than could be revealed by, for example, Fourier analysis. This highlights the dynamics of neural processes in the visual system.

Human visual processing for spatial 3-D imaging recognition: Cortical connectivity deficits in Parkinson's spatial perceptual visual pathway

Introduction Parkinson's disease (PD) is a Neurodegenerative disorder traditionally linked to motor impairment but recent findings have revealed, patients suffering from PD exhibit decline in visual-spatial skills [1] [2]. The aim of this study is to explore, neural structural connectivity between the superior parietal lobe (BA 5) and the primary visual cortex (BA 17, 18 & 19) in PD patients. Generally, this pathway modulates visual and spatial domains, hence if found to be impaired in PD patients could explain the dysfunction in visual-spatial skills. Methods In this research, we attempted to correlate the dorsal stream visual pathway in Parkinson's Patients. The study was carried out through ;“Diffusion Imaging Fibre Tractography;” which involves 120 DTI data sets from control and PD patients between 60 and 120 years with IRB approval. Observations It was observed that fibers in the females of the control group were higher (Fig 1) when compared to that of the males (Fig 2), and on completion, males displayed more prominent changes in numbers and volumes of where stream pathway;”(Fig 3). In males both right and left hemisphere were affected, whereas with female much difference is not noticed. The results are statistically significant at P Results Based on our observations, destruction in the where stream pathway were identified, and is believed as an underlying cause for deficiency in visual spatial recognition. However, for better understanding of this findings, functional and effective connectivity analysis must be recommended.

Color Constancy Algorithms Related to Human Visual Cognitive Process

Based on the cognition of the physiological structure of the human visual system, this paper considers that the mechanism of the human visual system to perceive the image color appearance includes the adaptive mechanism of the retinal photoreceptor to the ambient light and the spatial frequency response mechanism of the Neuronal receptive field in the visual pathway. In this paper, we first provide the computing framework of the image color appearance model related to human cognitive process, then propose using Gabor wavelet as the basis function of the visual nerve cells response to apply CIECAT02 model to the calculation of image color adaption and to simulate the multi-scale superposition of human visual spatial frequency tuning curves, and finally accomplish the development of the algorithm for predicting image color appearance. The results show that the prediction algorithm proposed in this paper is closer to the visual perception of human eyes than the similar algorithm.