Mechanism Of Human Visual System Research Articles

Bidirectional Feature Aggregation Network for Stereo Image Quality Assessment Considering Parallax Attention-Based Binocular Fusion

Inspired by the two-path visual information processing mechanism (i.e., a bottom-up path and a top-down path), we propose a bidirectional binocular feature aggregation based stereo image quality assessment (SIQA) network, which considers a two-path visual mechanism and realizes the binocular fusion based on parallax information. To better aggregate binocular features from different levels, a two-path feature aggregation structure, which simulates the bottom-up and top-down mechanism in human visual system (HVS), is proposed. It not only realizes the supplement of low-level detail information to high-level semantic in the bottom-up path, but also realizes the supplement of high-level semantic information to low-level detail in the top-down path. Simultaneously, because feature misalignment exists in binocular features of adjacent levels, a feature alignment module (FAM) based on deformable convolution is designed to integrate the binocular fusion features of adjacent levels. In addition, considering the importance role of parallax in guiding binocular fusion, a binocular fusion module (BFM) based on parallax attention mechanism, which is different with existing binocular fusion methods, is explicitly proposed to achieve the binocular fusion between the left and right view features. Extensive experiments are conducted on LIVE I, LIVE II, WIVC I and WIVC II databases to demonstrate the effectiveness of the proposed method.

Towards reliable object representation via sparse directional patches and spatial center cues

In the process of image understanding, the human visual system (HVS) performs multiscale analysis on various objects. HVS primarily focuses on marginally conspicuous image patches located within or around distinct objects rather than scanning the image pixels point by point. Inspired by the HVS mechanism, in this paper, we aimed to describe and exploit multiscale decomposition-based patch detection models for automatic visual feature representation and object localization in images. Our investigation into mimicking and modeling the HVS to capture conspicuous sparse patches and their spatial distribution clues makes a profound contribution to the automatic comprehension and characterization of images by machines. This study demonstrates that the sparse patch-based visual representation with spatial center cues is intrinsically tolerant to object positioning and understanding beyond object variations in spatial position, multiresolution, and chrominance, which has significant implications for many vision-based automatic object grabbing and perception applications, such as robotics, human‒machine interaction, and unmanned aerial vehicles (UAVs).

Prediction of motion sickness degree of stereoscopic panoramic videos based on content perception and binocular characteristics

The immersive viewing environment of Virtual Reality (VR) system inevitably brings motion sickness to users, and restricts the development of VR system to a certain extent. In order to solve the problem of motion sickness when users view, it is necessary to predicate the motion sickness degree of the visual content. However, the methods based on subjective questionnaires or physiological signals will become cumbersome and infeasible with the growth of demand and supply of VR content. Most of the existing prediction methods for motion sickness are designed for 2D panoramic video. In this paper, a motion sickness prediction method based on content perception and binocular characteristics is proposed for stereoscopic panoramic video. The proposed method mainly consists of two modules: feature extraction based on content analysis and time pooling. The feature extraction module takes into account the attention mechanism of human visual system and the multi-channel characteristics of retina, and simulates the sensory conflict and the three-stage process of binocular stereoscopic perception in virtual environment. The time pooling module takes into account the time memory effect of human eyes. The proposed method has achieved excellent prediction performance on stereoscopic panoramic video comfort database SPVCD, its predication results have a good correlation with the subjective scores, which shows that the proposed method can effectively predict the motion sickness degree of VR content.

Extraction and Data Processing of Freestyle Skiing Based on Motion Biomechanics

Objectives: Based on the principle of motion biomechanics, this paper studied the extraction and data processing of freestyle ski signals. Firstly, the algorithm of SLIC depth information and movement information was described in detail, and then the test and data collection methods were explained. Methods: Secondly, the algorithm was tested by this algorithm and traditional algorithm in the data set on the results of the comparison, the superiority of this algorithm was verified. Results: and starting from the characteristics of the underlying information of scene, consider in significant areas of the visual stimulus level. Conclusion: but at the same time the attention mechanism of human visual system was affected by the high-level features.

Saliency Maps-Based Convolutional Neural Networks for Facial Expression Recognition

Facial expression recognition (FER) is one of the important research contents in affective computing. It plays a key role in many application fields of human life. As a most common expression feature extraction method, the convolutional neural network (CNN) has the following main limitation. Due to the fact that the CNN network lacks the visual attention guidance, when it gets expression information it brings background noises, resulting in the lower recognition accuracy. In order to simulate the attention mechanism in human visual system, a salient feature extraction model is proposed, including the dilated inception module, the Difference of Gaussian (DOG) module, and the multi-indicator saliency prediction module. This model can effectively reflect the key facial information through the increase of the receptive field, the acquisition of multiscale features, and the simulation of human vision. In addition, a novel FER method for one single person is proposed. With the prior knowledge of saliency maps and the multilayer deep features in the CNN network, the recognition accuracy is improved by obtaining more targeted and more complete deep expression information. The experimental results of saliency prediction, action unit (AU) detection, and smile intensity estimation on the CAT2000, the CK+, and the BP4D databases prove that the proposed method improves the FER performance and is more effective than the existing approaches.

Blind Image Quality Assessment With Active Inference.

Blind image quality assessment (BIQA) is a useful but challenging task. It is a promising idea to design BIQA methods by mimicking the working mechanism of human visual system (HVS). The internal generative mechanism (IGM) indicates that the HVS actively infers the primary content (i.e., meaningful information) of an image for better understanding. Inspired by that, this paper presents a novel BIQA metric by mimicking the active inference process of IGM. Firstly, an active inference module based on the generative adversarial network (GAN) is established to predict the primary content, in which the semantic similarity and the structural dissimilarity (i.e., semantic consistency and structural completeness) are both considered during the optimization. Then, the image quality is measured on the basis of its primary content. Generally, the image quality is highly related to three aspects, i.e., the scene information (content-dependency), the distortion type (distortion-dependency), and the content degradation (degradation-dependency). According to the correlation between the distorted image and its primary content, the three aspects are analyzed and calculated respectively with a multi-stream convolutional neural network (CNN) based quality evaluator. As a result, with the help of the primary content obtained from the active inference and the comprehensive quality degradation measurement from the multi-stream CNN, our method achieves competitive performance on five popular IQA databases. Especially in cross-database evaluations, our method achieves significant improvements.

Multilevel Cross-Aware RGBD Indoor Semantic Segmentation for Bionic Binocular Robot

Scene understanding is one of the foundations for robots to achieve true artificial intelligence. Semantic segmentation that imitates the mechanism of human visual system can effectively promote the correctness of scene understanding. It conforms to the basic principle of human environment perception and enables robots to better serve human society. In this paper, we propose a multilevel cross-aware network (MCA-Net) for RGBD semantic segmentation. It utilizes basic residual structure to encode texture information and depth geometric information respectively. In view of the flow pattern of different visual features in the visual pathway of human brain, our MCA-Net attempts to jointly reason about 2D appearance and depth geometric information. Therefore, multilevel cross-aware fusion modules are designed to fuse multi-scale complementary features extracted from RGB and depth images. As the depth and color information are independent, the reasonable combination of depth and RGB images can improve the quality of semantic labeling. Experiments conducted on ScanNetv2 dataset show that the proposed network produces high quality segmentation results of RGBD images and outperforms state-of-the-art methods. Furthermore, the semantic labeling results on the bionic binocular robot in real-office scenes further demonstrate the effectiveness of the proposed MCA-Net.

Goal driven network pruning for object recognition

Pruning studies up to date focused on uncovering a smaller network by removing redundant units, and fine-tuning to compensate accuracy drop as a result. In this study, unlike the others, we propose an approach to uncover a smaller network that is competent only in a specific task, similar to top-down attention mechanism in human visual system. This approach doesn’t require fine-tuning and is proposed as a fast and effective alternative of training from scratch when the network focuses on a specific task in the dataset. Pruning starts from the output and proceeds towards the input by computing neuron importance scores in each layer and propagating them to the preceding layer. In the meantime, neurons determined as worthless are pruned. We applied our approach on three benchmark datasets: MNIST, CIFAR-10 and ImageNet. The results demonstrate that the proposed pruning method typically reduces computational units and storage without harming accuracy significantly.

An Improved Algorithm Inspired by Natural Biology for Image Enhancement Used in Space Station

In the space station, the uneven illumination of image brings significant challenges for image enhancement in vision system due to high dynamic range of light intensity and the light reflection of metal facilities. In order to solve this problem in monitoring, unattended operation and robot operation, Retinex algorithm was proposed to estimate the luminosity. However, the effectiveness of this method is highly dependent on the selected path and consumes huge computational cost. This paper proposes an algorithm inspired by the Archimedes spiral in natural environment to improve the processing time and enhance the images quality. First, the structure and mechanism of Human Visual System (HVS) are analyzed. Second, the original image is decomposed into the brightness and color layers based on the structural human eyes mechanism. Third, the brightness layer is processed by the improved Retinex algorithm to enhance the original image by merging it with the color layer. The proposed algorithm is validated with the experimental results.

Finding autofocus region in low contrast surveillance images using CNN-based saliency algorithm

How to automatically locate the focus region in low contrast image is a key issue for camera-equipped surveillance devices. Due to low signal to noise ratio, the performance of autofocus will seriously decline in low contrast image, making it quite difficult to recognize the focus region. To tackle this problem, we perform autofocus by conducting a salient object detection approach. A covariance based deep learning framework is proposed to evaluate the saliency of low contrast surveillance image. Based on the mechanism of human visual system, the autofocus region can be identified by the visual salient object. In this paper, low-level features of the low contrast images are first studied and extracted. Then the mutual covariances of the segmented blocks are trained via a 7-layers convolutional neural network (CNN). Next, the initial saliency map of the testing image can be obtained by estimating the saliency score of each block via the pre-trained CNN model. Finally, the resulting saliency map is refined by introducing the local-global difference and internal similarity approaches. Experimental results demonstrate that the proposed method outperforms existing ten state-of-the-art saliency models on three public datasets and a nighttime image dataset.

Combining Bottom-Up and Top-Down Visual Mechanisms for Color Constancy Under Varying Illumination.

Multi-illuminant-based color constancy (MCC) is quite a challenging task. In this paper, we proposed a novel model motivated by the bottom-up and top-down mechanisms of human visual system (HVS) to estimate the spatially varying illumination in a scene. The motivation for bottom-up based estimation is from our finding that the bright and dark parts in a scene play different roles in encoding illuminants. However, handling the color shift of large colorful objects is difficult using pure bottom-up processing. Thus, we further introduce a top-down constraint inspired by the findings in visual psychophysics, in which high-level information (e.g., the prior of light source colors) plays a key role in visual color constancy. In order to implement the top-down hypothesis, we simply learn a color mapping between the illuminant distribution estimated by bottom-up processing and the ground truth maps provided by the dataset. We evaluated our model on four datasets and the results show that our method obtains very competitive performance compared with the state-of-the-art MCC algorithms. Moreover, the robustness of our model is more tangible considering that our results were obtained using the same parameters for all the datasets or the parameters of our model were learned from the inputs, that is, mimicking how HVS operates. We also show the color correction results on some real-world images taken from the web.

A Method for Weak Target Detection Based on Human Visual Contrast Mechanism

In order to detect space weak targets in low signal-to-clutter ratio (SCR) environment, this letter presents an effective detection model. At the first stage, two-dimensional least-mean-square preprocessing part is applied to the original image, after which, the suspicious area can be obtained. At the second stage, a novel method based on the contrast mechanism of human visual system called neighborhood saliency map is applied, which improves the local contrast map algorithm and greatly increases the accuracy of neighborhood saliency estimation, meanwhile, effectively enhances targets and suppresses background. Then, a simple threshold segmentation is used to get real targets. Compared with other state-of-the-art algorithms, the proposed algorithm obtains the superior performance in terms of SCR gain, background suppression factor, and detection results (detection rate and false alarm rate). The model proposed in this letter can effectively detect spatial weak point targets in the case of average SCR ≈ 1 or even SCR <; 1 (with minimum SCR is about 0.55).

A Deep Spatial Contextual Long-Term Recurrent Convolutional Network for Saliency Detection.

Traditional saliency models usually adopt hand-crafted image features and human-designed mechanisms to calculate local or global contrast. In this paper, we propose a novel computational saliency model, i.e., deep spatial contextual long-term recurrent convolutional network (DSCLRCN), to predict where people look in natural scenes. DSCLRCN first automatically learns saliency related local features on each image location in parallel. Then, in contrast with most other deep network based saliency models which infer saliency in local contexts, DSCLRCN can mimic the cortical lateral inhibition mechanisms in human visual system to incorporate global contexts to assess the saliency of each image location by leveraging the deep spatial long short-term memory (DSLSTM) model. Moreover, we also integrate scene context modulation in DSLSTM for saliency inference, leading to a novel deep spatial contextual LSTM (DSCLSTM) model. The whole network can be trained end-to-end and works efficiently when testing. Experimental results on two benchmark datasets show that DSCLRCN can achieve state-of-the-art performance on saliency detection. Furthermore, the proposed DSCLSTM model can significantly boost the saliency detection performance by incorporating both global spatial interconnections and scene context modulation, which may uncover novel inspirations for studies on them in computational saliency models.

Analysis and research on the images of drivers and passengers wearing seat belt in traffic inspection

It is common to see drivers and passengers not wearing seat belt in China, which is a traffic violation, thereby to strengthen supervision by resorting to advanced technical means is imminent. In the existing research, the detection effect is favorable when there is visible difference between the seat belt and the background image. However, in the opposing situation, it can easily lead to missed detection or mis-detection, and greatly reducing the detection accuracy and making it difficult to meet the demands of intelligent and automatic traffic monitoring. Study on the images of drivers and passengers wearing seat belt in traffic inspection covers three aspects: vehicle target positioning, driver and passenger area positioning and seat belt wearing behavior decision. Vehicle target positioning studies the enhanced vehicle features and looks for a vehicle classification model that is suitable for enhancing the features, and obtains the determined vehicle area from the study detection. Driver and passenger area positioning studies a multi-level joint supervised learning method according to the inclusive relationship between the non-rigid human body and the rigid vehicle body, and gets the optimal driver and passenger area by designing the multi-level learning classifier. Seat belt wearing behavior decision studies from the method of linear target recognition and uses the perceptual grouping mechanism of human visual system for robust seat belt detection. In the experimental phase, we validated the validity and accuracy, and the experiment images were from the practical traffic junctions. The results show that the method proposed in this paper can carry out Seat Belt detection accurately and effectively. Compared with other Seat Belt target detection methods, it has high detection rate and low false positive rate, which can achieve the accurate detection of Seat Belt in intelligent transportation.

Salient objects detection in dynamic scenes using color and texture features

Visual saliency is an important research topic in the field of computer vision due to its numerous possible applications. It helps to focus on regions of interest instead of processing the whole image or video data. Detecting visual saliency in still images has been widely addressed in literature with several formulations. However, visual saliency detection in videos has attracted little attention, and is a more challenging task due to additional temporal information. A common approach for obtaining a spatio-temporal saliency map is to combine a static saliency map and a dynamic saliency map. In our work, we model the dynamic textures in a dynamic scene with local binary patterns to compute the dynamic saliency map, and we use color features to compute the static saliency map. Both saliency maps are computed using a bio-inspired mechanism of human visual system with a discriminant formulation known as center surround saliency, and are fused in a proper way. The proposed model has been extensively evaluated with diverse publicly available datasets which contain several videos of dynamic scenes, and comparison with state-of-the art methods shows that it achieves competitive results.

Depth estimation algorithm based on data-driven approach and depth cues for stereo conversion in three-dimensional displays

With the advances in three-dimensional (3-D) display technology, stereo conversion has attracted much attention as it can alleviate the problem of stereoscopic content shortage. In two-dimensional (2-D) to 3-D conversion, the most difficult and challenging problem is depth estimation from a single image. In order to recover a perceptually plausible depth map from a single image, a depth estimation algorithm based on a data-driven method and depth cues is presented. Based on the human visual system mechanism, which is sensitive to the foreground object, this study classifies the image into one of two classes, i.e., nonobject image and object image, and then leverages different strategies on the basis of image type. The proposed strategies efficiently extract the depth information from different images. Moreover, depth image-based rendering technology is utilized to generate stereoscopic views by combining 2-D images with their depth maps. The proposed method is also suitable for 2-D to 3-D video conversion. Qualitative and quantitative evaluation results demonstrate that the proposed depth estimation algorithm is very effective for generating stereoscopic content and producing visually pleasing and realistic 3-D views.

No-Reference Image Quality Assessment Using Image Saliency for JPEG Compressed Images

Image quality assessment (IQA) is one of the most important issues in the field of image processing. Traditional IQA methods usually assume that the &#x201C;reference&#x201D; or &#x201C;perfect&#x201D; image is given. Obviously, this assumption is limited because the reference image may not be available in most practical applications. In addition, the mechanisms of human visual system (HVS) have not been explicitly exploited in the majority of existing IQA methods. In this article, to reduce dependence on reference image while introducing one mechanism of HVS, we propose a new method, referred to as Image-Saliency-based No-reference Image Quality Index (ISNIQI), by incorporating image saliency derived from visual attention models into the quality assessment of JPEG compressed images. Since the high saliency image region attracts more visual attention, ISNIQI assigns larger weight to the quality of image region with higher saliency. Experimental results on JPEG compressed images demonstrate the effectiveness of the proposed method in terms of the correlation with subjective perception.

Using Saliency-Weighted Disparity Statistics for Objective Visual Comfort Assessment of Stereoscopic Images

Visual comfort assessment (VCA) for stereoscopic images is a particularly significant yet challenging task in 3D quality of experience research field. Although the subjective assessment given by human observers is known as the most reliable way to evaluate the experienced visual discomfort, it is time-consuming and non-systematic. Therefore, it is of great importance to develop objective VCA approaches that can faithfully predict the degree of visual discomfort as human beings do. In this paper, a novel two-stage objective VCA framework is proposed. The main contribution of this study is that the important visual attention mechanism of human visual system is incorporated for visual comfort-aware feature extraction. Specifically, in the first stage, we first construct an adaptive 3D visual saliency detection model to derive saliency map of a stereoscopic image, and then a set of saliency-weighted disparity statistics are computed and combined to form a single feature vector to represent a stereoscopic image in terms of visual comfort. In the second stage, a high dimensional feature vector is fused into a single visual comfort score by performing random forest algorithm. Experimental results on two benchmark databases confirm the superior performance of the proposed approach.

Quaternion discrete cosine transformation signature analysis in crowd scenes for abnormal event detection

In this paper, an abnormal event detection approach inspired by the saliency attention mechanism of human visual system is presented. Conventionally, statistics-based methods suffer from visual scale, complexity of normal events and insufficiency of training data, for the reason that a normal behavior model established from normal video data is used to detect unusual behaviors with an assumption that anomalies are events with rare appearance. Instead, we make the assumption that anomalies are events that attract human attention. Temporal and spatial anomaly saliency are considered consistently by representing the pixel value in each frame as a quaternion, with weighted components that composed of intensity, contour, motion-speed and motion-direction feature. For each quaternion frame, Quaternion Discrete Cosine Transformation (QDCT) and signature operation are applied. The spatio-temporal anomaly saliency map is developed by inverse QDCT and Gaussian smoothing. By multi-scale analyzing, abnormal events appear at those areas with high saliency score. Experiments on typical datasets show that our method can achieve high accuracy results.

Binocular perception based reduced-reference stereo video quality assessment method

A new reduced-reference (RR) stereo video quality assessment method is proposed in this paper by considering temporal characteristics of video and binocular perception in human visual system (HVS). Firstly, motion intensity is utilized to extract RR frames for the purpose of temporal characteristics in stereo video. Secondly, according to internal generative mechanism of HVS, fusion and rivalry in the process of binocular perception is modeled, and the RR frames are divided into binocular fusion portion and binocular rivalry portion. Then, RR frame quality indicators are computed for these two portions. Finally, the RR frame quality indicators of the original and distorted frames are compared. A temporal pooling strategy is utilized on these quality indicators to obtain final stereo video quality score, where the motion intensity is used for toning the pooling parameters. Experimental results show that the proposed method has better performances when compared to other state-of-the-art quality assessment methods.