Stereoscopic Saliency Research Articles

Adaptive Viewpoint Feature Enhancement-Based Binocular Stereoscopic Image Saliency Detection

Modeling 3D visual saliency has received great attention due to the development of emerging 3D display technologies. Traditional methods relying on low-level features may not be efficient in interpreting 3D visual content from high-level semantic perspective. Despite numerous efforts dedicated to this area, existing 3D visual saliency detection methods do not necessarily excel in exploring the stereoscopic image saliency driven by the intra-view and inter-view dependencies among left and right views. In this paper, we propose a visual saliency detection method for stereoscopic images grounded on adaptive viewpoint feature enhancement via binocular vision. More specifically, the correlation among left and right views is investigated through a delicately designed binocular stereoscopic saliency feature aggregation module, enabling the generation of more representative saliency features towards binocular vision. Subsequently, to further aggregate the saliency features in multiple scales, we design a progressive attention-based saliency feature pyramid extraction module to effectively integrate the features from top-level to down-level based on the network hierarchy mechanism. The saliency maps are ultimately produced for stereoscopic images by evaluating the obtained saliency features. In addition, we create a stereoscopic image saliency dataset (SIS-3D) that includes 1086 stereoscopic image pairs with various content and their corresponding human eye fixation annotations, aiming to further facilitate the research on visual saliency detection for stereoscopic images. Extensive experiments demonstrate that our proposed method improves CC by an average of 4.02% compared to representative counterparts on the newly built saliency dataset and another publicly available dataset.

Unsupervised Saliency Detection of Rail Surface Defects using Stereoscopic Images

Visual information is increasingly recognized as a useful method to detect rail surface defects due to its high efficiency and stability. However, it cannot sufficiently detect a complete defect in the complex background information. The addition of surface profiles can effectively improve this by including a 3-D information of defects. However, in high-speed detection, the traditional 3-D profile acquisition is difficult and separate from the image acquisition, which cannot satisfy the above-mentioned requirements effectively. Therefore, an unsupervised stereoscopic saliency detection method based on a binocular line-scanning system is proposed in this article. This method can simultaneously obtain a highly precise image as well as profile information while also avoids the decoding distortion of the structured light reconstruction method. In our method, a global low-rank nonnegative reconstruction algorithm with a background constraint is proposed. Unlike the low-rank recovery model, the algorithm has a more comprehensive low rank and background clustering properties. Furthermore, outlier detection based on the geometric properties of the rail surface is also proposed in this method. Finally, the image saliency results and depth outlier detection results are associated with the collaborative fusion, and a dataset (RSDDS-113) containing the rail surface defects is established for the experimental verification. The experimental results demonstrate that our method can obtain a mean absolute error of 0.09 and area under the ROC curve of 0.94, better than 15 state-of-the-art algorithms.

A Computational Model for Stereoscopic Visual Saliency Prediction

Depth information plays an important role in human vision as it provides additional cues that distinguish objects from their backgrounds. This paper explores depth information for analyzing stereoscopic saliency and presents a computational model that predicts stereoscopic visual saliency based on three aspects of human vision: 1) the pop-out effect; 2) comfort zones; and 3) background effects. Through an analysis of these three phenomena, we find that most of the stereoscopic saliency region can be explained. Our model comprises three modules, each describing one aspect of saliency distribution, and a control function that can be used to adjust the three models independently. The relationship between the three models is not mutually exclusive. One, two, or three phenomena may appear in one image. Therefore, to accurately determine which phenomena the image conforms to, we have devised a selection strategy that chooses the appropriate combination of models based on the content of the image. Our approach is implemented within a framework based on the multifeature analysis. The framework considers surrounding regions, color/depth contrast, and points of interest. The selection strategy can improve the performance of the framework. A series of experiments on two recent eye-tracking datasets shows that our proposed method outperforms several state-of-the-art saliency models.

Deep3DSaliency: Deep Stereoscopic Video Saliency Detection Model by 3D Convolutional Networks.

Stereoscopic saliency detection plays an important role in various stereoscopic video processing applications. However, conventional stereoscopic video saliency detection methods mainly use independent low-level features instead of extracting them automatically, and thus, they ignore the intrinsic relationship between the spatial and temporal information. In this paper, we propose a novel stereoscopic video saliency detection method based on 3D convolutional neural networks, namely Deep 3D Video Saliency (Deep3DSaliency). The proposed network consists of two sub-models: Spatiotemporal Saliency Model (STSM), and Stereoscopic Saliency Aware Model (SSAM). STSM directly takes three consecutive video frames as the input to extract visual spatiotemporal features, while SSAM attempts to further infer the depth and semantic features from the left and right video frames by shared parameters from STSM. The visual spatiotemporal features from STSM, and the depth and semantic features from SSAM are learned by an alternating optimization scheme. Finally, all these saliency-related features are combined together for the final stereoscopic saliency detection via 3D deconvolution. Experimental results show the superior performance of the proposed model over other existing ones in saliency estimation for 3D video sequences.

Disparity tuning guided stereoscopic saliency detection for eye fixation prediction

The development of emerging 3D display brings increasing attention of stereoscopic techniques such as quality assessment, re-targeting, and compression of 3D image, that require new saliency detection methods to deal with stereoscopic data. In this paper, we present a disparity tuning guided stereoscopic saliency (DTSS) model which apply the disparity tuning mechanism of visual cortical neurons into visual attention modeling. First, we investigate the rationality of converting features from physical quantity into perception quantity before computing saliency. Second, we discuss the biological principles that depth affects visual attention and consider both absolute and relative depth tuning mechanisms to model visual attention. Then, we propose a diffusion saliency feature combining depth and RGB features. Specifically, we use RGB contrast as primitive labels to diffuse a saliency map for depth map and using depth contrast as primitive labels to diffuse a saliency map for RGB image. Finally, an adaptively weighted fusion method is proposed for the integration of feature maps. Experiments demonstrate that the proposed model performs well against to the state-of-art methods on the task of 3D eye fixation prediction.

Stereoscopic saliency estimation with background priors based deep reconstruction

Abstract Studies have implied that depth is one of the important cues guiding visual attention. However, depth information has not been well explored in existing saliency estimation models. In this paper, we propose a model inspired by the observations in three-dimensional environment to better present the influence of depth. Firstly, we use depth to estimate each region’s probability of being background. Afterward, we sample pairs of surrounding and central patches from the possible background of each image to train an autoencoder-based network. As a network learned from background, it tends to describe the center-surround reconstruction pattern in background rather than foreground. Therefore, the detection of saliency can be formulated by measuring the reconstruction residual of the network. With emphasis on sampling from background, the proposed method can decrease the false positive rate in stereoscopic saliency estimation significantly. Experimental results demonstrate that the proposed method can outperform the state-of-the-art fixation prediction algorithms on several public data sets for stereoscopic saliency estimation. Additionally, it is efficiently used for proto-object extraction.

Stereoscopic saliency model using contrast and depth-guided-background prior

Abstract Many successful models of saliency have been proposed to detect salient regions for 2D images. Because stereopsis, with its distinctive depth information, influences human viewing, it is necessary for stereoscopic saliency detection to consider depth information as an additional cue. In this paper, we propose a 3D stereoscopic saliency model based on both contrast and depth-guided-background prior. First, a depth-guided-background prior is specifically detected from a disparity map apart from the conventional prior, assuming boundary super-pixels as background. Then, saliency based on disparity with the help of the proposed prior is proposed to prioritize the contrasts among super-pixels. In addition, a scheme to combine the contrast of disparity and the contrast of color is presented. Finally, 2D spatial dissimilarity features are further employed to refine the saliency map. Experimental results on the PSU stereo saliency benchmark dataset (SSB) show that the proposed method performs better than existing saliency models.

Stereoscopic visual saliency prediction based on stereo contrast and stereo focus

In this paper, we exploit two characteristics of stereoscopic vision: the pop-out effect and the comfort zone. We propose a visual saliency prediction model for stereoscopic images based on stereo contrast and stereo focus models. The stereo contrast model measures stereo saliency based on the color/depth contrast and the pop-out effect. The stereo focus model describes the degree of focus based on monocular focus and the comfort zone. After obtaining the values of the stereo contrast and stereo focus models in parallel, an enhancement based on clustering is performed on both values. We then apply a multi-scale fusion to form the respective maps of the two models. Last, we use a Bayesian integration scheme to integrate the two maps (the stereo contrast and stereo focus maps) into the stereo saliency map. Experimental results on two eye-tracking databases show that our proposed method outperforms the state-of-the-art saliency models.

Learning visual saliency from human fixations for stereoscopic images

In the previous years, a lot of saliency detection algorithms have been designed for saliency computation of visual content. Recently, stereoscopic display techniques have developed rapidly, which results in much requirement of stereoscopic saliency detection for emerging stereoscopic applications. Different from 2D saliency prediction, stereoscopic saliency detection methods have to consider depth factor. We design a novel stereoscopic saliency detection algorithm by machine learning technique. First, the features of luminance, color and texture are extracted to calculate the feature contract for predicting feature maps of stereoscopic images. Furthermore, the depth features are extracted for depth feature map computation. Sematic features including the center-bias factor and other top-down cues are also applied as the features in the proposed stereoscopic saliency detection method. Support Vector Regression (SVR) is applied to learn the saliency detection model of stereoscopic images. Experimental results obtained on a public large-scale eye tracking database demonstrate that the proposed method can predict better saliency results for stereoscopic images than other existing ones.

Saliency-based stereoscopic image retargeting

In this paper, we propose a new saliency based stereoscopic image retargeting method based on the characteristics of the Human Visual System (HVS). A new stereoscopic saliency detection method is designed by adopting low-level features of intensity, color, texture and depth. Besides, the viewing bias factors including center bias and depth bias existing in the HVS are adopted for stereoscopic visual attention modeling. Since the HVS is sensitive to edges in images, we fuse the saliency and edge maps to predict the visual significance of image pixels for image resizing. With the visual significance measure, we propose an image resizing method by minimizing the structure and depth distortion for stereoscopic image retargeting. Experimental results have shown that both our stereoscopic saliency detection and image retargeting methods can obtain better performance than the existing related methods on the public databases.

Stereoscopic Visual Attention Guided Seam Carving for Stereoscopic Image Retargeting

Stereoscopic image retargeting plays an important role in adaptive 3D stereoscopic displays. It aims to fit displays with various resolutions while preserving visually salient content and geometric consistency. We propose a stereoscopic image retargeting method based on stereoscopic visual attention guided seam carving. Firstly, stereoscopic saliency map is generated by combining 2D saliency and depth saliency maps, and significant energy map is generated by considering binocular disparity binocular and binocular just-noticeable-difference (BJND). Then, seam selection is applied to the left image based on stereoscopic saliency and energy maps, and seam replacement is performed for the occluded regions to prevent the geometry inconsistency. Finally, according to the matched left and right seams, the retargeted stereoscopic image is generated. In the experiments, subjective and objective analysis on three stereoscopic image databases shows that the proposed approach produces better seam carving results than the related existing methods.

Saliency detection for stereoscopic images.

Many saliency detection models for 2D images have been proposed for various multimedia processing applications during the past decades. Currently, the emerging applications of stereoscopic display require new saliency detection models for salient region extraction. Different from saliency detection for 2D images, the depth feature has to be taken into account in saliency detection for stereoscopic images. In this paper, we propose a novel stereoscopic saliency detection framework based on the feature contrast of color, luminance, texture, and depth. Four types of features, namely color, luminance, texture, and depth, are extracted from discrete cosine transform coefficients for feature contrast calculation. A Gaussian model of the spatial distance between image patches is adopted for consideration of local and global contrast calculation. Then, a new fusion method is designed to combine the feature maps to obtain the final saliency map for stereoscopic images. In addition, we adopt the center bias factor and human visual acuity, the important characteristics of the human visual system, to enhance the final saliency map for stereoscopic images. Experimental results on eye tracking databases show the superior performance of the proposed model over other existing methods.