Stereoscopic Image Retargeting Research Articles

A comprehensive review of image retargeting

With the development of display technologies, image retargeting plays a significant role in computer vision and pattern recognition communities currently. Image retargeting aims to display an image on a series of appliances with different resolutions and target aspect ratios. During the last decade, representative algorithms for image retargeting have been presented in the literature and achieved state-of-the-art performance. In this survey, we provide a comprehensive review of image retargeting, covering a wide variety of pioneering works for 2D image retargeting and stereoscopic image retargeting. 2D image retargeting focuses on preserving interesting regions when modifying the original image with arbitrary resolutions appropriately. Different from 2D image retargeting, stereoscopic image retargeting needs to preserve both the shape structure of salient objects and the depth consistency of 3D scenes simultaneously. In this survey, we start the first attempt to analyze the trends of 2D image retargeting and then summarize different types of stereoscopic image retargeting. Secondly, image retargeting quality assessment metrics for 2D images and stereoscopic images are introduced to evaluate retargeted images. Thirdly, we also investigate the evaluation datasets, and give the comparison results and analysis between different representative methods. Finally, the promising future research is thoroughly discussed to further improve the performance of 2D and stereoscopic image retargeting.

StereoARS: Quality Evaluation for Stereoscopic Image Retargeting With Binocular Inconsistency Detection

Many stereoscopic image retargeting (SIR) methods have been developed for automatically and intelligently resizing stereoscopic images and we cannot always rely on time-consuming subjective user studies to validate the performance of different SIR methods. It is therefore required to design reliable objective metrics for SIR quality evaluation. This paper extends our previous 2D aspect ratio similarity (ARS) metric to a stereo 3D version termed as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">StereoARS</i> where the key idea is to investigate into retargeting inconsistency between the original stereo correspondences. Our proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">StereoARS</i> operates via two stages: monocular quality estimation and binocular inconsistency detection. In the first stage, monocular quality estimation is performed by applying a modified ARS measure on the left and right views separately to quantify the quality degradation within each monocular view. In the second stage, binocular inconsistency detection is performed in both pixel-level and grid-level to characterize the influence of binocular rivalry and stereo visual discomfort on SIR quality. In addition, we also measure to what extent the original pixel visibility relation is preserved after SIR as another binocular quality factor. Finally, these monocular and binocular quality estimates are fused to produce an overall SIR quality score. Extensive experiments have demonstrated that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">StereoARS</i> achieves better alignment with human subjective ratings than the existing metrics by a large margin.

Combining Retargeting Quality and Depth Perception Measures for Quality Evaluation of Retargeted Stereopairs

Stereoscopic Image Retargeting (SIR) aims to adapt stereoscopic images and videos to 3D display devices with various aspect ratios by emphasizing the important content while retaining surrounding context with minimal visual distortion. To address the issue of SIR evaluation, this paper presents a new objective quality assessment method for retargeted stereopairs by combining image quality and depth perception measures. Specifically, the image quality measure is conducted between the source and retargeted intermediate views generated by the view synthesis method to characterize the geometric distortion and content loss of the retargeted stereopair, while several depth-aware features are extracted to measure the visual comfort&#x002F;discomfort and depth sensation when human views a 3D scene. Then, the extracted features are integrated into an overall perceptual quality prediction. Experiment results on NBU SIRQA and SIRD databases verify the superiority of our method.

List-Wise Rank Learning for Stereoscopic Image Retargeting Quality Assessment

Stereoscopic image retargeting (SIR) techniques attempt to display stereoscopic images on stereoscopic devices of various resolutions and aspect ratios to provide the users with better viewing experience. However, new quality perceptual problems emerge in the retargeted stereoscopic images generated by current SIR operators are quite different from those in the retargeted 2D images. In this paper, we dedicate to exploring the perceptual quality-related factors (e.g., shape preservation, object preservation and visual comfort.) of retargeted stereoscopic images, and propose a novel quality evaluation metric for SIR to achieve a more consistent evaluation with 3D perception and image degradation mechanism in the SIR process. Moreover, image quality features and 3D perceptual features are integrated into one representation for an overall perceptual quality prediction using a list-wise ranking approach, which gives priority to the ranking among the SIR results generated from the same stereoscopic source. Experimental results demonstrate that the proposed method outperforms most quality models developed for retargeted 2D/stereoscopic images.

Hierarchical visual comfort assessment for stereoscopic image retargeting

Stereoscopic Image Retargeting (SIR) has made it possible for the popularity of 3D application. Meanwhile, the adjustments brought to images may affect the visual comfort when enjoying 3D service. While for SIR, previous Visual Comfort Assessment (VCA) methods often cannot perform well, because they only analyze the influence of disparity on discomfort and do not take into account the effects from the unique and complex distortions of SIR. In this paper, we propose a Hierarchical Visual Comfort Assessment (Hi-VCA) scheme for SIR, considering hybrid distortions including structure, information, semantic distortions usually occurring in retargeting, and binocular incongruity existing in stereoscopic multimedia. Specifically, we first propose valid Local-SSIM and Dual Natural Scene Statistics (D-NSS) features to measure structural distortion and information loss. Considering disparity adjustments may brought by SIR, we design the binocular incongruity measurement by analyzing various binocular anomaly perception mechanisms of HVS. Finally, CNN-based feature is utilized to ensure the correct delivery of semantic information. Each measurement is complementary in describing visual comfort degradation and they are further aggregated. Extensive experiment results on published SIR database SIRD and two ordinary databases IEEE-SA and NBU 3D-VCA, demonstrate Hi-VCA has superior performance by better handling hybrid distortions compared to state-of-the-art schemes.

Unsupervised stereoscopic image retargeting via view synthesis and stereo cycle consistency losses

Stereoscopic image retargeting aims to manipulate the stereoscopic images to fit various devices with different resolutions and prescribed aspect ratios. With the development of various types of three-dimensional (3D) displays, stereoscopic image retargeting becomes increasingly popular in the field of computer graphics. In this paper, we propose an unsupervised stereoscopic image retargeting network (USIR-Net) to address the problem of stereoscopic image retargeting without label information. By exploring the inter-view correlation and disparity relationship of stereoscopic images, two unsupervised losses are developed to guide the learning of stereoscopic image retargeting model. First, in view of the inter-view correlation, a view synthesis loss is proposed to guarantee the generation of high-quality stereoscopic images with accurate inter-view relationship. Second, by exploiting the consistency of stereoscopic images before and after the retargeting, a stereo cycle consistency loss, which consists of a content consistency term and a disparity consistency term, is developed to preserve the structure information and prevent binocular disparity inconsistency. Quantitative and qualitative experimental results demonstrate that the proposed method achieves superior performance compared with state-of-the-art methods.

Stereoscopic Image Retargeting Based on Deep Convolutional Neural Network

Stereoscopic image retargeting aims at converting stereoscopic images to the target resolution adaptively. Different from 2D image retargeting, stereoscopic image retargeting needs to preserve both the shape structure of salient objects and depth consistency of 3D scenes. In this paper, we present a stereoscopic image retargeting method based on deep convolutional neural network to obtain high-quality retargeted images with both object shape preservation and scene depth preservation. First, a cross-attention extraction mechanism is constructed to generate attention map, which contains the valuable attention features of the left and right images and the common attention features between them. Second, since the disparity map can provide accurate depth information of objects in 3D scenes, a disparity-assisted 3D significance map generation module is utilized to further preserve the valuable depth information of stereoscopic images. Finally, in order to predict the retargeted stereoscopic images accurately, an image consistency loss is developed to preserve the geometric structure of salient objects, and a disparity consistency loss is introduced to eliminate depth distortions. Experimental results demonstrate that the proposed deep convolutional neural network can provide favorable stereoscopic image retargeting results.

Subjective and Objective Quality Assessment for Stereoscopic Image Retargeting

Binocular stereoscopic image retargeting (SIR) aims to adjust 3D images into target aspect ratios. In recent years, various SIR methods have been proposed, but there are few researches on visual quality assessment. As a consequence, we construct a benchmark stereoscopic image retargeting quality assessment database (NBU-SIRQA), which contains 720 stereoscopic retargeted images generated by eight representative SIR operators. Subjective test is conducted to obtain the mean opinion score (MOS) for each stereoscopic retargeted image. Additionally, we propose an objective SIRQA metric based on grid deformation and information loss (GDIL). The main idea of GDIL is to decompose the SIR operator into two transformations: monocular image retargeting transformation and viewpoint transformation. In each transformation, grid deformation and information loss are extracted simultaneously to represent image quality and 3D perception quality. Experimental results validated on our established NBU-SIRQA database show the superiority of our metric in measuring the quality of stereoscopic retargeted images over the existing approaches.

Seam Manipulator: Leveraging Pixel Fusion for Depth-Adjustable Stereoscopic Image Retargeting

To enhance the depth perception for stereoscopic image retargeting, this paper presents a depth-adjustable stereoscopic image retargeting method via pixel fusion technique. The traditional methods enforce the depth-preserving constraint on the left and right images with poor depth adaption. To solve this issue, we try to address two critical properties that how to change the widths of unmatched region to control the depth perception, and how to change the alignment of left and right seams to satisfy visual comfort limits, and thus propose a seam manipulator tool to optimize the process for scaling factor determination of stereoscopic seams. The important contribution of our method is to take scaling preservation energy and disparity consistency energy into optimization. The experimental results demonstrate that our method can achieve promising performances on depth perception and visual comfort adjustment for better visual experience.

Aesthetics-Driven Stereoscopic 3-D Image Recomposition With Depth Adaptation

Due to the availability and affordability of the stereoscopic equipment (e.g., stereo camera, lens, and display devices), stereoscopic image manipulation has been receiving considerable research attention in recent years. In this paper, we present a semiautomatic, aesthetic-driven stereoscopic image recomposition approach, which capacitates the change of the spatial position of the foreground object(s) in a given stereoscopic image to enhance human visual aesthetics. Our algorithm recomposes both the left and right stereo images simultaneously using a global optimization algorithm. To maximize image aesthetics, our algorithm minimizes a set of aesthetic quality errors, which is derived from selected photographic composition rules. In addition, depth adaptation is applied to the resized objects and change in vertical disparity of the resulting stereo image pair is minimized to ensure a pleasant three-dimensional (3-D) viewing experience. Our method can be used to perform stereoscopic image retargeting and recomposition simultaneously by providing the target image scale as the input. Empirical evaluations demonstrate the effectiveness of our approach in enhancing the aesthetics of stereoscopic 3-D images. Notably, depth adaptation is shown to play an important role in aesthetics enhancement.

QoE-Guided Warping for Stereoscopic Image Retargeting.

In the field of stereoscopic 3D (S3D) display, it is an interesting as well as meaningful issue to retarget the stereoscopic images to the target resolution, while the existing stereoscopic image retargeting methods do not fully take user's Quality of Experience (QoE) into account. In this paper, we have presented a QoE-guided warping method for stereoscopic image retargeting, which retarget the stereoscopic image and adapt its depth range to the target display while promoting user's QoE. Our method takes shape preservation, visual comfort preservation, and depth perception preservation energies into account, and simultaneously optimizes the 2D coordinates and depth information in 3D space. It also considers the specific viewing configuration in the visual comfort and depth perception preservation energy constraints. Experimental results on visually uncomfortable and comfortable stereoscopic images demonstrate that in comparison with the existing stereoscopic image retargeting methods, the proposed method can achieve a reasonable performance optimization among the QoE's factors of image quality, visual comfort, and depth perception, leading to promising overall S3D experience.

Saliency-based stereoscopic image retargeting

In this paper, we propose a new saliency based stereoscopic image retargeting method based on the characteristics of the Human Visual System (HVS). A new stereoscopic saliency detection method is designed by adopting low-level features of intensity, color, texture and depth. Besides, the viewing bias factors including center bias and depth bias existing in the HVS are adopted for stereoscopic visual attention modeling. Since the HVS is sensitive to edges in images, we fuse the saliency and edge maps to predict the visual significance of image pixels for image resizing. With the visual significance measure, we propose an image resizing method by minimizing the structure and depth distortion for stereoscopic image retargeting. Experimental results have shown that both our stereoscopic saliency detection and image retargeting methods can obtain better performance than the existing related methods on the public databases.

Visually preserving stereoscopic image retargeting using depth carving

This paper presents a method for retargeting a pair of stereoscopic images. Previous works have leveraged seam carving and image warping methods for two-dimensional image editing to address this issue. However, they did not consider the full advantages of the properties of stereoscopic images. Our approach offers substantial performance improvements over the state-of-the-art; the key insights driving the approach are that the input image pair can be decomposed into different depth layers according to the disparity and image segmentation, and the depth cues allow us to address the problem in a three-dimensional (3-D) space domain for best preserving objects. We propose depth carving that extends seam carving in a single image to resize the stereo image pair with disparity consistency. Our method minimizes the shape distortion and preserves object boundaries by creating new occlusions. As a result, the retargeted image pair preserves the stereoscopic quality and protects the original 3-D scene structure. Experimental results demonstrate that our method outperforms the previous methods.

Stereoscopic Visual Attention Guided Seam Carving for Stereoscopic Image Retargeting

Stereoscopic image retargeting plays an important role in adaptive 3D stereoscopic displays. It aims to fit displays with various resolutions while preserving visually salient content and geometric consistency. We propose a stereoscopic image retargeting method based on stereoscopic visual attention guided seam carving. Firstly, stereoscopic saliency map is generated by combining 2D saliency and depth saliency maps, and significant energy map is generated by considering binocular disparity binocular and binocular just-noticeable-difference (BJND). Then, seam selection is applied to the left image based on stereoscopic saliency and energy maps, and seam replacement is performed for the occluded regions to prevent the geometry inconsistency. Finally, according to the matched left and right seams, the retargeted stereoscopic image is generated. In the experiments, subjective and objective analysis on three stereoscopic image databases shows that the proposed approach produces better seam carving results than the related existing methods.

Content- and disparity-adaptive stereoscopic image retargeting

The paper proposes a content- and disparity-adaptive stereoscopic image retargeting. To simultaneously avoid the saliency content and disparity distortion, firstly, we calculate the image saliency region distortion difference, and conclude the factors causing visual distortion. Then, the proposed method via a convex quadratic programming can simultaneously avoid the distortion of the salient region and adjust disparity to a target area, by considering the relationship of the scaling factor of salient region and the disparity scaling factor. The experimental results show that the proposed method is able to successfully adapt the image disparity to the target display screen, while the salient objects remain undistorted in the retargeted stereoscopic image.

Object-Coherence Warping for Stereoscopic Image Retargeting

This paper addresses the topic of content-aware stereoscopic image retargeting. The key to this topic is consistently adapting a stereoscopic image to fit displays with various aspect ratios and sizes while preserving visually salient content. Most methods focus on preserving the disparities and shapes of visually salient objects through nonlinear image warping, in which distortions caused by warping are propagated to homogenous and low-significance regions. However, disregarding the consistency of object deformation sometimes results in apparent distortions in both the disparities and shapes of objects. An object-coherence warping scheme is proposed to reduce this unwanted distortion. The basic idea is to utilize the information of matched objects rather than that of matched pixels in warping. Such information implies object correspondences in a stereoscopic image pair, which allows the generation of an object significance map and the consistent preservation of objects. This strategy enables our method to consistently preserve both the disparities and shapes of visually salient objects, leading to good content-aware retargeting. In the experiments, qualitative and quantitative analyses of various stereoscopic images show that our results are better than those generated by related methods in terms of consistency of object preservation.