Image Retargeting Research Articles

StereoARS: Quality Evaluation for Stereoscopic Image Retargeting With Binocular Inconsistency Detection

Many stereoscopic image retargeting (SIR) methods have been developed for automatically and intelligently resizing stereoscopic images and we cannot always rely on time-consuming subjective user studies to validate the performance of different SIR methods. It is therefore required to design reliable objective metrics for SIR quality evaluation. This paper extends our previous 2D aspect ratio similarity (ARS) metric to a stereo 3D version termed as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">StereoARS</i> where the key idea is to investigate into retargeting inconsistency between the original stereo correspondences. Our proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">StereoARS</i> operates via two stages: monocular quality estimation and binocular inconsistency detection. In the first stage, monocular quality estimation is performed by applying a modified ARS measure on the left and right views separately to quantify the quality degradation within each monocular view. In the second stage, binocular inconsistency detection is performed in both pixel-level and grid-level to characterize the influence of binocular rivalry and stereo visual discomfort on SIR quality. In addition, we also measure to what extent the original pixel visibility relation is preserved after SIR as another binocular quality factor. Finally, these monocular and binocular quality estimates are fused to produce an overall SIR quality score. Extensive experiments have demonstrated that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">StereoARS</i> achieves better alignment with human subjective ratings than the existing metrics by a large margin.

Improved seam carving for structure preservation using efficient energy function

Improved seam carving for structure preservation using efficient energy function

Region Based Non-Uniform Scheme for Effective Retargeting

With the invent of displays of varied features like bendable displays, twistable displays and slant displays the feed of these displays need to be adjusted so that useful objects of the scene are not destroyed. In general, if we want to project a particular scene of specified size after capturing on the display of other dimensions, image resizing is done. In image resizing, the whole image will be converted into dimension of the target display. In the process the whole image, including all the objects may be expanded or squeezed depending on the size of the target display. But, in image retargeting, in addition to changing the size of the image to suit the target display, some additional intelligence will be added there by preserving or retaining useful objects and removing unnecessary background information. In this paper, the global significance map will drive the retargeting which is formed by merging the local significance maps. The local significance maps are derived after forming regions by considering the average intensity levels locally. The simulation results clearly show the superiority of the scheme proposed

Efficient and Accurate Stitching for 360° Dual-Fisheye Images and Videos.

Back-to-back dual-fisheye cameras are the most cost-effective devices to capture 360° visual content. However, image and video stitching for such cameras often suffer from the effect of fisheye distortion, photometric inconsistency between the two views, and non-collocated optical centers. In this paper, we present algorithms for geometric calibration, photometric compensation, and seamless stitching to address these issues for back-to-back dual-fisheye cameras. Specifically, we develop a co-centric trajectory model for geometric calibration to characterize both intrinsic and extrinsic parameters of the fisheye camera to fifth-order precision, a photometric correction model for intensity and color compensation to provide efficient and accurate local color transfer, and a mesh deformation model along with an adaptive seam carving method for image stitching to reduce geometric distortion and ensure optimal spatiotemporal alignment. The stitching algorithm and the compensation algorithm can run efficiently for 1920×960 images. Quantitative evaluation of geometric distortion, color discontinuity, jitter, and ghost artifact of the resulting image and video shows that our solution outperforms the state-of-the-art techniques.

Integration of Local and Global Features Using Gaussian Process Regression for Image Retargeting Quality Assessment

Integration of Local and Global Features Using Gaussian Process Regression for Image Retargeting Quality Assessment

Colourlab Image Database: Geometric Distortions

Over the years, a high number of different objective image quality metrics have been proposed. While some image quality metrics show a high correlation with subjective scores provided in different datasets, there still exists room for improvement. Different studies have pointed to evaluating the quality of images affected by geometrical distortions as a challenge for current image quality metrics. In this work, we introduce the Colourlab Image Database: Geometric Distortions (CID:GD) with 49 different reference images made specifically to evaluate image quality metrics. CID:GD is one of the first datasets which include three different types of geometrical distortions; seam carving, lens distortion, and image rotation. 35 state-ofthe-art image quality metrics are tested on this dataset, showing that apart from a handful of these objective metrics, most are not able to show a high performance. The dataset is available at &lt;ext-link ext-link-type="url" xlink:href="http://www.colourlab.no/cid"&gt;www.colourlab.no/cid&lt;/ext-link&gt;.

A Machine Learning Method for Detecting the Trace of Seam Carving

Image retargeting is a manipulation approach for resizing the images while aiming to keep the image distortion at a low level. Detecting image retargeting is of importance in image forensics or sometimes of importance in checking the originality. The aim of this paper is to introduce a new blind detection method for identifying retargeted images based on seam carving. For this purpose, a new method based on stripes at various numbers, Local Binary Pattern (LBP) transform, and energy map is introduced. The sub-images were obtained from square root of the energy map of LBP transform in the form of stripes for the feature extraction and these were evaluated in terms of several statistical features. The features extracted both from the natural and the seam carved images were used to train a Support Vector Machine (SVM) as a binary classifier. Experimental results were obtained using four-fold cross validation to improve the validity of the results during the evaluation process. According to the experiments, the proposed method produces improved accuracies when compared with the state-of-the-art solutions for the image retargeting detection based on seam carving.

Reduced Reference Quality Assessment for Image Retargeting by Earth Mover’s Distance

A reduced reference quality assessment algorithm for image retargeting by earth mover’s distance is proposed in this paper. In the reference image, all the feature points are extracted using scale invariant feature transform. Let the histograms of image patch around each feature point be local information, and the histograms of saliency feature as global information. Those feature information is extracted at the sender side and transmitted to the receiver side. After that, the same feature information extraction process is performed for the retargeted image at the receiver side. Finally, all feature information of the reference and retargeted images is used collectively to compute the quality of the retargeted image. An overall quality score is calculated from the local and global similarity measure using earth mover’s distance between reference and retargeted images. The key step in our algorithm is to provide an earth mover’s distance metric in a manner that indicates how the local and global information in the reference image is preserved in corresponding retargeted image. Experimental results show that the proposed algorithm can improve the image quality scores on four common criteria in the retargeted image quality assessment community.

Compressed domain image retargeting in hierarchical wavelet trees

This paper presents an efficient algorithm with content aware spatial scalability for effective displaying on various screens of different sizes. It combines the set partitioning in hierarchical trees (SPIHT) algorithm and the multiscale seam carving (MSC) algorithm in the compressed domain. As seams of insignificant wavelet trees can be discarded as early as possible, the burden of unnecessary computation and memory space is alleviated. Experimental results show that the proposed algorithm is preferable to other existing pixel-domain algorithms.

Multi-operator image retargeting in compressed domain by preserving aspect ratio of important contents

Multi-operator image retargeting in compressed domain by preserving aspect ratio of important contents

Combining Retargeting Quality and Depth Perception Measures for Quality Evaluation of Retargeted Stereopairs

Stereoscopic Image Retargeting (SIR) aims to adapt stereoscopic images and videos to 3D display devices with various aspect ratios by emphasizing the important content while retaining surrounding context with minimal visual distortion. To address the issue of SIR evaluation, this paper presents a new objective quality assessment method for retargeted stereopairs by combining image quality and depth perception measures. Specifically, the image quality measure is conducted between the source and retargeted intermediate views generated by the view synthesis method to characterize the geometric distortion and content loss of the retargeted stereopair, while several depth-aware features are extracted to measure the visual comfort&#x002F;discomfort and depth sensation when human views a 3D scene. Then, the extracted features are integrated into an overall perceptual quality prediction. Experiment results on NBU SIRQA and SIRD databases verify the superiority of our method.

Transformation-Aware Similarity Measurement for Image Retargeting Quality Assessment via Bidirectional Rewarping

Image retargeting is an effective way to adapt images for target displays with different aspect ratios and sizes. Meanwhile, effective image retargeting quality assessment (IRQA) is important for optimizing the image retargeting operations. In this paper, we propose a transform-aware similarity (TRASIM) measurement metric for IRQA, including bidirectional geometric distortion measurement, bidirectional information loss measurement, and global salient structure distortion measurement. The main innovation of the TRASIM is to build a universal framework to establish the similarity transformation via bidirectional rewarping to simulate different types of retargeting operators. Based on the similarity transformation, geometric distortion and content loss are measured to determine the retargeting quality. Experimental results on two widely used databases (CUHK and RetargetMe) indicate that the proposed TRASIM has higher consistency with subjective ranks, compared with the state-of-the-art IRQA metrics.

Seam Carving based visible watermarking robust to removal attacks

Visible watermarking offers direct copyright protection to the naked eye using visible pattern superimposed in the host image. However, due to their exposure, the watermark pattern becomes an easy target to remove by malicious entities that generates copyright-free images. Until now, in the scientific literature several semi-automatic visible watermark removal attacks have been proposed, showing enough visual quality of the recovered host image. In this paper, we propose a watermark design based on modified Seam Carving technique to provide robustness against removal attacks including inpainting-based attacks, Independent Component Analysis (ICA)-based attacks, and image matting-based attacks. Robustness against the attacks is measured using common image quality assessments and a watermark visibility assessment based on the Human Visual System (HVS). Experimental results show robustness of the proposed method against above mentioned watermark removal attacks under the most severe situation, in which adversary has the logotype used as a watermark pattern because it is public in Internet. The proposed method can be applied to any visible watermarking algorithms to make them robust against illegal removal attempts, without sacrificing their original functionality, such as reversibility for authorized users.

List-Wise Rank Learning for Stereoscopic Image Retargeting Quality Assessment

Stereoscopic image retargeting (SIR) techniques attempt to display stereoscopic images on stereoscopic devices of various resolutions and aspect ratios to provide the users with better viewing experience. However, new quality perceptual problems emerge in the retargeted stereoscopic images generated by current SIR operators are quite different from those in the retargeted 2D images. In this paper, we dedicate to exploring the perceptual quality-related factors (e.g., shape preservation, object preservation and visual comfort.) of retargeted stereoscopic images, and propose a novel quality evaluation metric for SIR to achieve a more consistent evaluation with 3D perception and image degradation mechanism in the SIR process. Moreover, image quality features and 3D perceptual features are integrated into one representation for an overall perceptual quality prediction using a list-wise ranking approach, which gives priority to the ranking among the SIR results generated from the same stereoscopic source. Experimental results demonstrate that the proposed method outperforms most quality models developed for retargeted 2D/stereoscopic images.

Hierarchical visual comfort assessment for stereoscopic image retargeting

Stereoscopic Image Retargeting (SIR) has made it possible for the popularity of 3D application. Meanwhile, the adjustments brought to images may affect the visual comfort when enjoying 3D service. While for SIR, previous Visual Comfort Assessment (VCA) methods often cannot perform well, because they only analyze the influence of disparity on discomfort and do not take into account the effects from the unique and complex distortions of SIR. In this paper, we propose a Hierarchical Visual Comfort Assessment (Hi-VCA) scheme for SIR, considering hybrid distortions including structure, information, semantic distortions usually occurring in retargeting, and binocular incongruity existing in stereoscopic multimedia. Specifically, we first propose valid Local-SSIM and Dual Natural Scene Statistics (D-NSS) features to measure structural distortion and information loss. Considering disparity adjustments may brought by SIR, we design the binocular incongruity measurement by analyzing various binocular anomaly perception mechanisms of HVS. Finally, CNN-based feature is utilized to ensure the correct delivery of semantic information. Each measurement is complementary in describing visual comfort degradation and they are further aggregated. Extensive experiment results on published SIR database SIRD and two ordinary databases IEEE-SA and NBU 3D-VCA, demonstrate Hi-VCA has superior performance by better handling hybrid distortions compared to state-of-the-art schemes.

Unsupervised stereoscopic image retargeting via view synthesis and stereo cycle consistency losses

Stereoscopic image retargeting aims to manipulate the stereoscopic images to fit various devices with different resolutions and prescribed aspect ratios. With the development of various types of three-dimensional (3D) displays, stereoscopic image retargeting becomes increasingly popular in the field of computer graphics. In this paper, we propose an unsupervised stereoscopic image retargeting network (USIR-Net) to address the problem of stereoscopic image retargeting without label information. By exploring the inter-view correlation and disparity relationship of stereoscopic images, two unsupervised losses are developed to guide the learning of stereoscopic image retargeting model. First, in view of the inter-view correlation, a view synthesis loss is proposed to guarantee the generation of high-quality stereoscopic images with accurate inter-view relationship. Second, by exploiting the consistency of stereoscopic images before and after the retargeting, a stereo cycle consistency loss, which consists of a content consistency term and a disparity consistency term, is developed to preserve the structure information and prevent binocular disparity inconsistency. Quantitative and qualitative experimental results demonstrate that the proposed method achieves superior performance compared with state-of-the-art methods.

Stereoscopic Image Retargeting Based on Deep Convolutional Neural Network

Stereoscopic image retargeting aims at converting stereoscopic images to the target resolution adaptively. Different from 2D image retargeting, stereoscopic image retargeting needs to preserve both the shape structure of salient objects and depth consistency of 3D scenes. In this paper, we present a stereoscopic image retargeting method based on deep convolutional neural network to obtain high-quality retargeted images with both object shape preservation and scene depth preservation. First, a cross-attention extraction mechanism is constructed to generate attention map, which contains the valuable attention features of the left and right images and the common attention features between them. Second, since the disparity map can provide accurate depth information of objects in 3D scenes, a disparity-assisted 3D significance map generation module is utilized to further preserve the valuable depth information of stereoscopic images. Finally, in order to predict the retargeted stereoscopic images accurately, an image consistency loss is developed to preserve the geometric structure of salient objects, and a disparity consistency loss is introduced to eliminate depth distortions. Experimental results demonstrate that the proposed deep convolutional neural network can provide favorable stereoscopic image retargeting results.

Context-aware saliency detection for image retargeting using convolutional neural networks

Image retargeting is the task of making images capable of being displayed on screens with different sizes. This work should be done so that high-level visual information and low-level features such as texture remain as intact as possible to the human visual system. At the same time, the output image may have different dimensions. Thus, simple methods such as scaling and cropping are not adequate for this purpose. In recent years, researchers have tried to improve the existing retargeting methods, and they have introduced new ones. However, a specific method cannot be utilized to retarget all types of images. In other words, different images require different retargeting methods. Image retargeting has a close relationship to image saliency detection, which is a relatively new image processing task. Earlier saliency detection methods were based on local and global but low-level image information. These methods are called bottom-up processes. On the other hand, newer approaches are top-down and mixed methods that consider the high level and semantic knowledge of the image too. In this paper, we introduce the proposed methods in both saliency detection and retargeting. For the saliency detection, the use of image context and semantic segmentation are examined, and a novel mixed bottom-up and top-down saliency detection method is introduced. After saliency detection, a modified version of an existing retargeting technique is utilized for retargeting the images. The results suggest that the proposed image retargeting pipeline has excellent performance compared to other tested methods. Also, the subjective evaluations on the Pascal dataset can be used as a retargeting quality assessment dataset for further research.

A Guided Edge-Aware Smoothing-Sharpening Filter Based on Patch Interpolation Model and Generalized Gamma Distribution

Smoothing and sharpening are two fundamental image processing operations. The latter is usually related to the former through the unsharp masking algorithm. In this paper, we develop a new type of filter which performs smoothing or sharpening via a tuning parameter. The development of the new filter is based on (1) a new Laplacian-based filter formulation which unifies the smoothing and sharpening operations, (2) a patch interpolation model similar to that used in the guided filter which provides edge-awareness capability, and (3) the generalized Gamma distribution which is used as the prior for parameter estimation. We have conducted detailed studies on the properties of two versions of the proposed filter (self-guidance and external guidance). We have also conducted experiments to demonstrate applications of the proposed filter. In the self-guidance case, we have developed adaptive smoothing and sharpening algorithms based on texture, depth and blurriness information extracted from an image. Applications include enhancing human face images, producing shallow depth of field effects, focus-based image enhancement, and seam carving. In the external guidance case, we have developed new algorithms for combining flash and no-flash images and for enhancing multi-spectral images using a panchromatic image.

Instance-Level Relative Saliency Ranking With Graph Reasoning.

Conventional salient object detection models cannot differentiate the importance of different salient objects. Recently, two works have been proposed to detect saliency ranking by assigning different degrees of saliency to different objects. However, one of these models cannot differentiate object instances and the other focuses more on sequential attention shift order inference. In this paper, we investigate a practical problem setting that requires simultaneously segment salient instances and infer their relative saliency rank order. We present a novel unified model as the first end-to-end solution, where an improved Mask R-CNN is first used to segment salient instances and a saliency ranking branch is then added to infer the relative saliency. For relative saliency ranking, we build a new graph reasoning module by combining four graphs to incorporate the instance interaction relation, local contrast, global contrast, and a high-level semantic prior, respectively. A novel loss function is also proposed to effectively train the saliency ranking branch. Besides, a new dataset and an evaluation metric are proposed for this task, aiming at pushing forward this field of research. Finally, experimental results demonstrate that our proposed model is more effective than previous methods. We also show an example of its practical usage on adaptive image retargeting.