Retargeting Quality Research Articles

Visual Attention Fusion Framework for Image Retargeting Quality Assessment

To adapt images for diversified digital devices, researchers have presented many image retargeting methods. However, the consistency between results of objective image retargeting quality assessment (IRQA) metrics and subjective perception is still low. In this paper, we propose a visual attention fusion (VAF) framework to assist IRQA metrics in better understanding the features of images such as image saliency, faces, and lines. First, we combine the results of multiple salient object detection algorithms to reduce the limitations of a single algorithm. Second, faces and lines are considered in our framework to measure deformations to these visually sensitive regions. Finally, we propose a saliency enhancement model to simulate human visual attention for IRQA. We combine the proposed VAF framework with some state-of-the-art IRQA metrics. Experimental results show that the proposed VAF framework can improve the consistency between the results of objective IRQA metrics and subjective opinion scores.

A Content-Aware Image Retargeting Quality Assessment Method Using Foreground and Global Measurement

Image retargeting methods aim to minimize the perceptual loss while changing sizes and aspect ratios of images. Since optimal retargeting methods for different images are generally not the same, the image retargeting quality assessment (IRQA) becomes a meaningful task. This paper proposes a content-aware image retargeting quality assessment method using foreground and global measurement to achieve better performance. In our proposed method, images are first divided into two categories according to the foreground object detection result, and then different corresponding measurements are designed for them. For those with obvious foreground object, both foreground and global measurement are applied. For others, only global measurement is conducted. Foreground measurement includes two complementary features: the high-level semantic similarity feature and the low-level size ratio feature. Global measurement includes another two features: an improved aspect ratio similarity (ARS) feature and edge group similarity (EGS) feature. Two public databases, i.e., the RetargetMe and CUHK, have been evaluated, and experimental results demonstrate that our method is quite effective, and it also provides state-of-the-art performance in the IRQA. aa Our code are available at https://github.com/SCUT-ML-GUO/IRQA.

Image retargeting quality assessment based on content deformation measurement

Content-aware based image retargeting methods aim at visualizing source images on diverse display devices with different aspect ratios while preventing the incurrence of visual distortions as far as possible. An objective retargeting assessment method, which has a high consistency with subjective perception, can measure the retargeting effects of different retargeting methods, thus promoting the development of retargeting technology. However, the correlation between the existing objective evaluation methods and subjective evaluation is not strong enough. In this paper, we propose an objective retargeting assessment method by fusing multiple features to quantify the content loss and structural deformation caused by retargeting. In order to measure the global deformation, there are two features are proposed which are elastic registration based on B-spline function and foreground retention ratio. Moreover, improved Aspect Ratio Similarity (IARS) and concentrated deletion ratio make for better detection of local deformation. These features are fused to obtain the objective retargeting assessment method. Compared with the state-of-art objective assessment methods on three public datasets, the proposed method has a better correlation with subjective perception.

Quality Assessment of Retargeted Images Using Hand-Crafted and Deep-Learned Features

Since the goal of image retargeting is to adapt source images on target displays with different sizes and aspect ratios, how to objectively evaluate the quality of retargeted images is particularly important to optimize the retargeting operations. In this paper, we proposed a new image retargeting quality assessment metric, which constructs the metric using both hand-crafted features and deep-learned features. To enhance the reliability and accuracy of the proposed method: 1) we use similarity transformation as local descriptor to extract hand-craft features, and measure structure distortion and content loss from the hand-craft features and 2) we use deep learning architecture to construct encoders and extract deep-learned features, and measure texture similarity and semantic similarity from the deep-learned features. We conduct experiments on two databases: RetargetMe and CUHK. Experimental results show that our method can achieve superior performance to the state-of-the-art metrics.

A bi-directional evaluation-based approach for image retargeting quality assessment

Image retargeting is a technique that adjusts input images into arbitrary dimensions (rows and columns) and simultaneously preserves regions of interest. Assess the image quality under varying aspect ratio is significantly more challenging since it requires content matching in addition to semantic content analysis. In this work, we propose an objective quality assessment algorithm for image retargeting, called bi-directional importance map similarity (BIMS). The key step in our approach is to assess quality in image retargeting through some features in a bi-directional way, all in a feature fusion framework. The motivation behind employing bi-directional features is because the nature of them is useful to estimate pertinent locations where we can analyze whenever relevant content is missing or any visual distortion arises. Our proposal was assessed on a well-known state-of-the-art dataset in which human viewers provided their personal opinions on the perceptual quality. Due to the experimental results obtained, we consider the BIMS is a good choice for quality assessment of retargeted images.

Multiple-Level Feature-Based Measure for Retargeted Image Quality.

Objective image retargeting quality assessment aims to use computational models to predict the retargeted image quality consistent with subjective perception. In this paper, we propose a multiple-level feature (MLF)-based quality measure to predict the perceptual quality of retargeted images. We first provide an in-depth analysis on the low-level aspect ratio similarity feature, and then propose a mid-level edge group similarity feature, to better address the shape/structure related distortion. Furthermore, a high-level face block similarity feature is designed to deal with sensitive region deformation. The multiple-level features are complementary as they quantify different aspects of quality degradation in the retargeted image, and the MLF measure learned by regression is used to predict the perceptual quality of retargeted images. Extensive experimental results performed on two public benchmark databases demonstrate that the proposed MLF measure achieves higher quality prediction accuracy than the existing relevant state-of-the-art quality measures.

Objective Quality Assessment of Image Retargeting by Incorporating Fidelity Measures and Inconsistency Detection.

The tremendous growth in mobile devices has resulted in huge generation and usage of digital images. Image quality assessment is thus an important issue for mobile media applications. In this paper, we focus on the quality evaluation of images generated by content-aware image retargeting, in which the reference and the distorted images are of different sizes. Through retargeting, many types of deformation inconsistency lead to shape distortion, deformation artifacts, and content information loss, worsening its perceptual quality. The deformation inconsistency occurs on different levels of the retargeted images. Limited by the accuracy of the alignment between the original and retargeted images, previous methods only focus on pixel-level and patch-level fidelity analyses and fail to detect deformation inconsistency. In this paper, we improve the alignment algorithm and propose a three-level representation of the retargeting process. Based on the analysis of this three-level representation, both fidelity measures and inconsistency detection are combined to determine the final retargeting quality. The proposed algorithm is validated on the public data sets RetargetMe and CUHK. Experimental results demonstrate that inconsistency detection contributes to accurately assessing the image retargeting perceptual quality. This inspires us to investigate more about deformation inconsistency to formulate the objective quality of image retargeting.

Full Reference Quality Assessment for Image Retargeting Based on Natural Scene Statistics Modeling and Bi-Directional Saliency Similarity

Image retargeting technology has been widely studied to adapt images for the devices with heterogeneous screen resolutions. Meanwhile effective objective retargeting quality assessment algorithms are also very important for optimizing and selecting favorable retargeting methods. Unlike previous assessment algorithms which rely on image local structure features and unidirectional prediction of information loss, we propose a bi-directional natural salient scene distortion model (BNSSD) including image natural scene statistics (NSS) measurement, salient global structure distortion measurement, and bi-directional salient information loss measurement. First, we propose a new NSS model in log-Gabor domain and verify its effectiveness in reflecting nature scene statistical distortions introduced during the retargeting process. Second, the concept of salient global structure distortion is proposed to measure the global structure uniformity in the corresponding salient regions between original and retargeted images. Finally, we propose a bidirectional salient information loss metric to measure the information loss between salient areas in original image and retargeted image. The effectiveness of the BNSSD model is verified on two widely recognized public databases, and the experimental results show that our method outperforms the state-of-the-art algorithms under different statistical assessment criteria.

A structure-aware axis-aligned grid deformation approach for robust image retargeting

Content-based image retargeting is a technique that resizes an input image to a given target resolution while minimizing distortions of important objects caused by aspect ratio variations. Conventional approaches have shared a similar methodology which aims to preserve salient regions as much as possible while allowing distortions of less important regions. Those methods have shown satisfactory results for input images whose objects are distinct and backgrounds are monotonous. However, their performance is not always guaranteed for images containing structural components such as straight lines, which are prone to be distorted after resizing and sensitive to human visual perception. In this paper, we propose a structure-aware axis-aligned grid deformation approach for robust image retargeting. Based on axis-aligned grid, our method finds the optimal grid for target image by quadratic optimization represented by two objective functions. The first one is the As-similar-as-possible (ASAP) energy function, which aims to preserve important regions while allowing distortions of trivial regions. The second one is the Adaptive Laplacian regularization (ALR) energy function, which aims to relieve structural distortions. Those two energy functions are combined into single quadratic optimization model ensuring the global convexity and solved by a quadratic programming solver for finding the optimal grid. Experimental results show that our method is robust to structural distortions while achieving the basic purpose of content-based image retargeting. For objective comparisons with other methods, we have provided objective evaluation scores by using a recent state-of-the-art image retargeting quality assessment scheme.

Learning Sparse Representation for Objective Image Retargeting Quality Assessment.

The goal of image retargeting is to adapt source images to target displays with different sizes and aspect ratios. Different retargeting operators create different retargeted images, and a key problem is to evaluate the performance of each retargeting operator. Subjective evaluation is most reliable, but it is cumbersome and labor-consuming, and more importantly, it is hard to be embedded into online optimization systems. This paper focuses on exploring the effectiveness of sparse representation for objective image retargeting quality assessment. The principle idea is to extract distortion sensitive features from one image (e.g., retargeted image) and further investigate how many of these features are preserved or changed in another one (e.g., source image) to measure the perceptual similarity between them. To create a compact and robust feature representation, we learn two overcomplete dictionaries to represent the distortion sensitive features of an image. Features including local geometric structure and global context information are both addressed in the proposed framework. The intrinsic discriminative power of sparse representation is then exploited to measure the similarity between the source and retargeted images. Finally, individual quality scores are fused into an overall quality by a typical regression method. Experimental results on several databases have demonstrated the superiority of the proposed method.

Learning quality assessment of retargeted images

Content-aware image resizing (or image retargeting) enables images to be fit to different display devices having different aspect ratios while preserving salient image content. There are many approaches to retargeting, although no “best” method has been agreed upon. Therefore, finding ways to assess the quality of image retargeting has become a prominent challenge. Traditional image quality assessment methods are not directly applicable to image retargeting because the retargeted image size is not same as the original one. In this paper, we propose an open framework for image retargeting quality assessment, where the quality prediction engine is a trained Radial Basis Function (RBF) neural network. Broadly, our approach is motivated by the observation that no single method can be expected to perform well on all types of content. We train the network on ten perceptually relevant features, including a saliency-weighted, SIFT-directed complex wavelet structural similarity (CW-SSIM) index, and a new image aesthetics evaluation method. These two features and eight other features are used by the neural network to learn to assess the quality of retargeted images. The accuracy of the new model is extensively verified by simulations.

Backward Registration-Based Aspect Ratio Similarity for Image Retargeting Quality Assessment

During the past few years, there have been various kinds of content-aware image retargeting operators proposed for image resizing. However, the lack of effective objective retargeting quality assessment metrics limits the further development of image retargeting techniques. Different from traditional image quality assessment (IQA) metrics, the quality degradation during image retargeting is caused by artificial retargeting modifications, and the difficulty for image retargeting quality assessment (IRQA) lies in the alternation of the image resolution and content, which makes it impossible to directly evaluate the quality degradation like traditional IQA. In this paper, we interpret the image retargeting in a unified framework of resampling grid generation and forward resampling. We show that the geometric change estimation is an efficient way to clarify the relationship between the images. We formulate the geometric change estimation as a backward registration problem with Markov random field and provide an effective solution. The geometric change aims to provide the evidence about how the original image is resized into the target image. Under the guidance of the geometric change, we develop a novel aspect ratio similarity (ARS) metric to evaluate the visual quality of retargeted images by exploiting the local block changes with a visual importance pooling strategy. Experimental results on the publicly available MIT RetargetMe and CUHK data sets demonstrate that the proposed ARS can predict more accurate visual quality of retargeted images compared with the state-of-the-art IRQA metrics.

Consistent Volumetric Warping Using Floating Boundaries for Stereoscopic Video Retargeting

The key to content-aware warping and cropping is adapting data to fit displays with various aspect ratios while preserving visually salient contents. Most previous studies achieve this objective by cropping insignificant contents near frame boundaries and consistently resizing frames through an optimization technique with various preservation constraints and fixed boundary conditions. These strategies significantly improve retargeting quality. However, warping under fixed boundary conditions may bound/limit the preservation of visually salient contents. Moreover, dynamic frame cropping and frame alignment may result in unnatural object/camera motions. In this paper, a floating boundary with volumetric warping and object-aware cropping is proposed to address these problems. In the proposed scheme, visually salient objects in the space-time domain are deformed as rigidly and as consistently as possible using information from matched objects and content-aware boundary constraints. The content-aware boundary constraints can retain visually salient contents in a fixed region with a desired resolution and aspect ratio, called critical region, during warping. Volumetric cropping with the fixed critical region is then performed to adjust stereoscopic videos to the desired aspect ratios. The strategies of warping and cropping using floating boundaries and spatiotemporal constraints enable our method to consistently preserve the temporal motions and spatial shapes of visually salient volumetric objects in the left and right videos as much as possible, thus leading to good content-aware retargeting. In addition, by considering shape, motion, and disparity preservation, the proposed scheme can be applied to various media, including images, stereoscopic images, videos, and stereoscopic videos. Qualitative and quantitative analyses of stereoscopic videos with diverse camera and considerable motions demonstrate a clear superiority of the proposed method over related methods in terms of retargeting quality.

A novel Genetic Algorithms and SURF-Based approach for image retargeting

Seam Carving is a method which aims at retargeting images, i.e., adjusting input images into arbitrary dimensions by removing pixel paths (seams) with minimal visual impact. The standard method is based on an exhaustive searching method for minimal cost seams according to a pixel relevance function also named energy function. In this work, we propose a new method to retarget images based on both Genetic Algorithms (GAs) and a high-performance scale and rotation-invariant interest point descriptor, the well-known method, Speed Up Robust Features (SURF). Along with our proposal, we present a content-aware fitness function, a new individual modeling, as well as a new index for image retargeting quality assessment. According to the simulations carried out, our proposal is very handy for image retargeting and outperforms the Seam Carving in terms of the image quality assessment index in most results.

Image retargeting quality assessment based on support vector regression

Image retargeting is to re-represent the image, usually with a different image size to cater for the display size of the end-user (e.g., computer, smartphone). In this paper, we propose an image retargeting quality assessment scheme based on four quality factors and support vector regression. The accounted quality factors are of two categorizes: 1) shape distortions, they are visual unpleasant artifacts introduced by retargeting, e.g., a straight line may be retargeted as a curve, a circle may be retargeted as an ellipse, etc.; 2) visual content changes, they mainly refer to the loss of visual information due to retargeting. Both spatial and frequency information are used toward a complete assessment. The spatial quality factors measure shape distortions and visual content changes separately and independently, and therefore it is possible to control their relative importance to the overall assessment. Since spatial quality factors need image matching which may cause measurement error, another quality factor in frequency domain is also used. The overall image retargeting quality is contributed by the four quality measures collectively and machine learning is used accounts for such a combination, corresponding discussions are also provided for each factor. The effectiveness of the proposed scheme is verified with the publicly-available subject-rated database (comprising of diverse images and retargeting methods) and better performance (in terms of both accuracy and complexity) is achieved in comparison with the state of the art schemes.

Image Retargeting Quality Assessment: A Study of Subjective Scores and Objective Metrics

This paper presents the result of a recent large-scale subjective study of image retargeting quality on a collection of images generated by several representative image retargeting methods. Owning to many approaches to image retargeting that have been developed, there is a need for a diverse independent public database of the retargeted images and the corresponding subjective scores to be freely available. We build an image retargeting quality database, in which 171 retargeted images (obtained from 57 natural source images of different contents) were created by several representative image retargeting methods. And the perceptual quality of each image is subjectively rated by at least 30 viewers, meanwhile the mean opinion scores (MOS) were obtained. It is revealed that the subject viewers have arrived at a reasonable agreement on the perceptual quality of the retargeted image. Therefore, the MOS values obtained can be regarded as the ground truth for evaluating the quality metric performances. The database is made publicly available (Image Retargeting Subjective Database, [Online]. Available: http://ivp.ee.cuhk.edu.hk/projects/demo/retargeting/index.html) to the research community in order to further research on the perceptual quality assessment of the retargeted images. Moreover, the built image retargeting database is analyzed from the perspectives of the retargeting scale, the retargeting method, and the source image content. We discuss how to retarget the images according to the scale requirement and the source image attribute information. Furthermore, several publicly available quality metrics for the retargeted images are evaluated on the built database. How to develop an effective quality metric for retargeted images is discussed through a specifically designed subjective testing process. It is demonstrated that the metric performance can be further improved, by fusing the descriptors of shape distortion and content information loss.

Image Retargeting Quality Assessment

AbstractContent‐aware image retargeting is a technique that can flexibly display images with different aspect ratios and simultaneously preserve salient regions in images. Recently many image retargeting techniques have been proposed. To compare image quality by different retargeting methods fast and reliably, an objective metric simulating the human vision system (HVS) is presented in this paper. Different from traditional objective assessment methods that work in bottom‐up manner (i.e., assembling pixel‐level features in a local‐to‐global way), in this paper we propose to use a reverse order (top‐down manner) that organizes image features from global to local viewpoints, leading to a new objective assessment metric for retargeted images. A scale‐space matching method is designed to facilitate extraction of global geometric structures from retargeted images. By traversing the scale space from coarse to fine levels, local pixel correspondence is also established. The objective assessment metric is then based on both global geometric structures and local pixel correspondence. To evaluate color images, CIE L*a*b* color space is utilized. Experimental results are obtained to measure the performance of objective assessments with the proposed metric. The results show good consistency between the proposed objective metric and subjective assessment by human observers.