Image Retargeting Quality Assessment Research Articles

A comprehensive review of image retargeting

With the development of display technologies, image retargeting plays a significant role in computer vision and pattern recognition communities currently. Image retargeting aims to display an image on a series of appliances with different resolutions and target aspect ratios. During the last decade, representative algorithms for image retargeting have been presented in the literature and achieved state-of-the-art performance. In this survey, we provide a comprehensive review of image retargeting, covering a wide variety of pioneering works for 2D image retargeting and stereoscopic image retargeting. 2D image retargeting focuses on preserving interesting regions when modifying the original image with arbitrary resolutions appropriately. Different from 2D image retargeting, stereoscopic image retargeting needs to preserve both the shape structure of salient objects and the depth consistency of 3D scenes simultaneously. In this survey, we start the first attempt to analyze the trends of 2D image retargeting and then summarize different types of stereoscopic image retargeting. Secondly, image retargeting quality assessment metrics for 2D images and stereoscopic images are introduced to evaluate retargeted images. Thirdly, we also investigate the evaluation datasets, and give the comparison results and analysis between different representative methods. Finally, the promising future research is thoroughly discussed to further improve the performance of 2D and stereoscopic image retargeting.

Integration of local and global features for image retargeting quality assessment

Integration of local and global features for image retargeting quality assessment

Image retargeting quality assessment: A survey

With the increasing variety of display devices, image retargeting has become an indispensable technology for adjusting the aspect ratio of images to adapt to different display terminals. Since the retargeting operation would cause geometric distortion and content loss of the image, the image retargeting quality assessment (IRQA) is necessary to guide the retargeting algorithm’s optimization, selection, and design. Our paper mainly works for systematically reviewing the state-of-the-art technologies in IRQA. And then, this paper further discusses image registration algorithms for matching the original image and the retargeted image. Next, we investigate the feature measurement methods for image retargeting quality evaluation. To facilitate the quantitative assessment of the IRQA methods, this paper gives a list of publicly open datasets and the performance of the mainstream methods. Finally, some promising research directions towards IRQA are pointed out. From this survey, engineers from the industry may find skills to improve their image retargeting systems, and researchers from academia may find ideas to conduct some innovative work.

LGGD+: Image Retargeting Quality Assessment by Measuring Local and Global Geometric Distortions

Numerous image retargeting algorithms have been proposed to achieve adaptive image resizing during the past years. To compare different image retargeting algorithms, reliable objective image retargeting quality assessment (IRQA) metrics are highly desired. Given that image retargeting usually introduces geometric distortions, this paper presents an objective IRQA metric by measuring both local and global geometric distortions (LGGD). Since human visual system perception is highly dependent on edges and the geometric distortions caused by image retargeting usually cause edge deformation, a sketch token-based local edge descriptor (ST-LED) is introduced to represent geometric-aware features in LGGD. First, ST-LED is first applied on both source and retargeted images for edge pattern representation. Second, pixel-level backward registration is conducted to enable estimating local geometric distortion (LGD) and a spatial pyramid-improved Bag-of-Token (BoT) model is built to enable estimating global geometric distortion (GGD). Since the proposed LGGD metric only focuses on geometric distortion while image retargeting quality is related with more aspects, we further fuse LGGD and an existing (EXT) IRQA metric to build a final version called LGGD+ for IRQA. Experiments on two benchmark databases demonstrate the superiority of LGGD+ and the excellent compatibility of our proposed LGGD for further improving a wide range of existing IRQA metrics (including both geometric distortion and non-geometric distortion metrics). In addition, the effectiveness of our LGGD metric is also demonstrated in another relevant task, i.e., quality evaluation of depth-image-based rendering (DIBR)-synthesized images, which also calls for accurate estimation of geometric distortion.

Integration of Deep Learned and Handcrafted Features for Image Retargeting Quality Assessment

This paper proposed an image retargeting quality assessment. In the proposed method, a deep convolution network is trained on the pixel displacement patterns of different image retargeting methods to produce a measure for evaluating the quality of output images. Also, the method extracts three other measures that assess the geometric changes of important objects in the image, the bending of block lines, and the extent of information loss during the retargeting process. The tests performed on two well-known databases, RetargetMe and CUHK, demonstrate the excellent performance, stability, and reliability of the proposed method compared to the existing methods. In this paper, we present a new method for evaluating the quality of retargeted images. The innovations of the proposed method are: Using a deep learning method to identify the important regions of the image that contain foreground objects and people Providing a quality evaluation measure, obtained by training a CNN with regression output on the pixel displacement patterns in different image retargeting methods Extracting the foreground objects in the original image and the corresponding objects in the retargeted image and determining the extent of geometric change in each object. Estimating the extent of information loss and distortion based on the extent to which the blocks of the original image have been bent. Estimating the quality score of the retargeted image using Gaussian process regression.

Image retargeting quality assessment based on saliency-driven classification

The overall geometric distortions of the visually prominent contents in resized images are sensitive to human perception. But the existing image retargeting quality assessment (IRQA) algorithms did not consider it, leading to a lower performance of evaluation algorithms. Additionally, the visually important region needs to be extracted from the retargeted images to measure their distortions. However, due to the size change and content alternation during retargeting, the visually prominent regions of the resized images obtained by the existing saliency detection approaches do not match the corresponding portions of the original photos. To address these issues, we propose an IRQA method based on saliency-driven classification, which uses various metrics to evaluate the qualities of different retargeted images. We also design a novel algorithm to extract the salient region of the retargeted image, which utilizes the original image to perform backward registration and position matching of the salient area. Besides, we present an improved aspect ratio similarity (ARS) metric to measure the overall geometric distortion occurring in the visually important region of the resized images. Experimental results show that the performance of the proposed IRQA method is superior to the other existing algorithms.

Integration of Local and Global Features Using Gaussian Process Regression for Image Retargeting Quality Assessment

Integration of Local and Global Features Using Gaussian Process Regression for Image Retargeting Quality Assessment

Reduced Reference Quality Assessment for Image Retargeting by Earth Mover’s Distance

A reduced reference quality assessment algorithm for image retargeting by earth mover’s distance is proposed in this paper. In the reference image, all the feature points are extracted using scale invariant feature transform. Let the histograms of image patch around each feature point be local information, and the histograms of saliency feature as global information. Those feature information is extracted at the sender side and transmitted to the receiver side. After that, the same feature information extraction process is performed for the retargeted image at the receiver side. Finally, all feature information of the reference and retargeted images is used collectively to compute the quality of the retargeted image. An overall quality score is calculated from the local and global similarity measure using earth mover’s distance between reference and retargeted images. The key step in our algorithm is to provide an earth mover’s distance metric in a manner that indicates how the local and global information in the reference image is preserved in corresponding retargeted image. Experimental results show that the proposed algorithm can improve the image quality scores on four common criteria in the retargeted image quality assessment community.

Transformation-Aware Similarity Measurement for Image Retargeting Quality Assessment via Bidirectional Rewarping

Image retargeting is an effective way to adapt images for target displays with different aspect ratios and sizes. Meanwhile, effective image retargeting quality assessment (IRQA) is important for optimizing the image retargeting operations. In this paper, we propose a transform-aware similarity (TRASIM) measurement metric for IRQA, including bidirectional geometric distortion measurement, bidirectional information loss measurement, and global salient structure distortion measurement. The main innovation of the TRASIM is to build a universal framework to establish the similarity transformation via bidirectional rewarping to simulate different types of retargeting operators. Based on the similarity transformation, geometric distortion and content loss are measured to determine the retargeting quality. Experimental results on two widely used databases (CUHK and RetargetMe) indicate that the proposed TRASIM has higher consistency with subjective ranks, compared with the state-of-the-art IRQA metrics.

List-Wise Rank Learning for Stereoscopic Image Retargeting Quality Assessment

Stereoscopic image retargeting (SIR) techniques attempt to display stereoscopic images on stereoscopic devices of various resolutions and aspect ratios to provide the users with better viewing experience. However, new quality perceptual problems emerge in the retargeted stereoscopic images generated by current SIR operators are quite different from those in the retargeted 2D images. In this paper, we dedicate to exploring the perceptual quality-related factors (e.g., shape preservation, object preservation and visual comfort.) of retargeted stereoscopic images, and propose a novel quality evaluation metric for SIR to achieve a more consistent evaluation with 3D perception and image degradation mechanism in the SIR process. Moreover, image quality features and 3D perceptual features are integrated into one representation for an overall perceptual quality prediction using a list-wise ranking approach, which gives priority to the ranking among the SIR results generated from the same stereoscopic source. Experimental results demonstrate that the proposed method outperforms most quality models developed for retargeted 2D/stereoscopic images.

Quality Evaluation for Image Retargeting With Instance Semantics

To meet the ever-increasing demand for devices with diversified displays, image retargeting has become a prevalent technique for adaptive image resizing. In practice, the retargeting operation inevitably causes impairments in the images; thus, image retargeting quality assessment (IRQA) is urgently needed and, can be used to guide algorithm optimization, selection and design. Unlike traditional image quality assessment, image retargeting introduces geometric distortions, which typically affect high-level image semantics. With this motivation, this paper presents a quality evaluation model for image retargeting based on INstance SEMantics (INSEM). Considering that the human visual system (HVS) perceives images highly dependent on apprehensible areas and that impairments in image retargeting mainly degrade the salient instances, an image instance is utilized as the basic semantic unit, and a top-down method is devised to extract instance-level semantic features for IRQA. In addition, taking into account the influence of semantic categories on the perception of retargeting quality, we further propose Semantic-based self-adaptive pooling (SSAP) to integrate instance-based semantic features. Finally, global features are incorporated to generate quality scores that are more consistent with people's perceptions. Extensive experiments and comparisons of three public databases, in terms of both intradatabase and cross-database settings, demonstrate the superiority of the proposed metric over state-of-the-art methods.

Subjective and Objective Quality Assessment for Stereoscopic Image Retargeting

Binocular stereoscopic image retargeting (SIR) aims to adjust 3D images into target aspect ratios. In recent years, various SIR methods have been proposed, but there are few researches on visual quality assessment. As a consequence, we construct a benchmark stereoscopic image retargeting quality assessment database (NBU-SIRQA), which contains 720 stereoscopic retargeted images generated by eight representative SIR operators. Subjective test is conducted to obtain the mean opinion score (MOS) for each stereoscopic retargeted image. Additionally, we propose an objective SIRQA metric based on grid deformation and information loss (GDIL). The main idea of GDIL is to decompose the SIR operator into two transformations: monocular image retargeting transformation and viewpoint transformation. In each transformation, grid deformation and information loss are extracted simultaneously to represent image quality and 3D perception quality. Experimental results validated on our established NBU-SIRQA database show the superiority of our metric in measuring the quality of stereoscopic retargeted images over the existing approaches.

Image Retargeting Quality Assessment Based on Registration Confidence Measure and Noticeability-Based Pooling

Nowadays, image retargeting approaches have been widely applied to adapt images of various resolutions to heterogenous display devices. To assess the quality of the retargeted images, image retargeting quality assessment (IRQA) has emerged as a critical problem in image quality assessment. In this paper, we address the IRQA problem with a newly proposed framework based on registration confidence measurement (RCM) and noticeability-based pooling (NBP). First, we define the RCM to evaluate the accuracy of image registration, which aligns scenes between the original and retargeted images. We then integrate the proposed RCM with the computed local fidelity of each image block to alleviate the negative influence of inaccurate registration on fidelity measurements. Meanwhile, we present a visual attention fusion (VAF) framework to enhance faces and lines in the saliency map, which are observed to be highly sensitive in the human visual system (HVS). Finally, we propose the NBP strategy, which aggregates the local fidelity of each image block into the overall quality of the retargeted image. Specifically, the NBP strategy sets larger quality ranges for the regions where the visual distortions are more accessible to HVS to reflect the easy noticeability of these regions. Experimental results on the MIT RetargetMe and CUHK datasets demonstrate that the proposed IRQA metric based on RCM and NBP outperforms the state-of-the-art IRQA metrics.

Retargeted image quality assessment based on multi-scale distortion-aware feature

Retargeted image quality assessment based on multi-scale distortion-aware feature

A Deep Evaluator for Image Retargeting Quality by Geometrical and Contextual Interaction

An image is compressed or stretched during the multidevice displaying, which will have a very big impact on perception quality. In order to solve this problem, a variety of image retargeting methods have been proposed for the retargeting process. However, how to evaluate the results of different image retargeting is a very critical issue. In various application systems, the subjective evaluation method cannot be applied on a large scale. So we put this problem in the accurate objective-quality evaluation. Currently, most of the image retargeting quality assessment algorithms use simple regression methods as the last step to obtain the evaluation result, which are not corresponding with the perception simulation in the human vision system (HVS). In this paper, a deep quality evaluator for image retargeting based on the segmented stacked AutoEnCoder (SAE) is proposed. Through the help of regularization, the designed deep learning framework can solve the overfitting problem. The main contributions in this framework are to simulate the perception of retargeted images in HVS. Especially, it trains two separated SAE models based on geometrical shape and content matching. Then, the weighting schemes can be used to combine the obtained scores from two models. Experimental results in three well-known databases show that our method can achieve better performance than traditional methods in evaluating different image retargeting results.

Ranking-Preserving Cross-Source Learning for Image Retargeting Quality Assessment.

Image retargeting techniques adjust images into different sizes and have attracted much attention recently. Objective quality assessment (OQA) of image retargeting results is often desired to automatically select the best results. Existing OQA methods train a model using some benchmarks (e.g., RetargetMe), in which subjective scores evaluated by users are provided. Observing that it is challenging even for human subjects to give consistent scores for retargeting results of different source images (diff-source-results), in this paper we propose a learning-based OQA method that trains a General Regression Neural Network (GRNN) model based on relative scores-which preserve the ranking-of retargeting results of the same source image (same-source-results). In particular, we develop a novel training scheme with provable convergence that learns a common base scalar for same-source-results. With this source specific offset, our computed scores not only preserve the ranking of subjective scores for same-source-results, but also provide a reference to compare the diff-source-results. We train and evaluate our GRNN model using human preference data collected in RetargetMe. We further introduce a subjective benchmark to evaluate the generalizability of different OQA methods. Experimental results demonstrate that our method outperforms ten representative OQA methods in ranking prediction and has better generalizability to different datasets.

Content-Aware Retargeted Image Quality Assessment

In targeting the low correlation between existing image scaling quality assessment methods and subjective awareness, a content-aware retargeted image quality assessment algorithm is proposed, which is based on the structural similarity index. In this paper, a similarity index, that is, a local structural similarity algorithm, which can measure different sizes of the same image is proposed. The Speed Up Robust Feature (SURF) algorithm is used to extract the local structural similarity and the image content loss degree. The significant area ratio is calculated by extracting the saliency region and the retargeted image quality assessment function is obtained by linear fusion. In the CUHK image database and the MIT RetargetMe database, compared with four representative assessment algorithms and other latest four kinds of retargeted image quality assessment algorithms, the experiment proves that the proposed algorithm has a higher correlation with Mean Opinion Score (MOS) values and corresponds with the result of human subjective assessment.

Visual Attention Fusion Framework for Image Retargeting Quality Assessment

To adapt images for diversified digital devices, researchers have presented many image retargeting methods. However, the consistency between results of objective image retargeting quality assessment (IRQA) metrics and subjective perception is still low. In this paper, we propose a visual attention fusion (VAF) framework to assist IRQA metrics in better understanding the features of images such as image saliency, faces, and lines. First, we combine the results of multiple salient object detection algorithms to reduce the limitations of a single algorithm. Second, faces and lines are considered in our framework to measure deformations to these visually sensitive regions. Finally, we propose a saliency enhancement model to simulate human visual attention for IRQA. We combine the proposed VAF framework with some state-of-the-art IRQA metrics. Experimental results show that the proposed VAF framework can improve the consistency between the results of objective IRQA metrics and subjective opinion scores.

A Content-Aware Image Retargeting Quality Assessment Method Using Foreground and Global Measurement

Image retargeting methods aim to minimize the perceptual loss while changing sizes and aspect ratios of images. Since optimal retargeting methods for different images are generally not the same, the image retargeting quality assessment (IRQA) becomes a meaningful task. This paper proposes a content-aware image retargeting quality assessment method using foreground and global measurement to achieve better performance. In our proposed method, images are first divided into two categories according to the foreground object detection result, and then different corresponding measurements are designed for them. For those with obvious foreground object, both foreground and global measurement are applied. For others, only global measurement is conducted. Foreground measurement includes two complementary features: the high-level semantic similarity feature and the low-level size ratio feature. Global measurement includes another two features: an improved aspect ratio similarity (ARS) feature and edge group similarity (EGS) feature. Two public databases, i.e., the RetargetMe and CUHK, have been evaluated, and experimental results demonstrate that our method is quite effective, and it also provides state-of-the-art performance in the IRQA. aa Our code are available at https://github.com/SCUT-ML-GUO/IRQA.

Image retargeting quality assessment based on content deformation measurement

Content-aware based image retargeting methods aim at visualizing source images on diverse display devices with different aspect ratios while preventing the incurrence of visual distortions as far as possible. An objective retargeting assessment method, which has a high consistency with subjective perception, can measure the retargeting effects of different retargeting methods, thus promoting the development of retargeting technology. However, the correlation between the existing objective evaluation methods and subjective evaluation is not strong enough. In this paper, we propose an objective retargeting assessment method by fusing multiple features to quantify the content loss and structural deformation caused by retargeting. In order to measure the global deformation, there are two features are proposed which are elastic registration based on B-spline function and foreground retention ratio. Moreover, improved Aspect Ratio Similarity (IARS) and concentrated deletion ratio make for better detection of local deformation. These features are fused to obtain the objective retargeting assessment method. Compared with the state-of-art objective assessment methods on three public datasets, the proposed method has a better correlation with subjective perception.