Image Retargeting Research Articles

Salient Region Detection Improved by Principle Component Analysis and Boundary Information

Salient region detection is useful for several image-processing applications, such as adaptive compression, object recognition, image retrieval, filter design, and image retargeting. A novel method to determine the salient regions of images is proposed in this paper. The L₀ smoothing filter and principle component analysis (PCA) play important roles in our framework. The L₀ filter is extremely helpful in characterizing fundamental image constituents, i.e., salient edges, and can simultaneously diminish insignificant details, thus producing more accurate boundary information for background merging and boundary scoring. PCA can reduce computational complexity as well as attenuate noise and translation errors. A local-global contrast is then used to calculate the distinction. Finally, image segmentation is used to achieve full-resolution saliency maps. The proposed method is compared with other state-of-the-art saliency detection methods and shown to yield higher precision-recall rates and F-measures.

Scalable Music: Automatic Music Retargeting and Synthesis

AbstractIn this paper we propose a method for dynamic rescaling of music, inspired by recent works on image retargeting, video reshuffling and character animation in the computer graphics community. Given the desired target length of a piece of music and optional additional constraints such as position and importance of certain parts, we build on concepts from seam carving, video textures and motion graphs and extend them to allow for a global optimization of jumps in an audio signal. Based on an automatic feature extraction and spectral clustering for segmentation, we employ length‐constrained least‐costly path search via dynamic programming to synthesize a novel piece of music that best fulfills all desired constraints, with imperceptible transitions between reshuffled parts. We show various applications of music retargeting such as part removal, decreasing or increasing music duration, and in particular consistent joint video and audio editing.

Content-Driven Retargeting of Stereoscopic Images

This letter proposes a novel warping-based method for the content-driven retargeting of stereoscopic images. Conventional algorithms in single image retargeting generally do not consider the scene depth saliency and disparity consistency when applied independently to left and right images . Therefore, the salient region and stereoscopic correlation of independently retargeted images can become corrupted. On the other hand, the proposed algorithm retains the stereo consistency of the retargeted images by matching the vertices of the grid and preserving the correspondence between them. Vertex disparity is propagated by a GPU interpolation to construct a sparse disparity map. To improve the capability of preserving visually important regions, the sparse disparity is used in conjunction with the image gradient in a saliency map computation. The experimental results show that the proposed method retains the correct disparity, while the salient objects remain undistorted in the retargeted stereoscopic image pairs.

Video Resizing Using Seam Carving and Object Detection

Video Resizing Using Seam Carving and Object Detection

Improved seam carving for stereo image resizing

Abstract When stereo images are shown in three-dimensional (3D) display devices of different aspect ratios, the resizing algorithm for single image could lead to shape and depth distortion of the stereo image’s main content. This paper aims to propose a novel method for retargeting stereo image pairs without distorting important objects in the scene while still maintaining the consistency between the left and right images. We extended seam carving algorithm to stereo images. The novelty of our method is that important objects are determined by jointly considering the intensities of gradients and visual fusion area. The retargeted stereo pair has a feasible 3D interpretation that is similar to the original one. Our method protected the important content and reduced the visual distortion in each of the images as well as the depth distortion. Experimental results are presented to demonstrate that the proposed method effectively guaranteed the geometric consistency of resized stereo images.

Efficient Image Compression based on Seam Carving for Arbitrary Resolution Display Devices

In mobile communications, image retargeting is generally required at the user end. The existing coding techniques do not support content-aware spatial scalability. The work presented in this paper addresses the increasing demand of compression techniques for arbitrary resolution display devices. This paper throws light on efficient content-aware compression of medical images. The principle of seam carving is incorporated into a wavelet codec. In the meantime, discrete wavelet transform (DWT) is performed and DWT coefficients are grouped. The coefficients are encoded and decoded using SPIHT. The bit stream is then transmitted in energy descending order and the image can be reconstructed in a content-aware manner. Experimental results show that the retargeted image generated by the proposed technique preserve sensitive image content and has been proven efficient for medical images by achieving excellent compression performances.

Stretchability-aware block scaling for image retargeting

This paper proposes an efficient approach to retarget images based on stretchability-aware block scaling. The image stretchability is first evaluated based on gradient, saliency and color features, and is used to generate the stretchable space. Then the optimal size of the stretched image is determined under the constraint of stretchable space and the same aspect ratio as the target image. Based on the analysis of image stretchability measures, the original image is partitioned into non-stretchable blocks and stretchable blocks, and their scaling factors are calculated based on their stretchability measures and the stretched image size, in order to possibly preserve non-stretchable blocks without distortion and reasonably resize stretchable blocks. Finally, the stretched image is uniformly scaled to generate the target image. Experimental results on a variety of images and the user study demonstrate that our approach achieves an overall better retargeting performance compared to the state-of-the-art image retargeting approaches.

Optimised image retargeting using aesthetic‐based cropping and scaling

Image retargeting is a critical technique in displaying images on devices with different resolutions. This study presents a new image retargeting algorithm based on aesthetic-based cropping and scaling. A composite measurement is first constructed under the guidelines of composition aesthetics in photographing. An aesthetic-based cropping is proposed to yield an optimal candidate retargeted image with maximum aesthetic value computed via a constructed composite measurement. The optimal candidate is uniformly scaled to obtain the retargeted image of target size. Some subjective and objective assessments demonstrate that the proposed scheme significantly improves the aesthetics of retargeted images while preserving the important objects. It also achieves better performance in terms of aesthetics than a number of conventional image retargeting approaches.

Matching-Area-Based Seam Carving for Video Retargeting

This paper presents a video retargeting method considering both spatial and temporal coherence for resizing videos. Our algorithm is based on a novel matching-area-based temporal energy adjustment that allows per-frame seam carving to remove the optimal pixels to achieve spatially and temporally continuous resized videos. The temporal energy adjustment allows the seam to track the object it previously carved, and avoid carving the seam on different objects in two consecutive frames to achieve both spatial and temporal coherence. Our method outperforms other state-of-the-art retargeting systems, as demonstrated in the results and widely supported by the conducted user study.

Patch-Based Image Warping for Content-Aware Retargeting

Image retargeting is the process of adapting images to fit displays with various aspect ratios and sizes. Most studies on image retargeting focus on shape preservation, but they do not fully consider the preservation of structure lines, which are sensitive to human visual system. In this paper, a patch-based retargeting scheme with an extended significance measurement is introduced to preserve shapes of both visually salient objects and structure lines while minimizing visual distortions. In the proposed scheme, a similarity transformation constraint is used to force visually salient content to undergo as-rigid-as-possible deformation, while an optimization process is performed to smoothly propagate distortions. These processes enable our approach to yield pleasing content-aware warping and retargeting. Experimental results and a user study show that our results are better than those generated by state-of-the-art approaches.

Detection of Content-Aware Image Resizing Using Seam Properties

Visually convincing content-aware image resizing, which preserves semantically important image content, has been actively researched in recent years. This paper proposes a resizing detector that reveals the trace of seam carving and seam insertion. To unveil the evidence of seam carving, we exploit energy bias of seam carved images. In addition, the correlation between adjacent pixels is analyzed to estimate the inserted seams. Empirical evidence from a large database of test images demonstrates the superior performance of the proposed detector under a variety of settings.

Improved real-time video resizing technique using temporal forward energy

A novel video resizing algorithm that preserves the dominant contents of video frames is proposed. Because a visual correlation (a similarity) exists on consecutive frames within an identical shot, the energy distribution of the neighboring frames is also correlated and similar, and then the seams in a frame are analogous to those of neighboring frames. Thus, the seams of the current frame are derived from a specified range considering the coordinates of the seams of the previous frame. The proposed method determines the two-dimensional connected paths for each frame by considering both the spatial and temporal correlations between frames to prevent jittering and jerkiness. For this purpose, a new temporal forward energy is proposed as the energy cost of a pixel. The proposed algorithm has a fast processing speed similar to the bilinear method, while preserving the main content of an image to the greatest extent possible. In addition, because the memory usage is remarkably small compared with the existing seam carving method, the proposed algorithm is usable in mobile devices, which have many memory restrictions. Computer simulation results indicate that the proposed technique provides a better objective performance, subjective image quality, and content conservation than conventional algorithms.

A Saliency Detection Model Using Low-Level Features Based on Wavelet Transform

Researchers have been taking advantage of visual attention in various image processing applications such as image retargeting, video coding, etc. Recently, many saliency detection algorithms have been proposed by extracting features in spatial or transform domains. In this paper, a novel saliency detection model is introduced by utilizing low-level features obtained from the wavelet transform domain. Firstly, wavelet transform is employed to create the multi-scale feature maps which can represent different features from edge to texture. Then, we propose a computational model for the saliency map from these features. The proposed model aims to modulate local contrast at a location with its global saliency computed based on the likelihood of the features, and the proposed model considers local center-surround differences and global contrast in the final saliency map. Experimental evaluation depicts the promising results from the proposed model by outperforming the relevant state of the art saliency detection models.

Improved Seam Merging for Content-Aware Image Resizing

In this paper, we propose an improved seam merging method for content-aware image resizing. This method merges a two-pixel-width seam element into one new pixel in image reduction and inserts a new pixel between the two pixels in image enlargement. To preserve important contents and structure, our method uses energy terms associated with importance and structure. Our method preserve the main structures by using a cartoon version of the original image when calculating the structure energy. In addition, we introduce a new energy term to suppress the distortion generated by excessive reduction or enlargement in iterated merger or insertion. Experimental results demonstrate that the proposed method can produce satisfactory results in both image reduction and enlargement.

Dynamic distortion maps for image retargeting

We present an image retargeting method that incorporates image content distortion into a mesh optimization process through the generation of dynamic distortion maps. The warping process is driven by the distortion produced by the warping process itself. We retarget the image through an iterative mesh optimization process to minimize the visual distortion. An adaptive distortion map is iteratively constructed to describe the visual distortion between the original image and the retargeted image. The mesh mapping from the source image to the retargeted image is optimized through an energy minimization process. The objective of the optimization is to allow the distortion produced by the retargeting process to be distributed to smooth and highly textured regions which cause less visual distortion while preserving the geometrical structure of the mesh at regions that may cause distortion after retargeting. Experimental evaluation of the algorithm is done both subjectively and objectively.

Comparative Study of Fixation Density Maps

Fixation density maps (FDM) created from eye tracking experiments are widely used in image processing applications. The FDM are assumed to be reliable ground truths of human visual attention and as such, one expects a high similarity between FDM created in different laboratories. So far, no studies have analyzed the degree of similarity between FDM from independent laboratories and the related impact on the applications. In this paper, we perform a thorough comparison of FDM from three independently conducted eye tracking experiments. We focus on the effect of presentation time and image content and evaluate the impact of the FDM differences on three applications: visual saliency modeling, image quality assessment, and image retargeting. It is shown that the FDM are very similar and that their impact on the applications is low. The individual experiment comparisons, however, are found to be significantly different, showing that inter-laboratory differences strongly depend on the experimental conditions of the laboratories. The FDM are publicly available to the research community.

Optimized image resizing using flow-guided seam carving and an interactive genetic algorithm

In this paper, we introduce a novel method for content-aware image resizing based on flow-guided seam carving. It extends the existing seam carving framework by replacing the conventional energy field with a "structure-aware" energy field that takes into account the feature orientations in the image. Guided by this new energy field, our approach excels in preserving (i.e., avoiding the distortion of) important structures in the image, such as shape boundaries. We also present a simple user interface to further optimize the resizing result based on the genetic selection process among multiple resizing operators such as scaling, cropping, and flow-guided seam carving. We show that such simple user interaction, coupled with the genetic algorithm, dramatically increases the chances of producing the user-desired outcome.

Image Retargeting Quality Assessment: A Study of Subjective Scores and Objective Metrics

This paper presents the result of a recent large-scale subjective study of image retargeting quality on a collection of images generated by several representative image retargeting methods. Owning to many approaches to image retargeting that have been developed, there is a need for a diverse independent public database of the retargeted images and the corresponding subjective scores to be freely available. We build an image retargeting quality database, in which 171 retargeted images (obtained from 57 natural source images of different contents) were created by several representative image retargeting methods. And the perceptual quality of each image is subjectively rated by at least 30 viewers, meanwhile the mean opinion scores (MOS) were obtained. It is revealed that the subject viewers have arrived at a reasonable agreement on the perceptual quality of the retargeted image. Therefore, the MOS values obtained can be regarded as the ground truth for evaluating the quality metric performances. The database is made publicly available (Image Retargeting Subjective Database, [Online]. Available: http://ivp.ee.cuhk.edu.hk/projects/demo/retargeting/index.html) to the research community in order to further research on the perceptual quality assessment of the retargeted images. Moreover, the built image retargeting database is analyzed from the perspectives of the retargeting scale, the retargeting method, and the source image content. We discuss how to retarget the images according to the scale requirement and the source image attribute information. Furthermore, several publicly available quality metrics for the retargeted images are evaluated on the built database. How to develop an effective quality metric for retargeted images is discussed through a specifically designed subjective testing process. It is demonstrated that the metric performance can be further improved, by fusing the descriptors of shape distortion and content information loss.

Context-Aware Saliency Detection

We propose a new type of saliency—context-aware saliency—which aims at detecting the image regions that represent the scene. This definition differs from previous definitions whose goal is to either identify fixation points or detect the dominant object. In accordance with our saliency definition, we present a detection algorithm which is based on four principles observed in the psychological literature. The benefits of the proposed approach are evaluated in two applications where the context of the dominant objects is just as essential as the objects themselves. In image retargeting, we demonstrate that using our saliency prevents distortions in the important regions. In summarization, we show that our saliency helps to produce compact, appealing, and informative summaries.

Content Aware Retargeting of Stego Images

Images play a vital role in steganography, as cover media to hide secret messages. Retargeting becomes mandatory while transmitting image in a heterogeneous network because sender and recipient may not have display devices of same resolution. Resizing images by scaling or cropping also lead to loss of visually important features. In steganography, resizing, compressing, and changing the color or format of stego images would result in loss of hidden information. In this paper we attempt to retarget stego images in a content preserving manner using a modified seam carving technique called PSC. Content in this context refers to the visually salient features in the cover image as well as the secret message hidden within it. The cover image is first subjected to PSC to extract as many seams leaving the segments containing ROIs intact. Based on the size of secret message segment(s) containing the ROIs are selected to encode the secret message and is transmitted to the receiver. The seams are also encoded separately in the reverse order of their extraction, so that on the receiver end the receiver would download all the image segments containing the ROIs and as many seams as required to constitute an image that fits into the target device. The retargeted image contains the secret message intact which could be decoded by the receiver.