Content-aware Image Retargeting Research Articles

SMART: Supervised Multi-class Image Retargeting Generative Model based on a Long-range Sampling Strategy

Content-aware image retargeting (CAIR) techniques are crucial in multimedia processing for displaying images on various devices while preserving visually salient contents with desirable visual effects. There are discrete and continuous algorithms. For the former, the artefacts happen when the foreground proportion is larger than the retargeting ratio; for the latter, the salient regions are prone to be squeezed. In this paper, we reformulate the retargeting process into sampling the salient signal and reconstruction under aesthetic supervision, the supervised multi-class image retargeting reconstruction (SMART) framework. The target images can be represented into complementary parts, the masked and unmasked ones, according to the saliency influences in the encoder phrase. The long-range sampling algorithm is proposed to calculate similarities through an 8-connected planar path while considering spatial distance and feature correlation. The sampled embeddings in latent space reconstruct the retargeted images under supervised signals for aesthetic quality. The semantic loss Lsem from the pretrained CLIP model can maintain consistency for both content and semantics. The supervised loss, Lir, is introduced to ensure the retargeted qualities are close to the preferred labels. Then, we release a new retargeting dataset comprising seven image classes (animal, building, car, flower, indoor, landscape and people) with supervised labels collected from designers for further aesthetic retargeting study. The ablation studies are conducted to confirm the effectiveness of the new dataset, and comparative experiments with state-of-the-art baselines demonstrate the advantages of the proposed method.

Supervised deep learning for content-aware image retargeting with Fourier Convolutions

Supervised deep learning for content-aware image retargeting with Fourier Convolutions

A shadow preservation framework for effective content-aware image retargeting process

<p>In the discipline of image retargeting, the structure of shadows in the image is required to be preserved since it gives structural information about the objects. Many traditional image retargeting techniques do not pay attention to preserve the shadow objects present in the image. In this paper a shadow-preservation framework is proposed in which the saliency map is derived using segmentation-based gPb-owt-ucm approach to emphasize the salient regions. To assess the quality of saliency map a fixation prediction analysis is conducted on each section of the image using the eye tracker technology. Second, to pay more attention on the shadow objects a shadow map is developed after executing image pre-processing, shadow detection, region growing, and region filling operations. To rectify the issue of misclassification of shadow pixels erosion and dilation operations are performed. To obtain an efficient importance map, a gradient map is prepared using canny operator which is further combined with the saliency map and shadow map. The obtained importance map is supplied to the seam diversion based image retargeting (SDIR) technique to resize the image in horizontal and vertical directions. To justify the efficiency of the proposed framework’s the obtained results are compared with the existing state-of-the-art. By preserving shadows, the proposed framework enhances the overall visual quality and maintains the integrity of objects in the retargeted images. The framework’s effectiveness is validated through quantitative evaluations and visual comparisons with existing methods. The results demonstrate its potential for improving content-aware image retargeting applications, paving the way for more realistic and visually appealing image resizing techniques in various domains.<em></em></p>

Content-aware image retargeting technique and iterated function system: frameworks, applications, and possible future advancements

Content-aware image retargeting technique and iterated function system: frameworks, applications, and possible future advancements

IR-WCO2C An Enhanced Multi Operators-Based Approach for Content-Aware Image Retargeting

IR-WCO2C An Enhanced Multi Operators-Based Approach for Content-Aware Image Retargeting

Deep Convolutional Neural Network for Identifying Seam-Carving Forgery

Seam carving is a representative content-aware image retargeting approach to adjust the size of an image. To preserve visually prominent content, seam-carving algorithms first calculate the connected path of pixels, referred to as the seam, according to a defined cost function and then adjust the size of an image by removing or duplicating repeatedly calculated seams. Seam carving is actively exploited to overcome diversity in the resolution of images between applications and devices; hence, detecting the distortion caused by seam carving has become important in image forensics. In this paper, we propose a convolutional neural network (CNN)-based approach to classifying seam-carving forgery. To attain the ability to learn low-level features, we designed a convolutional neural network (CNN) architecture comprising five types of network blocks specialized in capturing local artifacts caused by seam carving. An ensemble module is further adopted to both enhance performance and comprehensively analyze the features in the local areas. To validate the effectiveness of our work, extensive experiments based on various CNN-based baselines were conducted. Compared to the baselines, our work exhibits state-of-the-art performance in terms of three-class classification (original, seam inserted, and seam removed). The experimental results also demonstrate that our model with the ensemble module is robust for various unseen cases.

Structure preservation of image using an efficient content-aware image retargeting technique

The presence of a wide range of high- and low-resolution devices renders the images to experience changes in respect to aspect ratio and size for better adaptability. This paper proposes a novel and effective technique that vanquishes the problems encountered in the conventional seam carving method. To achieve minimum distortion in salient objects of the image, the proposed technique restricts intersection or overlapping of multiple seams in the horizontal and vertical direction and bypasses them to the neighboring low energy pixel. The proposed technique hinders the pixel selection from a single row or column beyond the defined threshold so as to save the image information and reduce distortion at a single location. To justify the effectiveness of the proposed technique, results have been presented in comparison with the five state-of-the-art image retargeting techniques. Compared with the conventional techniques, this technique shows remarkable results in terms of low distortion percentage. The proposed technique also produces excellent results for shrinkage and enlargement of a single image multiple times.

Conditional Random Field based salient proposal set generation and its application in content aware seam carving

Most of the existing generic object localization algorithms usually give the plausible object locations without taking into consideration the saliency ordering of the proposal set. This paper presents a novel object proposal generation which ranks the key objects according to their saliency score in the proposal pool. First, we formulate a Bayesian framework for generating a probabilistic edgemap which is used to assign a saliency value to the edgelets. A conditional random field is then learnt for edge-labeling by effectively combining the edge features with the relative spatial layout of the edge segments. Lastly, we propose an objectness score for the generated proposal set by analyzing the salient object edge density completely lying within the candidate boxes. Extensive experiments on the benchmark PASCAL VOC 2007 and 2012 datasets demonstrate that the proposed method provides competitive performance against popular generic object detection techniques while using fewer number of proposals. Additionally, we demonstrate the applicability of the generated proposal set for content aware image retargeting.

Image Retargetability

Real-world applications could benefit from the ability to automatically retarget an image to different aspect ratios and resolutions while preserving its visually and semantically important content. However, not all images can be equally processed. This study introduces the notion of image retargetability to describe how well a particular image can be handled by content-aware image retargeting. We propose to learn a deep convolutional neural network to rank photo retargetability, in which the relative ranking of photo retargetability is directly modeled in the loss function. Our model incorporates the joint learning of meaningful photographic attributes and image content information, which can facilitate the regularization of the complicated retargetability rating problem. To train and analyze this model, we collect a dataset that contains retargetability scores and meaningful image attributes assigned by six expert raters. The experiments demonstrate that our unified model can generate retargetability rankings that are highly consistent with human labels. To further validate our model, we show the applications of image retargetability in retargeting method selection, retargeting method assessment and generating a photo collage.

Accelerated seam carving for image retargeting

Display of images on different display devices having varied size and aspect ratio requires one to resize them. Many attempts have been made to perform content-aware image retargeting while generating an image compatible with a target display size. Seam carving is one of the image retargeting operators which alters the size of an image by removing least energy pixels. However, it requires high computational time in order to perform retargeting. In this study, the authors accelerate the naive seam carving process by removal or insertion of multiple pixel wide batch seam in a single iteration rather than a single pixel wide seam. Along with the energy of pixels to be removed, inserted energy after the removal of a batch seam is also minimised in order to prevent the inclusion of false edges. The width of a batch seam is a critical factor which is made adaptive during the retargeting process to preserve the energy of an image. They have shown a significant decrease in computational time with the increase in the width of a batch seam. They have compared the proposed technique with other state-of-the-art image retargeting operators using different quality assessment metrics and visual results.

A Content-Aware Image Retargeting Quality Assessment Method Using Foreground and Global Measurement

Image retargeting methods aim to minimize the perceptual loss while changing sizes and aspect ratios of images. Since optimal retargeting methods for different images are generally not the same, the image retargeting quality assessment (IRQA) becomes a meaningful task. This paper proposes a content-aware image retargeting quality assessment method using foreground and global measurement to achieve better performance. In our proposed method, images are first divided into two categories according to the foreground object detection result, and then different corresponding measurements are designed for them. For those with obvious foreground object, both foreground and global measurement are applied. For others, only global measurement is conducted. Foreground measurement includes two complementary features: the high-level semantic similarity feature and the low-level size ratio feature. Global measurement includes another two features: an improved aspect ratio similarity (ARS) feature and edge group similarity (EGS) feature. Two public databases, i.e., the RetargetMe and CUHK, have been evaluated, and experimental results demonstrate that our method is quite effective, and it also provides state-of-the-art performance in the IRQA. aa Our code are available at https://github.com/SCUT-ML-GUO/IRQA.

A Quad Edge-Based Grid Encoding Model for Content-Aware Image Retargeting.

In this paper, we present a novel grid encoding model for content-aware image retargeting. In contrast to previous approaches such as vertex-based and axis-aligned grid encoding models, our approach takes each horizontal/vertical distance between two adjacent vertices as an optimization variable. Upon this difference-based encoding scheme, every vertex position of a target grid is subsequently determined after optimizing the one-dimensional values. Our quad edge-based grid model has two major advantages for image retargeting. First, the model enables a grid optimization problem to be developed in a simple quadratic program while ensuring the global convexity of objective functions. Second, due to the independency of variables, spatial regularizations can be applied in a locally adaptive manner to preserve structural components. Based on this model, we propose three quadratic objective functions. Note that, in our work, their linear combination guides a grid deformation process to obtain a visually comfortable retargeting result by preserving salient regions and structural components of an input image. Comparative evaluations have been conducted with ten existing state-of-the-art image retargeting methods, and the results show that our method built upon the quad edge-based model consistently outperforms other previous methods both on qualitative and quantitative perspectives.

An optimized content-aware image retargeting method: toward expanding the perceived visual field of the high-density retinal prosthesis recipients

Objective. Retinal prosthesis devices have shown great value in restoring some sight for individuals with profoundly impaired vision, but the visual acuity and visual field provided by prostheses greatly limit recipients’ visual experience. In this paper, we employ computer vision approaches to seek to expand the perceptible visual field in patients implanted potentially with a high-density retinal prosthesis while maintaining visual acuity as much as possible. Approach. We propose an optimized content-aware image retargeting method, by introducing salient object detection based on color and intensity-difference contrast, aiming to remap important information of a scene into a small visual field and preserve their original scale as much as possible. It may improve prosthetic recipients’ perceived visual field and aid in performing some visual tasks (e.g. object detection and object recognition). To verify our method, psychophysical experiments, detecting object number and recognizing objects, are conducted under simulated prosthetic vision. As control, we use three other image retargeting techniques, including Cropping, Scaling, and seam-assisted shrinkability. Main results. Results show that our method outperforms in preserving more key features and has significantly higher recognition accuracy in comparison with other three image retargeting methods under the condition of small visual field and low-resolution. Significance. The proposed method is beneficial to expand the perceived visual field of prosthesis recipients and improve their object detection and recognition performance. It suggests that our method may provide an effective option for image processing module in future high-density retinal implants.

Objective Quality Assessment of Image Retargeting by Incorporating Fidelity Measures and Inconsistency Detection.

The tremendous growth in mobile devices has resulted in huge generation and usage of digital images. Image quality assessment is thus an important issue for mobile media applications. In this paper, we focus on the quality evaluation of images generated by content-aware image retargeting, in which the reference and the distorted images are of different sizes. Through retargeting, many types of deformation inconsistency lead to shape distortion, deformation artifacts, and content information loss, worsening its perceptual quality. The deformation inconsistency occurs on different levels of the retargeted images. Limited by the accuracy of the alignment between the original and retargeted images, previous methods only focus on pixel-level and patch-level fidelity analyses and fail to detect deformation inconsistency. In this paper, we improve the alignment algorithm and propose a three-level representation of the retargeting process. Based on the analysis of this three-level representation, both fidelity measures and inconsistency detection are combined to determine the final retargeting quality. The proposed algorithm is validated on the public data sets RetargetMe and CUHK. Experimental results demonstrate that inconsistency detection contributes to accurately assessing the image retargeting perceptual quality. This inspires us to investigate more about deformation inconsistency to formulate the objective quality of image retargeting.

Detecting image seam carving with low scaling ratio using multi-scale spatial and spectral entropies

Seam carving is the most popular content-aware image retargeting technique. However, it may also be used to correct poor photo composition in photography competition or to remove object from image for malicious purpose. A blind detection approach is presented for seam carved image with low scaling ratio (LSR). It exploits spatial and spectral entropies (SSE) on multi-scale images (candidate image and its down-sampled versions). We observe that when a few seams are deleted from an original image, its SSE distribution is greatly changed. Forty-two features are designed to unveil the statistical properties of SSE in terms of centralized tendency, dispersion tendency and distribution tendency. They are combined with the local binary pattern (LBP)-based energy features to form ninety-six features. Finally, support vector machine (SVM) is exploited as classifier to determine whether an image is original or suffered from seam carving. Experimental results show that the proposed approach achieves superior detection accuracy over the state-of-the-art works, especially for resized image by seam carving with LSRs. Moreover, it is robust against JPEG compression and seam insertion.

Codebook Guided Feature-Preserving for Recognition-Oriented Image Retargeting.

Traditional image resizing methods, such as uniform scaling and content-aware image retargeting, are designed to preserve the visually salient contents of an image while resizing it. In this paper, we propose a novel image resizing approach called recognition-oriented image retargeting. Its goal is to preserve the distinctive local features for recognition instead of the traditional visual saliency during resizing. Moreover, we also apply our approach to image matching and image retrieval applications to verify its performance. Meanwhile, using our approach to these applications is able to solve some of the challenging problems in their fields. In image matching application, we find that our approach shows promising preservation of local feature descriptors. In image retrieval task, extensive experiments on Oxford5K, Holidays, Paris, and Flickr100k data sets demonstrate that our approach consistently outperforms other image retargeting methods by large margins in the aspects of retrieval precision and query bits.

Comparative Analysis of Gradient Operators Based on Content Aware Image Retargeting

Comparative Analysis of Gradient Operators Based on Content Aware Image Retargeting

Backward Registration-Based Aspect Ratio Similarity for Image Retargeting Quality Assessment

During the past few years, there have been various kinds of content-aware image retargeting operators proposed for image resizing. However, the lack of effective objective retargeting quality assessment metrics limits the further development of image retargeting techniques. Different from traditional image quality assessment (IQA) metrics, the quality degradation during image retargeting is caused by artificial retargeting modifications, and the difficulty for image retargeting quality assessment (IRQA) lies in the alternation of the image resolution and content, which makes it impossible to directly evaluate the quality degradation like traditional IQA. In this paper, we interpret the image retargeting in a unified framework of resampling grid generation and forward resampling. We show that the geometric change estimation is an efficient way to clarify the relationship between the images. We formulate the geometric change estimation as a backward registration problem with Markov random field and provide an effective solution. The geometric change aims to provide the evidence about how the original image is resized into the target image. Under the guidance of the geometric change, we develop a novel aspect ratio similarity (ARS) metric to evaluate the visual quality of retargeted images by exploiting the local block changes with a visual importance pooling strategy. Experimental results on the publicly available MIT RetargetMe and CUHK data sets demonstrate that the proposed ARS can predict more accurate visual quality of retargeted images compared with the state-of-the-art IRQA metrics.

Hybrid image retargeting using optimized seam carving and scaling

We present a novel hybrid scheme for content-aware image retargeting that allows retargeting images into arbitrary dimensions while preserving visually prominent features and minimizing global info...

Detection of Seam Carved Image Based on Additional Seam Carving Behavior

Seam carving is a kind of content aware image retargeting algorithm and can be applied to resize and deliberately remove objects from digital images. Based on the observation that after applying an additional seam carving operation, the similarity, the energy relative error, and the difference of seam distance of original image are quite different from those of the seam-carved image, we propose and develop a new method for detecting seam carving or seam insertion of natural images without knowledge of the original image. First, we apply an additional seam carving operation to the testing image, then calculate similarity, energy relative error, and difference of seam distance between the testing image and its seam carved version. Last, we extract 11 dimensional features to detect seam carving operation to train a support vector machine classifier for recognizing whether an image is an original or it has been modified using seam-carving. Our experimental results demonstrate that our proposed forensic method achieves not only better detection rate but also lower dimensional features compared with other existing seam carved detection methods.