Retargeting Methods Research Articles

Deep‐Learning‐Based Facial Retargeting Using Local Patches

AbstractIn the era of digital animation, the quest to produce lifelike facial animations for virtual characters has led to the development of various retargeting methods. While the retargeting facial motion between models of similar shapes has been very successful, challenges arise when the retargeting is performed on stylized or exaggerated 3D characters that deviate significantly from human facial structures. In this scenario, it is important to consider the target character's facial structure and possible range of motion to preserve the semantics assumed by the original facial motions after the retargeting. To achieve this, we propose a local patch‐based retargeting method that transfers facial animations captured in a source performance video to a target stylized 3D character. Our method consists of three modules. The Automatic Patch Extraction Module extracts local patches from the source video frame. These patches are processed through the Reenactment Module to generate correspondingly re‐enacted target local patches. The Weight Estimation Module calculates the animation parameters for the target character at every frame for the creation of a complete facial animation sequence. Extensive experiments demonstrate that our method can successfully transfer the semantic meaning of source facial expressions to stylized characters with considerable variations in facial feature proportion.

Integration of Deep Learned and Handcrafted Features for Image Retargeting Quality Assessment

This paper proposed an image retargeting quality assessment. In the proposed method, a deep convolution network is trained on the pixel displacement patterns of different image retargeting methods to produce a measure for evaluating the quality of output images. Also, the method extracts three other measures that assess the geometric changes of important objects in the image, the bending of block lines, and the extent of information loss during the retargeting process. The tests performed on two well-known databases, RetargetMe and CUHK, demonstrate the excellent performance, stability, and reliability of the proposed method compared to the existing methods. In this paper, we present a new method for evaluating the quality of retargeted images. The innovations of the proposed method are: Using a deep learning method to identify the important regions of the image that contain foreground objects and people Providing a quality evaluation measure, obtained by training a CNN with regression output on the pixel displacement patterns in different image retargeting methods Extracting the foreground objects in the original image and the corresponding objects in the retargeted image and determining the extent of geometric change in each object. Estimating the extent of information loss and distortion based on the extent to which the blocks of the original image have been bent. Estimating the quality score of the retargeted image using Gaussian process regression.

StereoARS: Quality Evaluation for Stereoscopic Image Retargeting With Binocular Inconsistency Detection

Many stereoscopic image retargeting (SIR) methods have been developed for automatically and intelligently resizing stereoscopic images and we cannot always rely on time-consuming subjective user studies to validate the performance of different SIR methods. It is therefore required to design reliable objective metrics for SIR quality evaluation. This paper extends our previous 2D aspect ratio similarity (ARS) metric to a stereo 3D version termed as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">StereoARS</i> where the key idea is to investigate into retargeting inconsistency between the original stereo correspondences. Our proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">StereoARS</i> operates via two stages: monocular quality estimation and binocular inconsistency detection. In the first stage, monocular quality estimation is performed by applying a modified ARS measure on the left and right views separately to quantify the quality degradation within each monocular view. In the second stage, binocular inconsistency detection is performed in both pixel-level and grid-level to characterize the influence of binocular rivalry and stereo visual discomfort on SIR quality. In addition, we also measure to what extent the original pixel visibility relation is preserved after SIR as another binocular quality factor. Finally, these monocular and binocular quality estimates are fused to produce an overall SIR quality score. Extensive experiments have demonstrated that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">StereoARS</i> achieves better alignment with human subjective ratings than the existing metrics by a large margin.

Interactive multi‐character motion retargeting

AbstractA motion retargeting process is necessary as the body size and proportion of the actors are generally different from those of the target characters. However, the original spatial relationship between the multiple characters and the environment is easily broken when using previous motion retargeting methods, which are generally performed for each character independently. Therefore, time‐consuming manual adjustments by animators are usually required to obtain satisfactory results. To address these issues, we present a novel multicharacter motion retargeting method that preserves various types of spatial relationships between characters and environments. We establish a unified deformation‐based framework for the motion retargeting of multiple characters (more than two) or nonhuman characters with complex interactions. Also, an interactive motion editing interface with immediate feedback to the user is provided. We experimentally show that our method achieves a speedup when compared with previous motion retargeting methods.

A Shape-Aware Retargeting Approach to Transfer Human Motion and Appearance in Monocular Videos

Transferring human motion and appearance between videos of human actors remains one of the key challenges in Computer Vision. Despite the advances from recent image-to-image translation approaches, there are several transferring contexts where most end-to-end learning-based retargeting methods still perform poorly. Transferring human appearance from one actor to another is only ensured when a strict setup has been complied, which is generally built considering their training regime’s specificities. In this work, we propose a shape-aware approach based on a hybrid image-based rendering technique that exhibits competitive visual retargeting quality compared to state-of-the-art neural rendering approaches. The formulation leverages the user body shape into the retargeting while considering physical constraints of the motion in 3D and the 2D image domain. We also present a new video retargeting benchmark dataset composed of different videos with annotated human motions to evaluate the task of synthesizing people’s videos, which can be used as a common base to improve tracking the progress in the field. The dataset and its evaluation protocols are designed to evaluate retargeting methods in more general and challenging conditions. Our method is validated in several experiments, comprising publicly available videos of actors with different shapes, motion types, and camera setups. The dataset and retargeting code are publicly available to the community at: https://www.verlab.dcc.ufmg.br/retargeting-motion.

Context-aware saliency detection for image retargeting using convolutional neural networks

Image retargeting is the task of making images capable of being displayed on screens with different sizes. This work should be done so that high-level visual information and low-level features such as texture remain as intact as possible to the human visual system. At the same time, the output image may have different dimensions. Thus, simple methods such as scaling and cropping are not adequate for this purpose. In recent years, researchers have tried to improve the existing retargeting methods, and they have introduced new ones. However, a specific method cannot be utilized to retarget all types of images. In other words, different images require different retargeting methods. Image retargeting has a close relationship to image saliency detection, which is a relatively new image processing task. Earlier saliency detection methods were based on local and global but low-level image information. These methods are called bottom-up processes. On the other hand, newer approaches are top-down and mixed methods that consider the high level and semantic knowledge of the image too. In this paper, we introduce the proposed methods in both saliency detection and retargeting. For the saliency detection, the use of image context and semantic segmentation are examined, and a novel mixed bottom-up and top-down saliency detection method is introduced. After saliency detection, a modified version of an existing retargeting technique is utilized for retargeting the images. The results suggest that the proposed image retargeting pipeline has excellent performance compared to other tested methods. Also, the subjective evaluations on the Pascal dataset can be used as a retargeting quality assessment dataset for further research.

Weakly Supervised Reinforced Multi-Operator Image Retargeting

Image retargeting aims to adjust the resolution and aspect ratio to an arbitrary size while preserving important content of the image. Usually multi-operator image retargeting demonstrates better generalization than single operator scheme due to heterogeneous characteristics of different regions in the image. Most existing multi-operator retargeting methods search the optimal operator at each step with exponential complexity and with the possibility of falling into local optimum. Therefore, in order to produce better results with lower computational costs, we formulate the multi-operator retargeting as a Markov decision-making process and apply Reinforcement Learning (RL) to achieve global optimum. Instead of using traditional image-level measures, we design a high-level semantic and aesthetic reward function to better match human visual perception. With the priori in reward, we further propose a weakly supervised Semantics and Aesthetics aware Multi-operator Image Retargeting (SAMIR) framework. Particularly, the semantic part of the reward helps to constrain the severe deformations that may occur during retargeting process, while the aesthetic part guarantees the sensory quality, which can effectively measure the perceptual effects of different operators on various image content. The operator of each step is learned in an end-to-end manner. In addition, retargeting can be performed in arbitrary target size, step size, and direction. The experiment results on both representative aesthetic datasets and retargeting datasets consistently show that our model outperforms the state-of-the-art methods.

Extrapolation-Based Video Retargeting With Backward Warping Using an Image-to-Warping Vector Generation Network

Video retargeting is a technique used to transform a given video to a target aspect ratio. Current methods often cause severe visual distortion due to frequent temporal incoherence during the retargeting. In this study, we propose a new extrapolation-based video retargeting method using an image-to-warping vector generation network to maintain temporal coherence and prevent deformation of an input frame by extending the side area of an input frame. Backward warping-based extrapolation is performed using a displacement vector (DV) that is generated by a proposed convolutional neural network (CNN). The DV is defined as the displacement between the current hole to be filled in the extended area and a pixel in the input frame used to fill the hole. We also propose a technique to efficiently train the CNN including a method for ground-truth DV generation. After the extrapolation, we propose a technique for the maintenance of temporal coherence of the extended region and a distortion suppression scheme (DSC) for minimizing visual artifacts. The simulation results demonstrated that the proposed method improved bidirectional similarity (BDS) up to 3.69, which is a measure of the quality of video retargeting, compared with existing video retargeting methods.

Feature Map Retargeting to Classify Biomedical Journal Figures.

In this work, we propose a layer to retarget feature maps in Convolutional Neural Networks (CNNs). Our "Retarget" layer densely samples values for each feature map channel at locations inferred by our proposed spatial attention regressor. Our layer increments an existing saliency-based distortion layer by replacing its convolutional components with depthwise convolutions. This reformulation with the tuning of its hyper-parameters makes the Retarget layer applicable at any depth of feed-forward CNNs. Keeping in spirit with Content-Aware Image Resizing retargeting methods, we introduce our layers at the bottlenecks of three pre-trained CNNs. We validate our approach on the ImageCLEF2013, ImageCLEF2015, and ImageCLEF2016 document subfigure classification task. Our redesigned DenseNet121 model with the Retarget layer achieved state-of-the-art results under the visual category when no data augmentations were performed. Performing spatial sampling for each channel of the feature maps at deeper layers exponentially increases computational cost and memory requirements. To address this, we experiment with an approximation of the nearest neighbor interpolation and show consistent improvement over the baseline models and other state-of-the-art attention models. The code is available at https://github.com/VimsLab/CNN-Retarget.

Temporal Incoherence-Free Video Retargeting Using Foreground Aware Extrapolation

Video retargeting is a method of adjusting the aspect ratio of a given video to the target aspect ratio. However, temporal incoherence of video contents, which can occur frequently by video retargeting, is the most dominant factor that degrades the quality of retargeted videos. Current methods to maintain temporal coherence use the entire frames of the input videos; however, these methods cannot be implemented as on-time systems because of their tremendous computational complexity. As far as we know, there is no existing on-time video retargeting method that can avoid spatial distortion while perfectly maintaining temporal coherence. In this paper, we propose a novel on-time video retargeting method that can perfectly maintain temporal coherence and prevent the spatial distortion by using only two consecutive input frames. In our method, the maximum a posteriori-based foreground aware-block matching is used for the extrapolation that extends the side area of a given video to adjust its aspect ratio to the target. To maintain the temporal coherence of the extended area, the result of block matching for backward warping-based extrapolation of the start frame after the scene change occurs, is reused for the other frames until the next scene change occurs. In addition, we propose a scene scenario-adaptive fallback scheme to prevent severe distortions that can occur with reusing block matching results or extrapolation-based side extension. The simulation results showed that the proposed method greatly improved the bidirectional similarity value, which can measure the quality of video retargeting, by up to 10.26 compared with the existing on-time video retargeting methods.

Semantic Segmentation Guided Pixel Fusion for Image Retargeting

Image retargeting aims to obtain high visual quality of target images for human vision. Through semantic segmentation and understanding of input images, we can better preserve the important semantic regions, so as to effectively improve the performance of image retargeting. Benefit from the successful application of deep neural network in the field of semantic segmentation, in this paper, we propose a novel image retargeting approach using semantic segmentation and pixel fusion. Compared with existing image retargeting methods, our approach can effectively reduce geometric distortion during image retargeting by finely reallocating scaling factors for each region based on the semantic segmentation results. Experimental results demonstrate that the proposed approach can well preserve important semantic regions while leaving less unnatural geometric distortion. Our approach also shows the important role of semantic segmentation and understanding of scenes in image retargeting in detail.

A Deep Evaluator for Image Retargeting Quality by Geometrical and Contextual Interaction

An image is compressed or stretched during the multidevice displaying, which will have a very big impact on perception quality. In order to solve this problem, a variety of image retargeting methods have been proposed for the retargeting process. However, how to evaluate the results of different image retargeting is a very critical issue. In various application systems, the subjective evaluation method cannot be applied on a large scale. So we put this problem in the accurate objective-quality evaluation. Currently, most of the image retargeting quality assessment algorithms use simple regression methods as the last step to obtain the evaluation result, which are not corresponding with the perception simulation in the human vision system (HVS). In this paper, a deep quality evaluator for image retargeting based on the segmented stacked AutoEnCoder (SAE) is proposed. Through the help of regularization, the designed deep learning framework can solve the overfitting problem. The main contributions in this framework are to simulate the perception of retargeted images in HVS. Especially, it trains two separated SAE models based on geometrical shape and content matching. Then, the weighting schemes can be used to combine the obtained scores from two models. Experimental results in three well-known databases show that our method can achieve better performance than traditional methods in evaluating different image retargeting results.

Learning character-agnostic motion for motion retargeting in 2D

Analyzing human motion is a challenging task with a wide variety of applications in computer vision and in graphics. One such application, of particular importance in computer animation, is the retargeting of motion from one performer to another. While humans move in three dimensions, the vast majority of human motions are captured using video, requiring 2D-to-3D pose and camera recovery, before existing retargeting approaches may be applied. In this paper, we present a new method for retargeting video-captured motion between different human performers, without the need to explicitly reconstruct 3D poses and/or camera parameters. In order to achieve our goal, we learn to extract, directly from a video, a high-level latent motion representation, which is invariant to the skeleton geometry and the camera view. Our key idea is to train a deep neural network to decompose temporal sequences of 2D poses into three components: motion, skeleton, and camera view-angle. Having extracted such a representation, we are able to re-combine motion with novel skeletons and camera views, and decode a retargeted temporal sequence, which we compare to a ground truth from a synthetic dataset. We demonstrate that our framework can be used to robustly extract human motion from videos, bypassing 3D reconstruction, and outperforming existing retargeting methods, when applied to videos in-the-wild. It also enables additional applications, such as performance cloning, video-driven cartoons, and motion retrieval.

Object occlusion guided stereo image retargeting

The resizing of stereo images while preserving salient content and geometric consistency between the image pair is an essential problem to be addressed for 3D visualization. Existing stereo image retargeting techniques are observed to be incurring salient object deformation. In this paper, we formulate a seam carving method to perform stereo image retargeting using graph-cuts having node size as the number of pixels in one of the stereo images. We define the object map with depth ordering of each object from the camera. The seams are allowed to pass along the object-object occlusion boundary at depth discontinuity in order to prevent salient object deformation. We propose adaptive occlusion boundary weights as a function of an object area to be occluded to preserve small objects. The seam passing through object-object occlusion boundary in one image may not follow the exact boundary in the other image. We propose a refinement of the seam to follow object-object occlusion boundary in both the stereo images. By comparing the proposed method with other state-of-the-art stereo image retargeting methods, we show that the proposed method is able to preserve the salient object(s) along with geometric consistency between the stereo image pair.

Visual Attention Fusion Framework for Image Retargeting Quality Assessment

To adapt images for diversified digital devices, researchers have presented many image retargeting methods. However, the consistency between results of objective image retargeting quality assessment (IRQA) metrics and subjective perception is still low. In this paper, we propose a visual attention fusion (VAF) framework to assist IRQA metrics in better understanding the features of images such as image saliency, faces, and lines. First, we combine the results of multiple salient object detection algorithms to reduce the limitations of a single algorithm. Second, faces and lines are considered in our framework to measure deformations to these visually sensitive regions. Finally, we propose a saliency enhancement model to simulate human visual attention for IRQA. We combine the proposed VAF framework with some state-of-the-art IRQA metrics. Experimental results show that the proposed VAF framework can improve the consistency between the results of objective IRQA metrics and subjective opinion scores.

A Content-Aware Image Retargeting Quality Assessment Method Using Foreground and Global Measurement

Image retargeting methods aim to minimize the perceptual loss while changing sizes and aspect ratios of images. Since optimal retargeting methods for different images are generally not the same, the image retargeting quality assessment (IRQA) becomes a meaningful task. This paper proposes a content-aware image retargeting quality assessment method using foreground and global measurement to achieve better performance. In our proposed method, images are first divided into two categories according to the foreground object detection result, and then different corresponding measurements are designed for them. For those with obvious foreground object, both foreground and global measurement are applied. For others, only global measurement is conducted. Foreground measurement includes two complementary features: the high-level semantic similarity feature and the low-level size ratio feature. Global measurement includes another two features: an improved aspect ratio similarity (ARS) feature and edge group similarity (EGS) feature. Two public databases, i.e., the RetargetMe and CUHK, have been evaluated, and experimental results demonstrate that our method is quite effective, and it also provides state-of-the-art performance in the IRQA. aa Our code are available at https://github.com/SCUT-ML-GUO/IRQA.

Beyond Visual Retargeting: A Feature Retargeting Approach for Visual Recognition and Its Applications

The popularity of mobile applications has greatly enriched and facilitated our lives. However, the rapid increase of digital images and the problem of narrow bandwidth of the wireless network call for an appropriate approach to reduce the amount of data transmitted over the wireless network (i.e., low bit-rate transmission) while ensuring high recognition accuracy at the cloud. We propose a simple and effective feature retargeting (FR) approach for retargeting an image while preserving the representative local features (e.g., SIFT, SURF, and BRIEF) in the image. Our feature retargeting approach aims at low bit-rate visual recognition instead of high-quality visual perception that visual retargeting methods dedicate to. Our algorithm consists of two key novelties: estimating feature saliency and retargeting image: Estimating feature saliency focuses on predicting the relative importance of different features in an image by analyzing uniqueness in a specific context; Retargeting image aims at finding the optimal resolution for the retargeted image to maximize feature-saliency energy. We evaluate the proposed approach for two different applications in three large data sets and observe that our FR approach consistently outperforms state-of-the-art retargeting algorithms, resulting in both higher precision and lower bit-rates. We also demonstrate that even when the resolution of source image is reduced greatly, e.g., 1/7 original size, our algorithm produces superior results as compared with other approaches.

A Quad Edge-Based Grid Encoding Model for Content-Aware Image Retargeting.

In this paper, we present a novel grid encoding model for content-aware image retargeting. In contrast to previous approaches such as vertex-based and axis-aligned grid encoding models, our approach takes each horizontal/vertical distance between two adjacent vertices as an optimization variable. Upon this difference-based encoding scheme, every vertex position of a target grid is subsequently determined after optimizing the one-dimensional values. Our quad edge-based grid model has two major advantages for image retargeting. First, the model enables a grid optimization problem to be developed in a simple quadratic program while ensuring the global convexity of objective functions. Second, due to the independency of variables, spatial regularizations can be applied in a locally adaptive manner to preserve structural components. Based on this model, we propose three quadratic objective functions. Note that, in our work, their linear combination guides a grid deformation process to obtain a visually comfortable retargeting result by preserving salient regions and structural components of an input image. Comparative evaluations have been conducted with ten existing state-of-the-art image retargeting methods, and the results show that our method built upon the quad edge-based model consistently outperforms other previous methods both on qualitative and quantitative perspectives.

Image retargeting quality assessment based on content deformation measurement

Content-aware based image retargeting methods aim at visualizing source images on diverse display devices with different aspect ratios while preventing the incurrence of visual distortions as far as possible. An objective retargeting assessment method, which has a high consistency with subjective perception, can measure the retargeting effects of different retargeting methods, thus promoting the development of retargeting technology. However, the correlation between the existing objective evaluation methods and subjective evaluation is not strong enough. In this paper, we propose an objective retargeting assessment method by fusing multiple features to quantify the content loss and structural deformation caused by retargeting. In order to measure the global deformation, there are two features are proposed which are elastic registration based on B-spline function and foreground retention ratio. Moreover, improved Aspect Ratio Similarity (IARS) and concentrated deletion ratio make for better detection of local deformation. These features are fused to obtain the objective retargeting assessment method. Compared with the state-of-art objective assessment methods on three public datasets, the proposed method has a better correlation with subjective perception.

Composing Semantic Collage for Image Retargeting

Image retargeting has been applied to display images of any size via devices with various resolutions (e.g., cell phone, TV monitors). To fit an image with the target resolution, certain unimportant regions need to be deleted or distorted and the key problem is to determine the importance of each pixel. Existing methods predict pixel-wise importance in a bottom-up manner via eye fixation estimation or saliency detection. In contrast, the proposed algorithm estimates the pixel-wise importance based on a top-down criterion where the target image maintains the semantic meaning of the original image. To this end, several semantic components corresponding to foreground objects, action contexts, and background regions are extracted. The semantic component maps are integrated by a classification guided fusion network. Specifically, the deep network classifies the original image as object or scene-oriented, and fuses the semantic component maps according to classification results. The network output, referred to as the semantic collage with the same size as the original image, is then fed into any existing optimization method to generate the target image. Extensive experiments are carried out on the RetargetMe dataset and S-Retarget database developed in this work. Experimental results demonstrate the merits of the proposed algorithm over the state-of-the-art image retargeting methods.