Edit Propagation Research Articles

LF2MV: Learning an Editable Meta-View Towards Light Field Representation.

Light fields are 4D scene representations that are typically structured as arrays of views or several directional samples per pixel in a single view. However, this highly correlated structure is not very efficient to transmit and manipulate, especially for editing. To tackle this issue, we propose a novel representation learning framework that can encode the light field into a single meta-view that is both compact and editable. Specifically, the meta-view composes of three visual channels and a complementary meta channel that is embedded with geometric and residual appearance information. The visual channels can be edited using existing 2D image editing tools, before reconstructing the whole edited light field. To facilitate edit propagation against occlusion, we design a special editing-aware decoding network that consistently propagates the visual edits to the whole light field upon reconstruction. Extensive experiments show that our proposed method achieves competitive representation accuracy and meanwhile enables consistent edit propagation.

Edit propagation via color palettes

The ability to ease operation and real-time feedback to image editing has attracted increasing attention in our everyday lives. Existing edit propagation approaches typically formulate the color editing task as a quadratic energy optimization problem, which is computationally expensive and requires a lot of user interactions. To address this problem, in this paper, we propose a novel edit propagation method based on color palette. We first extract a color palette from the input image, and then calculate the mixing weights of the image pixels with respect to the extracted palette. Finally, we solve for an edited palette according to user edits, to propagate local edits to the whole image. Our main contributions include formulating the edit propagation as a palette-based optimization problem that can be solved efficiently, and requiring only fewer user interactions than existing methods. We have demonstrated our method for color editing on a wide range of examples. Compared to existing methods, our approach is more efficient and friendly for novice users due to its ease of interaction.

Identity-Aware and Shape-Aware Propagation of Face Editing in Videos.

The development of deep generative models has inspired various facial image editing methods, but many of them are difficult to be directly applied to video editing due to various challenges ranging from imposing 3D constraints, preserving identity consistency, ensuring temporal coherence, etc. To address these challenges, we propose a new framework operating on the StyleGAN2 latent space for identity-aware and shape-aware edit propagation on face videos. In order to reduce the difficulties of maintaining the identity, keeping the original 3D motion, and avoiding shape distortions, we disentangle the StyleGAN2 latent vectors of human face video frames to decouple the appearance, shape, expression, and motion from identity. An edit encoding module is used to map a sequence of image frames to continuous latent codes with 3D parametric control and is trained in a self-supervised manner with identity loss and triple shape losses. Our model supports propagation of edits in various forms: I. direct appearance editing on a specific keyframe, II. implicit editing of face shape via a given reference image, and III. existing latent-based semantic edits. Experiments show that our method works well for various forms of videos in the wild and outperforms an animation-based approach and the recent deep generative techniques.

Fast Edit Propagation for 360 Degree Panoramas Using Function Interpolation

Due to its intuitive stroke-based user interface, edit propagation has wide applications such as image recoloring and relighting, tone editing, image matting, etc. This paper presents a function interpolation method for efficient edit propagation on 360 degree panoramas. Given a panoramic image as input and user-specified strokes as edit hints, our approach adopts an adaptive sampling strategy based on the diversity of local color distribution. By selecting the most representative samples, this strategy can significantly reduce the number of samples. We assume that the projection of the input panorama is known, so that it can be mapped to a 3D unit sphere where the spherical distance metric can be defined. Then edit propagation is formulated as a radial basis function interpolation problem, which can be efficiently solved by using the non-negative linear least squares. We also utilize a multi-resolution strategy and a function look-up table for further acceleration. Experimental results show that our method can efficiently propagate edits on high-resolution panoramas and produce seam-free and smooth editing results.

Efficient propagation of sparse edits on 360[formula omitted] panoramas

We present an efficient method to propagate sparse user edits indicated by strokes on 360∘ panoramas. Our algorithm first projects each equirectangular pixel to its corresponding position on a 3D unit sphere, so each pixel can be characterized by a feature vector consisting of its 3D coordinates and RGB color values. We formulate edit propagation as an optimization problem that aims to satisfy the user edit constraints while preserving the manifold structure of the image at the same time. To solve the problem using a linear system efficiently, we first construct the K-D tree structure in the feature space to cluster pixels. Then we optimize the manifold structure where both the number of nearest neighbors and their corresponding weights are determined by the feature distributions. We further apply a multiresolution strategy to speedup the edit propagation. Our method is the first to perform interactive edit propagation on 360∘ panoramas. Experiments show that our method is able to generate seam-free and visually pleasing results, and users can receive instant feedback during interactive editing.

Interacting with Self-Similarity

Shape similarity is a fundamental problem in geometry processing, enabling applications such as surface correspondence, segmentation, and edit propagation. For example, a user may paint a stroke on one finger of a model and desire the edit to propagate to all fingers. Automatic approaches have difficulty matching user expectations, either due to an algorithm’s inability to guess the scale at which the user is intending to edit or due to underlying deficiencies in the similarity metric (e.g., semantic information not present in the geometry).We propose an approach to interactively design self-similarity maps. We investigate two primitive operations, useful in a variety of scenarios: region and curve similarity. Users select example similar and dissimilar regions. Starting with an automatically generated multi-scale shape signature, our approach solves for a scale parameter and thresholds that group the example regions as specified. We propose a new Smooth Shape Diameter Signature (SSDS) as a more efficient alternative to the Heat or Wave Kernel Signature. If no such parameters can be found, our approach modifies the shape signature itself. Given a curve drawn on the surface, we perform hybrid discrete/continuous optimization to find similar curves elsewhere.We apply our approach for interactive editing scenarios: propagating mesh geometry, patterns duplication, and segmentation.

PDE Learning of Filtering and Propagation for Task-Aware Facial Intrinsic Image Analysis.

Filtering and propagation are two basic operations in image analysis and rendering, and they are also widely used in computer graphics and machine learning. However, the models of filtering and propagation were based on diverse mathematical formulations, which have not been fully understood. This article aims to explore the properties of both filtering and propagation models from a partial differential equation (PDE) learning perspective. We propose a unified PDE learning framework based on nonlinear reaction-diffusion with a guided map, graph Laplacian, and reaction weight. It reveals that: 1) the guided map and reaction weight determines whether the PDE produces filtering or propagation diffusion and 2) the kernel of graph Laplacian controls the diffusion pattern. Based on the proposed PDE framework, we derive the mathematical relations between different models, including learning to diffusion (LTD) model, label propagation, edit propagation, and edge-aware filter. In practical verification, we apply the PDE framework to design diffusion operations with the adaptive kernel to tackle the ill-posed problem of facial intrinsic image analysis (FIIA). A flexible task-aware FIIA system is built to achieve various facial rendering effects, such as face image relighting and delighting, artistic illumination transfer, illumination-aware face swapping, or transfiguring. Qualitative and quantitative experiments show the effectiveness and flexibility of task-aware FIIA and provide new insights on PDE learning for visual analysis and rendering.

Learning a deep fully connected neural network from a single image for edit propagation

We introduce a simple but effective deep fully connected neural network (FNN) to solve the edit propagation as a multiclass pixel-level classification task. We construct the feature space using three- (or one-) dimensional normalized RGB (or grayscale) vectors and spatial coordinates. Our deep FNN-based model consists of color feature extraction, spatial feature extraction, feature combination, and classifier estimation. We train our model only using the feature within the region labeled by the user with strokes in a single image. Then our method directly outputs the edit propagation results after the forward pass without any refinement process. Our method automatically determines the importance of each image feature across the whole image jointly considering feature vectors from user strokes. Extensive experiments demonstrate that the proposed algorithm achieves superior performance over state-of-the-art methods, yet it remains simple and efficient.

Sparse Graph Regularized Mesh Color Edit Propagation.

Mesh color edit propagation aims to propagate the color from a few color strokes to the whole mesh, which is useful for mesh colorization, color enhancement and color editing, etc. Compared with image edit propagation, luminance information is not available for 3D mesh data, so the color edit propagation is more difficult on 3D meshes than images, with far less research carried out. This paper proposes a novel solution based on sparse graph regularization. Firstly, a few color strokes are interactively drawn by the user, and then the color will be propagated to the whole mesh by minimizing a sparse graph regularized nonlinear energy function. The proposed method effectively measures geometric similarity over shapes by using a set of complementary multiscale feature descriptors, and effectively controls color bleeding via a sparse ℓ1 optimization rather than quadratic minimization used in existing work. The proposed framework can be applied for the task of interactive mesh colorization, mesh color enhancement and mesh color editing. Extensive qualitative and quantitative experiments show that the proposed method outperforms the state-of-the-art methods.

Processing images for red\u2013green dichromats compensation via naturalness and information-preservation considered recoloring

Color vision deficiency (CVD) is caused by anomalies in the cone cells of the human retina. It affects approximately 200 million individuals throughout the world. Although previous studies have proposed compensation methods, contrast and naturalness preservation have not been adequately and simultaneously addressed in the state-of-the-art studies. This paper focuses on red–green dichromats’ compensation and proposes a recoloring algorithm that combines contrast enhancement and naturalness preservation in a unified optimization model. In this implementation, representative color extraction and edit propagation methods are introduced to maintain global and local information in the recolored image. The quantitative evaluation results showed that the proposed method is competitive with state-of-the-art methods. A subjective experiment was also conducted and the evaluation results revealed that the proposed method obtained the best scores in preserving both naturalness and information for individuals with severe red–green CVD.

Joint learning of visual and spatial features for edit propagation from a single image

In this paper, we regard edit propagation as a multi-class classification problem and deep neural network (DNN) is used to solve the problem. We design a shallow and fully convolutional DNN that can be trained end-to-end. To achieve this, our method uses combinations of low-level visual features, which are extracted from the input image, and spatial features, which are computed through transforming user interactions, as input of the DNN, which efficiently performs a joint learning of visual and spatial features. We then train the DNN on many of such combinations in order to build a DNN-based pixel-level classifier. Our DNN is also equipped with patch-by-patch training and whole image estimation, speeding up learning and inference. Finally, we improve classification accuracy of the DNN by employing a fully connected conditional random field. Experimental results show that our method can respond to user interactions well and generate precise results compared with the state-of-art edit propagation approaches. Furthermore, we demonstrate our method on various applications.

Edit Propagation Using Deep Neural Network from a Single Image

Edit Propagation Using Deep Neural Network from a Single Image

Instant Edit Propagation on Images Based on Bilateral Grid

Instant Edit Propagation on Images Based on Bilateral Grid

Lighting transfer across multiple views through local color transforms

We present a method for transferring lighting between photographs of a static scene. Our method takes as input a photo collection depicting a scene with varying viewpoints and lighting conditions. We cast lighting transfer as an edit propagation problem, where the transfer of local illumination across images is guided by sparse correspondences obtained through multi-view stereo. Instead of directly propagating color, we learn local color transforms from corresponding patches in pairs of images and propagate these transforms in an edge-aware manner to regions with no correspondences. Our color transforms model the large variability of appearance changes in local regions of the scene, and are robust to missing or inaccurate correspondences. The method is fully automatic and can transfer strong shadows between images. We show applications of our image relighting method for enhancing photographs, browsing photo collections with harmonized lighting, and generating synthetic time-lapse sequences.

Efficient manifold-preserving edit propagation using quad-tree data structures

In this paper, we propose an edit propagation algorithm using quad-tree data structures for image manipulation. First, we use a quad-tree to adaptively group all pixels into clusters. Then, we build a manifold-preserving propagation function based on clusters using locally linear embedding for improved distance. Moreover, we employ an adaptive weight function built on cell corners instead of individual pixels. Because the number of corners is smaller than the number of individual pixels, it results in runtime performance improvement. Finally, the edits of all pixels can be computed by interpolating the edits solved from the clusters. Compared with previous approaches, our method requires less time without sacrificing the visualization quality. Experimental results demonstrate two applications of our algorithm: grayscale image colorization and color image recoloring.

A novel manifold preserving edit propagation based on K nearest neighbors and locally linear embedding

Manifold preserving edit propagation provides a robust way for propagating sparse user edits to a whole image, which preserves the manifold structure during edit propagation. In this paper, we propose to use k nearest neighbors (KNN) and locally linear embedding (LLE) to perform manifold preserving edit propagation. We make innovations from two aspects. First, KNN is adopted to find the neighbors of the similar features with the sample pixel. Secondly, we represent each pixel as a linear combination of its neighbors in feature space according to LLE. Finally, the image will be reconstructed by using the weights computed according to LLE. We have demonstrated our manifold preserving edit propagation on various applications including matting, colorization and color replacement. The experimental results have proved the effectiveness of the proposed algorithm.

Spatio-angular consistent editing framework for 4D light field images

This paper presents a practical framework for various light field editing algorithms, such as segmentation, sparse edit propagation, and inpainting. A novel framework is introduced with its user interface to perform the light field editing algorithms. Unlike single-image algorithms, the coherence between light field subaperture images is fully considered. Instead of processing all light field subaperture images independently, the proposed framework performs editing in the cluster image domain. The edit result in the cluster image is propagated back to each light field subaperture image using 2D-to-4D light field edit propagation. Experimental results on test images captured by a Lytro off-the-shelf light field camera confirm that the proposed method provides robust and consistent results of edited light field subaperture images.

DeepProp: Extracting Deep Features from a Single Image for Edit Propagation

AbstractEdit propagation is a technique that can propagate various image edits (e.g., colorization and recoloring) performed via user strokes to the entire image based on similarity of image features. In most previous work, users must manually determine the importance of each image feature (e.g., color, coordinates, and textures) in accordance with their needs and target images. We focus on representation learning that automatically learns feature representations only from user strokes in a single image instead of tuning existing features manually. To this end, this paper proposes an edit propagation method using a deep neural network (DNN). Our DNN, which consists of several layers such as convolutional layers and a feature combiner, extracts stroke‐adapted visual features and spatial features, and then adjusts the importance of them. We also develop a learning algorithm for our DNN that does not suffer from the vanishing gradient problem, and hence avoids falling into undesirable locally optimal solutions. We demonstrate that edit propagation with deep features, without manual feature tuning, can achieve better results than previous work.

Learn Sparse Dictionaries for Edit Propagation.

With the increasing availability of high-resolution images, videos, and 3D models, the demand for scalable large data processing techniques increases. We introduce a method of sparse dictionary learning for edit propagation of large input data. Previous approaches for edit propagation typically employ a global optimization over the whole set of pixels (or vertexes), incurring a prohibitively high memory and time-consumption for large input data. Rather than propagating an edit pixel by pixel, we follow the principle of sparse representation to obtain a representative and compact dictionary and perform edit propagation on the dictionary instead. The sparse dictionary provides an intrinsic basis for input data, and the coding coefficients capture the linear relationship between all pixels and the dictionary atoms. The learned dictionary is then optimized by a novel scheme, which maximizes the Kullback-Leibler divergence between each atom pair to remove redundant atoms. To enable local edit propagation for images or videos with similar appearance, a dictionary learning strategy is proposed by considering range constraint to better account for the global distribution of pixels in their feature space. We show several applications of the sparsity-based edit propagation, including video recoloring, theme editing, and seamless cloning, operating on both color and texture features. Our approach can also be applied to computer graphics tasks, such as 3D surface deformation. We demonstrate that with an atom-to-pixel ratio in the order of 0.01% signifying a significant reduction on memory consumption, our method still maintains a high degree of visual fidelity.

Multiple Facial Image Editing Using Edge-Aware PDE Learning

This paper introduces a novel facial editing tool, called edge-aware mask, to achieve multiple photo-realistic rendering effects in a unified framework. The edge-aware masks facilitate three basic operations for adaptive facial editing, including region selection, edit setting and region blending. Inspired by the state-of-the-art edit propagation and partial differential equation PDE learning method, we propose an adaptive PDE model with facial priors for masks generation through edge-aware diffusion. The edge-aware masks can automatically fit the complex region boundary with great accuracy and produce smooth transition between different regions, which significantly improves the visual consistence of face editing and reduce the human intervention. Then, a unified and flexible facial editing framework is constructed, which consists of layer decomposition, edge-aware masks generation, and layer/mask composition. The combinations of multiple facial layers and edge-aware masks can achieve various facial effects simultaneously, including face enhancement, relighting, makeup and face blending etc. Qualitative and quantitative evaluations were performed using different datasets for different facial editing tasks. Experiments demonstrate the effectiveness and flexibility of our methods, and the comparisons with the previous methods indicate that improved results are obtained using the combination of multiple edge-aware masks.