Image Editing Tasks Research Articles

Computational Illusion Knitting

Illusion-knit fabrics reveal distinct patterns or images depending on the viewing angle. Artists have manually achieved this effect by exploiting "microgeometry," i.e., small differences in stitch heights. However, past work in computational 3D knitting does not model or exploit designs based on stitch height variation. This paper establishes a foundation for exploring illusion knitting in the context of computational design and fabrication. We observe that the design space is highly constrained, elucidate these constraints, and derive strategies for developing effective, machine-knittable illusion patterns. We partially automate these strategies in a new interactive design tool that reduces difficult patterning tasks to familiar image editing tasks. Illusion patterns also uncover new fabrication challenges regarding mixed colorwork and texture; we describe new algorithms for mitigating fabrication failures and ensuring high-quality knit results.

Weather Translation via Weather-Cue Transferring.

In this article, the weather translation task is proposed, which aims to transfer the weather type of the image from one category to another. Weather translation is a complicated image weather editing task that changes the weather cue of an image across multiple weather types, and it is related to image restoration, image editing, and photographic style transfer tasks. Although lots of approaches have been developed for traditional image translation and restoration tasks, only few of them are capable of handling the multicategory weather types problem with a single network due to the rich categories and highly complicated semantic structures of weather images. Especially, it is difficult to change the weather cue while preserving the weather-invariant area. To solve these issues, we developed a weather-cue guided multidomain translation approach based on StarGAN v2, termed WeatherGAN. In the proposed model, the core generator is redesigned to transfer the weather cue according to the target weather type. The weather segmentation module is first introduced to acquire the weather semantic structure of images in a weakly supervised multitask manner. In addition, a weather clues module is presented to reprocess the weather segmentation into a weather-specific clues map, which identifies the weather-invariant and weather-cue areas clearly. Extensive studies and evaluations show that our approach outperforms the state of the art. The data and source code will be publicly available soon after the manuscript is accepted.

Spatial-Contextual Discrepancy Information Compensation for GAN Inversion

Most existing GAN inversion methods either achieve accurate reconstruction but lack editability or offer strong editability at the cost of fidelity. Hence, how to balance the distortion-editability trade-off is a significant challenge for GAN inversion. To address this challenge, we introduce a novel spatial-contextual discrepancy information compensation-based GAN-inversion method (SDIC), which consists of a discrepancy information prediction network (DIPN) and a discrepancy information compensation network (DICN). SDIC follows a ``compensate-and-edit'' paradigm and successfully bridges the gap in image details between the original image and the reconstructed/edited image. On the one hand, DIPN encodes the multi-level spatial-contextual information of the original and initial reconstructed images and then predicts a spatial-contextual guided discrepancy map with two hourglass modules. In this way, a reliable discrepancy map that models the contextual relationship and captures fine-grained image details is learned. On the other hand, DICN incorporates the predicted discrepancy information into both the latent code and the GAN generator with different transformations, generating high-quality reconstructed/edited images. This effectively compensates for the loss of image details during GAN inversion. Both quantitative and qualitative experiments demonstrate that our proposed method achieves the excellent distortion-editability trade-off at a fast inference speed for both image inversion and editing tasks. Our code is available at https://github.com/ZzqLKED/SDIC.

Confusable facial expression recognition with geometry-aware conditional network

Many facial expression recognition (FER) methods have now achieved satisfactory results. However, some facial expressions have similar muscle deformation, making them easy to confuse. These confusable facial expressions are a key challenge to accurately recognizing facial expressions. In addition, most current FER methods rely on the convolution operation, but convolution is a building block that processes one local neighborhood at a time; thus, it fails to capture the geometric patterns that are important for facial muscle deformation. To address this issue, considering the problems of pose variations and insufficient training data, this paper proposes a geometry-aware conditional network (GACN) that captures long-range dependencies for simultaneous pose-invariant facial expression editing and geometry-aware FER. Specifically, the GACN can complete a pose-invariant image editing task with long-range dependency by introducing conditional self-attention operations to a generative adversarial network. Moreover, the GACN presents non-local operations as building blocks of the classifier to capture the texture and geometry patterns simultaneously. Finally, these two tasks can further boost each other’s performances through our GACN, and confusable facial expressions can be effectively distinguished. And we overcome the effect of pose variations while expanding and enriching the training set. Our proposed algorithm is evaluated on both the in-the-lab and in-the-wild datasets and outperforms the state-of-the-art methods.

Patch-based stochastic attention for image editing

Attention mechanisms have become of crucial importance in deep learning in recent years. These non-local operations, which are similar to traditional patch-based methods in image processing, complement local convolutions. However, computing the full attention matrix is an expensive step with heavy memory and computational loads. These limitations curb network architectures and performances, in particular for the case of high resolution images. We propose an efficient attention layer based on the stochastic algorithm PatchMatch, which is used for determining approximate nearest neighbors. We refer to our proposed layer as a “Patch-based Stochastic Attention Layer” (PSAL). Furthermore, we propose different approaches, based on patch aggregation, to ensure the differentiability of PSAL, thus allowing end-to-end training of any network containing our layer. PSAL has a small memory footprint and can therefore scale to high resolution images. It maintains this footprint without sacrificing spatial precision and globality of the nearest neighbors, which means that it can be easily inserted in any level of a deep architecture, even in shallower levels. We demonstrate the usefulness of PSAL on several image editing tasks, such as image inpainting, guided image colorization, and single-image super-resolution. Our code is available at: https://github.com/ncherel/psal.

Optimal transport-based unsupervised semantic disentanglement: A novel approach for efficient image editing in GANs

The latent space of pre-trained generative adversarial networks (GANs) is rich in semantic information, which often becomes highly entangled. It is crucial to identify semantic directions within this latent space, as these directions correlate with image attributes and are vital for image editing tasks. Existing methods for semantic discovery usually involve labor-intensive procedures such as manual labeling and training attribute classifiers, which limits their practicality. In response to this issue, the paper proposes the Optimal Transport-based Unsupervised Semantic Disentanglement (OTUSD) algorithm. This novel method efficiently uncovers semantic directions in the latent space of GANs by utilizing the concepts of manifold learning and optimal transport (OT) theory. OTUSD applies singular value decomposition (SVD) to the OT matrix that links latent codes to generated images. This process yields singular vectors that correspond to semantically meaningful directions. Unlike traditional methods, OTUSD bypasses the need for time-consuming labeling and training processes, thus enhancing efficiency and revealing a wider array of semantically meaningful directions. Experimental results demonstrate the effectiveness of OTUSD in discovering semantic directions from several state-of-the-art GAN models, including StyleGAN, StyleGAN2, and BigGAN. This performance emphasizes the potential applicability of OTUSD to image editing and other related tasks, and illuminates its value in harnessing the manifold learning and OT mapping capabilities inherent in GANs for semantic disentanglement. The implementation code is available at https://github.com/LuckAlex/OTUSD.

An Interactive Image Segmentation Method Based on Multi-Level Semantic Fusion.

Understanding and analyzing 2D/3D sensor data is crucial for a wide range of machine learning-based applications, including object detection, scene segmentation, and salient object detection. In this context, interactive object segmentation is a vital task in image editing and medical diagnosis, involving the accurate separation of the target object from its background based on user annotation information. However, existing interactive object segmentation methods struggle to effectively leverage such information to guide object-segmentation models. To address these challenges, this paper proposes an interactive image-segmentation technique for static images based on multi-level semantic fusion. Our method utilizes user-guidance information both inside and outside the target object to segment it from the static image, making it applicable to both 2D and 3D sensor data. The proposed method introduces a cross-stage feature aggregation module, enabling the effective propagation of multi-scale features from previous stages to the current stage. This mechanism prevents the loss of semantic information caused by multiple upsampling and downsampling of the network, allowing the current stage to make better use of semantic information from the previous stage. Additionally, we incorporate a feature channel attention mechanism to address the issue of rough network segmentation edges. This mechanism captures richer feature details from the feature channel level, leading to finer segmentation edges. In the experimental evaluation conducted on the PASCAL Visual Object Classes (VOC) 2012 dataset, our proposed interactive image segmentation method based on multi-level semantic fusion demonstrates an intersection over union (IOU) accuracy approximately 2.1% higher than the currently popular interactive image segmentation method in static images. The comparative analysis highlights the improved performance and effectiveness of our method. Furthermore, our method exhibits potential applications in various fields, including medical imaging and robotics. Its compatibility with other machine learning methods for visual semantic analysis allows for integration into existing workflows. These aspects emphasize the significance of our contributions in advancing interactive image-segmentation techniques and their practical utility in real-world applications.

The Study of Evolution and Application Related to the Chat-GPT

Chat-GPT has become increasingly popular and has provided tremendous help to people in their daily life. The fundamental working principle of Chat-GPT involves the incorporation of various methods such as Natural Language Processing (NLP), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), Transformer and Reinforcement Learning from Human Feedback model (RLHF). These methods assist Chat-GPT in understanding, predicting and computing the desired outcomes for the users. NLP is used to help machines understand and process human language. RNN is employed to facilitate the machine in comprehending the input's logic. LSTM enables the control of memory elements, allowing the machine to combine unrelated elements in memory. RLHF serves as a switch button, controlling the machine's output and improving the accuracy of results. Chat-GPT can be used for image editing, coding, translation, paper editing and other tasks. However, it is critical to exercise caution while using Chat-GPT and not to become overly reliant on it, given the potential problems it may create.

ImageEye: Batch Image Processing using Program Synthesis

This paper presents a new synthesis-based approach for batch image processing. Unlike existing tools that can only apply global edits to the entire image, our method can apply fine-grained edits to individual objects within the image. For example, our method can selectively blur or crop specific objects that have a certain property. To facilitate such fine-grained image editing tasks, we propose a neuro-symbolic domain-specific language (DSL) that combines pre-trained neural networks for image classification with other language constructs that enable symbolic reasoning. Our method can automatically learn programs in this DSL from user demonstrations by utilizing a novel synthesis algorithm. We have implemented the proposed technique in a tool called ImageEye and evaluated it on 50 image editing tasks. Our evaluation shows that ImageEye is able to automate 96% of these tasks.

The Regularised Epsilon-derivative for Image Reintegration

Many image-editing tasks are carried out in the gradient domain. Suppose that for an image I the gradient ∇I consists of a pair of fields (p;q); then some image “reintegration” scheme is tasked with converting derivative fields (p;q) back to imagespace I; typically, a Poisson equation solver is used for this task. But what if we have altered (p;q) so that this pair (p;q) is no longer integrable? Then we have to project onto integrable gradient data that will indeed reintegrate to an approximation of the original image. For example, we may wish to alter (p;q) so as to emphasize or de-emphasize some aspects of the image, e.g. ameliorating wrinkles in skin images (or indeed enhancing them in the case of ageing a face image). Here, we propose a new gradient kernel that retains part of the original image, regularising the reintegration back into the image domain. We compare our approach with the Screened-Poisson method which includes a term λ times a “screen” term that moves the solution image back closer to the input image. Effectively, we are doing a similar adjustment, but we show that the results are a good deal better than using Screened-Poisson, which tends to overly blur the output. Moreover, in Screened-Poisson one must choose a value for λ, which may be different for every image – here we determine that our new kernel method does not need to adapt to each image yet delivers comparable or better results.

E2Style: Improve the Efficiency and Effectiveness of StyleGAN Inversion.

This paper studies the problem of StyleGAN inversion, which plays an essential role in enabling the pretrained StyleGAN to be used for real image editing tasks. The goal of StyleGAN inversion is to find the exact latent code of the given image in the latent space of StyleGAN. This problem has a high demand for quality and efficiency. Existing optimization-based methods can produce high-quality results, but the optimization often takes a long time. On the contrary, forward-based methods are usually faster but the quality of their results is inferior. In this paper, we present a new feed-forward network "E2Style" for StyleGAN inversion, with significant improvement in terms of efficiency and effectiveness. In our inversion network, we introduce: 1) a shallower backbone with multiple efficient heads across scales; 2) multi-layer identity loss and multi-layer face parsing loss to the loss function; and 3) multi-stage refinement. Combining these designs together forms an effective and efficient method that exploits all benefits of optimization-based and forward-based methods. Quantitative and qualitative results show that our E2Style performs better than existing forward-based methods and comparably to state-of-the-art optimization-based methods while maintaining the high efficiency as well as forward-based methods. Moreover, a number of real image editing applications demonstrate the efficacy of our E2Style. Our code is available at https://github.com/wty-ustc/e2style.

SR-Inpaint: A General Deep Learning Framework for High Resolution Image Inpainting

Recently, deep learning has enabled a huge leap forward in image inpainting. However, due to the memory and computational limitation, most existing methods are able to handle only low-resolution inputs, typically less than 1 K. With the improvement of Internet transmission capacity and mobile device cameras, the resolution of image and video sources available to users via the cloud or locally is increasing. For high-resolution images, the common inpainting methods simply upsample the inpainted result of the shrinked image to yield a blurry result. In recent years, there is an urgent need to reconstruct the missing high-frequency information in high-resolution images and generate sharp texture details. Hence, we propose a general deep learning framework for high-resolution image inpainting, which first hallucinates a semantically continuous blurred result using low-resolution inpainting and suppresses computational overhead. Then the sharp high-frequency details with original resolution are reconstructed using super-resolution refinement. Experimentally, our method achieves inspiring inpainting quality on 2K and 4K resolution images, ahead of the state-of-the-art high-resolution inpainting technique. This framework is expected to be popularized for high-resolution image editing tasks on personal computers and mobile devices in the future.

Learning joint latent representations based on information maximization

Learning disentangled and interpretable representations is an important aspect of information understanding. In this paper, we propose a novel deep learning model representing both discrete and continuous latent variable spaces which can be used in either supervised or unsupervised learning. The proposed model is trained using an optimization function employing the mutual information maximization criterion. For the unsupervised learning setting we define a lower bound to the mutual information between the joint distribution of the latent variables corresponding to the real data and those generated by the model. The maximization of this lower bound during the training induces the learning of disentangled and interpretable data representations. Such representations can be used for attribute manipulation and image editing tasks.

UniFaceGAN: A Unified Framework for Temporally Consistent Facial Video Editing.

Recent research has witnessed advances in facial image editing tasks including face swapping and face reenactment. However, these methods are confined to dealing with one specific task at a time. In addition, for video facial editing, previous methods either simply apply transformations frame by frame or utilize multiple frames in a concatenated or iterative fashion, which leads to noticeable visual flickers. In this paper, we propose a unified temporally consistent facial video editing framework termed UniFaceGAN. Based on a 3D reconstruction model and a simple yet efficient dynamic training sample selection mechanism, our framework is designed to handle face swapping and face reenactment simultaneously. To enforce the temporal consistency, a novel 3D temporal loss constraint is introduced based on the barycentric coordinate interpolation. Besides, we propose a region-aware conditional normalization layer to replace the traditional AdaIN or SPADE to synthesize more context-harmonious results. Compared with the state-of-the-art facial image editing methods, our framework generates video portraits that are more photo-realistic and temporally smooth.

PyMatting: A Python Library for Alpha Matting

An important step of many image editing tasks is to extract specific objects from an image in order to place them in a scene of a movie or compose them onto another background. Alpha matting describes the problem of separating the objects in the foreground from the background of an image given only a rough sketch. We introduce the PyMatting package for Python which implements various approaches to solve the alpha matting problem. Our toolbox is also able to extract the foreground of an image given the alpha matte. The implementation aims to be computationally efficient and easy to use. The source code of PyMatting is available under an open-source license at https://github.com/pymatting/pymatting.

Diversifying Semantic Image Synthesis and Editing via Class‐ and Layer‐wise VAEs

AbstractSemantic image synthesis is a process for generating photorealistic images from a single semantic mask. To enrich the diversity of multimodal image synthesis, previous methods have controlled the global appearance of an output image by learning a single latent space. However, a single latent code is often insufficient for capturing various object styles because object appearance depends on multiple factors. To handle individual factors that determine object styles, we propose a class‐ and layer‐wise extension to the variational autoencoder (VAE) framework that allows flexible control over each object class at the local to global levels by learning multiple latent spaces. Furthermore, we demonstrate that our method generates images that are both plausible and more diverse compared to state‐of‐the‐art methods via extensive experiments with real and synthetic datasets in three different domains. We also show that our method enables a wide range of applications in image synthesis and editing tasks.

Image Editing via Segmentation Guided Self-Attention Network

Image editing is one of the most popular directions in computer vision. Recently, many methods have benefited from the advances in deep learning, showing promising performance in the image editing task by inpainting the editing areas. These methods take advantage of edge information as user guidance to generate the desired content. However, they are suffering from generating color discrepancy and inconsistent boundaries. In this letter, we propose a deep image editing method based on a self-attention network which copies information for each of the small patches from distant spatial locations. The proposed method smooths the image, computes segmentation maps, and utilizes the segmentation information for guiding the self-attention layers to explicitly leverage image features from surrounding areas with similar appearances. Experimental results show that the proposed method achieves better performance, is flexible for different purposes, and is fast for implementation.

Unsupervised Meta-Learning of Figure-Ground Segmentation via Imitating Visual Effects

This paper presents a “learning to learn” approach to figureground image segmentation. By exploring webly-abundant images of specific visual effects, our method can effectively learn the visual-effect internal representations in an unsupervised manner and uses this knowledge to differentiate the figure from the ground in an image. Specifically, we formulate the meta-learning process as a compositional image editing task that learns to imitate a certain visual effect and derive the corresponding internal representation. Such a generative process can help instantiate the underlying figure-ground notion and enables the system to accomplish the intended image segmentation. Whereas existing generative methods are mostly tailored to image synthesis or style transfer, our approach offers a flexible learning mechanism to model a general concept of figure-ground segmentation from unorganized images that have no explicit pixel-level annotations. We validate our approach via extensive experiments on six datasets to demonstrate that the proposed model can be end-to-end trained without ground-truth pixel labeling yet outperforms the existing methods of unsupervised segmentation tasks.

Semantic photo manipulation with a generative image prior

Despite the recent success of GANs in synthesizing images conditioned on inputs such as a user sketch, text, or semantic labels, manipulating the high-level attributes of an existing natural photograph with GANs is challenging for two reasons. First, it is hard for GANs to precisely reproduce an input image. Second, after manipulation, the newly synthesized pixels often do not fit the original image. In this paper, we address these issues by adapting the image prior learned by GANs to image statistics of an individual image. Our method can accurately reconstruct the input image and synthesize new content, consistent with the appearance of the input image. We demonstrate our interactive system on several semantic image editing tasks, including synthesizing new objects consistent with background, removing unwanted objects, and changing the appearance of an object. Quantitative and qualitative comparisons against several existing methods demonstrate the effectiveness of our method.

Knowing That You Know What I Know Helps?

To accomplish collaborative work, collaborators need to know one another's states of work for coordination. Similarly, in situations of knowledge transfer where experts need to instruct novices by passing implicit knowledge and personal experience about the task, one challenge lies in the knowledge gap between experts and novices. Both parties need to gain an awareness of each other's state of task knowledge for effective knowledge transfer. In this paper, we aim to investigate how online knowledge transfer is supported by a design called knowledge transparency of a shared document workspace, which allows explicitly annotating and sharing ones' knowledge state to make it visible to their collaborators. We conducted a laboratory study to examine how knowledge transparency affects knowledge transfer among experts and novices using an image editing task over an online document. The results shed light on how to support knowledge co-construction and communication for the purpose of knowledge transfer through interface design.