Inpainting Technology Research Articles

Research on image inpainting methods based on machine learning

The technique of restoring sections of a picture that have been lost or damaged is known as "image inpainting." In light of recent developments in machine learning, academics have begun investigating the possibility of using deep learning methods to the process of picture inpainting. However, the current body of research does not include a comprehensive review of the many different inpainting methods that are based on machine learning, nor does it compare and contrast these methods. This article provides an overview of some of the most advanced and common machine learning based image restoration techniques that are currently available. These techniques include Multivariate inpainting technology and Unit inpainting technology, such as Context-Encoder Network, Generative Adversarial Network (GAN), and U-Net Network. We examine not just the benefits and drawbacks of each method, but also the ways in which it might be used in a variety of settings. At the conclusion of the piece, we predict that machine learning-based inpainting will continue to gain popularity and application in the years to come.

Robust Image Inpainting Forensics by Using an Attention-Based Feature Pyramid Network

Deep learning has injected a new sense of vitality into the field of image inpainting, allowing for the creation of more realistic inpainted images that are difficult to distinguish from the original ones. However, this also means that the malicious use of image inpainting technology to tamper with images could lead to more serious consequences. In this paper, we use an attention-based feature pyramid network (AFPN) to locate the inpainting traces left by deep learning. AFPN employs a feature pyramid to extract low- and high-level features of inpainted images. It further utilizes a multi-scale convolution attention (MSCA) module to optimize the high-level feature maps. The optimized high-level feature map is then fused with the low-level feature map to detect inpainted regions. Additionally, we introduce a fusion loss function to improve the training effectiveness. The experimental results show that AFPN exhibits remarkable precision in deep inpainting forensics and effectively resists JPEG compression and additive noise attacks.

A Frequency Attention-Based Dual-Stream Network for Image Inpainting Forensics

The rapid development of digital image inpainting technology is causing serious hidden danger to the security of multimedia information. In this paper, a deep network called frequency attention-based dual-stream network (FADS-Net) is proposed for locating the inpainting region. FADS-Net is established by a dual-stream encoder and an attention-based blue-associative decoder. The dual-stream encoder includes two feature extraction streams, the raw input stream (RIS) and the frequency recalibration stream (FRS). RIS directly captures feature maps from the raw input, while FRS performs feature extraction after recalibrating the input via learning in the frequency domain. In addition, a module based on dense connection is designed to ensure efficient extraction and full fusion of dual-stream features. The attention-based associative decoder consists of a main decoder and two branch decoders. The main decoder performs up-sampling and fine-tuning of fused features by using attention mechanisms and skip connections, and ultimately generates the predicted mask for the inpainted image. Then, two branch decoders are utilized to further supervise the training of two feature streams, ensuring that they both work effectively. A joint loss function is designed to supervise the training of the entire network and two feature extraction streams for ensuring optimal forensic performance. Extensive experimental results demonstrate that the proposed FADS-Net achieves superior localization accuracy and robustness on multiple datasets compared to the state-of-the-art inpainting forensics methods.

Generative adversarial network based image inpainting

Image inpainting, which is the repair of pixels in damaged areas of an image to make it look as much like the original image as possible. Deep learning-based image inpainting technology is a prominent area of current research interest. This paper focuses on a systematic and comprehensive study of GAN-based image inpainting and presents an analytical summary. Firstly, this paper introduces GAN, which includes the principle of GAN and its mathematical expression. Secondly, the recent GAN-based image inpainting algorithms are summarized, and the advantages and disadvantages of each algorithm are listed. After that, the evaluation metrics, and common datasets of deep learning-based image inpainting are listed. Finally, the existing image inpainting methods are summarized and summarized, and the ideas for future key research directions are presented and prospected.

Structure-guided virtual restoration for defective silk cultural relics

The digital restoration of damaged silk relics is a type of blind restoration problem that relies on limited prior information. Existing image restoration techniques face significant difficulties in repainting missing patterns of both structural and content. Silk cultural relics are prone to decay, mold, moth, fading, pollution, and adhesion due to long-term exposure to external factors such as temperature, humidity, and microorganisms. Thus, restoring cultural relic patterns is crucial for archeological and design industries. Virtual restoration, based on image inpainting technology, can tackle this problem. However, existing inpainting algorithms are not effective in defective areas lacking a structural trend and only relying on prior information. Moreover, the unique and irregular patterns of silk cultural relics pose a further challenge. To address these issues, we propose a structure-guided virtual restoration method. Concretely, to maintain global image consistency, we introduce an adaptive curve fitting algorithm to reconstruct missing structural lines. We then establish a new priority function to improve the filling order of patches to be repaired, and an adaptive selection of the sample block size is adopted based on the sparsity of the structure. Our method outperforms existing techniques by reconstructing the structural lines according to the pattern design, thus guiding image inpainting to avoid erroneous fillings and block effects. Results show that the restored images achieved an average SSIM of 0.977 and an average PSNR of 39.16. The proposed method for virtual restoration, utilizing curve fitting and image inpainting guided by the artifact's structure, has demonstrated significant improvements in the preservation of structure continuity and faithful reproduction of original patterns on the silk cultural relics.

A Criminisi-DnCNN Model-Based Image Inpainting Method

Existing image inpainting methods achieve unideal results in dealing with centralized inpainting areas. For this reason, in this study, a Criminisi-DnCNN model-based image inpainting method is proposed. Inspired by the manual inpainting technology, the pointwise mutual information (PMI) algorithm was adopted to obtain the marginal structural map of the images to be repaired. Then, the Criminisi algorithm was used to restore the marginal structure to obtain the complete marginal structure image guided by the superficial linear structure. Finally, the problem of texture inpainting was converted into the counterpart of image denoising through the separation of variables by using the denoising convolutional neural network image denoiser (DnCNN). Compared with the existing inpainting methods, this model has improved the clarity of the marginal structure and reduced the blurring of the area to be repaired.

Overview of Image Inpainting and Forensic Technology

At present, digital image is the most significant information carrier on the Internet, which provides convenience for people to exchange information. With the progress of image processing theory, digital image inpainting technology has also developed rapidly. It not only enhances the image expression but also provides tools for image forgery. Misusing the tools may trigger a crisis of trust in the image. To eliminate the negative impact of image forgery, many scholars have conducted in-depth research on it and proposed a series of detection methods called image forensics for institutions or communities to identify such forged images. Current forensic technology still has many limitations, such as relying on specific features and data distribution which makes forensic technology lag behind image inpainting technology. So far, academia still lacks a unified understanding of image forgery and detection technology, and the research architecture of inpainting and forensic technology is not clear. This article reviews the development of image inpainting and forensic technology and systematically summarizes and scientifically classifies the current research work. Finally, the forensic technology is summarized and the future research direction is prospected to guide subsequent scholars to further promote the progress of image forensic technology.

Image inpainting based on deep learning: A review

Image inpainting aims to restore the pixel features of damaged parts in incomplete image and plays a key role in many computer vision tasks. Image inpainting technology based on deep learning is a major current research hotspot. To deeply understand related methods and technologies, this article combs and summarizes the latest research status in this field. Firstly, we summarize inpainting methods of different types of neural network structure based on deep learning, then analyze and study important technical improvement mechanisms. In addition, various algorithms are comprehensively reviewed from the aspects of model network structure and restoration methods. And we select some representative image inpainting methods for comparison and analysis. Finally, the current problems of image inpainting are summarized, and the future development trend and research direction are prospected.

Intelligent Monitoring Method of Short‐Distance Swimming Physical Function Fatigue Limit Mobile Calculation

The detection and classification of moving targets have always been a key technology in intelligent video surveillance. Current detection and classification algorithms for moving targets still face many difficulties, mainly because of the complexity of the monitoring environment and the limitations of target characteristics. Therefore, this article conducts corresponding research on moving target detection and classification in intelligent video surveillance. According to the Gaussian Mixture Background Model and Frame Difference Method, this paper proposes a moving target detection method based on GMM (Gaussians Mixture Model) and Frame Difference Method. This method first proposes a new image combination algorithm that combines GMM and frame difference method, which solves the problems of noise and voids inside the target caused by the fusion of traditional GMM and frame difference method. The moving target detection method can effectively solve the problems of incomplete moving target detection, target internal gap, and noise, and it plays a vital role in the subsequent moving target classification process. Then, the method adds image inpainting technology to compensate the moving target in space and obtain a better target shape. The innovation of this paper is that in order to solve the multiobject classification problem, a binary tree decision support vector machine based on statistical learning is constructed as a classifier for moving object classification. Improve the learning efficiency of the classifier, solve the competitive classification problem of the traditional SVM, and increase the efficiency of the mobile computing intelligent monitoring method by more than 70%.

An Intelligent Forensics Approach for Detecting Patch-Based Image Inpainting

Image inpainting algorithms have a wide range of applications, which can be used for object removal in digital images. With the development of semantic level image inpainting technology, this brings great challenges to blind image forensics. In this case, many conventional methods have been proposed which have disadvantages such as high time complexity and low robustness to postprocessing operations. Therefore, this paper proposes a mask regional convolutional neural network (Mask R-CNN) approach for patch-based inpainting detection. According to the current research, many deep learning methods have shown the capacity for segmentation tasks when labeled datasets are available, so we apply a deep neural network to the domain of inpainting forensics. This deep learning model can distinguish and obtain different features between the inpainted and noninpainted regions. To reduce the missed detection areas and improve detection accuracy, we also adjust the sizes of the anchor scales due to the inpainting images and replace the original nonmaximum suppression single threshold with an improved nonmaximum suppression (NMS). The experimental results demonstrate this intelligent method has better detection performance over recent approaches of image inpainting forensics.

Generative image inpainting for link prediction

Link prediction is a fundamental task that predicts whether a link exists between two nodes based on the currently observed network. Existing approaches such as heuristic-based algorithms assume that two nodes are likely to have a link in a network. In fact, they limit algorithm effectiveness when the assumptions are not correct. Moreover, these link prediction algorithms lack generalization ability, which indicates that they have different effects on different types of networks. For example, the common neighbours algorithm works well on social networks, but it shows poor performance on electric power networks. Inspired by the image inpainting technology of generative adversarial networks and the adjacency matrix representation of networks, we propose a new framework for link prediction based on the image inpainting method, named the Generative Image Inpainting for Link Prediction algorithm (GIILP), to address these problems. The key idea of the GIILP is that the network can be converted into an image (the image is a form of the adjacency matrix (two dimensions)). Pixel values represent the likelihood of two nodes. Thus, the problem of predicting a possible link between nodes is converted into filling missing pixels in an image. The link information of the network can be expressed by the image information, which means that our algorithm does not need assumptions such as heuristics, and works regardless of the dataset type. The experimental results on multiple link prediction public datasets demonstrate that our algorithm has an advantage over other algorithms, including heuristic-based and deep learning methods.

A Detection Approach Using LSTM-CNN for Object Removal Caused by Exemplar-Based Image Inpainting

Exemplar-based image inpainting technology is a “double-edged sword”. It can not only restore the integrity of image by inpainting damaged or removed regions, but can also tamper with the image by using the pixels around the object region to fill in the gaps left by object removal. Through the research and analysis, it is found that the existing exemplar-based image inpainting forensics methods generally have the following disadvantages: the abnormal similar patches are time-consuming and inaccurate to search, have a high false alarm rate and a lack of robustness to multiple post-processing combined operations. In view of the above shortcomings, a detection method based on long short-term memory (LSTM)-convolutional neural network (CNN) for image object removal is proposed. In this method, CNN is used to search for abnormal similar patches. Because of CNN’s strong learning ability, it improves the speed and accuracy of the search. The LSTM network is used to eliminate the influence of false alarm patches on detection results and reduce the false alarm rate. A filtering module is designed to eliminate the attack of post-processing operation. Experimental results show that the method has a high accuracy, and can resist the attack of post-processing combination operations. It can achieve a better performance than the state-of-the-art approaches.

An Adaptive Face Image Inpainting Algorithm Based on Feature Symmetry

Face image inpainting technology is an important research direction in image restoration. When the current image restoration methods repair the damaged areas of face images with weak texture, there are problems such as low accuracy of face image decomposition, unreasonable restoration structure, and degradation of image quality after inpainting. Therefore, this paper proposes an adaptive face image inpainting algorithm based on feature symmetry. Firstly, we locate the feature points of the face, and segment the face into four feature parts based on the feature point distribution to define the feature search range. Then, we construct a new mathematical model, introduce feature symmetry to improve priority calculation, and increase the reliability of priority calculation. After that, in the process of searching for matching blocks, we accurately locate similar feature blocks according to the relative position and symmetry criteria of the target block and various feature parts of the face. Finally, we introduced the HSV (Hue, Saturation, Value) color space to determine the best matching block according to the chroma and brightness of the sample, reduce the repair error, and complete the face image inpainting. During the experiment, we firstly performed visual evaluation and texture analysis on the inpainting face image, and the results show that the face image inpainting by our algorithm maintained the consistency of the face structure, and the visual observation was closer to the real face features. Then, we used the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as objective evaluation indicators; among the five sample face images inpainting results given in this paper, our method was better than the reference methods, and the average PSNR value improved from 2.881–5.776 dB using our method when inpainting 100 face images. Additionally, we used the time required for inpainting the unit pixel to evaluate the inpainting efficiency, and it was improved by 12%–49% with our method when inpainting 100 face images. Finally, by comparing the face image inpainting experiments with the generative adversary network (GAN) algorithm, we discuss some of the problems with the method in this paper based on graphics in repairing face images with large areas of missing features.

A 360-degree Panoramic Image Inpainting Network Using a Cube Map

Inpainting has been continuously studied in the field of computer vision. As artificial intelligence technology developed, deep learning technology was introduced in inpainting research, helping to improve performance. Currently, the input target of an inpainting algorithm using deep learning has been studied from a single image to a video. However, deep learning-based inpainting technology for panoramic images has not been actively studied. We propose a 360-degree panoramic image inpainting method using generative adversarial networks (GANs). The proposed network inputs a 360-degree equirectangular format panoramic image converts it into a cube map format, which has relatively little distortion and uses it as a training network. Since the cube map format is used, the correlation of the six sides of the cube map should be considered. Therefore, all faces of the cube map are used as input for the whole discriminative network, and each face of the cube map is used as input for the slice discriminative network to determine the authenticity of the generated image. The proposed network performed qualitatively better than existing single-image inpainting algorithms and baseline algorithms.

Image Inpainting Research Based on Deep Learning

Abstract With the rapid development of computer technology, image inpainting has become a research hotspot in the field of deep learning. Image inpainting belongs to the intersection of computer vision and computer graphics, and is an image processing technology between image editing and image generation. The proposal of generative adversarial network effectively improves the problems of poor image inpainting effect and large difference between the inpainting image and the target image, and promotes the development of image inpainting technology. In this paper, the image inpainting is based on the generation of confrontation networks. Its network structure establishes two repair paths, namely the reconstruction path and the generation path, and the two paths correspond to two groups of networks. The encoder and generator in the network respectively complete the encoding and decoding tasks based on the residual network. The discriminator also uses the patch block discriminator on the basis of the residual network to discriminate the authenticity of the image. This paper uses Places2 data set to verify the algorithm, and uses PSNR and SSIM two objective evaluation methods to evaluate the quality of the repaired image. Experiments show that the algorithm inpainting effect is better.

Research on texture image inpainting of jacquard fabric based on non-single vision

For the texture image inpainting of jacquard fabric with a large damaged region but complete data information around it, traditional exemplar-based image inpainting algorithms are not only have rapid falling traditional confidence value, but also the matching precision is greatly limited due to the lack of robustness, which results in a wrong guided direction and an unsatisfactory inpainting effect. To solve the above-mentioned problems, a novel digital image acquisition system for damaged fabric was designed and one set of texture image inpainting algorithms of jacquard fabric based on non-single vision was developed. In this image inpainting method, the images to be inpainted and target images were obtained through the non-single-vision imaging method. By using the image registration and Poisson image blending technology, the damaged region was filled by block matching first and then patches from the target image to the data missing region were copied. The experimental results clearly showed that the proposed method achieves better results in visual appearance and inpainting quality, compared with traditional approaches for inpainting. In addition, this method shows broad potential application prospects in the virtual restoration of ancient fabrics using image inpainting technology.

Virtual Restoration of Stained Chinese Paintings Using Patch-Based Color Constrained Poisson Editing with Selected Hyperspectral Feature Bands

Stains, as one of most common degradations of paper cultural relics, not only affect paintings’ appearance, but sometimes even cover the text, patterns, and colors contained in the relics. Virtual restorations based on common red–green–blue images (RGB) which remove the degradations and then fill the lacuna regions with the image’s known parts with the inpainting technology could produce a visually plausible result. However, due to the lack of information inside the degradations, they always yield inconsistent structures when stains cover several color materials. To effectively remove the stains and restore the covered original contents of Chinese paintings, a novel method based on Poisson editing is proposed by exploiting the information inside the degradations of selected three feature bands as the auxiliary information to guide the restoration since the selected feature bands captured fewer stains and could expose the covered information. To make the Poisson editing suitable for stain removal, the feature bands were also exploited to search for the optimal patch for the pixels in the stain region, and the searched patch was used to construct the color constraint on the original Poisson editing to ensure the restoration of the original color of paintings. Specifically, this method mainly consists of two steps: feature band selection from hyperspectral data by establishing rules and reconstruction of stain contaminated regions of RGB image with color constrained Poisson editing. Four Chinese paintings (‘Fishing’, ‘Crane and Banana’, ‘the Hui Nationality Painting’, and ‘Lotus Pond and Wild Goose’) with different color materials were used to test the performance of the proposed method. Visual results show that this method can effectively remove or dilute the stains while restoring a painting’s original colors. By comparing values of restored pixels with nonstained pixels (reference of their same color materials), images processed by the proposed method had the lowest average root mean square error (RMSE), normalized absolute error (NAE), and average differences (AD), which indicates that it is an effective method to restore the stains of Chinese paintings.

Interactive Colorization via Multi-Cues Manipulation

Colorization plays an important role in image processing, which aims to change the original image color according to specific image color characteristics. In this article, the authors present a regional color editing method based on multi-cues manipulation, including interactive segmentation, inpainting and gradient-preserving optimization. Firstly, the user draws strokes specifying the target region which needs to change color, through which the target image can be segmented by the K-means color clustering method. Then the example-based inpainting technology is applied to achieve natural transition along the boundary. After color propagation, they apply an optimization algorithm to preserve the gradient. The experiment results demonstrate that the proposed approach can not only achieve a visual satisfactory local color propagation results, but also preserve the texture details well.

Inpainting for Remotely Sensed Images With a Multichannel Nonlocal Total Variation Model

Filling dead pixels or removing uninteresting objects is often desired in the applications of remotely sensed images. In this paper, an effective image inpainting technology is presented to solve this task, based on multichannel nonlocal total variation. The proposed approach takes advantage of a nonlocal method, which has a superior performance in dealing with textured images and reconstructing large-scale areas. Furthermore, it makes use of the multichannel data of remotely sensed images to achieve spectral coherence for the reconstruction result. To optimize the proposed variation model, a Bregmanized-operator-splitting algorithm is employed. The proposed inpainting algorithm was tested on simulated and real images. The experimental results verify the efficacy of this algorithm.

Two-Phase Image Inpainting: Combine Edge-Fitting with PDE Inpainting

AbstractThe digital image inpainting technology based on partial differential equations (PDEs) has become an intensive research topic over the last few years due to the mature theory and prolific numerical algorithms of PDEs. However, PDE based models are not effective when used to inpaint large missing areas of images, such as that produced by object removal. To overcome this problem, in this paper, a two-phase image inpainting method is proposed. First, some edges which cross the damaged regions are located and the missing parts of these edges are fitted by using the cubic spline interpolation. These fitted edges partition the damaged regions into some smaller damaged regions. Then these smaller regions may be inpainted by using classical PDE models. Experiment results show that the inpainting results by using the proposed method are better than those of BSCB model and TV model.