Image Inpainting Research Articles

Image inpainting algorithm based on double curvature-driven diffusion model with P-Laplace operator.

The method of partial differential equations for image inpainting achieves better repair results and is economically feasible with fast repair time. Addresses the inability of Curvature-Driven Diffusion (CDD) models to repair complex textures or edges when the input image is affected by severe noise or distortion, resulting in discontinuous repair features, blurred detail textures, and an inability to deal with the consistency of global image content, In this paper, we have the CDD model of P-Laplace operator term to image inpainting. In this method, the P-Laplace operator is firstly introduced into the diffusion term of CDD model to regulate the diffusion speed; then the improved CDD model is discretized, and the known information around the broken region is divided into two weighted average iterations to get the inpainting image; finally, the final inpainting image is obtained by weighted averaging the two image inpainting images according to the distancing. Experiments show that the model restoration results in this paper are more rational in terms of texture structure and outperform other models in terms of visualization and objective data. Comparing the inpainting images with 150, 1000 and 100 iterations respectively, Total Variation(TV) model and the CDD model inpainting algorithm always has inpainting traces in details, and TV model can't meet the visual connectivity, but the algorithm in this paper can remove the inpainting traces well, TV model and the CDD model inpainting algorithm always have inpainting traces in details, and TV model can't meet the visual connectivity, but the algorithm in this paper can remove the inpainting traces well. Of the images used for testing, the highest PSNR reached 38.7982, SSIM reached 0.9407, and FSIM reached 0.9781, the algorithm not only inpainting the effect and, but also has fewer iterations.

STNet: Structure and texture-guided network for image inpainting

Generative Adversarial Network (GAN) has made great progress in image inpainting due to its strong generation capability. However, the previous methods based on GAN cannot understand the structure and texture of damaged images simultaneously, which leads to inconsistent structure and blurred details of inpainting results. To address this issue, we propose a Structure and Texture-guided Network for image inpainting (STNet), which consists of three networks, i.e. structure reconstruction network, texture reconstruction network, and image refinement network. STNet can restore the coherent structure and fine textures of damaged images in the first two networks, and fuse them to complete image inpainting in the last network. Experiments on three standard datasets CelebA, Places2, and Pairs StreetView show STNet is effective and outperforms other related methods. In particular, we also conducted experiments on the damaged images from tomb murals of Tang Dynasty in China, whose results also verify that our method can be applied to the restoration of historical relics. Our code is available at https://github.com/nwuAI/STNet.

Improved medical image inpainting using automatic multi-task learning driven deep learning approach

Distorted medical images can drastically reduce diagnosis accuracy using computer-aided diagnostic (CAD) systems. The objective of medical image classification is to improve diagnostic imaging precision and restore regions degraded by image inpainting. Deep Learning (DL) methods like Generative Adversarial Networks (GANs) are being used to solve these medical image inpainting issues. Recently, various improved GAN models were proposed for improved image inpainting, but challenges like computational inefficiency, unreliable restoration quality, and lack of correct analysis of discriminator block. To overcome these problems, a novel GAN model is proposed with modified Generator and Discriminator blocks for improved medical image inpainting. To improve image inpainting quality with minimum computational complexity, the Multi-Task Learning (MTL)-based generator block is designed. The MTL encoder is connected with three decoders for three tasks edge detection, organ boundary generation, and image completion. The MTL-based encoder and decoders are modified with an optimum number of layers and regularization functions to improve the outcomes. The output of the generator block is input into the discriminator block, where we designed the relevant feedback strategy to decrease generator loss by automatically changing the weights on wrong classifications. The simulation results revealed the efficiency of the proposed medical image inpainting approach and discriminator block performances over the state-of-the-art. Due to the modified MTL-based generator and discriminator block in the proposed approach, the PSNR is improved by 14.31 %, SSIM is improved by 8.56%, UQI is improved by 9.45%, and MSE is reduced by 22.78% using the proposed model compared to the state-of-the-arts.

Enhancing Visual Odometry with Estimated Scene Depth: Leveraging RGB-D Data with Deep Learning

Advances in visual odometry (VO) systems have benefited from the widespread use of affordable RGB-D cameras, improving indoor localization and mapping accuracy. However, older sensors like the Kinect v1 face challenges due to depth inaccuracies and incomplete data. This study compares indoor VO systems that use RGB-D images, exploring methods to enhance depth information. We examine conventional image inpainting techniques and a deep learning approach, utilizing newer depth data from devices like the Kinect v2. Our research highlights the importance of refining data from lower-quality sensors, which is crucial for cost-effective VO applications. By integrating deep learning models with richer context from RGB images and more comprehensive depth references, we demonstrate improved trajectory estimation compared to standard methods. This work advances budget-friendly RGB-D VO systems for indoor mobile robots, emphasizing deep learning’s role in leveraging connections between image appearance and depth data.

Image inpainting based on tensor ring decomposition with generative adversarial network

Image inpainting based on tensor ring decomposition with generative adversarial network

Reconstruction of the Chemical Gas Concentration Distribution Using Partial Convolution-Based Image Inpainting.

An interpolation method, which estimates unknown values with constrained information, is based on mathematical calculations. In this study, we addressed interpolation from an image-based perspective and expanded the use of image inpainting to estimate values at unknown points. When chemical gas is dispersed through a chemical attack or terrorism, it is possible to determine the concentration of the gas at each location by utilizing the deployed sensors. By interpolating the concentrations, we can obtain the contours of gas concentration. Accurately distinguishing the contours of a contaminated region from a map enables the optimal response to minimize damage. However, areas with an insufficient number of sensors have less accurate contours than other areas. In order to achieve more accurate contour data, an image inpainting-based method is proposed to enhance reliability by erasing and reconstructing low-accuracy areas in the contour. Partial convolution is used as the machine learning approach for image-inpainting, with the modified loss function for optimization. In order to train the model, we developed a gas diffusion simulation model and generated a gas concentration contour dataset comprising 100,000 contour images. The results of the model were compared to those of Kriging interpolation, one of the conventional spatial interpolation methods, finally demonstrating 13.21% higher accuracy. This suggests that interpolation from an image-based perspective can achieve higher accuracy than numerical interpolation on well-trained data. The proposed method was validated using gas concentration contour data from the verified gas dispersion modeling software Nuclear Biological Chemical Reporting And Modeling System (NBC_RAMS), which was developed by the Agency for Defense Development, South Korea.

AFTLNet: An efficient adaptive forgery traces learning network for deep image inpainting localization

Deep-learning-based image inpainting repairs a region with visually believable content, leaving behind imperceptible traces. Since deep image inpainting approaches can malevolently remove key objects and erase visible copyright watermarks, the desire for an effective method to distinguish the inpainted regions has become urgent. In this work, we propose an adaptive forgery trace learning network (AFTLN), which consists of two subblocks: the adaptive block and the Densenet block. Specifically, the adaptive block exploits an adaptive difference convolution to maximize the forgery traces by iteratively updating its weights. Meanwhile, the Densenet block improves the feature weights and reduces the impact of noise on the forgery traces. An image-inpainting detector, namely AFTLNet, is designed by integrating AFTLN with neural architecture search, and global and local attention modules, which aims to find potential tampered regions, enhance feature consistency, and reduce intra-class differences, respectively. The experimental results present that our proposed AFTLNet exceeds existing inpainting detection approaches. Finally, an inpainting dataset of 26K image pairs is constructed for future research. The dataset is available at https://pan.baidu.com/s/10SRJeQBNnTHJXvxl8xzHcg with password: 1234.

RectanglingGAN: Deep rectangling for stitched image via image inpainting

The deep rectangling task aims to transform edge-irregular stitched images into standardised rectangular formats using deep learning. Existing deep rectangling solutions rely on mesh-based segmentation and deformation, which introduce global distortions detrimental to subsequent analysis and processing. Therefore, we propose RectanglingGAN to handle missing regions in a distortion-free manner via image inpainting. Unlike generic inpainting models, RectanglingGAN focuses on preserving the quality of edge regions. In particular, we present an adaptive distance transform module to improve the attention of the feature maps to the mask boundaries. The module dynamically selects the appropriate distance transformation scheme according to the image content. Moreover, we introduce the distance-weighted decaying reconstruction loss into the pixel-wise reconstruction, which introduces the spatial location information between the inpainting pixels and the mask boundaries by inverting the distance-transformed mask. By incorporating this loss, the model pays more attention to the quality of generation pixels far from the mask boundary. Extensive experiments validate the effectiveness of RectanglingGAN compared to state-of-the-art methods, highlighting its significant advantages in terms of both the quality and the fidelity of the edge region in generation images.

DIVA: Deep unfolded network from quantum interactive patches for image restoration

This paper presents a deep neural network called DIVA unfolding a baseline adaptive denoising algorithm (DeQuIP), relying on the theory of quantum many-body physics. Furthermore, it is shown that with very slight modifications, this network can be enhanced to solve more challenging image restoration tasks such as image deblurring, super-resolution and inpainting. Despite a compact and interpretable (from a physical perspective) architecture, the proposed deep learning network outperforms several recent algorithms from the literature, designed specifically for each task. The key ingredients of the proposed method are on one hand, its ability to handle non-local image structures through the patch-interaction term and the quantum-based Hamiltonian operator, and, on the other hand, its flexibility to adapt the hyperparameters patch-wisely, due to the training process.

Cell-Impermeable Buffering Fluorogenic Probes for Live-Cell Super-Resolution Imaging of Plasma Membrane Morphology Dynamics.

Super-resolution fluorescence imaging has emerged as a potent tool for investigating the nanoscale structure and function of the plasma membrane (PM). Nevertheless, the challenge persists in achieving super-resolution imaging of PM dynamics due to limitations in probe photostability and issues with cell internalization staining. Herein, we report assembly-mediated buffering fluorogenic probes BMP-14 and BMP-16 exhibiting fast PM labeling and extended retention time (over 2 h) on PM. The incorporation of alkyl chains proves effective in promoting the aggregation of BMP-14 and BMP-16 into nonfluorescent nanoparticles to realize fluorogenicity and regulate the buffering capacity to rapidly replace photobleached probes ensuring stable long-term super-resolution imaging of PM. Utilizing these PM-buffering probes, we observed dynamic movements of PM filopodia and continuous shrinkage, leading to the formation of extracellular vesicles (EVs) using structured illumination microscopy (SIM). Furthermore, we discovered two distinct modes of EV fusion: one involving fusion through adjacent lipids and the other through filamentous lipid traction. The entire process of EV fusion outside the PM was dynamically tracked. Additionally, BMP-16 exhibited a unique capability of inducing single-molecule fluorescence blinking when used for cell membrane staining. This property makes BMP-16 suitable for the PAINT imaging of cell membranes.

Bidirectional interaction of CNN and Transformer for image inpainting

Image inpainting requires filling the corrupted regions with plausible contents and fitting seamlessly with the surroundings. Currently, CNN-dominated inpainting models still produce artifacts inconsistent with surroundings when facing large-scale corrupted regions. This is attributed to the locality and weight sharing of convolutions, which limits the ability of CNNs to learn the correlations between corrupted contents and distant contexts explicitly. In contrast, Transformer-dominated models are experts in modeling such long-range correlations, but they often deteriorate local details due to the lack of locality. A potential inpainting model should incorporate the advantages of CNN and Transformer. To do this, we propose an inpainting-oriented concurrent hybrid network, termed bidirectional interaction dual-stream network (BIDS-Net), which can sufficiently integrate the advantages of CNN and Transformer. Specifically, we adopt a CNN stream to capture rich local patterns to achieve detail reconstruction and refinement, and a Transformer stream to model long-range contextual correlations to fully borrow the relevant information within a global field of view. More importantly, we devise a bidirectional feature interaction (BFI) module based on a selective feature fusion method, which aims to enhance the locality of the transformer stream and the long-range correlation awareness of the CNN stream to realize full complementary advantages. In addition, we build both of the streams upon a hierarchical encoder–decoder structure to facilitate multi-scale context reasoning and improve computational efficiency. The comprehensive experimental results on three public datasets demonstrate the superiority of the proposed BIDS-Net in image inpainting task.

Oil painting image style recognition based on ResNet-NTS network

With the improvement of economic level, cultural and artistic pursuits have become the new life style of people. For painting exhibition style appreciation, the study uses deep learning and neural networks to classify and recognize the style identification of oil painting images. Firstly, the classification technology of residual networks is used to collect features and enhance regions in images of Western oil painting art. Secondly, the residual network is optimized and improved. Finally, attention mechanism and improved activation function are introduced to achieve the recognition effect of oil painting image style. Through comparative experiments, the results show that the improved residual network has an accuracy of 69.38% in style recognition of oil painting datasets. After optimizing with cosine annealing decay strategy, the network testing system model has the highest recognition accuracy of 74.41% for oil painting style data. Finally, the residual network and the network test system are respectively used in the style recognition of Xizang Thangka art dataset, and the accuracy of style recognition of the improved residual network and the network test system is 87.39% and 95.28%, respectively. Therefore, it proves that the residual network and network test system model based on the residual network and network test system model have superiority in the effect of oil painting style recognition, and improve technical support for art dissemination and management.

On the long-time behavior of the continuous and discrete solutions of a nonlocal Cahn–Hilliard type inpainting model

In this paper, we study the analytical and numerical long-time stability of a nonlocal Cahn–Hilliard model with a fidelity term for image inpainting introduced in our previous work (Jiang et al., 2024). First, we establish the uniform boundedness of the continuous problem in both L2 and H1 spaces, which is obtained by using the Gagliardo–Nirenberg inequality and the uniform Grönwall lemma. Then, for the temporal semi-discrete scheme, the uniform estimates in L2 and H1 spaces are derived with the aid of the discrete uniform Grönwall lemma under a suitable assumption on the nonlinear potential. This demonstrates the long-time stability of the proposed scheme in L2 and H1 spaces. Finally, we validate the long-time stability and the applicability of our method in signal reconstruction and image inpainting. These numerical experiments demonstrate the high effectiveness of our proposed model.

Deep learning adversarial attacks and defenses on license plate recognition system

The breakthroughs in Machine learning and deep neural networks have revolutionized the handling of critical practical challenges, achieving state-of-the-art performance in various computer vision tasks. Notably, the application of deep neural networks in optical character recognition (OCR) has significantly enhanced the performance of OCR systems, making them a pivotal preprocessing component in text analysis pipelines for crucial applications such as license plate recognition (LPR) systems, where the efficiency of OCR is paramount. However, despite the advancements, the integration of deep neural networks in OCR introduces inherent security vulnerabilities, particularly susceptibility to adversarial examples. Adversarial examples in LPR systems are crafted by introducing perturbations to original license plate images, which can effectively compromise the integrity of the license plate recognition process, leading to erroneous license plate number identification. Given that the primary goal of OCR in this context is to accurately recognize license plate numbers, even a single misinterpreted character can significantly impact the overall performance of the LPR system. The vulnerability of LPR systems to adversarial attacks underscores the urgent need to address the security weaknesses inherited from deep neural networks. In response to these challenges, the exploration of alternative defense mechanisms, such as image denoising and in-painting, presents a compelling approach to bolstering the resilience of LPR systems against adversarial attacks. By prioritizing practical implementation and integration of image denoising and inpainting techniques align with the operational requirements of real-world LPR systems. These methods can be seamlessly integrated into existing pipelines, offering pragmatic and accessible means of enhancing security without imposing significant computational overhead. By embracing a multi-faceted approach that combines the strengths of traditional image processing techniques, the research endeavors to develop comprehensive and versatile defense strategies tailored to the specific vulnerabilities and requirements of LPR systems. This holistic approach aims to fortify LPR systems against adversarial threats, thereby fostering increased trust and reliability in the deployment of OCR and LPR technologies across various domains and applications.

EndoSRR: a comprehensive multi-stage approach for endoscopic specular reflection removal.

Specular reflections in endoscopic images not only disturb visual perception but also hamper computer vision algorithm performance. However, the intricate nature and variability of these reflections, coupled with a lack of relevant datasets, pose ongoing challenges for removal. We present EndoSRR, a robust method for eliminating specular reflections in endoscopic images. EndoSRR comprises two stages: reflection detection and reflection region inpainting. In the reflection detection stage, we adapt and fine-tune the segment anything model (SAM) using a weakly labeled dataset, achieving an accurate reflection mask. For reflective region inpainting, we employ LaMa, a fast Fourier convolution-based model trained on a 4.5M-image dataset, enabling effective inpainting of arbitrarily shaped reflection regions. Lastly, we introduce an iterative optimization strategy for dual pre-trained models to refine the results of specular reflection removal, named DPMIO. Utilizing the SCARED-2019 dataset, our approach surpasses state-of-the-art methods in both qualitative and quantitative evaluations. Qualitatively, our method excels in accurately detecting reflective regions, yielding more natural and realistic inpainting results. Quantitatively, our method demonstrates superior performance in both segmentation evaluation metrics (IoU, E-measure, etc.) and image inpainting evaluation metrics (PSNR, SSIM, etc.). The experimental results underscore the significance of proficient endoscopic specular reflection removal for enhancing visual perception and downstream tasks. The methodology and results presented in this study are poised to catalyze advancements in specular reflection removal, thereby augmenting the accuracy and safety of minimally invasive surgery.

ColorNetVis: An Interactive Color Network Analysis System for Exploring the Color Composition of Traditional Chinese Painting.

In the field of digital humanities, color research aims to discover explanations for painting history and color usage habits. However, researchers analyzing color relationships is challenging and time-consuming, as it requires color extraction and a detailed review of many painting images for reference and comparison of color relationships. In our work, we propose ColorNetVis, an interactive color network analysis tool that enables researchers to explore color relationships through color networks. The core of ColorNetVis is a bipartite network model that establishes a bipartite relationship between colors and Chinese painting within a scope based on color difference measurement. It constructs a one-mode color network through projection algorithms and similarity calculation methods to discover the relationship between colors. We propose a coordinated set of views to demonstrate the combination of determined color networks with painting types and real-world attributes. We use color space view, color attribute distribution view, and single color query components to assist researchers in conducting detailed color analysis and validation. Through case studies, researcher reviews, and user studies, we demonstrate that ColorNetVis can effectively help researchers discover knowledge of color relationships and potential color research directions.

Correction: AGG: attention-based gated convolutional GAN with prior guidance for image inpainting

Correction: AGG: attention-based gated convolutional GAN with prior guidance for image inpainting

Алгоритм проведения профилактических и лечебных курсов арктической ландшафтной терапии для купирования различных типов стресса.

In connection with the need of non-drug biotechnologies for stress relief in the extreme conditions of the Far North, an algorithm was created and a preliminary testing of the method of Arctic landscape therapy was carried out. The method is personalized, non-invasive and, thanks to the use of virtual and painted images of natural objects, can be used at any time of the year. Due to the steady decrease in situational and personal anxiety of patients, it contributes to the solution of a wide range of various social problems

A painting authentication method based on multi-scale spatial-spectral feature fusion and convolutional neural network

The scientific authentication of paintings holds significant importance within the realm of art collection. Employing convolutional neural networks for the classification of authentic and counterfeit painting images based on color images is a viable but suboptimal choice. This study investigates the potential for authenticating paintings by incorporating high-spectral images alongside high-resolution spatial images. High-resolution and high-spectral images of genuine and counterfeit paintings were acquired using a push-broom digital scanning system. The processing methods presented in this approach for the acquired images are: 1) The study utilized the circular local binary pattern (LBP) and principal component analysis (PCA) to extract surface texture and spectral data from Chinese character images in paintings, encompassing both spatial and spectral dimensions. 2) The technique utilizing non-subsampling Shearlet transform (NSST) and pulse-coupled neural network (PCNN) was employed to integrate the spatial and spectral characteristics of the images into a pseudo-color image, producing a dataset of feature data for genuine and counterfeit paintings. 3) The experiments aimed to achieve the authenticity of artworks using EfficientNet v2-s, the hyperparameters of which were fine-tuned accordingly. The experimental findings demonstrate that this approach attained a 90.8 % accuracy on the test dataset, representing a 3.5 % enhancement over the existing top-performing three-dimensional convolutional neural network (3D-CNN).

“Sparse + Low-Rank” tensor completion approach for recovering images and videos

Recovering color images and videos from highly undersampled data is a fundamental and challenging task in face recognition and computer vision. By the multi-dimensional nature of color images and videos, in this paper, we propose a novel tensor completion approach, which is able to efficiently explore the sparsity of tensor data under the discrete cosine transform (DCT). Specifically, we introduce two “sparse + low-rank” tensor completion models as well as two implementable algorithms for finding their solutions. The first one is a DCT-based sparse plus weighted nuclear norm induced low-rank minimization model. The second one is a DCT-based sparse plus p-shrinking mapping induced low-rank optimization model. Moreover, we accordingly propose two implementable augmented Lagrangian-based algorithms for solving the underlying optimization models. A series of numerical experiments including color image inpainting and video data recovery demonstrate that our proposed approach performs better than many existing state-of-the-art tensor completion methods, especially for the case when the ratio of missing data is high.