Image Inpainting Research Articles

The use of abstract animations and a graphical body image for assessing pain outcomes among adults with sickle cell disease

Painimation, a novel digital pain assessment tool, allows patients to communicate their pain quality, intensity, and location using abstract animations and a paintable body image. This study determined the construct validity of pain animations and body image measures by testing correlations with validated pain outcomes in adults with sickle cell disease (SCD). Analyses used baseline data from a multisite randomized trial of 359 adults with SCD and chronic pain. Participants completed questionnaires on demographics, pain severity, frequency and interference, catastrophizing, opioid use, mood and quality of life, plus the Painimation app. Participants were categorized by selected pain animations, and were split into groups based on the proportion of painted body image. The “shooting” pain animation and greater body image scores associated with poorer pain outcomes in univariate analyses, except “happy” mood days. Potential confounding was evaluated by age, gender, race, education, disability, site, depression, and anxiety. Only depression scores significantly covaried in multivariate models, accounting for the effect of greater body image score and shooting animation on all outcomes except daily pain intensity. Both pain animations and body image measures correlated with validated pain outcomes, quality of life and mental health measures. This demonstrates animations and body image data can assess SCD pain severity, potentially with more accuracy than a 0-10 scale. In exploratory analyses, depression scores accounted for the association between Painimation and other pain outcomes. Future research will explore whether Painimation can differentiate biological and psychosocial pain components. PerspectiveThis article presents the preliminary construct validity of Painimation in sickle cell disease (SCD) by examining the associations of “pain animations” and body area image data with daily e-diary and traditional self-report pain outcomes.

Image Recognition of New Year Pictures based on Machine Learning

New Year pictures are an essential part of Chinese traditional culture, with profound historical deposits and unique artistic value. However, with the rapid development of society, the inheritance and protection of traditional New Year pictures are facing many challenges. One of them is the recognition of the New Year painting images. This paper introduces image recognition technology based on machine learning, including research background, method steps, result discovery, and advantage limitations. Image recognition is an essential means to protect and inherit the traditional New Year painting culture and machine learning technology can improve recognition accuracy and efficiency. This paper presents deep learning implemented through data collection, feature extraction, and classification. The experimental results show that the method can effectively identify New Year images with high accuracy and recall.

Traditional landscape painting and art image restoration methods based on structural information guidance

Abstract In the field of traditional landscape painting and art image restoration, traditional restoration methods have gradually revealed limitations with the development of society and technological progress. In order to enhance the restoration effects of Chinese landscape paintings, an innovative image restoration algorithm is designed in this research, combining edge restoration with generative adversarial networks (GANs). Simultaneously, a novel image restoration model with embedded multi-scale attention dilated convolution is proposed to enhance the modeling capability for details and textures in landscape paintings. To better preserve the structural features of artistic images, a structural information-guided art image restoration model is introduced. The introduction of adversarial networks into the repair model can improve the repair effect. The art image repair model adds a multi-scale attention mechanism to handle more complex works of art. The research results show that the image detection model improves by 0.20, 0.07, and 0.06 in the Spearman rank correlation coefficient, Pearson correlation coefficient, and peak signal-to-noise ratio (PSNR), respectively, compared to other models. The proposed method outperforms mean filtering, wavelet denoising, and median filtering algorithms by 6.3, 9.1, and 15.8 dB in PSNR and by 0.06, 0.12, and 0.11 in structural similarity index. In the image restoration task, the structural similarity and information entropy indicators of the research model increase by approximately 9.3 and 3%, respectively. The image restoration method proposed in this study is beneficial for preserving and restoring precious cultural heritage, especially traditional Chinese landscape paintings, providing new technological means for cultural relic restoration.

TSFormer: Tracking Structure Transformer for Image Inpainting

TSFormer: Tracking Structure Transformer for Image Inpainting

Structure-Guided Image Inpainting Based on Multi-Scale Attention Pyramid Network

Current single-view image inpainting methods often suffer from low image information utilization and suboptimal repair outcomes. To address these challenges, this paper introduces a novel image inpainting framework that leverages a structure-guided multi-scale attention pyramid network. This network consists of a structural repair network and a multi-scale attention pyramid semantic repair network. The structural repair component utilizes a dual-branch U-Net network for robust structure prediction under strong constraints. The predicted structural view then serves as auxiliary information for the semantic repair network. This latter network exploits the pyramid structure to extract multi-scale features of the image, which are further refined through an attention feature fusion module. Additionally, a separable gated convolution strategy is employed during feature extraction to minimize the impact of invalid information from missing areas, thereby enhancing the restoration quality. Experiments conducted on standard datasets such as Paris Street View and CelebA demonstrate the superiority of our approach over existing methods through quantitative and qualitative comparisons. Further ablation studies, by incrementally integrating proposed mechanisms into a baseline model, substantiate the effectiveness of our multi-view restoration strategy, separable gated convolution, and multi-scale attention feature fusion.

Curvature-Dependent Elastic Bending Total Variation Model for Image Inpainting with the SAV Algorithm

Curvature-Dependent Elastic Bending Total Variation Model for Image Inpainting with the SAV Algorithm

Appearance flow based structure prior guided image inpainting

Appearance flow based structure prior guided image inpainting

Image Inpainting via Correlated Multi-Resolution Feature Projection.

With the advancement in image editing applications, image inpainting is gaining more attention due to its ability to recover corrupted images efficiently. Also, the existing methods for image inpainting either use two-stage coarse-to-fine architectures or single-stage architectures with a deeper network. On the other hand, shallow network architectures lack the quality of results and the methods with remarkable inpainting quality have high complexity in terms of number of parameters or average run time. Despite the improvement in the inpainting quality, these methods still lack the correlated local and global information. In this work, we propose a single-stage multi-resolution generator architecture for image inpainting with moderate complexity and superior outcomes. Here, a multi-kernel non-local (MKNL) attention block is proposed to merge the feature maps from all the resolutions. Further, a feature projection block is proposed to project features of MKNL to respective decoder for effective reconstruction of image. Also, a valid feature fusion block is proposed to merge encoder skip connection features at valid region and respective decoder features at hole region. This ensures that there will not be any redundant feature merging while reconstruction of image. Effectiveness of the proposed architecture is verified on CelebA-HQ Liu, et al. 2015, Karras et al. 2017, and Places2 Zhou et al. 2018 datasets corrupted with publicly available NVIDIA mask dataset Liu et al. 2018. The detailed ablation study, extensive result analysis, and application of object removal prove the robustness of the proposed method over existing state-of-the-art methods for image inpainting.

MILG: Realistic lip-sync video generation with audio-modulated image inpainting

Existing lip synchronization (lip-sync) methods generate accurately synchronized mouths and faces in a generated video. However, they still confront the problem of artifacts in regions of non-interest (RONI), e.g., background and other parts of a face, which decreases the overall visual quality. To solve these problems, we innovatively introduce diverse image inpainting to lip-sync generation. We propose Modulated Inpainting Lip-sync GAN (MILG), an audio-constraint inpainting network to predict synchronous mouths. MILG utilizes prior knowledge of RONI and audio sequences to predict lip shape instead of image generation, which can keep the RONI consistent. Specifically, we integrate modulated spatially probabilistic diversity normalization (MSPD Norm) in our inpainting network, which helps the network generate fine-grained diverse mouth movements guided by the continuous audio features. Furthermore, to lower the training overhead, we modify the contrastive loss in lip-sync to support small-batch-size and few-sample training. Extensive experiments demonstrate that our approach outperforms the existing state-of-the-art of image quality and authenticity while keeping lip-sync.

A Mumford-Shah Model-based Method for Inpainting Markers in Ultrasound Images

Abstract During ultrasound diagnosis in clinics, clinicians often mark the lesion area, resulting in the presence of unnecessary objects in the collected ultrasound images. These markers potentially affect subsequent image analysis. To address this problem, we proposed an image inpainting method that combines image processing with Mumford-Shah algorithm to remove markers from ultrasound images. The proposed method consists of two parts. First, the input ultrasound images were processed by several image processing algorithms, including contrast enhancement, image binarization, edge detection, hole filling, and connected domain marking. These algorithms were combined to extract the mask of the manual marker in the ultrasound images. Second, the original ultrasound images and mask images were put into the Mumford-Shah model to implement ultrasound image inpainting by minimizing the following energy functional. The continuous iteration of Euler-Lagrange equation was employed to update the iteration item. If the difference between the two output results was less than the pre-set threshold, the output result could be regarded as the repaired ultrasound image. Experimental results show that the designed image processing method accurately extracted the binary mask, whether they are colourful or binary, number or line in ultrasound images. Meanwhile, the improved Mumford-Shah model effectively implemented the ultrasound image inpainting, with the accuracy of 96.55%. Therefore, the proposed method can be used to remove unnecessary objects in ultrasound images, which is beneficial to improve the accuracy of the artificial intelligence model.

Mapping the Landscape of VFX and AI: A PRISMA-guided Systematic Review

Objective: This systematic literature review aims to comprehensively synthesize existing evidence on Visual Effects (VFX) and Artificial Intelligence (AI) using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The primary objective is to map the landscape of VFX and AI, examining the historical development of VFX techniques and practices in the context of AI advancements. Methods: Following PRISMA guidelines, a thorough search of five electronic databases—IEEE Xplore, Scopus, Taylor & Francis Online, ACM Digital Library, and ScienceDirect—was conducted to identify articles published up to April 18, 2024. Inclusion criteria encompassed English-language journal articles. Independent reviewers screened articles for eligibility, resolving discrepancies through consensus. Data extraction and synthesis were performed systematically. Results: A total of 41 studies met the inclusion criteria and were included in the review. These studies covered a wide range of topics related to VFX and AI, with various study designs. The findings are presented in a narrative synthesis. Discussion: The review indicates that AI development in VFX is comprehensive, covering topics such as image processing, inpainting, video processing, 3D modelling, rendering, deepfake technology, style transfer, tracking, animation, simulation, rigging, texture compression, and depth map enhancement. These advancements highlight the transformative role of AI in enhancing creative expression, production efficiency, and the quality of visual effects. Conclusion: This systematic literature review provides a comprehensive overview of the current state of knowledge on AI and VFX. The integration of AI into VFX is profoundly transforming the industry, enhancing creative expression, production efficiency, and the quality of visual effects. Continued advancements in AI technologies promise to further revolutionize VFX practices, offering new possibilities and efficiencies for future productions.

Advancing image inpainting efficiency: An exploration of pixel and channel split operations

Deep learning-based image inpainting techniques have achieved unprecedented results using encoder–decoder structures to recover complex missing areas of an image. Recent inpainting models use additional information or networks (e.g., landmarks, edges, styles, and filters) to realize improved restoration performance, but at the cost of increased computational resources. To improve the relationship between inpainting performance and the number of model parameters, researchers have investigated efficient structural approaches such as recurrent and residual connection structures. However, these methods are difficult to apply in the general encoder–decoder structure. In this study, we explored the downsampling and upsampling operations associated with an encoder–decoder structure. We propose two novel split operations: the pixel-split operation (PSO) and channel-split operation (CSO). The proposed PSO transfers image features from high to low resolution with two dilation rate effects and a similar number of parameters as existing downsampling operations. Conversely, the proposed CSO increases the image resolution using only one-fourth the number of parameters of existing upsampling operations. The restoration performance and efficiency of the proposed model were evaluated in terms of five metrics on public datasets, e.g., the Places2 and CelebA datasets, to validate our proposed operations’ contribution to inpainting performance. We achieved state-of-the-art performance and reduced the size of the parameters by 20%. An ablation study was conducted to confirm the effect of each operation on the CelebA-HQ dataset. Results indicated that these split operations exhibit an advanced relationship between inpainting performance and optimization of the model parameters. The corresponding codes are available online (https://github.com/MrCAIcode/Split_operation_for_inpainting).

Image inpainting algorithm based on inference attention module and two-stage network

Current image inpainting techniques often yield distorted structures or blurred textures that clash with the surrounding context, particularly when addressing extensive missing regions or highly textured images. These methods often struggle to reconstruct realistic and coherent image structures. To address this challenge, we introduce a two-stage network image restoration approach that leverages an inferential attention mechanism. In the first stage, an edge generation network is employed to produce plausible phantom edge information. Subsequently, an image complementation network is utilized to complete the image restoration process. To enhance the visual realism and restoration accuracy of the generated images, we incorporate an inferential attention mechanism within the image complementation network. This mechanism effectively mitigates inconsistencies in the generated features, leading to the production of more meaningful and effective information. We evaluate our proposed method on benchmark datasets, including Places2 dataset, CelebFaces Attributes dataset, and Paris StreetView dataset. The experimental results can demonstrate that the proposed method achieves better Structural Similarity Index and Peak Signal-to-noise Ratio than others. These metrics indicate superior performance compared to existing image inpainting methods, delivering higher image inpainting accuracy and more realistic image reconstruction effect.

Comprehensively Modulated Sub-Attojoule Operated Optoelectronic Synapses for Image Encryption and Inpainting.

High-performance optoelectronic synaptic transistors play a crucial role in developing and emulating artificial visual systems. However, due to the predominant use of single-structure material modulation in optimizing optoelectronic synapses, their energy consumption significantly trails behind that of electronic synapses by several orders of magnitude. Herein, polymer dielectric layers and optimized contact strategies are adopted to realize the ultralow consumption optoelectronic synapses. Integration of polyimide dielectric significantly enhances photogenerated charge carrier dissociation, leading to substantial improvements in photoresponsivity (1.5 × 106 A·W-1), photodetectivity (6.9 × 1012 Jones), and external quantum efficiency (4.0 × 108%). Additionally, optimized contact properties augment their appeal for ultralow energy consumption in optoelectronic synapse applications. Excitatory postsynaptic current is triggered at an incredibly low voltage of 5 μV and boosts an impressively low energy consumption of 0.05 aJ, ranking among the best-reported results in this field. Next, we demonstrate an integrated system combining the MoS2 optoelectronic synapses with a recurrent neural network enabling 100% accurate recognition of optical signals, particularly in scenarios with aJ-leveled energy consumption. Finally, an image encryption system has been developed, in which images are encrypted by photoelectronic conversion of synapse arrays with random voltage settings and decrypted according to the recurrent neural network-based accuracy. More importantly, once partially damaged images are encrypted, through the decryption image inpainting can be realized due to the high accuracy. The proposed innovative approach holds promise for advancing artificial intelligence applications with improved energy efficiency, information security, and computational capabilities.

Brain Magnetic Resonance Image Inpainting via Deep Edge Region-based Generative Adversarial Network

Brain Magnetic Resonance Image Inpainting via Deep Edge Region-based Generative Adversarial Network

Image inpainting by bidirectional information flow on texture and structure

Image inpainting aims to recover damaged regions of a corrupted image and maintain the integrity of the structure and texture within the filled regions. Previous popular approaches have restored images with both vivid textures and structures by introducing structure priors. However, the structure prior-based approaches meet the following main challenges: (1) the fine-grained textures suffer from adverse inpainting effects because they do not fully consider the interaction between structures and textures, (2) the features of the multi-scale objects in structural and textural information cannot be extracted correctly due to the limited receptive fields in convolution operation. In this paper, we propose a texture and structure bidirectional generation network (TSBGNet) to address the above issues. We first reconstruct the texture and structure of corrupted images; then, we design a texture-enhanced-FCMSPCNN (TE-FCMSPCNN) to optimize the generated textures. We also conjoin a bidirectional information flow (BIF) module and a detail enhancement (DE) module to integrate texture and structure features globally. Additionally, we derive a multi-scale attentional feature fusion (MAFF) module to fuse multi-scale features. Experimental results demonstrate that TSBGNet effectively reconstructs realistic contents and significantly outperforms other state-of-the-art approaches on three popular datasets. Moreover, the proposed approach yields promising results on the Dunhuang Mogao Grottoes Mural dataset.

A text guided cross modal joint inpainting algorithm for ancient murals

Existing deep learning algorithms for ancient mural inpainting only consider prior information of mural images to guide the inpainting of damaged areas, lacking guidance from textual semantic information, which often leads to problems such as inconsistent semantics and disordered textures in the inpainting results. In this work, we propose a text guided cross modal joint inpainting algorithm for ancient murals. Firstly, a cross modal joint inpainting network for ancient murals was constructed, which introduced text semantic information as the control guidance for mural inpainting, providing contextual information for mural image inpainting and guiding the inpainting of damaged areas. Then, design a text encoder and an image encoder to extract text and image features respectively, including word and sentence features. Input the sentence features into the conditional enhancement module, resample the text sentences, and solve the problem of text sparsity. Secondly, design a text decoupling module that utilizes a multimodal attention mechanism to extract the semantic information of the missing parts of damaged murals, and design a text semantic enhancement module to improve the semantic representation ability of the text, achieve the repair of local details of murals, and solve the problem of texture disorder. Finally, the reconstruction and repair of murals are completed through a residual discriminator game and a spectral normalized Markov dual discriminator against the mural generation network. Through inpainting experiments on real Dunhuang murals, it has been shown that the proposed method is more effective than comparative methods in repairing damaged murals, achieving better visual perception and coordination in reconstruction, and outperforming comparative algorithms in both subjective and objective evaluations.

Ancient Painting Inpainting Based on Multi-Layer Feature Enhancement and Frequency Perception

Image inpainting aims to restore the damaged information in images, enhancing their readability and usability. Ancient paintings, as a vital component of traditional art, convey profound cultural and artistic value, yet often suffer from various forms of damage over time. Existing ancient painting inpainting methods are insufficient in extracting deep semantic information, resulting in the loss of high-frequency detail features of the reconstructed image and inconsistency between global and local semantic information. To address these issues, this paper proposes a Generative Adversarial Network (GAN)-based ancient painting inpainting method using multi-layer feature enhancement and frequency perception, named MFGAN. Firstly, we design a Residual Pyramid Encoder (RPE), which fully extracts the deep semantic features of ancient painting images and strengthens the processing of image details by effectively combining the deep feature extraction module and channel attention. Secondly, we propose a Frequency-Aware Mechanism (FAM) to obtain the high-frequency perceptual features by using the frequency attention module, which captures the high-frequency details and texture features of the ancient paintings by increasing the skip connections between the low-frequency and the high-frequency features, and provides more frequency perception information. Thirdly, a Dual Discriminator (DD) is designed to ensure the consistency of semantic information between global and local region images, while reducing the discontinuity and blurring differences at the boundary during image inpainting. Finally, extensive experiments on the proposed ancient painting and Huaniao datasets show that our proposed method outperforms competitive image inpainting methods and exhibits robust generalization capabilities.

Representation of classical painting images in an advertising product as a form of dialogue between elite and mass culture (using the example of Volkswagen AG’s 2008 advertising campaign)

Representation of classical painting images in an advertising product as a form of dialogue between elite and mass culture (using the example of Volkswagen AG’s 2008 advertising campaign)

Quantum deep generative prior with programmable quantum circuits

Exploiting the utility of near-term quantum devices is a long-standing challenge whereas hybrid quantum machine learning emerges as a promising candidate. Here we propose a quantum-enhanced deep generative algorithm based on programmable quantum circuit-induced quantum latent codes. To validate the effectiveness of the algorithm, we conduct optical ghost imaging experiments, collecting dataset under varying physical sampling rates. Leveraging a physically enhanced loss function and a pretrained neural network, the quantum algorithm exhibits superior reconstruction performance compared to conventional algorithms. We observe the learnable quantum latent space enhances the out-of-distribution generalization capability of the pretrained model. In the context of computer vision problems, numerical results indicate that quantum latent space can increase the generation diversity of the pretrained model. Furthermore, the algorithm outperforms classical counterparts, particularly in image inpainting and colorization, by a significant margin. Our study demonstrates the utility of hybrid quantum-classical algorithms in enhancing generalization capability, highlighting the potential of near-term quantum devices in large-scale generative artificial intelligence.