Inpainted Video Research Articles

Video Inpainting with Local and Global Refinement

Video inpainting is a crucial task in computer vision and video editing, involving in removing unnecessary objects and restoration of missing or corrupted regions within a video sequence. One approach that has gained prominence in recent years is the combination of local and global refinement techniques. This innovative strategy leverages the strengths of both local and global information to produce high-quality inpainted videos. In the context of video inpainting, local refinement focuses on accurately restoring missing or damaged regions by considering nearby pixels or frames. Encoder-decoder network can be employed to fill in gaps with content that seamlessly blends with the surrounding context. On the other hand, global refinement seeks to ensure temporal consistency and smooth transitions between frames, preventing noticeable artifacts or jittering in the inpainted video. In conclusion, by seamlessly blending local and global inpainting strategies, these methods can effectively remove unwanted elements from videos while preserving both spatial and temporal coherence. This technology finds applications in video editing, restoration of damaged archival footage, and even in the entertainment industry for special effects and scene corrections, ultimately contributing to the improvement of video quality and aesthetics. Keywords: Video Inpainting, Local and Global Refinement, Encoder-Decoder network, Recurrent Flow Completion, mask-guided sparse video Transformer, dual-domain propagation.

Transformer-Guided Video Inpainting Algorithm Based on Local Spatial-Temporal joint

INTRODUCTION: Video inpainting is a very important task in computer vision, and it’s a key component of various practical applications. It also plays an important role in video occlusion removal, traffic monitoring and old movie restoration technology. Video inpainting is to obtain reasonable content from the video sequence to fill the missing region, and maintain time continuity and spatial consistency.OBJECTIVES: In previous studies, due to the complexity of the scene of video inpainting, there are often cases of fast motion of objects in the video or motion of background objects, which will lead to optical flow failure. So the current video inpainting algorithm hasn’t met the requirements of practical applications. In order to avoid the problem of optical flow failure, this paper proposes a transformer-guided video inpainting model based on local Spatial-temporal joint.METHODS: First, considering the rich Spatial-temporal relationship between local flows, a Local Spatial-Temporal Joint Network (LSTN) including encoder, decoder and transformer module is designed to roughly inpaint the local corrupted frames, and the Deep Flow Network is used to calculate the local bidirectional corrupted flows. Then, the local corrupted optical flow map is input into the Local Flow Completion Network (LFCN) with pseudo 3D convolution and attention mechanism to obtain a complete set of bidirectional local optical flow maps. Finally, the roughly inpainted local frame and the complete bidirectional local optical flow map are sent to the Spatial-temporal transformer and the inpainted video frame is output.RESULTS: Experiments show that the algorithm achieves high quality results in the video target removal task, and has a certain improvement in indicators compared with advanced technologies.CONCLUSION: Transformer-Guided Video Inpainting Algorithm Based on Local Spatial-Temporal joint can obtain high-quality optical flow information and inpainted result video.

Consistent Video Inpainting Using Axial Attention-Based Style Transformer

Maintaining spatial and temporal consistency in the inpainted video area of the video is a challenging problem. Recent research focuses on flow information for synthesizing temporally smooth pixels while neglecting semantic structural coherence across the video frames. Thus, it suffers from over-smoothing and shadowy outlines that significantly degrade the inpainted video quality. We propose an end-to-end consistent video inpainting model that will substantially improve the inpainted video region to overcome this problem. The model employs a deep encoder (DE), axial attention block (AAB), style transformer, and decoder to enhance video inpainting with a realistic structure. A deep encoder (DE) encodes features effectively while the axial attention block (AAB) recreates all retrieved attributes by merging recoverable multi-scale characteristics with local spatial structures. Then, a novel-style transformer with the style manipulation block (SMB) fills the missing area with rich visual elements and temporal coherence. We use two publicly available benchmark datasets to assess the model's performance. Experimental results demonstrate that our method performs better than the state-of-the-art methods by a large margin. Besides, an extensive ablation study validates the model's performance.

Spatio-Temporal Inference Transformer Network for Video Inpainting

Video inpainting aims to complete in a visually pleasing way the missing regions in video frames. Video inpainting is an exciting task due to the variety of motions across different frames. The existing methods usually use attention models to inpaint videos by seeking the damaged content from other frames. Nevertheless, these methods suffer due to irregular attention weight from spatio-temporal dimensions, thus giving rise to artifacts in the inpainted video. To overcome the above problem, Spatio-Temporal Inference Transformer Network (STITN) has been proposed. The STITN aligns the frames to be inpainted and concurrently inpaints all the frames, and a spatio-temporal adversarial loss function improves the STITN. Our method performs considerably better than the existing deep learning approaches in quantitative and qualitative evaluation.

Joint Subtitle Extraction and Frame Inpainting for Videos with Burned-In Subtitles

Subtitles are crucial for video content understanding. However, a large amount of videos have only burned-in, hardcoded subtitles that prevent video re-editing, translation, etc. In this paper, we construct a deep-learning-based system for the inverse conversion of a burned-in subtitle video to a subtitle file and an inpainted video, by coupling three deep neural networks (CTPN, CRNN, and EdgeConnect). We evaluated the performance of the proposed method and found that the deep learning method achieved high-precision separation of the subtitles and video frames and significantly improved the video inpainting results compared to the existing methods. This research fills a gap in the application of deep learning to burned-in subtitle video reconstruction and is expected to be widely applied in the reconstruction and re-editing of videos with subtitles, advertisements, logos, and other occlusions.

Improvement in video inpainting in presence of moving subjects

Inpainting or completion is used with the purpose of restoring damaged images and video frames. This paper proposes an applicable algorithm to inpaint corrupted subjects in video frames. To begin with, background and foreground (moving subject) are separated from each other in each frame, with the aim of getting to a more visually pleasant result. Static background inpainting is done using a patch-based method. To inpaint the corrupted moving subject, a subject-based method that is an improvement on the rigid object-based method is used in this study to consider the special issue of inpainting the human body. To fill in the holes created by occluding subjects the most appropriate template is found using a similarity measure which is based on both contour and pixel values. The inpainted video is acquired by superimposing the completed foreground on the inpainted background

Video Inpainting Using a Contour-Based Method in Presence of More than One Moving Objects

To restore corrupted images and video frames, a technique named inpainting or completion is used. This paper introduces an approach which is used to inpaint damaged objects in video frames. These objects are damaged whether partially or totally. In order to get to a more visually pleasant result, the moving objects and the background are separated from each other. To inpaint the stationary background, a patch-based method is used. Missing moving objects are inpainted using an object-based method with the help of a contour-based method. Presence of more than one moving object makes a problem; how to separate the goal object from other moving objects. This paper proposes a new method based on contour to solve this problem. Finally, the inpainted foreground and background are superimposed; the result is the inpainted video.

Auto-inpainting heritage scenes: a complete framework for detecting and infilling cracks in images and videos with quantitative assessment

The need for preservation of cultural heritage has necessitated the research on digitally repairing the photographs of damaged monuments. In this paper, we first propose a technique for automatically detecting the cracked regions in photographs of monuments. Unlike the usual practice of manually selecting the mask for inpainting, the detected regions are supplied to an inpainting algorithm. Thus, the process of digitally repairing the cracked regions that physical objects have, using inpainting, is completely automated. The detection of cracked regions is based on comparison of patches, for which we use a measure derived from the edit distance, which is a popular string metric used in the area of text mining. Further, we extend this method to perform inpainting of video frames by making use of the scale-invariant feature transform and homography. We consider the camera to move while capturing video of the heritage site, as such videos are typically captured by novices, hobbyists and tourists. Finally, we also propose a video quality measure to quantify the temporal consistency of the inpainted video. Experiments have been carried out on videos captured from the heritage site at Hampi, India.