Depth Image Based Rendering Research Articles

TRRHA: A two-stream re-parameterized refocusing hybrid attention network for synthesized view quality enhancement

In multi-view video systems, the decoded texture video and its corresponding depth video are utilized to synthesize virtual views from different perspectives using the depth-image-based rendering (DIBR) technology in 3D-high efficiency video coding (3D-HEVC). However, the distortion of the compressed multi-view video and the disocclusion problem in DIBR can easily cause obvious holes and cracks in the synthesized views, degrading the visual quality of the synthesized views. To address this problem, a novel two-stream re-parameterized refocusing hybrid attention (TRRHA) network is proposed to significantly improve the quality of synthesized views. Firstly, a global multi-scale residual information stream is applied to extract the global context information by using refocusing attention module (RAM), and the RAM can detect the contextual feature and adaptively learn channel and spatial attention feature to selectively focus on different areas. Secondly, a local feature pyramid attention information stream is used to fully capture complex local texture details by using re-parameterized refocusing attention module (RRAM). The RRAM can effectively capture multi-scale texture details with different receptive fields, and adaptively adjust channel and spatial weights to adapt to information transformation at different sizes and levels. Finally, an efficient feature fusion module is proposed to effectively fuse the extracted global and local information streams. Extensive experimental results show that the proposed TRRHA achieves significantly better performance than the state-of-the-art methods. The source code will be available at https://github.com/647-bei/TRRHA.

Implementation Paper on Automated 2D Image to 3D Model Generation

With the rise of digital world, traditional learning methods, while time-tested, often fall short in meeting the dynamic educational needs of today’s learners. They are constrained by several drawbacks that impede effective comprehension and engagement. This abstract sheds light on the limitations of traditional learning approaches and introduces how the integration of 3D model reconstruction can mitigate these challenges, thereby enhancing the learning experience. We proposed deep Conventional techniques, which create a virtual view from a reference picture, use two distinct steps: depth image-based rendering (DIBR) with estimated depth and depth (or disparity) estimation for the reference image and based on these steps we can create 3D model. Which allows educators to create immersive and interactive learning environments that engage students on a profound level. By converting abstract concepts into tangible and visually stimulating representations, 3D models bridge the gap between theory and practice, making complex subjects more accessible. Keywords: 2D Image, 3D Model, CNN, DIBR

Coarse- and Fine-Grained Fusion Hierarchical Network for Hole Filling in View Synthesis.

Depth image-based rendering (DIBR) techniques play an essential role in free-viewpoint videos (FVVs), which generate the virtual views from a reference 2D texture video and its associated depth information. However, the background regions occluded by the foreground in the reference view will be exposed in the synthesized view, resulting in obvious irregular holes in the synthesized view. To this end, this paper proposes a novel coarse and fine-grained fusion hierarchical network (CFFHNet) for hole filling, which fills the irregular holes produced by view synthesis using the spatial contextual correlations between the visible and hole regions. CFFHNet adopts recurrent calculation to learn the spatial contextual correlation, while the hierarchical structure and attention mechanism are introduced to guide the fine-grained fusion of cross-scale contextual features. To promote texture generation while maintaining fidelity, we equip CFFHNet with a two-stage framework involving an inference sub-network to generate the coarse synthetic result and a refinement sub-network for refinement. Meanwhile, to make the learned hole-filling model better adaptable and robust to the "foreground penetration" distortion, we trained CFFHNet by generating a batch of training samples by adding irregular holes to the foreground and background connection regions of high-quality images. Extensive experiments show the superiority of our CFFHNet over the current state-of-the-art DIBR methods. The source code will be available at https://github.com/wgc-vsfm/view-synthesis-CFFHNet.

Cross-Layer Video Synthesizing and Antenna Allocation Scheme for Multi-View Video Provisioning Under Massive MIMO Networks

Due to the growing need for bandwidth starving Multi-View Videos (MVV) in virtual reality, TV, and education, effectively allocating the resources of next-generation wireless technologies for MVV streams becomes increasingly crucial. To achieve high utility for MVV users, this paper proposes a cross-layer resource allocation mechanism to leverage video synthesizing schemes (such as Depth-Image-Based Rendering (DIBR) for efficient MVV streaming with massive MIMO). First, we formulate a new problem, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">antenna allocation with video synthesis</i> (AAVS), and prove its NP-hardness. Then, we design an approximation algorithm named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Utility-based Multi-View Synthesis</i> (UMVS) with the analytical performance provided, and dynamic scenarios are addressed by augmenting UMVS with deep reinforcement learning. Data-driven simulation results show that UMVSoutperforms existing antenna allocation schemes by at least 10%, and the DRL extension provides an additional 6% improvement in system utility under congested scenarios.

Adaptable 2D to 3D Stereo Vision Image Conversion Based on a Deep Convolutional Neural Network and Fast Inpaint Algorithm.

Algorithms for converting 2D to 3D are gaining importance following the hiatus brought about by the discontinuation of 3D TV production; this is due to the high availability and popularity of virtual reality systems that use stereo vision. In this paper, several depth image-based rendering (DIBR) approaches using state-of-the-art single-frame depth generation neural networks and inpaint algorithms are proposed and validated, including a novel very fast inpaint (FAST). FAST significantly exceeds the speed of currently used inpaint algorithms by reducing computational complexity, without degrading the quality of the resulting image. The role of the inpaint algorithm is to fill in missing pixels in the stereo pair estimated by DIBR. Missing estimated pixels appear at the boundaries of areas that differ significantly in their estimated distance from the observer. In addition, we propose parameterizing DIBR using a singular, easy-to-interpret adaptable parameter that can be adjusted online according to the preferences of the user who views the visualization. This single parameter governs both the camera parameters and the maximum binocular disparity. The proposed solutions are also compared with a fully automatic 2D to 3D mapping solution. The algorithm proposed in this work, which features intuitive disparity steering, the foundational deep neural network MiDaS, and the FAST inpaint algorithm, received considerable acclaim from evaluators. The mean absolute error of the proposed solution does not contain statistically significant differences from state-of-the-art approaches like Deep3D and other DIBR-based approaches using different inpaint functions. Since both the source codes and the generated videos are available for download, all experiments can be reproduced, and one can apply our algorithm to any selected video or single image to convert it.

AFD-Former: A Hybrid Transformer With Asymmetric Flow Division for Synthesized View Quality Enhancement

Recently, CNN-based post-processing has shown great potential in Synthesized View Quality Enhancement (SVQE). However, due to the limited receptive field of convolution, it is ineffective in explicitly modeling long-range dependencies, which are critical to eliminate the distortion induced by Depth Image Based Rendering (DIBR) in synthesized views. Although transformers exhibit tremendous success at learning global contextual information, it is weak at extracting local texture information. To take full advantages of the CNN and transformer, we present a novel U-shaped hybrid transformer with asymmetric flow division to collaboratively capture global-local information for SVQE, termed as AFD-former. Specifically, the AFD-former utilizes the Transformer-CNN Block (TCB) as encoder and decoder, in which several Dynamic Hybrid Attention Blocks (DHABs) are designed to simultaneously model long-range interactions and retain texture details. Then, considering that the deeper layers of the U-shaped network play more roles in capturing global information while shallow layers more in extracting local information, an Asymmetric Flow Division Unit (AFDU) is embedded into each DHAB to assign different contributions of global-local contextual information to the transformer and CNN branches across different layers. Finally, a dynamic learnable modulator is incorporated into two branches to help model effectively feature representation learning. That can be viewed as the dynamic process of adjusting the weight for each channel of the input feature based on contextual cues. Extensive experiments demonstrate that the proposed AFD-former can significantly enhance perceptual quality of synthesized views with similar SVQE speed compared with the related state-of-the-art SVQE methods. The source code will be available at https://github.com/House-yuyu/AFD-former.

As-Deformable-As-Possible Single-Image-Based View Synthesis Without Depth Prior

Depth-image-based rendering (DIBR) technologies have been widely employed to synthesize novel realistic views from a single image in 3D video applications. However, DIBR-oriented approaches heavily rely on the accuracy of depth maps, usually requiring the depth GT as a prior. Despite that, there might exist extensive float precision losses and invalid holes in the synthesized view due to warping error and occlusion. In this paper, we propose an end-to-end as-deformable-as-possible (ADAP) single-image-based view synthesis solution without depth prior. It addresses the above issues through two stages: alignment and reconstruction, where we first transform the input image to the latent feature space and then reconstruct the novel view in the image domain. In the first stage, the input image is deformed to align with the synthesized view at feature level. To this end, we propose an ADAP alignment mechanism through pixel-level warping to error-level quantization to feature-level alignment, progressively improving the deformable capability in handling challenging motion conditions in real-world scenes. In the second stage, we exploit an occlusion-aware reconstruction module to recover the content details from the deformed feature at pixel level. Extensive experiments demonstrate that our alignment-reconstruction approach is robust to the depth map. Even with a coarsely estimated depth map, our solution outperforms other SoTA schemes in the popular benchmarks.

Multi-layer and Multi-scale feature aggregation for DIBR-Synthesized image quality assessment

Depth-Image-Based Rendering (DIBR) is one of the main fundamental techniques for generating new viewpoints in 3D video applications such as multi-viewpoint video (MVV), free viewpoint video (FVV) and virtual reality (VR). Due to the imperfections of color images, depth maps or texture restoration techniques, several types of distortions occur in synthesized views. However, most of related works evaluated the quality of DIBR-synthesized views by only detecting a specific type of distortion, such as stretching, black holes, blurring, etc., which were unable to accurately evaluate the quality of DIBR-synthesized views. In this paper, a new no-reference image quality assessment method is proposed to evaluate the quality of DIBR-synthesized images by combining multi-layer and multi-scale features of images. To be specific, the distortions introduced by different stages of virtual viewpoint synthesis are first analyzed, and then multi-layer and multi-scale features are extracted to estimate the degree of texture and structure distortions. As a result, individual quality scores associated with two types of distortions (e.g., structural distortion and texture distortion) are aggregated to an overall image quality. Experimental results on two publicly available DIBR datasets show that the method has better performance than the state-of-the-art models.Index Terms: image quality assessment, DIBR-synthesized image, distortion correction, BIQA.

A bilateral attention based generative adversarial network for DIBR 3D image watermarking

This paper presents a bilateral attention based generative adversarial network (BAGAN) for depth-image-based rendering (DIBR) 3D image watermarking to protect the image copyright. Convolutional block operations are employed to extract main image features for robust watermarking, but embedding watermark into some features will degrade image quality much. To relieve this kind of image distortion, the bilateral attention module (BAM) is utilized by mining correlations of the center view and the depth map to compute attention of the 3D image for guiding watermark to distribute over different image regions. Since a modality gap exists between the center view and the depth map, a cross-modal feature fusion module (CMFFM) is designed for BAM to bridge the cross-view gap. Because the depth map has lots of flat background information including many redundant features, to prune them, the depth redundancy elimination module (DREM) is used for cross-view feature fusion. In the decoder, two extractors with the same structure are built to recover watermark from the center view and the synthesized view, respectively. In addition, the discriminator is supposed to build a competitive relationship with the encoder to increase the image quality. The noise sub-network is used to train different image attacks for robustness. Extensive experimental results have demonstrated that the proposed BAGAN can obtain higher watermarking invisibility and robustness compared with existing DIBR 3D watermarking methods. Ablation experiments have also proven the effectiveness of DREM, CMFFM and BAM on BAGAN.

Quality Prediction of View Synthesis Based on Curriculum-Style Structure Generation

Existing quality metrics of view synthesis usually perform on synthesized images, which are produced based on a computationally expensive depth-image-based rendering (DIBR) process. Moreover, current metrics quantify quality by extracting hand-crafted features, which may fail to fully capture the complex distortion characteristics. With the success of deep learning on numerous computer vision tasks, it has become possible to utilize convolutional neural networks to predict the quality of DIBR-synthesized images. In this letter, we propose a deep model to predict the quality of view synthesis based on Curriculum-style Structure Generation without conducting the DIBR process. Specifically, considering that the distortion of view synthesis is mainly manifested in the destruction of image structure, a structure generation network is first built to learn the structure of the new view from the original one by curriculum-style training. Then, we transfer the prior knowledge learned from the last phase into the quality prediction network for measuring the structure distortion, based on which a regressor is introduced to produce the quality score. Experimental results prove the advantages of the proposed model.

Stereoscopic view synthesis with progressive structure reconstruction and scene constraints.

Depth image-based rendering (DIBR) is an important technology in the process of 2D-to-3D conversion. It uses texture images and related depth maps to render virtual views. While there are still some challenging problems in the current DIBR systems, such as disocclusion occurrences. Inpainting methods based on deep learning have recently shown significant improvements and generated plausible images. However, most of these methods may not deal well with the disocclusion holes in the synthesized views, because on the one hand they only treat this issue as generative inpainting after 3D warping, rather than following the full DIBR processing procedures. While on the other hand the distributions of holes on the virtual views are always around the transition regions of foreground and background, which makes them more difficult to distinguish without special constraints. Motivated by these observations, this paper proposes a novel learning-based method for stereoscopic view synthesis, in which the disocclusion regions are restored by a progressive structure reconstruction strategy instead of direct texture inpainting. Additionally, some special cues in the synthesized scenes are further exploited as constraints for the network to alleviate hallucinated structure mixtures among different layers. Extensive empirical evaluations and comparisons validate the strengths of the proposed approach and demonstrate that the model is more suitable for stereoscopic synthesis in the 2D-to-3D conversion applications.

Deep Learning-Based Synthesized View Quality Enhancement with DIBR Distortion Mask Prediction Using Synthetic Images.

Recently, deep learning-based image quality enhancement models have been proposed to improve the perceptual quality of distorted synthesized views impaired by compression and the Depth Image-Based Rendering (DIBR) process in a multi-view video system. However, due to the lack of Multi-view Video plus Depth (MVD) data, the training data for quality enhancement models is small, which limits the performance and progress of these models. Augmenting the training data to enhance the synthesized view quality enhancement (SVQE) models is a feasible solution. In this paper, a deep learning-based SVQE model using more synthetic synthesized view images (SVIs) is suggested. To simulate the irregular geometric displacement of DIBR distortion, a random irregular polygon-based SVI synthesis method is proposed based on existing massive RGB/RGBD data, and a synthetic synthesized view database is constructed, which includes synthetic SVIs and the DIBR distortion mask. Moreover, to further guide the SVQE models to focus more precisely on DIBR distortion, a DIBR distortion mask prediction network which could predict the position and variance of DIBR distortion is embedded into the SVQE models. The experimental results on public MVD sequences demonstrate that the PSNR performance of the existing SVQE models, e.g., DnCNN, NAFNet, and TSAN, pre-trained on NYU-based synthetic SVIs could be greatly promoted by 0.51-, 0.36-, and 0.26 dB on average, respectively, while the MPPSNRr performance could also be elevated by 0.86, 0.25, and 0.24 on average, respectively. In addition, by introducing the DIBR distortion mask prediction network, the SVI quality obtained by the DnCNN and NAFNet pre-trained on NYU-based synthetic SVIs could be further enhanced by 0.02- and 0.03 dB on average in terms of the PSNR and 0.004 and 0.121 on average in terms of the MPPSNRr.

Blind Quality Prediction for View Synthesis Based on Heterogeneous Distortion Perception.

The quality of synthesized images directly affects the practical application of virtual view synthesis technology, which typically uses a depth-image-based rendering (DIBR) algorithm to generate a new viewpoint based on texture and depth images. Current view synthesis quality metrics commonly evaluate the quality of DIBR-synthesized images, where the DIBR process is computationally expensive and time-consuming. In addition, the existing view synthesis quality metrics cannot achieve robustness due to the shallow hand-crafted features. To avoid the complicated DIBR process and learn more efficient features, this paper presents a blind quality prediction model for view synthesis based on HEterogeneous DIstortion Perception, dubbed HEDIP, which predicts the image quality of view synthesis from texture and depth images. Specifically, the texture and depth images are first fused based on discrete cosine transform to simulate the distortion of view synthesis images, and then the spatial and gradient domain features are extracted in a Two-Channel Convolutional Neural Network (TCCNN). Finally, a fully connected layer maps the extracted features to a quality score. Notably, the ground-truth score of the source image cannot effectively represent the labels of each image patch during training due to the presence of local distortions in view synthesis image. So, we design a Heterogeneous Distortion Perception (HDP) module to provide effective training labels for each image patch. Experiments show that with the help of the HDP module, the proposed model can effectively predict the quality of view synthesis. Experimental results demonstrate the effectiveness of the proposed model.

DIBR Zero-Watermarking Based on Invariant Feature and Geometric Rectification

Despite the success of watermarking technique for protecting depth image-based rendering (DIBR) 3-D videos, existing methods still can hardly ensure the robustness against geometric attacks, lossless video quality, and distinguishability between different videos simultaneously. In this article, we propose a novel zero-watermarking scheme to address this challenge. Specifically, we design CT-SVD features to ensure both distinguishability and robustness against signal processing and DIBR conversion attacks. In addition, a logistic–logistic chaotic system is utilized to encrypt features for the enhanced security. Moreover, a rectification mechanism based on salient map detection and SIFT matching is designed to resist geometric attacks. Finally, we establish an attention-based fusion mechanism to explore the complementary robustness of rectified and unrectified features. Experimental results demonstrate that our proposed method outperforms the existing schemes in terms of losslessness, distinguishability, and robustness against geometric attacks.

LGGD+: Image Retargeting Quality Assessment by Measuring Local and Global Geometric Distortions

Numerous image retargeting algorithms have been proposed to achieve adaptive image resizing during the past years. To compare different image retargeting algorithms, reliable objective image retargeting quality assessment (IRQA) metrics are highly desired. Given that image retargeting usually introduces geometric distortions, this paper presents an objective IRQA metric by measuring both local and global geometric distortions (LGGD). Since human visual system perception is highly dependent on edges and the geometric distortions caused by image retargeting usually cause edge deformation, a sketch token-based local edge descriptor (ST-LED) is introduced to represent geometric-aware features in LGGD. First, ST-LED is first applied on both source and retargeted images for edge pattern representation. Second, pixel-level backward registration is conducted to enable estimating local geometric distortion (LGD) and a spatial pyramid-improved Bag-of-Token (BoT) model is built to enable estimating global geometric distortion (GGD). Since the proposed LGGD metric only focuses on geometric distortion while image retargeting quality is related with more aspects, we further fuse LGGD and an existing (EXT) IRQA metric to build a final version called LGGD+ for IRQA. Experiments on two benchmark databases demonstrate the superiority of LGGD+ and the excellent compatibility of our proposed LGGD for further improving a wide range of existing IRQA metrics (including both geometric distortion and non-geometric distortion metrics). In addition, the effectiveness of our LGGD metric is also demonstrated in another relevant task, i.e., quality evaluation of depth-image-based rendering (DIBR)-synthesized images, which also calls for accurate estimation of geometric distortion.

Learning-Based Stereoscopic View Synthesis with Cascaded Deep Neural Networks

Depth image-based rendering (DIBR) is an important technique in the 2D to 3D conversion process, which renders virtual views with a texture image and the associated depth map. However, certain problems, such as disocclusion, still exist in current DIBR systems. In this study, a new learning-based framework that models conventional DIBR synthesis pipelines is proposed to solve these problems. The proposed model adopts a coarse-to-fine approach to realize virtual view prediction and disocclusion region refinement sequentially in a unified deep learning framework that includes two cascaded joint filter block-based convolutional neural networks (CNNs) and one residual learning-based generative adversarial network (GAN). An edge-guided global looping optimization strategy is adopted to progressively reconstruct the scene structures on the novel view, and a novel directional discounted reconstruction loss is proposed for better training. In this way, our framework performs well in terms of virtual view quality and is more suitable for 2D to 3D conversion applications. The experimental results demonstrate that the proposed method can generate visually satisfactory results.

Multiple-feature-based zero-watermarking for robust and discriminative copyright protection of DIBR 3D videos

Zero-watermarking is a key technique for achieving lossless and flexible copyright protection of depth image-based rendering (DIBR) videos. Existing approaches extract features of both 2D frames and depth maps via a single mechanism to protect them simultaneously. However, it is difficult for these schemes to fully satisfy the copyright protection requirements of the two components, including the remarkable discriminative capability of 3D videos and robustness against various attacks. Hence, in this paper, we propose a novel multiple-feature-based zero-watermarking scheme to protect the copyright of DIBR 3D videos. To the best of our knowledge, this is the first scheme that integrates multiple features to improve both the discriminative capability and robustness against various attacks. Specifically, dual-tree complex wavelet transform and discrete cosine transform features enhance the robustness against DIBR conversion and noise addition, respectively, while ring-partition statistical residual features ensure robustness against geometric attacks and provide sufficient discriminative capacity. In addition, we use a logistic-logistic chaotic system to encrypt these multiple features for enhanced security and design an attention-based fusion approach to offer an optimal copyright protection solution. Extensive experimental results demonstrate that our proposed scheme has stronger robustness and discriminative capacity compared to state-of-the-art zero-watermarking methods.

Quality Assessment of View Synthesis Based on Visual Saliency and Texture Naturalness

Depth-Image-Based-Rendering (DIBR) is one of the core techniques for generating new views in 3D video applications. However, the distortion characteristics of the DIBR synthetic view are different from the 2D image. It is necessary to study the unique distortion characteristics of DIBR views and design effective and efficient algorithms to evaluate the DIBR-synthesized image and guide DIBR algorithms. In this work, the visual saliency and texture natrualness features are extracted to evaluate the quality of the DIBR views. After extracting the feature, we adopt machine learning method for mapping the extracted feature to the quality score of the DIBR views. Experiments constructed on two synthetic view databases IETR and IRCCyN/IVC, and the results show that our proposed algorithm performs better than the compared synthetic view quality evaluation methods.

A comprehensive survey on robust image watermarking

With the rapid development and popularity of the Internet, multimedia security has become a general essential concern. Especially, as manipulation of digital images gets much easier, the challenges it brings to authentication certification are increasing. As part of the solution, digital watermarking has made significant contributions to image content security and has attracted increasing attention. In this paper, we present a comprehensive review on digital image watermarking methods that were published in recent years illustrating the conventional schemes in different domains. We provide an overview of geometric invariant techniques and emerging watermarking methods for novel medias, such as depth image based rendering (DIBR), high dynamic range (HDR), screen content images (SCIs), and point cloud model. Particularly, as deep learning has achieved a great success in the field of image processing, and has also successfully been used in the field of digital watermarking, learning-based watermarking methods using various neural networks are summarized according to the utilization of neural networks in the single stage training (SST) and double stage training (DST). Finally, we provide an analysis and summary on those methods, and suggest some future research directions.

Novel view synthesis in embedded virtual reality devices

Virtual Reality and Free Viewpoint navigation require high-quality rendered images to be realistic. Current hardware assisted raytracing methods cannot reach the expected quality in real-time and are also limited by the 3D mesh quality. An alternative is Depth Image Based Rendering (DIBR) where the input only consists of images and their associated depth maps for synthesizing virtual views to the Head Mounted Display (HMD). The MPEG Immersive Video (MIV) standard uses such DIBR algorithm called the Reference View Synthesizer (RVS). We have first implemented a GPU version, called the Realtime accelerated View Synthesizer (RaViS), that synthesizes two virtual views in real-time for the HMD. In the present paper, we explore the differences between desktop and embedded GPU platforms, porting RaViS to an embedded HMD without the need for a separate, discrete desktop GPU. The proposed solution gives a first insight into DIBR View Synthesis techniques in embedded HMDs using OpenGL and Vulkan, a cross-platform 3D rendering library with support for embedded devices.