3D Video Communication Research Articles

Watermarking and Encryption for Holographic Communication

Holographic communication is a three-dimensional (3D) video communication technology based on computer-generated holograms (CGHs) which has the potential to give users a more realistic visual perception. As this is an emerging field, the encrypted encoding and decoding methods in holographic communication have not been widely studied. In this work, a watermarking and encryption method for holographic communication is proposed. A watermark is inserted into the original image using the discrete cosine transform before the calculation of the CGH, while a secret key is employed to produce the encrypted CGH during the holographic calculation. Through the proposed watermarking and encryption method, the signal of holographic communication is difficult to decrypt. Even if the signal is decrypted, the source of the leak is easy to trace due to the existence of the watermark. The watermarking and encryption method can provide a practical solution for the privacy protection and copyright protection of 3D video communication.

Classification of Real 3D and Fake 3D Video

The tremendous growth in communication and media technology and the wide availability of cheaper end using devices have made 3D video communication very popular due to its immersive experience. It has been observed that a 3D video can be produced either by direct accusation using a 3D camera (say real 3D video) or by rendering from a set of 2D images (say fake 3D video). There are several occasions where it is required to distinguish between such real and fake 3D video sequences. In this paper, an algorithm is proposed which can distinguish the real 3D video from the fake one. A set of distinguishing features has been identified which are primarily based on the vertical parallax and sharpness peculiarities of object edges due to 3D acquisition process and rendering. Finally, two different supervised learning classifiers (Support Vector Machines and Linear discriminant analysis), are being trained using these features to detect the fake 3D video sequences. A comprehensive set of experiments has been carried out to justify the applicability of the proposed detection scheme over the recent existing scheme.

Holo Reality: Real-time low-bandwidth 3D range video communications on consumer mobile devices with application to augmented reality

An increasing number of mobile devices are equipped to acquire 3D range data along with color texture (e.g., iPhone X). As these devices are adopted, more people will have direct access to 3D imaging devices, bringing advanced applications, such as mobile 3D video calls and remote 3D telemedicine, within reach. This paper introduces Holo Reality, a novel platform that enables real-time, wireless 3D video communications to and from todayâ–™s mobile (e.g., iPhone, iPad) devices. The major contributions are (1) a modular platform for performing 3D video acquisition, encoding, compression, transmission, decompression, and visualization entirely on consumer mobile devices and (2) a demonstration system that successfully delivered 3D video content from one mobile device to another, in real-time, over standard wireless networks. Our demonstration system uses augmented reality to visualize received 3D video content within the userâ–™s natural environment, highlighting the platformâ–™s potential to enable advanced applications for telepresence and telecollaboration. This technology also has the potential to realize new applications within areas such as mechatronics and telemedicine.

Improved joint algorithms for reliable wireless transmission of 3D color-plus-depth multi-view video

Error control techniques like Error Resilience (ER) and Error Concealment (EC) are preferred techniques to ameliorate the lost Macro-Blocks (MBs) in the 3D Video (3DV) communication systems. In this paper, we present different enhanced ER-EC algorithms for intra-frame images for 3DV and Depth (3DV + D) communication through wireless networks. At the encoder, the slice structured coding, explicit flexible macro-block ordering, and context adaptive variable length coding are utilized. At the decoder, a hybrid approach comprising spatial circular scan order interpolation algorithm and temporal partitioning motion compensation algorithm is suggested to reconstruct the Disparity Vectors (DVs) and Motion Vectors (MVs) of the erroneous color images. For the corrupted depth images, a depth-assisted EC algorithm is proposed. Then, the optimum concealment MVs and DVs are chosen by employing the weighted overlapping block motion and disparity compensation algorithm. Furthermore, the Bayesian Kalman Filter (BKF) is utilized as an amelioration tool due to its efficiency to smooth the remnant inherent corruptions in the formerly optimally chosen color and depth DVs and MVs to obtain a good video quality. Simulation results on several 3DV streams show that the suggested algorithms have extremely adequate subjective and objective video quality performance compared to the traditional methods, particularly at high Packet Loss Rates (PLRs).

Proposed dynamic error control techniques for QoS improvement of wireless 3D video transmission

SummaryError control techniques like error resilience (ER) and error concealment (EC) are efficient techniques to ameliorate the lost macroblocks (MBs) in the 3D video (3DV) communication system. In this paper, we propose efficient and adaptive hybrid ER‐EC algorithms for 3DV transmission over error‐prone wireless channels. At the encoder, adaptive preprocessing ER mechanisms are proposed through using the context adaptive variable length coding entropy, slice structured coding modes, and explicit flexible macroblock ordering mapping. They are used to assist the suggested EC techniques at the decoder to accurately reconstruct the erroneous MBs and frames. At the decoder, an efficient postprocessing EC technique with multiproposition methods is proposed to dynamically select the convenient EC hypothesis method based on the size of the lost MBs, the faulty view, and the frame type. It conceals the received erroneous MBs of intra‐encoded and inter‐encoded frames of the transmitted 3DV by exploiting the temporal, spatial, and inter‐view correlations among frames and views. To further improve the decoded 3DV quality, a weighted overlapping block motion and disparity compensation technique is used to reinforce the performance of the suggested ER‐EC techniques. Experimental results on various 3DV streams prove that the suggested techniques have considerably acceptable subjective and objective 3DV performance. They achieve an improved average peak signal‐to‐noise ratio gain by almost 2.85 dB compared to the conventional error control algorithms at a packet loss rate = 40%.

Recursive Bayesian Filtering-Based Error Concealment Scheme for 3D Video Communication Over Severely Lossy Wireless Channels

The three-dimensional video (3DV) is composed of variable-length stream sequences captured via diversified cameras surrounding an object. Thus, it is an urgent task to accomplish sufficient encoding to be compatible with incoming bandwidth demands, while achieving a recommended 3DV reception performance. In the 3DV compression framework, the lost macro-blocks (MBs) might propagate into the following frames and the adjoining views. Therefore, it is obligatory to avoid error propagation by concealing the corrupted MBs at the decoder through the utilization of appropriate post-processing error concealment (EC) techniques. The existing EC algorithms fundamentally exploit the temporal, inter-view, and spatial matching within the 3DV frames and views to reconstruct the disparity vectors (DVs) and motion vectors (MVs) of the corrupted MBs. Unluckily, in the state of high severe corruptions and heavily erroneous MBs, these concealment algorithms are predominantly unreliable and might give unreliable 3DV quality. Thence, in this work, we suggest the utilization of the outer block boundary matching algorithm to estimate the MVs and the directional interpolation EC algorithm to estimate the DVs of the erroneous MBs. After that, the Bayesian Kalman filter (BKF) is employed because of its efficiency to filter out the inherent errors in the previously predicted DVs and MVs to accomplish better 3D video performance. Experimental results on standard 3DV sequences demonstrate that the suggested BKF-based EC scheme is more powerful with heavy losses. It subjectively and objectively outperforms the traditional concealment techniques at severely random and bursty packet loss rates (PLRs).

Holostream: High-Accuracy, High-Speed 3D Range Video Encoding and Streaming Across Standard Wireless Networks

Holostream is a novel platform which enables high-quality 3D video communication on mobile devices (e.g., iPhones, iPads) using existing standard wireless networks. The major contributions are: (1) a novel high-quality 3D video compression method that drastically reduces both 3D geometry and color texture data sizes in order to transmit them within the bandwidths provided by existing wireless networks; (2) a novel pipeline for 3D video recording, encoding, compression, decompression, visualization and interaction; and (3) a demonstration system which successfully delivered video-rate, photorealistic 3D video content through a standard wireless network to mobile devices. The novel platform improves the quality and expands upon the capabilities of popular applications already utilizing real-time 3D data delivery, such as teleconferencing and telepresence. This technology could also enable emerging applications which may require highresolution, high-accuracy 3D video data delivery, such as remote robotic surgery and telemedicine.

Performance evaluation of enhanced error correction algorithms for efficient wireless 3D video communication systems

SummaryIn the Three‐Dimensional Multi‐View Video (3D MVV) coding prediction structure, the coded bit‐streams may be dropped down due to error transmissions. So, it is compulsory to recover the corrupted Macro‐Blocks (MBs) at the decoder utilizing efficient Error Concealment (EC) post‐processing techniques. The EC methods exploit the temporal, inter‐view, and spatial matching among 3D MVV views and frames without any alterations in the encoder software or hardware or in the transmission rate. They retrieve the Disparity Vectors (DVs) and Motion Vectors (MVs) of the erroneous MBs. To enhance the received 3D MVV quality, we suggest several optimized schemes to reconstruct the lost and erroneous MBs of inter‐encoded and intra‐encoded frames. A hybrid method combining the Circular Scan Order Interpolation Algorithm (CSOIA) and Partitioning Motion Compensation Algorithm (PMCA) is proposed for intra‐frame EC. For inter‐corrupted MBs, a hybrid method comprising Directional Textural Motion Coherence Algorithm (DTMCA) and Directional Interpolation Error Concealment Algorithm (DIECA) is suggested. Experimental outcomes on several 3D MVV frames demonstrate that the suggested EC algorithms have more robustness to both heavily random and slice losses. They objectively and subjectively work better than the state‐of‐the‐art EC techniques. The proposed work achieves high 3D MVV quality with an improved average Peak Signal‐to‐Noise Ratio (PSNR) gain up to 2.45 to 4.55 dB compared with traditional EC algorithms at Packet Loss Rates (PLRs) of 40%.

3D Video Communication System by Using Kinect and Head Mounted Displays

Existing video communication systems, used in private business or video teleconference, show a part of the body of users on display only. This situation does not have realistic sensation because of showing a part of body on display and showing other users by 2D. This makes users feel communicating with the other users at a long distance without realistic sensation. Furthermore, although these existing communication systems have file transfer function such as sending and receiving file data, it does not use intuitive manipulation. It uses mouse or touching display only. In order to solve these problems, we propose 3D communication system supported by Kinect and head mounted display (HMD) to provide users communications with realistic sensation and intuitive manipulation. This system is able to show whole body of users on HMD as if they were in the same room by 3D reconstruction. It also enables users to transfer and share information by intuitive manipulation to augmented reality (AR) objects toward the other users in the future. The result of this paper is a system that extracts human body by using Kinect, reconstructs extracted human body on HMD, and also recognizes users' hands to be able to manipulate AR objects by a hand.

DEVELOPMENT A FILE TRANSFER APPLICATION BY HANDOVER FOR 3D VIDEO COMMUNICATION SYSTEM IN SYNCHRONIZED AR SPACE

Existing video communication systems, used in business or private life, provides file data transfer function. However, these systems need many manipulation steps by using mouse and keyboard. These steps are not easy to transfer files for the PC beginners. Furthermore, it is not intuitive action from the point of view of handing of things. In order to solve these issues, we have proposed 3D video communication system by using Kinect and Head Mounted Display (HMD). This system provides users communications with realistic sensation and intuitive manipulation. In this system, users can see the other user’s body part through HMD and communicate in AR (Augmented Reality) space. In this paper, we provide a intuitive file data transfer application by handing of AR objects using this system.

Robust Adaptive Intra Refresh for Multiview Video

Transmission error propagation in wireless multimedia communication systems has become a recurring problem. This persistent problem has led to grave consequences on the visual quality of the decoded video. It is against this backdrop that, we present an adaptive intra refresh (AIR) error -resilient coding tool to mitigate the effect of transmission error propagation in 3D video communications. This work utilizes periodic insertion of intra macroblocks in badly error -infected frames temporally as well as related frames in the multi view video scheme. Our objective is to maxim ize the transmission efficiency while ensuring the transmission robustness of the coded bitstream. The selection of periodic macroblocks is based on areas with high motion above a pre-set threshold. The coding modes of the macroblocks are based on the distortion expectation due to transmission errors. Extensive simulation results show significant improvement in both objective and subjective video quality at different intra refresh rates.

Edge-preserving Intra mode for efficient depth map coding based on H.264/AVC

Depth-image-based-rendering (DIBR) algorithms for 3D video communication systems based on the “multi-view video plus depth” format are very sensitive to the accuracy of depth information. Specifically, edge regions in the depth data should be preserved in the coding/decoding process to ensure good view synthesis performance, which directly affects the overall system performance. This paper proposes a novel scheme for edge-aware Intra depth compression based on the H.264/AVC framework enabled on both Intra (I) and Inter (P) slices. The proposed scheme includes a new Intra mode specifically targeted to depth macroblocks with arbitrarily shaped edges, which are typically not predicted well by the standard Intra modes of H.264/AVC and result in high rate–distortion costs. The proposed algorithm segments edge macroblocks into two regions each approximated by a flat surface. A binary mask identifying the two regions is defined and encoded by means of context-coding with adaptive template selection. As a novel contribution, the proposed mode allows exploiting the correlation with causal neighboring edge macroblocks to improve the performance of context-coding of binary masks and allow significant bit rate savings. The proposed method has been exhaustively compared with different state-of-the-art algorithms for edge-aware depth coding and the results highlight significant improvements in most of the cases, both in terms of reconstructed depth quality, view synthesis performance, and overall texture plus depth rate–distortion performance.

Visual Perception Based Objective Stereo Image Quality Assessment for 3D Video Communication

Stereo image quality assessment is a crucial and challenging issue in 3D video communication. One of major difficulties is how to weigh binocular masking effect. In order to establish the assessment mode more in line with the human visual system, Watson model is adopted, which defines visibility threshold under no distortion composed of contrast sensitivity, masking effect and error in this study. As a result, we propose an Objective Stereo Image Quality Assessment method (OSIQA), organically combining a new Left-Right view Image Quality Assessment (LR-IQA) metric and Depth Perception Image Quality Assessment (DP-IQA) metric. The new LR-IQA metric is first given to calculate the changes of perception coefficients in each sub-band utilizing Watson model and human visual system after wavelet decomposition of left and right images in stereo image pair, respectively. Then, a concept of absolute difference map is defined to describe abstract differential value between the left and right view images and the DP-IQA metric is presented to measure structure distortion of the original and distorted abstract difference maps through luminance function, error sensitivity and contrast function. Finally, an OSIQA metric is generated by using multiplicative fitting of the LR-IQA and DP-IQA metrics based on weighting. Experimental results shows that the proposed method are highly correlated with human visual judgments (Mean Opinion Score) and the correlation coefficient and monotony are more than 0.92 under five types of distortions such as Gaussian blur, Gaussian noise, JP2K compression, JPEG compression and H.264 compression.

Robust Adaptive Intra Refresh for Multiview Video

Transmission error propagation in wireless multimedia communication systems has become a recurring problem. This persistent problem has led to grave consequences on the visual quality of the decoded video. It is against this backdrop that, we present an adaptive intra refresh (AIR) error-resilient coding tool to mitigate the effect of transmission error propagation in 3D video communications. This work utilizes periodic insertion of intra macroblocks in badly error-infected frames temporally as well as related frames in the multi view video scheme. Our objective is to maximize the transmission efficiency while ensuring the transmission robustness of the coded bitstream. The selection of periodic macroblocks is based on areas with high motion above a pre-set threshold. The coding modes of the macroblocks are based on the distortion expectation due to transmission errors. Extensive simulation results show significant improvement in both objective and subjective video quality at different intra refresh rates.

Fostering cooperation between overlay applications and underlying networks

Considering the demanding nature of future high definition (HD) and three-dimensional (3D) video communications, application overlays will need increased awareness of underlying network capabilities. We propose a new cross-layer solution to enhance overlay/ISP interactions by providing a comprehensive, media aware, open, and standardized interface between ISPs and application overlays. We further discuss how cooperation between the layers will be improved and describe an example use case to demonstrate the benefit of such interworking.

Real-time 3D video communication in 3D virtual worlds: Technical realization of a new communication concept

After a failed initial attempt in the 1950s to bring three dimensionality into the entertainment sector, 3D technologies today have reached a point where media production, transportation, as well as consumption are of sufficient speed and quality to finally advance the complete media chain from two dimensional (2D) to three dimensional information. Production is based either on stereo- or multi-scopic video capturing techniques or on synthetic environments that can mimic or adopt real-life behavior. The resulting 3D media can be transported by providers over the existing telecommunication infrastructure to be consumed by users using single- or multi-view 3D devices such as TVs, goggles, or projectors. The Visual Communication Department in Bell Labs' Application Research Domain has taken on the research challenge to replace a user's avatar in a 3D virtual environment with a real time representation of the user derived from his streaming 3D video image. With this, the tedious process of avatar creation becomes obsolete as people are able to gesture naturally and convey facial expressions within these synthetic worlds. This paper highlights the technical challenges around building and implementing an architecture for innovative and immersive communication systems. We also discuss the technical limitations of the current approach, and pinpoint major open issues.

3D video communications: Challenges and opportunities

AbstractThis paper surveys major techniques in 3D communications area, which covers the whole pipeline of the 3D video communication framework, including 3D content creation, data representation, compression, delivery, decompression, post‐processing, and 3D scene rendering stages. Both the current state‐of‐the‐art, stereo 3D, and future trend, free‐viewpoint 3D, are demonstrated in details. On the other hand, the paper highlights a few features in the emerging 4G wireless systems that are critical for 3D communications system design. At the end, the topics with potential but challenges, for example 3D over 4G networks, distributed 3D video coding, 3D multi‐user communication, scalability and universal 3D access, are discussed and pointed out to audiences for further investigation. Copyright © 2010 John Wiley & Sons, Ltd.

Video encoder design for high-definition 3D video communication systems

VLSI realization of video compression is the key to real-time high-definition 3D communication systems. The newly established multiview video coding standard, as an extension profile of H.264/AVC, draws more and more attention for its high compression ratio and free-viewpoint support. Besides providing the 3D experience, multiview video can also give users complete scene perception. However, the multiple-viewpoint throughput requirement of MVC increase the complexity and hardware cost dramatically. The system memory bandwidth, on-chip memory size, and processing data throughput of each module all need to be optimized in an MVC encoder. Therefore, efficient hardware solutions for MVC architecture design are needed. In this article an overview of 3D video coding standards developments and design challenges of an MVC encoder are discussed. Then the algorithm and architecture optimization schemes are proposed. For the trade-off between system memory bandwidth and on-chip memory size, a cache-based prediction engine is proposed to ease both design challenges. Moreover, the hybrid open-close loop intra prediction scheme and the frame-parallel pipeline-doubled dual CABAC solve the throughput requirement problem. At the end of this article, based on all the proposed solutions, a prototype single-chip MVC encoder design with processing ability of 4096 × 2160 single-view to 1280 × 720 seven-view is presented.

Computer-Generated Holograms and 3-D Visual Communication

Summary form only given. 3D visual communication is one of the most topical issues in the development of the modern information society. There are no doubts that the ultimate solution for 3-D visualization is holographic imaging. This is the only method that is capable of reproducing, in the most natural viewing conditions, 3-D images that have all visual properties of the original objects including full parallax, and are visually separated from the display device. 3-D visual communication and display can be achieved through generating, at the viewer side, of holograms out of data that contain all relevant information regarding the scene to be viewed. Digital computers are ideal means for converting data on 3-D scenes into optical holograms for visual perception. Basic ideas of using computer generated holograms for 3D visualization date back to 1960-th .1970-th. However, at that time there was no an appropriate technological base for the implementation of these ideas. Recent advances in computer engineering, electro optics and nano-photonics allow making a decisive breakthrough in this respect. The goal of the tutorial is to assist new generation of researches in the adaptation of these ideas to the new emerging technical means and their implementation in 3D displays, 3D television and 3D video communication. It covers all main relevant issues, from basics of holography to principles of synthesis and encoding of computer generated holograms and methods of 3D visualization using computer generated holograms.

A Novel H.264 Based Motion Vector Recovery Method for 3D Video Transmission

The performance of 3D video transmission over error-prone channels is limited by the channel noise. Error concealment is an effective method to combat these channel errors. In H.264 video coding, motion vectors (MVs) are highly significant for the matching of the missing video block. Thus how to recover the lost MV is an important issue for error concealment in 3D video communications. In this paper, we propose a new method to recover the corrupted MVs for 3D video transmission, which is based on the traditional boundary matching algorithm (BMA) for the 2D video. Our proposed depth based BMA (DBMA) method will make full use of the relationships between the 2D video and depth map to recover the lost MV based on BMA. Experimental results show that it is highly effective and significantly outperforms other existing MV recovery methods for 3D video error concealment at different block loss ratios (BLRs)1.