Multi-view Coding Research Articles

MPCNet: Compressed multi-view video restoration via motion-parallax complementation network

The performance in restoring compressed multi-view video (MVV) of the existing learning-based methods is limited because they only utilize information of temporally adjacent frames or parallax neighboring views. However, the compression artifacts caused by multi-view coding (MVC) may be related to the reference errors of intra-frame, inter-frame, and inter-view. In this paper, with delicately utilizing the stereo information from both temporal and parallax domains, a motion-parallax complementation network (MPCNet) is proposed to restore the quality of compressed MVV more efficiently. First, we introduce a motion-parallax complementation strategy consisting of a coarse stage and a fine stage. By mutually compensating the feature extracted from multiple domains, useful multi-frame information can be efficiently preserved and aggregated step by step. Second, an attention-based feature filtering and modulation module (AFFM) is proposed, which provides an efficient fusion method for two features by suppressing misleading information. By deploying it in most submodules of the proposed approach, the representational ability of MPCNet can be improved, resulting in a more substantial restoration performance. Experimental results prove the effectiveness of MPCNet by an average increase of 1.978 dB in PSNR, and 0.0282 in MS-SSIM. The BD-rate reduction can reach 47.342% on average. The subjective quality is greatly improved and lots of compression distortions are eliminated. Meanwhile, this work also benefits the accuracy improvement for high-level vision tasks, e.g., mIoU of semantic segmentation and mAP of object detection achieve 0.352 and 51.71, respectively. Quantitative and qualitative analyses demonstrate that MPCNet outperforms state-of-the-art approaches.

Information theory-guided heuristic progressive multi-view coding

Multi-view representation learning aims to capture comprehensive information from multiple views of a shared context. Recent works intuitively apply contrastive learning to different views in a pairwise manner, which is still scalable: view-specific noise is not filtered in learning view-shared representations; the fake negative pairs, where the negative terms are actually within the same class as the positive, and the real negative pairs are coequally treated; evenly measuring the similarities between terms might interfere with optimization. Importantly, few works study the theoretical framework of generalized self-supervised multi-view learning, especially for more than two views. To this end, we rethink the existing multi-view learning paradigm from the perspective of information theory and then propose a novel information theoretical framework for generalized multi-view learning. Guided by it, we build a multi-view coding method with a three-tier progressive architecture, namely Information theory-guided heuristic Progressive Multi-view Coding (IPMC). In the distribution-tier, IPMC aligns the distribution between views to reduce view-specific noise. In the set-tier, IPMC constructs self-adjusted contrasting pools, which are adaptively modified by a view filter. Lastly, in the instance-tier, we adopt a designed unified loss to learn representations and reduce the gradient interference. Theoretically and empirically, we demonstrate the superiority of IPMC over state-of-the-art methods.

CSI: Contrastive data Stratification for Interaction prediction and its application to compound-protein interaction prediction.

Accurately predicting the likelihood of interaction between two objects (compound-protein sequence, user-item, author-paper, etc.) is a fundamental problem in Computer Science. Current deep-learning models rely on learning accurate representations of the interacting objects. Importantly, relationships between the interacting objects, or features of the interaction, offer an opportunity to partition the data to create multi-views of the interacting objects. The resulting congruent and non-congruent views can then be exploited via contrastive learning techniques to learn enhanced representations of the objects. We present a novel method, Contrastive Stratification for Interaction Prediction (CSI), to stratify (partition) a dataset in a manner that can be exploited via Contrastive Multiview Coding to learn embeddings that maximize the mutual information across congruent data views. CSI assigns a key and multiple views to each data point, where data partitions under a particular key form congruent views of the data. We showcase the effectiveness of CSI by applying it to the compound-protein sequence interaction prediction problem, a pressing problem whose solution promises to expedite drug delivery (drug-protein interaction prediction), metabolic engineering, and synthetic biology (compound-enzyme interaction prediction) applications. Comparing CSI with a baseline model that does not utilize data stratification and contrastive learning, and show gains in average precision ranging from 13.7% to 39% using compounds and sequences as keys across multiple drug-target and enzymatic datasets, and gains ranging from 16.9% to 63% using reaction features as keys across enzymatic datasets. Code and dataset available at https://github.com/HassounLab/CSI.

Deep In-Loop Filtering via Multi-Domain Correlation Learning and Partition Constraint for Multiview Video Coding

The deep learning-based in-loop filtering methods have greatly improved the coding efficiency for High Efficiency Video Coding (HEVC). However, directly applying these HEVC-orientated in-loop filtering methods to multiview video coding may not obtain satisfactory performance due to the characteristics of multiview video. In this paper, a deep in-loop filtering method based on multi-domain correlation learning and partition constraint network (MDP-Net) is proposed to boost the multiview video coding performance. To the best of our knowledge, this work is the first attempt at deep in-loop filtering for multiview video coding. Specifically, a multi-domain correlation learning module is presented to restore the high-frequency details of the distorted frame by exploring the multi-domain correlations. Besides, based on the block partition information generated in video coding, a partition-constrained reconstruction module is proposed to better attenuate the compression artifacts by designing a partition loss. Finally, the proposed MDP-Net is integrated into 3D-HEVC reference software, and the experimental results demonstrate that the proposed method achieves considerable performance improvement compared with 3D-HEVC.

Image Segmentation Methods for Subpicture Partitioning in the VVC Video Encoder

The emergence of the new generation video coding standard, Versatile Video Coding (VVC), has brought along novel features rendering the new standard more efficient and flexible than its predecessors. Aside from efficient compression of 8 k or higher camera-captured content, VVC also supports a wide range of applications, including computer-generated content, high dynamic range (HDR) content, multilayer and multi-view coding, video region extraction, as well as 360° video. One of the newly introduced coding tools in VVC, offering extraction and independent coding of rectangular sub-areas within a frame, is called Subpicture. In this work, we turn our attention to frame partitioning using Subpictures in VVC, and more particularly, a content-aware partitioning is considered. To achieve that, we make use of image segmentation algorithms and properly modify them to operate on a per Coding Tree Unit (CTU) basis in order to render them compliant with the standard’s restrictions. Additionally, since subpicture boundaries need to comply with slice boundaries, we propose two methods for properly partitioning a frame using tiles/slices aiming to avoid over-partitioning of a frame. The proposed algorithms are evaluated regarding both compression efficiency and image segmentation effectiveness. Our evaluation results indicate that the proposed partitioning schemes have a negligible impact on compression efficiency and video quality

RHMC: Modeling consistent information from deep multiple views via Regularized and Hybrid Multiview Coding

Modeling multiple views by unsupervised representation learning is a challenging problem, since multi-view data features are different from each other, and it is unclear how to perform representation learning to fully explore discriminative information from multiple views. Recent works focus on learning consistent information by contrasting high-dimensional features, but amounts of useful view-consistent information of low-dimensional representation is ignored, and such information can benefit the learned representation on downstream tasks. In this paper, we propose a novel method called Regularized and Hybrid Multiview Coding (RHMC) for comprehensively modeling consistent information between multiple views. We induce the discriminative improvement of features by constraining the learned multi-view representations in the latent space with diversified self-supervised learning tasks. Specifically, RHMC proposes a hybrid MI estimation, which jointly considers the mutual information between the aggregated high-dimensional feature and low-dimensional representation of views. RHMC enables the learned representation to model the consistent information that is hidden in low-dimensional representations under the direction of the aggregated high-dimensional feature. In hidden space, the problem of multi-view domain shift degenerates the performance of mutual information estimation. To tackle the issue, RHMC imposes globally aligned structures of the learned representations in the latent space, that is aligning the probability distribution of different views by utilizing the Wasserstein distance-based view alignment regularization. Empirically, our experimental validation supports the ability of RHMC to outperform the state-of-the-art self-supervised methods by a significant margin, which proves that RHMC can indeed model consistent and discriminative information from multi-view data.

Disparity-Aware Reference Frame Generation Network for Multiview Video Coding.

Multiview video coding (MVC) aims to compress the multiview video through the elimination of video redundancies, where the quality of the reference frame directly affects the compression efficiency. In this paper, we propose a deep virtual reference frame generation method based on a disparity-aware reference frame generation network (DAG-Net) to transform the disparity relationship between different viewpoints and generate a more reliable reference frame. The proposed DAG-Net consists of a multi-level receptive field module, a disparity-aware alignment module, and a fusion reconstruction module. First, a multi-level receptive field module is designed to enlarge the receptive field, and extract the multi-scale deep features of the temporal and inter-view reference frames. Then, a disparity-aware alignment module is proposed to learn the disparity relationship, and perform disparity shift on the inter-view reference frame to align it with the temporal reference frame. Finally, a fusion reconstruction module is utilized to fuse the complementary information and generate a more reliable virtual reference frame. Experiments demonstrate that the proposed reference frame generation method achieves superior performance for multiview video coding.

Contrastive Multiview Coding With Electro-Optics for SAR Semantic Segmentation

In the training of deep learning models, how the model parameters are initialized greatly affects the model performance, sample efficiency, and convergence speed. Representation learning for model initialization has recently been actively studied in the remote sensing field. In particular, the appearance characteristics of the imagery obtained using the a synthetic aperture radar (SAR) sensor are quite different from those of general electro-optical (EO) images, and thus representation learning is even more important in remote sensing domain. Motivated from contrastive multiview coding, we propose multi-modal representation learning for SAR semantic segmentation. Unlike previous studies, our method jointly uses EO imagery, SAR imagery, and a label mask. Several experiments show that our approach is superior to the existing methods in model performance, sample efficiency, and convergence speed.

Developments in International Video Coding Standardization After AVC, With an Overview of Versatile Video Coding (VVC)

In the last 17 years, since the finalization of the first version of the now-dominant H.264/Moving Picture Experts Group-4 (MPEG-4) Advanced Video Coding (AVC) standard in 2003, two major new generations of video coding standards have been developed. These include the standards known as High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC). HEVC was finalized in 2013, repeating the ten-year cycle time set by its predecessor and providing about 50% bit-rate reduction over AVC. The cycle was shortened by three years for the VVC project, which was finalized in July 2020, yet again achieving about a 50% bit-rate reduction over its predecessor (HEVC). This article summarizes these developments in video coding standardization after AVC. It especially focuses on providing an overview of the first version of VVC, including comparisons against HEVC. Besides further advances in hybrid video compression, as in previous development cycles, the broad versatility of the application domain that is highlighted in the title of VVC is explained. Included in VVC is the support for a wide range of applications beyond the typical standard- and high-definition camera-captured content codings, including features to support computer-generated/screen content, high dynamic range content, multilayer and multiview coding, and support for immersive media such as 360° video.

An efficient rate–distortion optimization method for dependent view in MV-HEVC based on inter-view dependency

Rate–distortion optimization (RDO) is utilized to select the optimal coding parameters in multi-view video coding (MVC), which employs a Lagrange multiplier to balance the relationship between the distortion and the bitrate. In this paper, an efficient RDO method for the dependent view (DV) in multi-view video (MVV) is proposed based on inter-view dependency. First of all, by investigating the sources of the distortion in the DV, a new distortion model for the DV is established. In addition, based on the proposed distortion model, an efficient Lagrangian multiplier decision for B frame is proposed by considering the inter-view dependency. Finally, the optimized Lagrangian multiplier for P frame is designed using the scaling factor which is deduced to have a linear relationship with the disparity between I frame and P frame. Experiment results demonstrate that compared with the original HTM-16.0 encoder, the proposed overall method reduces 12.19% BD-rate for the DV on average, bringing 0.40 dB BD-PSNR gain.

Compressed Sensing Based Low-Power Multi-View Video Coding and Transmission in Wireless Multi-Path Multi-Hop Networks

Wireless Multimedia Sensor Network (WMSN) is increasingly being deployed for surveillance, monitoring and Internet-of-Things (IoT) sensing applications where a set of cameras capture and compress local images and then transmit the data to a remote controller. Such captured local images may also be compressed in a multi-view fashion to reduce the redundancy among overlapping views. In this paper, we present a novel paradigm for compressed-sensing-enabled multi-view coding and streaming in WMSN. We first propose a new encoding and decoding architecture for multi-view video systems based on Compressed Sensing (CS) principles, composed of cooperative sparsity-aware block-level rate-adaptive encoders, feedback channels and independent decoders. The proposed architecture leverages the properties of CS to overcome many limitations of traditional encoding techniques, specifically massive storage requirements and high computational complexity. Then, we present a modeling framework that exploits the aforementioned coding architecture. The proposed mathematical problem minimizes the power consumption by jointly determining the encoding rate and multi-path rate allocation subject to distortion and energy constraints. Extensive performance evaluation results show that the proposed framework is able to transmit multi-view streams with guaranteed video quality at lower power consumption.

A novel fast search method to find disparity vectors in multiview video coding

In multiview/3D video, the amount of data to be transmitted to the decoder increases proportionally with the number of cameras. One way to efficiently compress such video is to use Multiview Video Coding (MVC) which simultaneously reduces temporal and spatial redundancy within the same view and among multiple views. But, existing disparity estimation methods used to extract the redundancy among views are typically too computationally complex. In this paper, we propose a method for fast disparity estimation in multiview/3D video. First, for each block of the predicted frame, the view-dependent geometry and the depth information are used to find the corresponding block in the reference views. Then, a fast and greedy search method is proposed to search the surrounding areas to find the most similar block. Simulation results show that our proposed method achieves better bitrate and quality with much lower computational complexity compared to state-of-the-art methods.

Flexible Complexity Optimization in Multiview Video Coding

Flexible optimization on video coding computational complexity is imperative to adapt to diverse terminals in video streaming, especially with high definition videos and increasingly comprehensive encoders. Until now, few efforts have been contributed to the encoder of multiview videos, which complicates its encoding structure with duplex frame display. To address this issue, we propose a flexible complexity optimization framework in this paper. In the first place, the proposed algorithm flexibly reduces the encoder complexity at different external-defined complexity constraints; furthermore, it is also committed to compression efficiency optimization at different complexity levels. The framework is achieved by a hybrid approach: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">complexity allocation</i> , which allocates the external-defined complexity constraint to all views, hierarchical layers and frames based on linear programming; and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">complexity regulation</i> , which dynamically adjusts local candidate partitions to fulfil the targeted local complexity constraint, with a probability-driven Alternate Partition Cost (APC) minimization. The overall algorithm is implemented on the popular multiview encoder, Multiview High Efficiency Video Coding (MV-HEVC), with promising Rate-Distortion (RD) performances at different complexity constraints, which are also superior to two recent complexity optimization algorithms.

A Bit Allocation Method Based on Inter-View Dependency and Spatio-Temporal Correlation for Multi-View Texture Video Coding

Due to the limited storage capacity and network bandwidth, an efficient rate control (RC) algorithm becomes more and more critical for multi-view video coding (MVC). Based on inter-view dependency and spatio-temporal correlation, a novel bit allocation method for multi-view texture video coding is proposed in this article. Firstly, considering that the distortion in the base view (BV) is directly transmitted to the dependent view (DV) by inter-view skip mode, a joint multi-view RD model is built based on the inter-view dependency. Based on the proposed joint multi-view RD model, a precise power model is derived to represent the target bitrates relationship between the BV and the DV. Secondly, since the P frame in the DV (P-DV) is mainly predicted from the corresponding I frame in the BV (I-BV) by disparity compensated prediction (DCP), the constant proportional relationship between the ratio of the average bitrates of the P-DV to the corresponding I-BV and the ratio of the total bitrates of the DV to the BV is discovered. Based on this discovery, a novel linear model is developed to assign the target bitrates of the P-DV. Finally, considering the spatio-temporal correlation, a new parameter prediction method is proposed for the R-$\lambda $ model in coding tree unit (CTU) level. Extensive experimental results show that the proposed overall method outperforms other state-of-the-art algorithms in terms of RD performance.

1D representation of Laplacian eigenmaps and dual k ‐nearest neighbours for unified video coding

This study proposes a framework of video coding based on Laplacian eigenmaps (LEM) and its related embedding and reconstruction algorithm (ERA). Firstly, a one-dimensional (1D) representation of LEM is adopted to achieve an extremely low bit per pixel (BPP). Secondly, dual k-nearest neighbours, which keeps neighbour relationships both in high-dimensional data space and low-dimensional representation space and overcomes the disadvantage of classical non-linear dimensionality reduction methods which cannot preserve the neighbour properties in both of the spaces, based ERA of LEM is employed to gain extraordinarily high peak-signal-to-noise ratio (PSNR). Thirdly, a unified framework of video coding is fit for intra-frame, inter-frame and multi-view video coding. Finally, it is evaluated by simulation experiments that, in the situation of low bitrate transmission, the proposed method can attain better performance of BPP and PSNR than that of the state-of-the-art methods, such as highly efficient video coding.

Low Power FPGA Implementation of Multi-View Video Coding with Hybrid Compression and Decompression Algorithm

The video is one of the most useful and most appealing medium to represent some information. More usage of digital multi-media via communications media, wireless communications, intranet, internet and cellular mobile leads to the uncontrollable growth of data in media. The video compression technique is used in this research work to improve the processing speed of the entire system. In this work, Low Cost - Multi Video Coding - Hybrid Compression and Decompression (LC-MVC-HCD) method is used to reduce computation complexity. The combinational of Discrete Wavelet Transform (DWT) and Discrete Cosine Transform (DCT) algorithms are denoted as hybrid algorithm. Based on this hybrid algorithm, the compression process is performing which improves the video coding efficiency of MVC. The LC-MVC-HCD methodwas implemented in the Matlab, Xilinx and Cadence tool. In Application Specific Integrated Circuit (ASIC) implementation, the area, power, and delay minimized by using the cadence encounter tool with 180nm and 45nm technology. In Field Programmable Gate Array (FPGA) implementation, the number of Lookup Tables (LUTs), slice, and flip-flop are minimized based on two different kinds of Virtex devices such as Virtex -6 and Virtex-7. In Matlab, Peak Signal to Noise Ratio (PSNR), computational time and bit error rate were analyzed for the LC-MVC-HCD method. The experimental outcome showed that the proposed methodology has improved performance ASIC and FPGA up to 2-3% compared to existing methods like Direct mode decision MVC and LC-MVC-DWT.

Random access prediction structures for light field video coding with MV-HEVC

Computational imaging and light field technology promise to deliver the required six-degrees-of-freedom for natural scenes in virtual reality. Already existing extensions of standardized video coding formats, such as multi-view coding and multi-view plus depth, are the most conventional light field video coding solutions at the moment. The latest multi-view coding format, which is a direct extension of the high efficiency video coding (HEVC) standard, is called multi-view HEVC (or MV-HEVC). MV-HEVC treats each light field view as a separate video sequence, and uses syntax elements similar to standard HEVC for exploiting redundancies between neighboring views. To achieve this, inter-view and temporal prediction schemes are deployed with the aim to find the most optimal trade-off between coding performance and reconstruction quality. The number of possible prediction structures is unlimited and many of them are proposed in the literature. Although some of them are efficient in terms of compression ratio, they complicate random access due to the dependencies on previously decoded pixels or frames. Random access is an important feature in video delivery, and a crucial requirement in multi-view video coding. In this work, we propose and compare different prediction structures for coding light field video using MV-HEVC with a focus on both compression efficiency and random accessibility. Experiments on three different short-baseline light field video sequences show the trade-off between bit-rate and distortion, as well as the average number of decoded views/frames, necessary for displaying any random frame at any time instance. The findings of this work indicate the most appropriate prediction structure depending on the available bandwidth and the required degree of random access.

An Improved High Capacity and Efficient Data Hiding Scheme for 3D Videos Without Distortion Drift

Compared to traditional images and 2D videos, 3D videos are more likely to cause distortion drift problems in data hiding. In this paper, an improved scheme for high capacity and efficient data hiding in 3D videos based on Multi-view coding (MVC) standard is proposed, which avoids the problems of distortion drift. To improve the visual quality of data hiding, two selection modes are provided to limit the distortion drift. By modifying the selected coefficients of $4\times 4$ quantized discrete cosine transforms (QDCT) in macroblocks and hiding data into b4-frames, the proposed scheme will effectively prevent distortion drift which caused by intra prediction and inter prediction. Moreover, the proposed algorithm maintains great randomness by using two random sequences. Several benchmark 3D video sequences of different resolutions and diverse contents are used for experimental evaluation. It is experimentally proved that the algorithm has greater embedding capacity and higher efficient than the previous methods.

Efficient Content Adaptive Plenoptic Video Coding

In this paper, a content adaptive coding method for plenoptic video is proposed to reduce both of the spatial and temporal redundancy. Based on the spatial correlations among subapertures, the plenoptic video is divided into two subaperture groups: the central view videos and the residual videos. Coding methods, multiview coding method based on spatial correlation(S-MVC) and multiview coding method based on temporal correlation(T-MVC), are adopted for residual videos according to correlation and residual energy analysis, while the central view videos are always encoded by S-MVC. Correlation analysis is performed first, and residual energy analysis is further performed if necessary. Coding the central view videos and the residual videos separately reduces bitstream, choosing different encoding methods for residual videos makes full use of the characteristics of different video content. Both the central view videos and the residual videos are compressed by MV-HEVC. The experimental results demonstrate that the proposed method outperforms a pseudo-sequence-based MVC method for plenoptic video by an average of 25.06% bitrate saving.

Texture-Distortion-Constrained Joint Source-Channel Coding of Multi-View Video Plus Depth-Based 3D Video

A novel joint source and channel coding scheme tailored to 3D video is proposed in this paper to minimize the end-to-end view synthesis distortion within a given total bit rate for both texture and depth as well as a maximum tolerable distortion constraint for texture. First, we formulate a joint texture and depth coding mode selection strategy for error-resilient source coding of multi-view video plus depth-based 3D video through using the Lagrange multiplier method. Then, by considering the effect of residual errors after channel coding, we evolve to a more general formulation that jointly optimizes error-resilient source coding and channel coding in an integrated manner for unequal error protection between texture and depth, for which a theoretic solution using a proposed dual-trellis is derived. Finally, we extend the general formulation by including the texture distortion constraint. We show how to optimize the view synthesis quality while simultaneously catering to the texture quality constraint. Experimental results demonstrate the proposed algorithm has much better performance than existing related work.