Depth Modeling Modes Research Articles

A Novel Coding Scheme for Large-Scale Point Cloud Sequences Based on Clustering and Registration

Due to the huge volume of point cloud data, storing and transmitting it is currently difficult and expensive in autonomous driving. Learning from the high-efficiency video coding (HEVC) framework, we propose a novel compression scheme for large-scale point cloud sequences, in which several techniques have been developed to remove the spatial and temporal redundancy. The proposed strategy consists mainly of three parts: intracoding, intercoding, and residual data coding. For intracoding, inspired by the depth modeling modes (DMMs), in 3-D HEVC (3-D-HEVC), a cluster-based prediction method is proposed to remove the spatial redundancy. For intercoding, a point cloud registration algorithm is utilized to transform two adjacent point clouds into the same coordinate system. By calculating the residual map of their corresponding depth image, the temporal redundancy can be removed. Finally, the residual data are compressed either by lossless or lossy methods. Our approach can deal with multiple types of point cloud data, from simple to more complex. The lossless method can compress the point cloud data to 3.63% of its original size by intracoding and 2.99% by intercoding without distance distortion. Experiments on the KITTI dataset also demonstrate that our method yields better performance compared with recent well-known methods. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article deals with the problem of efficient compression of point cloud sequences that come from light detection and ranging (LiDARs) mounted on autonomous mobile robots. The vast amount of point cloud data could be an important bottleneck for transmission and storage. Inspired by the HEVC algorithm, we develop a novel coding architecture for the point cloud sequence. The scans are divided into intraframe and interframe, which are encoded separately using different techniques. Our method can be used for the compression of LiDAR point cloud sequences or dense LiDAR point cloud map and will significantly reduce the transmission bandwidth and storage spaces. We have to admit that although our method is less effective for real-time solutions, it can be highly efficient for off-line applications. Future studies will concentrate on further optimizing the coding algorithm to reduce the computational complexity and trying to find a balance between them.

Fast 3D-HEVC Depth Intra Coding Based on Boundary Continuity

The encoding format of the 3D extension of high efficiency video coding (3D-HEVC) consists of a multiview color texture and an associated depth map. Because of the unique characteristics of the depth map, advanced coding techniques are designed for depth map coding at the expense of computational complexity. In this paper, fast algorithms are conceived to accelerate the intra coding time of the depth map based on boundary continuity. First, the proposed fast prediction unit (PU) mode decision reduces the number of conventional intra prediction modes based on calculating the total sum of squares (TSS) of the PU boundaries. Second, the proposed fast depth modeling mode (DMM) decision makes use of the variances of the boundary pixels to determine the execution of the DMM. Third, the proposed coding unit (CU) early termination algorithm decides whether to further split the current CU by utilizing the thresholds of the TSS and the rate-distortion cost (RD-cost). The experimental results show that the proposed algorithm provides better performance in terms of coding speed and bitrate than the algorithm in previous work. The coding time of the depth map is reduced by 56.08%, while the Bjontegaard delta bitrate (BD-BR) is only increased by 0.32% for the synthesis view.

Efficient Depth Intra Frame Coding in 3D-HEVC by Corner Points.

To improve the coding performance of depth maps, 3D-HEVC includes several new depth intra coding tools at the expense of increased complexity due to a flexible quadtree Coding Unit/Prediction Unit (CU/PU) partitioning structure and a huge number of intra mode candidates. Compared to natural images, depth maps contain large plain regions surrounded by sharp edges at the object boundaries. Our observation finds that the features proposed in the literature either speed up the CU/PU size decision or intra mode decision and they are also difficult to make proper predictions for CUs/PUs with the multi-directional edges in depth maps. In this work, we reveal that the CUs with multi-directional edges are highly correlated with the distribution of corner points (CPs) in the depth map. CP is proposed as a good feature that can guide to split the CUs with multi-directional edges into smaller units until only single directional edge remains. This smaller unit can then be well predicted by the conventional intra mode. Besides, a fast intra mode decision is also proposed for non-CP PUs, which prunes the conventional HEVC intra modes, skips the depth modeling mode decision, and early determines segment-wise depth coding. Furthermore, a two-step adaptive corner point selection technique is designed to make the proposed algorithm adaptive to frame content and quantization parameters, with the capability of providing the flexible tradeoff between the synthesized view quality and complexity. Simulation results show that the proposed algorithm can provide about 66% time reduction of the 3D-HEVC intra encoder without incurring noticeable performance degradation for synthesized views and it also outperforms the previous state-of-the-art algorithms in term of time reduction and ∆ BDBR.

A Novel Coding Architecture for Multi-Line LiDAR Point Clouds Based on Clustering and Convolutional LSTM Network

Light detection and ranging (LiDAR) plays an indispensable role in autonomous driving technologies, such as localization, map building, navigation and object avoidance. However, due to the vast amount of data, transmission and storage could become an important bottleneck. In this article, we propose a novel compression architecture for multi-line LiDAR point cloud sequences based on clustering and convolutional long short-term memory (LSTM) networks. LiDAR point clouds are structured, which provides an opportunity to convert the 3D data to 2D array, represented as range images. Thus, we cast the 3D point clouds compression as a range image sequence compression problem. Inspired by the high efficiency video coding (HEVC) algorithm, we design a novel compression framework for LiDAR data that includes two main techniques: intra-prediction and inter-prediction. For intra-frames, inspired by the depth modeling modes (DMM) adopted in 3D-HEVC, we develop a clustering-based intra-prediction technique, which can utilize the spatial structure characteristics of point clouds to remove the spatial redundancy. For inter-frames, we design a prediction network model using convolutional LSTM cells. The network model is capable of predicting future inter-frames using the encoded intra-frames. As a result, temporal redundancy can be removed. Experiments on the KITTI dataset demonstrate that the proposed method achieves an impressive compression ratio (CR), with 4.10% at millimeter precision, which means the point clouds can compress to nearly 1/25 of their original size. Additionally, compared with the well-known octree, Google Draco, and MPEG TMC13 methods, our algorithm yields better performance in compression ratio.

6WR: A Hardware Friendly 3D-HEVC DMM-1 Algorithm and its Energy-Aware and High-Throughput Design

This brief presents the Six Wedgelets and six Refinements (6WR), a hardware friendly algorithm targeting the Depth Modeling Mode 1 (DMM-1) encoding tool of the 3D-High Efficiency Video Coding (3D-HEVC) standard. This brief also presents the high-throughput and energy-aware hardware design for the 6WR. The 6WR algorithm reduces 98.5% of the evaluated wedge lets by exploring the edges gradients, with average coding efficiency losses between 1.2% and 2.8%. The hardware design implements the Bresenham algorithm to avoid the use of memory. The synthesis results show that the 6WR architecture can process up to nine views in 3D full HD 1080p videos at 30 frames per second, with a power dissipation of 263.7 mW. When compared with related works, the 6WR architecture reached the highest throughput and the best results of coding efficiency and energy efficiency when the same target throughput is considered.

Fast depth map intra-mode selection for 3D-HEVC intra-coding

As the last effort of ISO/IEC MPEG and ITU-T Video Coding Experts Group for 3D video coding, high-efficiency video coding (HEVC)-based 3D video coding (3D-HEVC) is the new international emerging standard to compress multi-view texture videos plus depth maps data. In the test model of 3D-HEVC, the conventional HEVC intra-prediction modes and new additional intra-coding tools called depth modeling modes (DMMs) are exploited for depth map intra-coding. These intra-modes provide important coding improvement. However, the improvement of the coding efficiency is obtained at the expense of a computational complexity increase, which obstructs 3D-HEVC from practical real-world applications. Previously, we propose a depth map intra-model decision using tensor features as homogeneity determination. In this paper, we propose an amelioration of the previously proposed depth map intra-model decision using an efficient homogeneity determination. In most cases, the angular intra-coding and DMM could be skipped if some specific conditions are satisfied. The experimental results show that the proposed intra-model decision can reduce the encoding time significantly, with negligible loss of rate distortion performance.

Fast depth intra coding based on spatial correlation and rate distortion cost in 3D-HEVC

3D-HEVC video coding standard adopts many sophisticated algorithms to improve the coding performance, which results in heavy calculation complexity. To reduce the computation burden, a complexity reduction scheme for depth map based on spatial correlation and the rate–distortion (RD) cost is proposed. Firstly, A maximum depth layer decision (MDLD) algorithm is proposed to predict the maximum depth layer (MDL) of each coding tree unit (CTU) accurately. Using the frequency distribution histogram theory in the statistics, the frequency distribution characteristics (FDC) of the RDcost J0 of the CTU level in depth layer 0 is investigated under different MDL. The initial threshold for different MDL is designed based on the FDC of the J0 under allowable false rate (FR). An exponential relationship between the threshold and QP is discovered. Secondly, A fast depth intra mode decision (FDIMD) algorithm is proposed to skip unnecessary prediction modes. Considering the spatial correlation, the FDC of the RDcost JORM of the optimal rough mode (ORM) in rough mode decision (RMD) is analyzed under different optimal prediction mode. Based on the FDC of the JORM, the mode-skipping threshold THst is designed to skip angular modes and depth modeling modes (DMMs). An exponential model is built to represent the relationship between the THst and QP. Finally, a fast wedgelet pattern decision method based on K-Means is developed to reduce the complexity of DMM1. Experimental results show that the proposed algorithm achieves an average reduction of 58.9% on intra coding time, with negligible degradation of coding performance. The proposed algorithm performs better in terms of complexity and performance than other approaches.

3D-HEVC Bipartition Modes Encoder and Decoder Design Targeting High-Resolution Videos

This paper presents hardware designs for the encoder and decoder of the 3D-High Efficiency Video Coding (3D-HEVC) bipartition modes targeting real-time processing of high-resolution videos. These hardware designs include the depth modeling modes 1 (DMM-1) and 4 (DMM-4). The encoder was designed using a simplification in the DMM-1 algorithm with the advantage of better using the hardware resources and without coding efficiency loss. The encoder architecture was designed in a scalable structure supporting different block sizes and reaching different throughputs according to the application requirements. The decoder was designed sharing resources between DMM-1 and DMM-4 execution and supporting all available block sizes. Both architectures were synthesized for 65- and 28-nm technologies, being capable of achieving real-time processing for ${1920}\times {1080}$ videos at 30 frames/s.

Multi-strategy depth intra mode decision algorithm in 3D-HEVC

The new generation of 3D video coding standard, denoted as 3D High Efficiency Video Coding (3D-HEVC), uses Multi-view Video Plus Depth (MVD) format to represent stereo videos. Due to the reason that the fringe information of depth maps has great influence on the video quality of synthesized views, Depth Modeling Modes (DMMs) are introduced to maintain the quality of depth maps, while causing computational complexity increase tremendously. Therefore, a multi-strategy depth intra mode decision algorithm is proposed in this paper. It incorporates early rough mode decision (RMD) termination strategy, candidate mode reduction strategy and fast DMM decision strategy. The experimental results indicate that the proposed algorithm can obtain an average encoding time reduction of 34.38% for depth map coding, with just 0.43% BDBR increase.

A low-latency DMM-1 encoder for 3D-HEVC

Depth modeling mode 1 (DMM-1) is vital for 3D high-efficiency video coding, since it results in low distortion and is suitable for sharp boundaries. When designing DMM-1 encoders, traditional work used serial methods to calculate all the wedgelets, leading to large latency; they may meet the requirements of 30 fps 1080P 3D video at most, but may be powerless for higher frame rate or 3D video resolution. In this paper, we propose a flexible parallel architecture for DMM-1 encoder. It can simultaneously evaluate all the wedgelets, saving encoding time without increasing distortion; it can also be configured into a partial-parallel architecture to save area. Experiments show that our method can save 33.6–94.3% encoding time for different test schemes. Synthesis results in SMIC 55 nm show that the VLSI design for proposed parallel architecture can meet the requirements of 1080P and higher-resolution video processing in real time.

A Novel Point Cloud Compression Algorithm Based on Clustering

Due to the enormous volume of point cloud data, transmitting and storing the data requires large bandwidth and storage space. It could be a critical bottleneck, especially in tasks such as autonomous driving. In this letter, we propose a novel point cloud compression algorithm based on clustering. The proposed scheme starts with a range image-based segmentation step, which segments the three-dimensional (3-D) range data into ground and main objects. Then, it introduces a novel prediction method according to the segmented regions’ shape. This prediction method is inspired by the depth modeling modes used in 3-D high-efficiency video coding for depth map coding. Finally, the few prediction residual is efficiently compressed with several lossless or lossy data compression techniques. Experimental results show that the proposed algorithm can largely eliminate the spatial redundant information of the point cloud data. The lossless compression scheme reaches a compression ratio of nearly 5%, which means that the point cloud is compressed to 5% of its original size without any distance distortion. Compared with other methods, the proposed compression algorithm also shows better performance.

Fast Depth Map Intra Coding for 3D Video Compression Based Tensor Feature Extraction and Data Analysis

3D high-efficiency video coding (3D-HEVC) is the latest standard for 3D video compression created by the ISO/IEC MPEG and ITU-T Video Coding Experts Group (VCEG) based on a new video format called multiview texture videos plus depth maps (MVDs). In 3D-HEVC depth map intra prediction, the test model uses the conventional HEVC intra modes and new supplementary coding tools called depth modeling modes (DMMs) to better preserve the sharp edges of depth maps. These intra prediction modes fundamentally enhance the depth map intra coding efficiency. Although this process enables high coding efficiency, it also results in a very high encoding complexity, which limits the use of the 3D-HEVC encoder in practical and real-world applications. For most cases, the conventional intra coding and DMMs can be skipped if the current region is classified as a homogenous region. Bringing this intuitive approach to practice, this paper proposes fast depth map intra coding based on tensor feature extraction and data analysis. The experimental results show that the proposed intra model decision brings a good complexity reduction with negligible loss of rate distortion performance.

Fast Mode Decision Based on Gradient Information in 3D-HEVC

The 3D extension of High Efficiency Video Coding (3D-HEVC) introduce Depth Modeling Mode (DMM) and 35 conventional intra modes to enhance the quality of coding, while bringing unacceptable computational complexity during the process of rough mode decision (RMD) and most probable mode (MPM). In this paper, we proposed a fast mode decision algorithm for texture map and depth map coding based on gradient information. Firstly, analyzing the characteristics of predict units (PU) in different intra mode applications that obtain the lowest RD-cost,then extracting the gradient information of PU to classify the PU into three types of gradient blocks and selecting appropriate candidate modes for the PU,thereby avoiding search each mode in coding process and skip the process of RMD and MPM early. Experimental results show that the proposed algorithm can achieve average 30.6% time saving with negligible reduction of coding performance.

Fast intra mode decision and fast CU size decision for depth video coding in 3D-HEVC

3D-HEVC is the new coding standard for both texture and associated depth video. In addition to several new coding tools that are designed for texture video with improved coding efficiency, some tools are specified for depth video, such as depth modeling modes (DMMs), segment-wise DC (SDC) mode and depth intra skip mode. This paper proposes techniques that accelerate the encoding of depth video, including a fast intra mode decision and a fast CU size decision. A hierarchical mode decision is proposed for the fast intra mode decision, based on a statistical analysis of mode distribution. The intra mode can be predicted at a very early stage if the hierarchical comparison of the rate–distortion (R–D) cost for selected candidate modes is satisfied for a given condition. For the fast CU size decision, smaller CU sizes are not evaluated if the current CU has the desired properties. The CU size of the collocated texture is also considered to avoid over-splitting of the current depth CU. The experimental results show that the proposed techniques achieve an average 62.5% time saving with 0.81% increase in the bitrate for the synthesized views under the all intra configuration, compared with the reference software HTM 15.0.

Parallelism exploration for 3D high-efficiency video coding depth modeling mode one

This article presents a parallelism exploration over the depth modeling mode 1 (DMM-1) encoding algorithm of the 3D high-efficiency video coding (3D-HEVC) standard and applied the proposed solutions in a multicore central processing unit (CPU) and two graphics processor unities (GPU). The article evaluates efficient parallel algorithms for DMM-1, which also take advantage of simplifications proposed in our previous works. We demonstrate that DMM-1 can obtain a scalable speedup when running in systems with several available cores even when simplifications are being applied. Experimental results for 1920 × 1088 resolution videos show that the proposed parallel algorithms achieved up to 2 frames per second (fps) in a four-cores (with eight threads) CPU and more than 30 fps in two different GPUs. Therefore, the speedup attained with GPU enables real-time 3D-video encoding applying the proposed parallelism strategies together with the DMM-1 proposed simplifications.

Efficient Intra Mode Selection for Depth-Map Coding Utilizing Spatiotemporal, Inter-Component and Inter-View Correlations in 3D-HEVC.

3D-high efficiency video coding (HEVC) is developed for the compression of the multi-view video plus depth format, which is based on the latest generation of video coding standard, HEVC. It further adopts several new intra prediction modes, depth-modeling modes (DMMs) in intra candidate modes for a better representation of edges in depth maps, which introduces a drastic increase in the computational complexity. The procedure of depth intra mode decision together with DMMs and existing intra modes is a very time consuming part due to huge complexity of full rate distortion (RD) cost calculation. In this paper, a low complexity intra mode selection algorithm is proposed to reduce complexity of depth intra prediction in both intra-frames and inter-frames. An experimental analysis is first performed to study the inter-view correlation and the inter-component (texture video and its associated depth) correlation in intra coding information such as the intra mode and RD cost. All intra modes available in 3D-HEVC are classified into three activity classes assigned with different mode-weight factors, and the coding mode complexity of a coding unit (CU) is defined according to the intra mode information from available spatiotemporal, inter-view, and inter-component neighboring coded CUs. The coding mode complexity analysis is utilized to assign different candidate intra modes for different types of CUs. The optimal intra prediction mode and the RD cost value in current CU depth level are further used to skip unnecessary intra prediction sizes. Experimental results show that the proposed fast depth intra coding algorithm achieves 61% complexity reduction on intra prediction, while incurring a 0.2% Bjontegaard metric increase for coded and synthesized views compared to the test model of 3D-HEVC.

Low complexity intra prediction mode decision for 3D-HEVC depth coding

The 3D High Efficiency Video Coding (3D-HEVC) is the latest 3D extension of the HEVC video coding standard. It supports multi-view videos plus depth (MVD), which is a sophisticated format for enhanced 3D content. Depth modeling modes (DMM) are adopted in the 3D-HEVC for better sharp edge encoding. However, employing DMM causes important increase of computational complexity. In this paper, we propose a three steps scheme for fast intra depth coding in 3D-HEVC. Based on the kirsch edge detection algorithm, a fast angular mode decision algorithm is performed. In order to avoid unnecessary evaluation of the DMMs, a new early termination approach (ET) based on the smoothness of the processed block is adopted. Besides, based on the enhanced shuffled frog leaping algorithm (E-SFLA), a new heuristic method to search the optimal wedgelet pattern for DMM1 Depth mode is introduced. The proposed algorithm is implemented on the top of four different versions of HTM. It outperforms HTM-16.0, the latest used version, and four different work from the literature, in terms of encoding time with similar coding and quality efficiency.

A reduced computational effort mode-level scheme for 3D-HEVC depth maps intra-frame prediction

To encode the depth maps efficiently, 3D-HEVC introduced three intra-frame prediction tools: (i) Depth Intra Skip (DIS), (ii) Depth Modeling Modes (DMMs), and (iii) Segment-wise DC Coding (SDC) that raise the encoding effort. Therefore, we analyzed the most time-consuming steps at the intra-frame prediction and proposed a model-level scheme for reducing the encoding time. The most time-consuming encoding steps were the DMM-1 wedgelet search and the Rate-Distortion (RD) list evaluation in Transform-Quantization and SDC flows. Consequently, we proposed a scheme composed of two solutions for speeding up the DMM-1 and one solution for reducing the RD-list size, accelerating the RD-list evaluation. The DMM-1 speeding up solutions use the information of neighbor-encoded blocks and the data contained in the border of the encoding block to accelerate the wedgelet search. The proposed scheme reduces 31.9% the encoding time with an impact of 0.272% in the Bjontegaard Delta-rate (BD-rate), surpassing the related works.

Probability-Based Depth Intra-Mode Skipping Strategy and Novel VSO Metric for DMM Decision in 3D-HEVC

Multiview video plus depth format has been adopted as the emerging 3D video representation recently. It includes a limited number of textures and depth maps to synthesize additional virtual views. Since the quality of depth maps influences the view synthesis process, their sharp edges should be well preserved to avoid mixing foreground with background. To address this issue, 3D-High Efficiency Video Coding (HEVC) introduces new coding tools, a partition-based intra mode [depth modeling mode (DMM)], a residual description technique [segmentwise depth coding (SDC)], and a more complex rate-distortion (RD) evaluation with view synthesis optimization (VSO), to provide more accurate predictions and achieve higher compression rate. However, these new techniques introduce a lot of possible candidates, and each of them requires complicated RD calculation in the process of intra-mode decision. They lead to unacceptable computational burden in a 3D-HEVC encoder. Therefore, in this paper, we raise two efficient techniques for depth intra-mode decision. First, by investigating the statistical characteristics of variance distributions in the two partitions of DMM, a simple but efficient criterion based on the squared Euclidean distance of variances (SEDV) is suggested to evaluate RD costs of the DMM candidates instead of the time-consuming VSO process. Second, a probability-based early depth intra-mode decision is proposed to select only the most promising mode and make the early determination of using SDC based on the low-complexity RD cost in rough mode decision. Experimental results show that the proposed algorithm with these two new techniques provides 33%–48% time reduction with little drop in the coding performance compared with the state-of-the-art algorithms.

Fast mode decision and PU size decision algorithm for intra depth coding in 3D-HEVC

In 3D video extension of High-Efficiency-Video-Coding (3D-HEVC), intra depth coding involves huge computational complexity for the added depth modeling modes (DMMs) and the new complex processing flow. This paper proposes an algorithm including the fast mode and prediction unit (PU) size decisions to accelerate the intra depth coding. For the mode decision, after a modification of the processing flow, three strategies are adopted: (1) evaluation results for intra conventional modes (ICMs) are utilized to determine whether DMMs are skipped; (2) with a predicted direction from ICMs, a golden ratio method is adopted to simplify the DMMs searching; and (3) fast Planar mode decision is made by analyzing the relationship of DMMs and Planar mode. For the fast PU size decision, evaluation costs of ICMs and DMMs, with combination of some trained thresholds, are used to make decisions. Experimental results show the overall algorithm reduces 63.00% computational complexity, with 1.04% BD-rate increasing.