Merge Mode Research Articles

Efficient Geometry Surface Coding in V-PCC

In recent video-based point cloud compression (V-PCC), 3D point clouds are projected onto 2D images and compressed by High-Efficiency Video Coding (HEVC). However, HEVC was originally designed for natural visual signals, which is a suboptimal framework for point clouds. Therefore, there are still problems in geometry information compression in V-PCC: (1) The distortion based on the sum of squared error (SSE) in the existing rate-distortion optimization (RDO) is inconsistent with the geometric quality measurement; (2) The existing prediction cannot explore the fixed relationship between the corresponding far layer and near layer depth, which means that the far layer depth can be always not less than the corresponding near layer depth. In this paper, we present an efficient geometry surface coding (EGSC) method for V-PCC to address the problems. Firstly, an error projection (EP) model is designed to establish the relationship between the SSE-based distortion and the geometry quality metric. Secondly, an EP-based RDO is employed to improve the geometry information compression by estimating the point normals with gradients. Finally, an occupancy-map driven scheme is proposed to improve the prediction accuracy of merge modes. Experimental results show that the proposed method achieves an average of over 10% bit-rate saving compared with the V-PCC reference software.

On searching the best mode for forex forecasting: bidirectional long short-term memory default mode is not enough

Presently, the Forex market has become the world’s largest financial market with more than US$5 trillion daily volume. Therefore, it attracts many researchers to learn its traded currency pairs characteristics and predict their future values. Here, we propose simple three layers Bidirectional long short-term memory (Bi-LSTM) networks for Forex forecasting with four different merge modes. Moreover, the proposed model is also compared to the conventional long short-term memory (LSTM) networks with the same architecture. Five major Forex currency pairs, namely AUD/USD, EUR/USD, GBP/USD, USD/CHF, and USD/JPY, with more than ten years of historical records are considered in this study. It is revealed from the experimental results that among four available merge modes, the concatenation mode as the default merge mode in Bi-LSTM networks is actually the less preferred mode for Forex forecasting (Root mean square error 0.30685517, mean absolute error 0.27442235, mean absolute percentage error 0.827108%). Moreover, Bi-LSTM average mode gets the highest R2 score that could achieve 89.579%. Therefore, the proposed three layers Bi-LSTM networks could provide a baseline result for developing a good trading strategy in Forex forecasting.

TransMeta simultaneously assembles multisample RNA-seq reads.

Assembling RNA-seq reads into full-length transcripts is crucial in transcriptomic studies and poses computational challenges. Here we present TransMeta, a simple and robust algorithm that simultaneously assembles RNA-seq reads from multiple samples. TransMeta is designed based on the newly introduced vector-weighted splicing graph model, which enables accurate reconstruction of the consensus transcriptome via incorporating a cosine similarity–based combing strategy and a newly designed label-setting path-searching strategy. Tests on both simulated and real data sets show that TransMeta consistently outperforms PsiCLASS, StringTie2 plus its merge mode, and Scallop plus TACO, the most popular tools, in terms of precision and recall under a wide range of coverage thresholds at the meta-assembly level. Additionally, TransMeta consistently shows superior performance at the individual sample level.

Motion Vector Coding and Block Merging in the Versatile Video Coding Standard

This paper overviews the motion vector coding and block merging techniques in the Versatile Video Coding (VVC) standard developed by the Joint Video Experts Team (JVET). In general, inter-prediction techniques in VVC can be classified into two major groups: “whole block-based inter prediction” and “subblock-based inter prediction”. In this paper, we focus on techniques for whole block-based inter prediction. As in its predecessor, High Efficiency Video Coding (HEVC), whole block-based inter prediction in VVC is represented by adaptive motion vector prediction (AMVP) mode or merge mode. Newly introduced features purely for AMVP mode include symmetric motion vector difference and adaptive motion vector resolution. The features purely for merge mode include pairwise average merge, merge with motion vector difference, combined inter-intra prediction and geometric partitioning mode. Coding tools such as history-based motion vector prediction and bidirectional prediction with coding unit weights can be applied on both AMVP mode and merge mode. This paper discusses the design principles and the implementation of the new inter-prediction methods. Using objective metrics, simulation results show that the methods overviewed in the paper can jointly achieve 6.2% and 4.7% BD-rate savings on average with the random access and low-delay configurations, respectively. Significant subjective picture quality improvements of some tools are also reported when comparing the resulting pictures at same bitrates.

Ensemble application of bidirectional LSTM and GRU for aspect category detection with imbalanced data

E-commerce websites produce a large number of online reviews, posts, and comments about a product or service. These reviews are used to assist consumers in buying or recommending a product. However, consumers are expressing their views on a specific aspect category of a product. In particular, aspect category detection is one of the subtasks of aspect-based sentiment analysis, and it classifies a given text into a set of predefined aspects. Naturally, a class imbalance problem occurs in real-world applications. The class imbalance is studied over the last two decades using machine learning algorithms. However, there is very little empirical research in deep learning with the class imbalance problem. In this paper, we propose bidirectional LSTM and GRU networks to deal with imbalance aspect categories. The proposed method applies a data-level technique to reduce class imbalance. Specifically, we employ the stratified sampling technique to deal with imbalanced classes. Moreover, we create word vectors with the corpus-specific word embeddings and pre-trained word embeddings. This word representations fed into the proposed method and their merge modes such as addition, multiplication, average, and concatenation. The performance of this method is evaluated with a confusion matrix, precision, recall, F1-score with micro-average, macro-average, and weighted average. The experimental result analysis suggests that the proposed method outperforms with pre-trained word embeddings.

Regression-Based Motion Vector Field for Video Coding

In this paper, we study a method for compensating the non-translational motion behavior in video coding. The proposed method models the motion field of a prediction block based on the motion information of the neighboring blocks by using a linear regression approach. In order to provide a finer granularity of motion vectors the Regression-based Motion Vector Field (RMVF) method derives the motion field in ${4}\times {4}$ sub-block accuracy. Such approach generates a smooth and more realistic motion vector field inside the prediction block. The motion field generated with RMVF is then used as a new merge mode along with other merge modes in VTM-2.0 test model of the Versatile Video Coding (H.266/VVC) standard. The conducted experiments with JVET CTC sequences illustrate that the proposed RMVF method provides 0.77%, 0.19% and 0.41% bitrate reductions with random access (RA), low delay B (LDB) and low delay P (LDP) configurations, respectively. Furthermore, this method provides on average 0.65% bitrate saving for the 360° sequences in equirectangular projection format (ERP) with RA configuration.

CUPSEED - A combined use of prediction syntax elements to embed data in SHVC video.

With the rapid advancement in digital technologies, video rises to become one of the most effective communication tools that continues to gain popularity and importance. As a result, various proposals are put forward to manage videos, and one of them is data embedding. Essentially, data embedding inserts data into the video to serve a specific purpose, including proof of ownership via watermark, covert communication in steganography, and authentication via fragile watermark. However, most conventional methods embed data by using only one type of syntax element defined in the video coding standard, which may suffer from large bit rate overhead, quality degradation, or low payload. Therefore, this work aims to explore the combined use of multiple prediction syntax elements in SHVC for the purpose of data embedding. Specifically, the intra prediction mode, motion vector predictor, motion vector difference, merge mode and coding block structure are collectively manipulated to embed data. The experimental results demonstrate that, in comparison to the conventional single-venue data embedding methods, the combined use of prediction syntax elements can achieve higher payload while preserving the perceptual quality with minimal bit rate variation. In the best case scenario, a total of 556.1 kbps is embedded into the video sequence PartyScene with a drop of 0.15 dB in PSNR while experiencing a bit rate overhead of 7.4% when all prediction syntax elements are utilized altogether. A recommendation is then put forward to choose specific types of syntax element for data embedding based on the characteristics of the video.

Intra Block Copy Mirror Mode for Screen Content Coding in Versatile Video Coding

Intra Block Copy (IBC) cannot work well for blocks with symmetry or mirror characteristic, because it is based on the translational block matching technique. In order to solve this problem, in this paper, an adequately designed IBC-Mirror mode is proposed for Versatile Video Coding (VVC), which extends our previous work. The newly proposed IBC-Mirror mode consists of two sub-modes, including IBC-Mirror Advanced Motion Vector Prediction (AMVP) mode and IBC-Mirror Merge mode. In IBC-Mirror AMVP mode, various Motion Estimation (ME) methods are applied, including previously proposed spatial search, and newly designed methods, i.e., Hash-based, Cache-based, Predictor-based and History-based IBC-Mirror search. As for IBC-Mirror Merge mode, Motion Information, composed of Block Vector (BV) and mirror flipping information, is added to the merge list to improve coding efficiency. Compared to the traditional IBC mode in VTM-10.0, the latest test model, the proposed mode can achieve 1~2% BD-rate saving. Compared to our previous work, methods in this paper can further achieve 0.5% coding gains and reduce 4% coding time. Higher coding efficiency and lower complexity show the superiority of extensions and improvements in this paper.

A CNN-Based Fast Inter Coding Method for VVC

The Versatile Video Coding (VVC) achieves superior coding efficiency as compared with the High Efficiency Video Coding (HEVC), while its excellent coding performance is at the cost of several high computational complexity coding tools, such as Quad-Tree plus Multi-type Tree (QTMT)-based Coding Units (CUs) and multiple inter prediction modes. To reduce the computational complexity of VVC, a CNN-based fast inter coding method is proposed in this paper. First, a multi-information fusion CNN (MF-CNN) model is proposed to early terminate the QTMT-based CU partition process by jointly using the multi-domain information. Then, a content complexity-based early Merge mode decision is proposed to skip the time-consuming inter prediction modes by considering the CU prediction residuals and the confidence of MF-CNN. Experimental results show that the proposed method reduces an average of 30.63% VVC encoding time, and the Bjøontegaard Delta Bit Rate (BDBR) increases about 3%.

Decoder-Side Motion Vector Refinement in VVC: Algorithm and Hardware Implementation Considerations

This paper presents an overview of the decoder-side motion vector refinement (DMVR) algorithm in the Versatile Video Coding (VVC) standard. The proposed DMVR algorithm aims to increase the prediction accuracy of the blocks coded in merge mode using the bilateral matching-based refinement method. Compared with previous decoder-side motion vector derivation approaches, the proposed method significantly increases the coding efficiency without signaling additional side information. Furthermore, the hardware implementation considerations of the DMVR design are particularly focused in this study. This paper details and analyzes the novel features of DMVR contributing to the increase in coding efficiency and the reduction in computational complexity and implementation difficulty. Experimental results based on the VVC test model version 8.0 demonstrate that average Bjøntegaard Delta rate savings of 0.80 % and 2.81 % are achieved for the “tool-off” and “tool-on” test configurations, respectively. Moreover, 4 % additional decoding time and negligible additional external memory bandwidth requirements of DMVR based on the common test conditions for VVC are reported.

Novel skip motion estimation for efficient inter coding in HEVC

Video coding is necessary to transmit large amounts of video information. HEVC, which is the latest video coding standard, includes many advanced tools to achieve higher coding efficiency than previous standards, such as MPEG-1/2/4 and H.264/AVC. In particular, new techniques in inter prediction, such as skip, merge, and advanced motion vector prediction (AMVP) modes, drastically improve coding performance by reducing temporal redundancies between consecutive frames within a video. However, the skip mode, which does not perform a motion estimation, can deteriorate the coding performance, because of inaccurate motion vector prediction in videos with fast and complex motions. In order to solve this problem, a novel skip motion estimation method is proposed for efficient inter coding in HEVC. Unlike the conventional skip mode, the proposed skip mode performs the motion estimation, and then transmits the associated motion information. Experimental results demonstrate that the proposed method achieves the high coding performance with the low decoding complexity.

Deep Video Prediction Network-Based Inter-Frame Coding in HEVC

In this paper, we propose a novel Convolutional Neural Network (CNN) based video coding technique using a video prediction network (VPN) to support enhanced motion prediction in High Efficiency Video Coding (HEVC). Specifically, we design a CNN VPN to generate a virtual reference frame (VRF), which is synthesized using previously coded frames, to improve coding efficiency. The proposed VPN uses two sub-VPN architectures in cascade to predict the current frame in the same time instance. The VRF is expected to have higher temporal correlation than a conventional reference frame, and, thus it is substituted for a conventional reference frame. The proposed technique is incorporated into the HEVC inter-coding framework. Particularly, the VRF is managed in a HEVC reference picture list, so that each prediction unit (PU) can choose a better prediction signal through Rate-Distortion optimization without any additional side information. Furthermore, we modify the HEVC inter-prediction mechanisms of Advanced Motion Vector Prediction and Merge modes adaptively when the current PU uses the VRF as a reference frame. In this manner, the proposed technique can exploit the PU-wise multi-hypothesis prediction techniques in HEVC. Since the proposed VPN can perform both the video interpolation and extrapolation, it can be used for Random Access (RA) and Low Delay B (LD) coding configurations. It is shown in experimental results that the proposed technique provides -2.9% and -5.7% coding gains, respectively, in RA and LD coding configurations as compared to the HEVC reference software, HM 16.6 version.

Efficient Prediction Methods With Enhanced Spatial-Temporal Correlation for HEVC

The state-of-the-art high efficiency video coding (HEVC) standard provides a significant improvement relative to H.264/AVC, with almost 50% bitrate reduction. However, there are still requirements to further improve the coding performance of HEVC. In this paper, we reveal some problems in both intra and inter prediction by giving detailed observations, and solve them by further exploring spatial and temporal correlations. First, it is noteworthy that large boundary distortions in intra prediction blocks arise from the simplistic extrapolation operation with partial reference samples. To reduce these boundary distortions, we propose an enhanced prediction boundary filtering method based on generalized Gaussian distribution functions, and extend this method to merge prediction. Second, we combine intra and inter prediction to create a novel prediction method, in which spatial and temporal correlations are simultaneously considered. In addition, the existing merge mode can only provide accurate prediction results for simple movements, owing to the lack of motion estimation. To improve the flexibility, we optimize the existing merge mode by intensifying the temporal correlation with a simple motion search process. The experimental results show that the proposed methods achieve consistent bitrate savings, 1.3% for all intra, 1.5% for random access, 2.5% (up to 4.2%) for low delay P, and 1.2% for low delay B configuration. The encoding time only rises by 70% - 80%, and the decoding time increases negligibly.

Euclidean Distance-Based Weighted Prediction for Merge Mode in HEVC

Merge mode can achieve a considerable coding gain because of reducing the cost of coding motion information in video codecs. However, the simple adoption of the motion information from the neighbouring blocks may not achieve the optimal performance as the motion correlation between the pixels and the neighbouring block decreases with their distance increasing. To address this problem, the paper proposes a Euclidean distance-based weighted prediction algorithm as an additional candidate in the merge mode. First, several predicted blocks are generated by motion compensation prediction (MCP) with the motion information from available neighbouring blocks. Second, an additional predicted block is generated by a weighted average of the predicted blocks above, where the weighted coefficient is related to Euclidean distances from the neighbouring candidate to the pixel points in the current block. Finally, the best merge mode is selected by the rate distortion optimization (RDO) among the original merge candidates and the additional candidate. Experimental results show that, on the joint exploration test model 7.0 (JEM 7.0), the proposed algorithm achieves better coding performance than the original merge mode under all configurations including random access (RA), low delay B (LDB), and low delay P (LDP), with a slight coding complexity increase. Especially for the LDP configuration, the proposed method achieves 1.50% bitrate saving on average.

Fast Depth and Inter Mode Prediction for Quality Scalable High Efficiency Video Coding

The scalable high efficiency video coding (SHVC) is an extension of high efficiency video coding (HEVC). It introduces multiple layers and inter-layer prediction, thus significantly increases the coding complexity on top of the already complicated HEVC encoder. In inter prediction for quality SHVC, in order to determine the best possible mode at each depth level, a coding tree unit can be recursively split into four depth levels, including merge mode, inter2N×2N, inter2N×N, interN×2N, interN×N, inter2N×nU, inter2N×nD, internL×2N and internRx×2N, intra modes and inter-layer reference (ILR) mode. This can obtain the highest coding efficiency, but also result in very high coding complexity. Therefore, it is crucial to improve coding speed while maintaining coding efficiency. In this research, we have proposed a new depth level and inter mode prediction algorithm for quality SHVC. First, the depth level candidates are predicted based on inter-layer correlation, spatial correlation and its correlation degree. Second, for a given depth candidate, we divide mode prediction into square and non-square mode predictions respectively. Third, in the square mode prediction, ILR and merge modes are predicted according to depth correlation, and early terminated whether residual distribution follows a Gaussian distribution. Moreover, ILR mode, merge mode and inter2N×2N are early terminated based on significant differences in Rate Distortion (RD) costs. Fourth, if the early termination condition cannot be satisfied, non-square modes are further predicted based on significant differences in expected values of residual coefficients. Finally, inter-layer and spatial correlations are combined with residual distribution to examine whether to early terminate depth selection. Experimental results have demonstrated that, on average, the proposed algorithm can achieve a time saving of 71.14%, with a bit rate increase of 1.27%.

Fast Integer Motion Estimation With Bottom-Up Motion Vector Prediction for an HEVC Encoder

Although advanced motion vector prediction (AMVP) modes based on motion estimation (ME) are selected significantly less due to the merge mode newly adopted in the high-efficiency video coding (HEVC), integer ME (IME) still occupies a large amount of computation in HEVC because the HEVC supports a highly flexible block partitioning structure. Introduction of the merge mode in HEVC substantially affects the optimal search algorithm of IME. Therefore, this gives a chance to reduce the computational complexity of ME with marginal drop of the compression efficiency. This paper proposes a new IME algorithm that significantly reduces its search ranges. Computational complexity of the proposed algorithm is reduced without serious degradation in coding efficiency by obtaining additional accurate motion vector predictors (MVPs) in a bottom-up order and searching narrow regions around multiple MVPs. These bottom-up MVPs are obtained from the prediction units (PUs) in the coding units (CUs) in the hierarchically lower level, which share the pixel area either completely or partially with the current PU. Although the encoder should perform the proposed IME in a bottom-up order from the smallest CUs to the larger CUs, several other processes, such as fractional ME and merges mode are performed in a top-down order to exploit several fast algorithms already adopted in the HEVC test model encoder. To keep compatibility with HEVC standard, the bottom-up MVPs are utilized for only IME. The bitstream is generated using standard AMVP. According to the simulation results, it is confirmed that the proposed algorithm improves the accuracy of MVPs, which leads to the reduction of IME computational complexity up to 81.95% on average with the Bjontegaard delta bitrate of 0.39%.

Motion and disparity vectors early determination for texture video in 3D-HEVC

3D-HEVC is the state-of-the-art video coding standard for 3D video, and it is an extension of high efficiency video coding (HEVC) standard. Besides the original HEVC coding tools, 3D-HEVC adopts some advanced coding tools, such as disparity vector (DV), inter-view prediction and inter-component prediction. However, these advanced tools lead to extremely high encoding complexity at the same time, thus it cannot be well applied in real-time multimedia systems. In this paper, we propose a motion and disparity vectors early determination algorithm to reduce 3D-HEVC computational complexity. First, based on the statistical analyses, the spatial and temporal motion vector (MV) candidates are adaptively reduced for the prediction unit (PU) with the Merge mode. Then, for the PU with the Inter mode, the combination of spatial and temporal candidates is used to early determine the final MV. Finally, an adaptive optimization algorithm is adopted to select the valid inter-view disparity vectors (DV) candidates. Moreover, if the difference between candidate vectors is within a conditional range, current PU will be encoded with the Merge mode to skip unnecessary coding process. Experimental results show that for the texture views encoding, the proposed algorithm achieves an average of 33.03% encoding time saving, and an average of 0.47% BD-Rate increases.

Fast inter-prediction algorithm based on motion vector information for high efficiency video coding

High Efficiency Video Coding (HEVC/H.265) is the latest international video coding standard, which achieves better compression ratio and supports higher resolution than Advanced Video Coding (H.264/AVC). However, HEVC/H.265 increases the computational burden. To reduce the coding complexity of the HEVC encoder, this paper proposes a fast inter-prediction algorithm to speed up coding time. We collect the average rate-distortion costs (RD-cost) of Skip modes and Merge modes to accelerate prediction unit (PU) mode decisions. In addition, we also acquire and analyze the motion vector range from Merge modes and Inter 2N × 2N modes to decide whether to execute Merge and advanced motion vector prediction (AMVP) of other PUs. The experimental results show that the proposed algorithm provides 48.54% time saving on average in random-access configuration and maintains good rate-distortion performance and video quality at the same time. The proposed algorithm also outperforms previous works.

Probability Model-Based Early Merge Mode Decision for Dependent Views Coding in 3D-HEVC

As a 3D extension to the High Efficiency Video Coding (HEVC) standard, 3D-HEVC was developed to improve the coding efficiency of multiview videos. It inherits the prediction modes from HEVC, yet both Motion Estimation (ME) and Disparity Estimation (DE) are required for dependent views coding. This improves coding efficiency at the cost of huge computational costs. In this article, an early Merge mode decision approach is proposed for dependent texture views and dependent depth maps coding in 3D-HEVC based on priori and posterior probability models. First, the priori probability model is established by exploiting the hierarchical and interview correlations from those previously encoded blocks. Second, the posterior probability model is built by using the Coded Block Flag (CBF) of the current coding block. Finally, the joint priori and posterior probability model is adopted to early terminate the Merge mode decision for both dependent texture views and dependent depth maps coding. Experimental results show that the proposed approach saves 45.2% and 30.6% encoding time on average for dependent texture views and dependent depth maps coding while maintaining negligible loss of coding efficiency, respectively.

Algorithm optimization and hardware implementation for Merge mode in HEVC

Merge mode is a new tool for improving inter-frame coding efficiency in high-efficiency video coding. This tool can save the bitrate for the motion vector by sharing this vector with neighboring blocks. Merge is a process that selects a candidate motion vector by calculating the cost of rate-distortion. However, this process requires a large number of complex computations and memory access, thereby resulting in the low efficiency of hardware implementation. This paper proposes a new Merge candidate decision scheme that determines the most favorable Merge candidate from a full list of candidates by comparing the sum of absolute transformed difference with the weighted header bit instead of performing a complex calculation for sum of squared difference and entropy coding process in HM16.7. The simulation results show that the performance of the proposed algorithm is close to that of HM16.7 and increases the BD-rate only by 0.22–1.21%. The multilevel pipelines architecture is also exploited in the hardware design. The weighted header bit operation is performed by using the look-up table, which reduces both the complexity and encoding clock cycle. The designed system is implemented with a register transfer level code. The synthesis results from the Design Compiler show that compared with other architecture, the proposed architecture offers great advantages in resource utilization and can process 1920 × 1080 at 353 frame/s for P-slices with a clock frequency of 1057 MHz and logic gate count of 285.2 K.