Sample Adaptive Offset Research Articles

Hardware Efficient Integrated In-loop Filter for HEVC Encoder

The deblocking filter (DF) and the sample adaptive offset (SAO) filter, which aids in enhancing the subjective quality of the image, make up the in-loop filter of the high-efficiency video coding (HEVC) encoder and decoder. The in-loop filter significantly increases the computational load on the HEVC encoder. It is challenging to design an in-loop filter on hardware that can handle intensive computations while using the least amount of on-chip memory, taking external memory traffic and dependencies simultaneously delivering high throughput to support Ultra HD video applications. The proposed design employs the following strategies to address these issues. This work proposes an address generation technique for pipelined horizontal and vertical filtering in DF, that avoids a transpose buffer which otherwise is required. This enables easy pipelining and parallelization thus improving throughput while reducing the on-chip memory utilization. A simplified SAO filter with parallel-pipelined processing is included in the design. These features enable the design to support ultra-HD 7680 × 4320 @ 40 fps video applications. The proposed hardware architecture has a total gate count of 7.73 K LUTs and 2.8 K slice registers, and it is implemented on a 28 nm field programmable gate array (FPGA) platform.

Chaotic moving video quality enhancement based on deep in-loop filtering

The Joint Video Experts Team (JVET) has announced the latest generation of the Versatile Video Coding(VVC, H.266) standard. The in-loop filter in VVC inherits the De-Blocking Filter (DBF) and Sample Adaptive Offset (SAO) of High Efficiency Video Coding (HEVC, H.265), and adds the Adaptive Loop Filter (ALF) to minimize the error between the original sample and the decoded sample. However, for chaotic moving video encoding with low bitrates, serious blocking artifacts still remain after in-loop filtering due to the severe quantization distortion of texture details. To tackle this problem, this paper proposes a Convolutional Neural Network(CNN) based VVC in-loop filter for chaotic moving video encoding with low bitrates. First, a blur-aware attention network is designed to perceive the blurring effect and to restore texture details. Then, a deep in-loop filtering method is proposed based on the blur-aware network to replace the VVC in-loop filter. Finally, experimental results show that the proposed method could averagely save 8.3% of bit consumption at similar subjective quality. Meanwhile, under close bit rate consumption, the proposed method could reconstruct more texture information, thereby significantly reducing the blocking artifacts and improving the visual quality.

Asymmetric Learned Image Compression With Multi-Scale Residual Block, Importance Scaling, and Post-Quantization Filtering

Recently, deep learning-based image compression has made significant progresses, and has achieved better rate-distortion (R-D) performance than the latest traditional method, H.266/VVC, in both MS-SSIM metric and the more challenging PSNR metric. However, a major problem is that the complexities of many leading learned schemes are too high. In this paper, we propose an efficient and effective image coding framework, which achieves similar R-D performance with lower complexity than the state of the art. First, we develop an improved multi-scale residual block (MSRB) that can expand the receptive field and capture global information more efficiently, which further reduces the spatial correlation of the latent representations. Second, an importance scaling network is introduced to directly scale the latents to achieve content-adaptive bit allocation without sending side information, which is more flexible than previous importance map methods. Third, we apply a post-quantization filter (PQF) to reduce the quantization error, motivated by the Sample Adaptive Offset (SAO) filter in video coding. Moreover, our experiments show that the performance of the system is less sensitive to the complexity of the decoder. Therefore, we design an asymmetric paradigm, in which the encoder employs three stages of MSRBs to improve the learning capacity, whereas the decoder only uses one stage of MSRB, which reduces the decoder complexity and still yields satisfactory performance. Experimental results show that compared to the state-of-the-art method, the encoding and decoding time of the proposed method are about 17 times faster, and the R-D performance is only reduced by about 1% on both Kodak and Tecnick-40 datasets, which is still better than H.266/VVC(4:4:4) and other leading learning-based methods. Our source code is publicly available at https://github.com/fengyurenpingsheng.

Adaptive and Efficient Hybrid In-loop Filter Based on Enhanced Generative Adversarial Networks with Sample Adaptive Offset Filter for HEVC/H-265

In this manuscript, an Adaptive and Efficient Hybrid In-loop Filter based on Enhanced Generative Adversarial Network Deblocking Filter (EGANDF) with Sample Adaptive Offset filter (EGANDF-SAO-HEVC) is proposed for High Efficiency Video Coding (HEVC)/H-265. In this, the proposed hybrid in-loop filter involves EGANDF and Sample Adaptive Offset (SAO) filter that lessens the blocking artifacts caused by block-wise processing for coding unit (CU), which is mainly used for improving the video quality. Initially, EGANDF is proposed for HEVC/H-265 for removing blocking artifacts along low computation. Here, the output of EGANDF is given to the SAO filter for reducing ringing artifacts by diminishing high-frequency components during quantization. Thus, the proposed method efficiently reduces artifacts for improving video quality performance. The proposed EGANDF-SAO-HEVC method is implemented in the working platform of HEVC reference software with MATLAB. Finally, the proposed EGANDF-SAO-HEVC model has attained 27.26%, 29.65%, 12.45% higher accuracy, 33.56%, 31.8%, 28.7% higher sensitivity, 34.7%, 33.5%, 32.6% higher specificity, 46.92%, 35.7%, 41.3% lower MSE, 25.7%, 29.7%, 35.6% higher PSNR, and 25.6%, 28.9%, 13.6% higher SSIM for using basketball video sequence when compared to the existing methods.

Optimized Sample Adaptive Offset Filter in HEVC

The high-efficiency video coding (HEVC) standard incorporates sample adaptive offset (SAO) as an in-loop filtering technique to reduce distortion. The fundamental tenet of SAO is to classify every reconstructed pixel and then simply add the offset to each group of pixels. The previous existing technique employs 32 bands in band offset (BO) calculation to determine the proper offset value to minimize the distortion in the SAO estimate algorithm for statistics collection and parametric determination procedure. The suggested method will reduce the necessary band count by 16 bands. It does not alter how edge offset is determined in any way. It offers enough and efficient room for creating superior band group predictions and selections, as well as an ideal decrease in the implemented area. Thus, reducing the number of bands to 16 also decreases the amount of storage needed to hold each pixel’s information in binary files by up to 0.65%.

Gated fusion network for SAO filter and inter frame prediction in Versatile Video Coding

In order to achieve higher coding efficiency, the Versatile Video Coding (VVC) standard includes several new components at the expense of an increase in decoder computational complexity. These technologies often create ringing and contouring effects on the reconstructed frames at a low bit rate and introduce blurring and distortion. To smooth those visual artifacts, the H.266/VVC framework supports four post-processing filter operations. The state-of-the-art CNN-based in-loop filters prefer to deploy multiple networks for various quantization parameters and frame resolutions, which increases training resources and subsequently becomes overhead at decoder frame reconstruction. This paper presents a single deep-learning-based model for sample adaptive off-set (SAO) non-linear filtering operation on the decoder side, uses feature correlation among adjacent frames, and substantiates the merits of intra–inter frame quality enhancement. We introduced a variable filter size dual multi-scale convolutional neural network (D-MSCNN) to attenuate the compression artifact and incorporated strided deconvolution to restore the high-frequency details on the distorted frame. Our model follows sequential training across all QP values and updates the model weights. Using data augmentation, weight fusion, and residual learning, we demonstrated that our model could be trained effectively by transferring the convolution prior feature indices to the decoder to produce a dense output map. The Objective measurements demonstrate that the proposed method outperforms the baseline VVC method in PSNR, MS-SSIM, and VMAF metrics and achieves an average of 5.16% bit rate saving on different test sequence categories.

Vision‐oriented algorithm for fast decision in 3D video coding

This paper designs a novel method to reduce the coding complexity of 3D-HEVC encoder by utilizing the properties of human visual perception. Two vision-oriented edge detections are proposed: for colour texture detection, the authors adopt the Just-Noticeable Distortion (JND); for depth map, the authors combine the Sample Adaptive Offset (SAO) and the Just Noticeable Depth Difference (JNDD) model. The authors also analyse the properties of colour texture and depth map to classify the coding tree unit (CTU) into various kinds of types, including complex-edge CTU, moderate-edge CTU and homogeneous CTU. Besides, fast mode decisions and early termination criteria are performed individually on each type of CTUs according to their characteristics. Especially for those CTUs with more edge information, the proposed projection-based fast mode decision and residual-based early termination preserve important colour texture while speeding up the coding at the same time. The proposed vision-oriented algorithm reduces 31.981% of the overall average coding time with only 1.580% BD-Bitrate increase. Experimental results show that the proposed algorithm can provide considerable time-saving while still maintain the video quality, which outperforms the previous researches.

Deep learning based HEVC in-loop filter and noise reduction

The lossy compression techniques at low bit rate often create ringing and contouring effects on the output images and introduce various blurring and distortion at block bounders. To overcome those compression artifacts different neural network based post-processing techniques have been experimented with over the last few years. The traditional loop-filter methods in the HEVC frame-work support two post-processing operations namely a de-blocking filter followed by a sample adaptive offset (SAO) filter. These operations usually introduce extra signaling bits and become overhead to the network with high-resolution video processing. In this study, we came up with a new deep learning-based algorithm for SAO filtering operations and substantiated the merits of the proposed method. We introduced a variable filter size sub-layered dense CNN (SDCNN) to improve the denoising operation and incorporated large stride deconvolution layers for further computation improvement. We demonstrate that our deconvolution model can effectively be trained by leveraging the high-frequency edge features learned in a shallow network using residual learning and data augmentation techniques. Extensive experiments show that our approach outperformed other state-of-the-art approaches in terms of SSIM, Bjøntegaard delta bit-rate (BD-BR), BD-PSNR measurements on the standard video test set and achieves an average of 8.73 % bit rate saving compared to HEVC baseline.

A fast integrated deblocking filter and sample-adaptive-offset parameter estimation architecture for HEVC

Low power hardware acceleration cores for integration into real-time High Efficiency Video Coding (HEVC) codec for smartphones, tablets, camcorders, and televisions are in great demand. This demand motivates one for an efficient approximation of important power-consuming modules of HEVC including in-loop filters. This paper presents a hardware-efficient implementation of integrated deblocking filter (DBF) and sample adaptive offset (SAO) parameter estimation architecture for 16×16, 32×32, and 64×64 coding tree units (CTU) in HEVC. When the architecture is extended to HEVC-Test-Model (HM) Software, the luminance peak-signal-to-noise-ratio gets increased by 0.02 decibel, and the execution time of DBF and SAO gets decreased by at most 35% and 38%, respectively while compared to the reference algorithm. Moreover, it delivers rate–distortion performance comparable to the HEVC standard and reports mean-squared-error, structural-similarity (SSIM) index, and multi-scale SSIM (MS-SSIM) index of values 0.15, 0.9984, and 1, respectively for 4K video sequences. The architecture consumes minimum power, area, and energy equal to 9.83 milliwatts, 162 kilo-gate-equivalents, and 44 picojoules, respectively while supporting up to forty-six 8K frames per second. Additionally, it reports 78% smaller area and requires 75% less clock-cycles-per-largest-coding-unit as compared to the separate implementation of DBF and SAO with reference to HEVC. Such designs with low power, area, and energy can be integrated into a real-time HEVC codec for portable HEVC-compliant consumer electronic devices.

Detection of transcoded HEVC videos based on in-loop filtering and PU partitioning analyses

With the increasing maturity of video editing technology, forgers are more inclined to transcode videos to High Efficiency Video Coding (HEVC) videos, as HEVC not only enables people to enjoy high-definition videos but also allows broadcasters to stream it more efficiently across networks. Therefore, to verify the originality and authenticity, it is of great significance to propose an algorithm for detecting transcoded HEVC videos. In this paper, a theoretical model of video transcoding is first constructed, and a novel transcoding detection algorithm based on In-loop Filtering and Prediction Units (PU) Partition (IFPP) is proposed. By analyzing the statistical characteristics of strong and normal filtering modes in deblocking filtering and calculating offset values in Sample Adaptive Offset (SAO) filtering, the transcoding traces in inter-coded frames can be captured. In addition, PU partition statistics are also extracted to make full use of traces in intra-coded frames. By fusing these subfeatures, the proposed IFPP feature with 17 dimensions can be obtained, which is further fed to the Support Vector Machine (SVM) classifier. Finally, experiments are conducted on datasets with various coding parameters. Results show that the proposed algorithm outperforms existing algorithms and has better robustness.

Fast Sample Adaptive Offset Jointly Based on HOG Features and Depth Information for VVC in Visual Sensor Networks.

Visual sensor networks (VSNs) can be widely used in multimedia, security monitoring, network camera, industrial detection, and other fields. However, with the development of new communication technology and the increase of the number of camera nodes in VSN, transmitting and compressing the huge amounts of video and image data generated by video and image sensors has become a major challenge. The next-generation video coding standard—versatile video coding (VVC), can effectively compress the visual data, but the higher compression rate is at the cost of heavy computational complexity. Therefore, it is vital to reduce the coding complexity for the VVC encoder to be used in VSNs. In this paper, we propose a sample adaptive offset (SAO) acceleration method by jointly considering the histogram of oriented gradient (HOG) features and the depth information for VVC, which reduces the computational complexity in VSNs. Specifically, first, the offset mode selection (select band offset (BO) mode or edge offset (EO) mode) is simplified by utilizing the partition depth of coding tree unit (CTU). Then, for EO mode, the directional pattern selection is simplified by using HOG features and support vector machine (SVM). Finally, experimental results show that the proposed method averagely saves 67.79% of SAO encoding time only with 0.52% BD-rate degradation compared to the state-of-the-art method in VVC reference software (VTM 5.0) for VSNs.

RECENT TECHNICAL OF ADDED SAMPLE ADAPTIVE OFFSET FOR HEVC

Video compression uses encoding to convert video files into smaller files for efficient storage and transmission in image processing. A decoder reconstructs the received data into a representation of the original video. This research presents recently added in-loop as a filtering technique for extension of sample adaptive offset (SAO) in High-Efficiency Video Coding (HEVC).The principle of SAO is to lessen pixel errors caused by the compression process. Through this an average data reduction up to 1.43% in Low delay configuration and 1.03% in Random access configuration

A CNN-Based Post-Processing Algorithm for Video Coding Efficiency Improvement

Lossy compression algorithms are widely used in video coding. However, lossy compressed videos exist some annoying distortion and artifacts, such as blocking, blurring, and ringing. Thus, coding efficiency improvement is a steady-state topic in the domain of video coding. High Efficiency Video Coding (HEVC), a recent video standard, adopts two in-loop filters for the improvement of the coding efficiency, including deblocking (DB) and sample adaptive offset (SAO). In a certain extent, traditional in-loop filters reduce the distortion and improve the video quality. But the reduction of the distortion is a nonlinear problem that is difficult to be solved by traditional linear filters. Recently, the progress of deep learning shows the possibility to settle the complex problems in the computer vision field. Meanwhile, according to the compressive sensing theory, the post-processing method at the decoder end can further enhance the coding efficiency. In this paper, we propose a variable-filter-size Residue-learning convolutional neural network with batch normalization layer (VRCNN-BN). Our model is an end-to-end model. We feed the decoded pictures to the model at the decoder end. Different from previous methods, we apply the model to luma pictures and chroma pictures, respectively. In order to comprehensively evaluate the coding performance of both luma and chroma components, the color-sensitivity-based combined PSNR (CS-PSNR) is exploited to measure the effectiveness of the proposed method. Compared to HEVC baseline, our approach achieves an average BD-rate reduction of 10.3%, 8.9%, 13.1% and 11.8% in terms of CS-PSNR for random access, all intra, low delay P and low delay B configurations, respectively. Abundant experimental results indicate that our method is better than existing similar methods.

Fast sample adaptive offset algorithm for 360-degree video coding

360-degree video is becoming popular with the development of the virtual reality(VR) technology in recent years. For this kind of video, pictures are projected to a two-dimensional plane. Standard video encoders are used for compression. Due to the projection mapping of the spherical video to the two-dimensional plane video, many conventional coding tools can be adjusted properly for better performance. As the 360-degree videos are high resolution videos, usually 8K resolution videos, the high complexity for encoding 360-degree videos is a problem to be solved urgently, especially for the latest video standard H.265/HEVC. Sample adaptive offset (SAO) is a new tool adopted in H.265/HEVC. It can reduce ringing artifacts and achieve higher coding efficiency. But it also introduces high computational complexity, especially on high resolution videos. In this paper, a fuzzy control based fast SAO method is proposed to reduce the high computational complexity in SAO encoding process according to the characteristics of the ERP format video. Compared with the original algorithm in the reference software HM 16.14, experimental results show that the proposed method can reduce about 73% SAO encoding time on average under the common test conditions with negligible loss.

A Fast Sample Adaptive Offset Algorithm for 360∘ Video

The increased demand for virtual reality brings more challenges to 360∘ video coding. 360∘ video needs to be projected as a planar video before encoding. This process will produce projection distortion. The degree of projection distortion depends on the location of the pixel. Traditional coding algorithms cannot respond to this feature efficiently enough. In this paper, a fast Sample Adaptive Offset (SAO) algorithm for 360∘ video is proposed. The proposed algorithm improves the SAO process. On the basis of retaining the whole SAO process, a simplified SAO process is added. First, the coding tree unit (CTU) of performing the simplified SAO process is filtered according to the rate distortion cost (RD-cost) of the intra- or inter-prediction and the location of the CTU. Subsequently, the CTU is sampled at intervals according to the equirectangular projection (ERP) characteristics, and the CTU of performing the simplified SAO process is determined. Experimental results show that the proposed algorithm achieves 60% time of SAO process reduction, with only 0.29% luma Bjontegaard delta rate (BD-rate) increases on average.

Design of High-Speed H.265 Sample Adaptive Offset Estimation for Ultra-Hd TV Encoding using Clock Synchronization Code

Sample adaptive offset (SAO) is a recently presented in-circle separating segment in H.265/High Efficiency Video Coding (HEVC). SAO adds to a striking coding effectiveness enhancement; the estimation of SAO parameters commands the multifaceted nature of in-circle sifting in HEVC encoding. Double clock engineering that procedures statistics collection (SC) and parameter decision (PD), the two principle useful squares of SAO estimation, at high-and low-speed timekeepers, separately. Such a technique decreases the general zone by 56% by tending to the heterogeneous information streams of SC and PD. This exploration work endeavors to ad lib the working clock speed by adjusting a double clock synchronizer VLSI structure of H.265 ultra HD encoder and control minimization. To additionally enhance the territory and power productivity, calculation engineering co-improvements are connected, including a coarse range selection (CRS) and an accumulator bit width reduction (ABR). They together may accomplish another 25% territory decrease. The proposed VLSI configuration is fit for handling 8k at 120-outlines/s encoding.

Attention-Based Dual-Scale CNN In-Loop Filter for Versatile Video Coding

As the upcoming video coding standard, Versatile Video Coding (i.e., VVC) achieves up to 30% Bjontegaard delta bit-rate (BD-rate) reduction compared with High Efficiency Video Coding (H.265/HEVC). To eliminate or alleviate different kinds of compression artifacts like blocking, ringing, blurring and contouring effects, three in-loop filters, i.e. de-blocking filter (DBF), sample adaptive offset (SAO) and adaptive loop filter (ALF), have been involved in VVC. Recently, Convolutional Neural Network (CNN) has attracted tremendous attention and shows great potential in many tasks in image processing. In this work, we design a CNN-based in-loop filter as an integrated single-model solution which is adaptive to almost any scenarios in video coding. An architecture named as ADCNN (i.e., Attention based Dual-scale CNN) with an attention based processing block is proposed to reduce artifacts of I frames and B frames, which take advantage of informative priors such as the quantization parameter (QP) and partitioning information. Different from existing CNN-based filtering methods, which are mainly designed for the luma component and may need to train different models for different QPs, the proposed filter is adapted to different QPs and different frame types, and all the components (i.e., both luma and chroma) are processed simultaneously with feature exchange and fusion between components for information supplementary. Experimental results show that the proposed ADCNN filter can achieve 6.54%, 13.27%, 15.72% BD-rate savings for Y, U, V respectively under the all intra configuration and 2.81%, 7.86%, 8.60% BD-rate savings under the random access configuration. It can be used to replace all the conventional in-loop filters and also outperforms them without increase in encoding time.

Parallel In-Loop Filtering in HEVC Encoder on GPU

In-loop filtering is an important part of high efficiency video coding (HEVC), which consists of deblocking filter and sample adaptive offset (SAO) filter. It can not only improve the compression efficiency of HEVC, but also improve the visual quality of the reconstructed videos significantly. However, the high computational complexity hampers its applications for real-time encoding scenarios. In this paper, we propose a parallel strategy for in-loop filtering in HEVC encoder on graphics processing unit (GPU). In the proposed strategy, the pipeline structure for HEVC encoding by parallel processing deblocking filter and SAO on GPU is described first. Then, the joint optimization for deblocking filter and SAO on GPU is detailed by parallel processing of deblocking filter and parallel processing of SAO separately. The joint optimization can improve the degree of parallelism and ease the computational burden of the CPU. Experimental results demonstrate that the proposed method can achieve about 47% (up to 67%) time saving on average for test sequences.

Residual Highway Convolutional Neural Networks for in-loop Filtering in HEVC.

High efficiency video coding (HEVC) standard achieves half bit-rate reduction while keeping the same quality compared with AVC. However, it still cannot satisfy the demand of higher quality in real applications, especially at low bit rates. To further improve the quality of reconstructed frame while reducing the bitrates, a residual highway convolutional neural network (RHCNN) is proposed in this paper for in-loop filtering in HEVC. The RHCNN is composed of several residual highway units and convolutional layers. In the highway units, there are some paths that could allow unimpeded information across several layers. Moreover, there also exists one identity skip connection (shortcut) from the beginning to the end, which is followed by one small convolutional layer. Without conflicting with deblocking filter (DF) and sample adaptive offset (SAO) filter in HEVC, RHCNN is employed as a high-dimension filter following DF and SAO to enhance the quality of reconstructed frames. To facilitate the real application, we apply the proposed method to I frame, P frame, and B frame, respectively. For obtaining better performance, the entire quantization parameter (QP) range is divided into several QP bands, where a dedicated RHCNN is trained for each QP band. Furthermore, we adopt a progressive training scheme for the RHCNN where the QP band with lower value is used for early training and their weights are used as initial weights for QP band of higher values in a progressive manner. Experimental results demonstrate that the proposed method is able to not only raise the PSNR of reconstructed frame but also prominently reduce the bit-rate compared with HEVC reference software.

CABAC-based selective encryption for HEVC using RC6 in different operation modes

This paper proposes a selective encryption (SE) algorithm for the almost recently video compressing technique that is called High-Efficiency Video Coding (HEVC). The improvement of our HEVC SE algorithm is to secure real-time HEVC streaming using low computational overhead, same bit rate and ensuring the video format compliance. This contribution is achieved with low computational RC6 algorithm to encrypt subset of binstrings that is binarized using the bypass binary arithmetic coding mode in the context adaptive binary arithmetic coding (CABAC) process of HEVC. This binstrings subset is the non-zero discrete cosine transform (DCT) coefficients sign bits, motion vector difference (MVD) sign bits, remaining absolute values suffix of DCT, MVD absolute values suffix, Sample adaptive offset (SAO) sign bit, residual size, reference picture index and the delta Quantization Parameter (QP). Also, this paper investigates the performance evaluation of using the RC6 with its operation modes within the proposed HEVC CABAC SE. This investigation is done for choosing the best operation mode for RC6 to be suitable for being used in the HEVC real-time application. The security analysis like histogram analysis, correlation coefficients test and key sensitivity test are presented to make sure the protection of HEVC CABAC SE algorithm against brute force and statistical attacks. The performance analysis results prove that the HEVC CABAC SE is highly secure and can be used in real-time HEVC applications.