Adaptive Video Coding Research Articles

An End-to-End Video Coding Method via Adaptive Vision Transformer

Deep learning-based video coding methods have demonstrated superior performance compared to classical video coding standards in recent years. The vast majority of the existing deep video coding (DVC) networks are based on convolutional neural networks (CNNs), and their main drawback is that since CNNs are affected by the size of the receptive field, they cannot effectively handle long-range dependencies and local detail recovery. Therefore, how to better capture and process the overall structure as well as local texture information in the video coding task is the core issue. Notably, the transformer employs a self-attention mechanism that captures dependencies between any two positions in the input sequence without being constrained by distance limitations. This is an effective solution to the problem described above. In this paper, we propose end-to-end transformer-based adaptive video coding (TAVC). First, we compress the motion vector and residuals through a compression network built on the vision transformer (ViT) and design the motion compensation network based on ViT. Second, based on the requirement of video coding to adapt to different resolution inputs, we introduce a position encoding generator (PEG) as adaptive position encoding (APE) to maintain its translation invariance across different resolution video coding tasks. The experiment shows that for multiscale structural similarity index measurement (MS-SSIM) metrics, this method exhibits significant performance gaps compared to conventional engineering codecs, such as [Formula: see text], [Formula: see text], and VTM-15.2. We also achieved a good performance improvement compared to the CNN-based DVC methods. In the case of peak signal-to-noise ratio (PSNR) evaluation metrics, TAVC also achieves good performance.

Adaptive Video Coding Framework with Spatial-Temporal Fusion for Optimized Streaming in Next-Generation Networks

Predicting future frames and improving inter-frame prediction are ongoing challenges in the field of video streaming. By creating a novel framework called STreamNet (Spatial-Temporal Video Coding), fusing bidirectional long short-term memory with temporal convolutional networks, this work aims to address the issue at hand. The development of STreamNet, which combines spatial hierarchies with local and global temporal dependencies in a seamless manner, along with sophisticated preprocessing, attention mechanisms, residual learning, and effective compression techniques, is the main contribution. Significantly, STreamNet claims to provide improved video coding quality and efficiency, making it suitable for next-generation networks. STreamNet has the potential to provide reliable and optimal streaming in high-demand network environments, as shown by preliminary tests that show a performance advantage over existing methods.

UE-TV: User-Centric Energy-Efficient HDTV Broadcast over LTE and Wi-Fi

This paper presents an innovative multi-faceted architecture, called UE-TV, for user-centric, revenue aware, and energy-efficient TV broadcast. Scalable high-efficiency video coded high definition television (HDTV) content is broadcast over LTE multicast/broadcast single frequency network (MBSFN) along with the available Wi-Fi access points (APs). The proposed framework adaptively encodes the TV content and allocates radio resource based on current network. Stackelberg two-stage game theoretic approach discerns an optimal transmit power at the LTE base stations (eNodeBs) and the proportion of subscribers that are respectively served by eNodeBs and available Wi-Fi APs. Varied user equipment (UE) resolutions, users' energy/price sensitivities, and channel conditions govern the service options and user satisfaction. Our analysis and simulations show that, in comparison with the broadcast schemes over MBSFN without or with adaptive video coding, UE-TV framework significantly enhances the user satisfaction via optimized price/quality trade-off as well as energy saving at the eNodeBs and UEs.

HEVC-Based Perceptually Adaptive Video Coding Using a DCT-Based Local Distortion Detection Probability Model.

Discrete Cosine Transform (DCT)-based just noticeable difference (JND) profiles have widely been applied into human perception-based video coding in order to reduce perceptual redundancy, which is one of the main goals of perceptual video coding (PVC). However, there are two problems for this approach: 1) the JND value of each transform coefficient is estimated for a fixed-sized DCT kernel (e.g., 8 × 8), but flexible coding structures with variable-sized transform units have been utilized in standard video coding frameworks, such high efficiency video coding (HEVC) and 2) the DCT transform coefficients are suppressed by the amounts of JND values for the removal of perceptual redundancy, but the DCT transform coefficients of residues are not sufficiently suppressed due to many small transform coefficient values in mid- and high-frequency regions below the JND values. In order to solve these problems, we propose a more generalized visibility model in the DCT domain, called the DCT-based local distortion detection probability (LDDP) model that can estimate a degree of distortion visibility for any distribution of the transform coefficients of any sized DCT kernel for residues. Furthermore, we propose an HEVC-compliant LDDP-based PVC scheme where transform coefficients are sufficiently suppressed based on the LDDP model. The proposed PVC scheme is implemented in the HEVC Test Model (HM 11.0) reference software to show the effectiveness of the LDDP-based PVC scheme. Objective and subjective tests for encoded test sequences are performed. The experimental results show that the LDDP-based PVC scheme achieves a significant performance improvement of bitrate reduction at the similar visual quality levels compared with the original HM 11.0.

Adaptive General Scale Interpolation Based on Weighted Autoregressive Models

The autoregressive (AR) model has been widely used in signal processing for its effective estimation, especially in image processing. Many dedicated $2\times $ interpolation algorithms adopt the AR model to describe the strong correlation between low-resolution (LR) pixels and high-resolution (HR) pixels. However, these AR model-based methods closely depend on the fixed relative position between LR pixels and HR pixels that are nonexistent in the general scale interpolation. In this paper, we present an adaptive general scale interpolation algorithm that is capable of arbitrary scaling factors considering the nonstationarity of natural images. Different from other dedicated $2\times $ interpolation methods, the proposed AR terms are modeled by pixels with their adjacent unknown HR neighbors. To compensate for the information loss caused by mismatches of AR models, we consider a weighting scheme suitable for general scale situations based on the pixel similarity to increase accuracy of the estimation. Comprehensive experiments demonstrate the effectiveness of the proposed method on general scaling factors. The maximum gain of peak signal-to-noise ratio is 2.07 dB compared with segment adaptive gradient angle in $1.5 \times $ enlargements. To evaluate the performance in resolution adaptive video coding, we have also tested our method on Joint Scalable Video Model codec and obtained better subjective quality and rate-distortion performance.