Video Coding Research Articles

High-Capacity Video Steganography Based on Chaotic Maps for High-Efficiency Video Coding (HEVC)

High-Capacity Video Steganography Based on Chaotic Maps for High-Efficiency Video Coding (HEVC)

Reference-based In-loop Filter with Robust Neural Feature Transfer for Video Coding

Reference-based In-loop Filter with Robust Neural Feature Transfer for Video Coding

DCMR: Degradation compensation and multi-dimensional reconstruction based pre-processing for video coding

The rapid growth of video data poses a serious challenge to the limited bandwidth. Video coding pre-processing technology can remove coding noise without changing the architecture of the codec. Therefore, it can improve the coding efficiency while ensuring a high degree of compatibility with existing codec. However, the existing pre-processing methods have the problem of feature redundancy, and lack an effective mechanism to recover high-frequency details. In view of these problems, we propose a Degradation Compensation and Multi-dimensional Reconstruction (DCMR) pre-processing method for video coding to improve compression efficiency. Firstly, we develop a degradation compensation model, which aims at filtering the coding noise in the original video and relieving the frame quality degradation caused by transmission. Secondly, we construct a lightweight multi-dimensional feature reconstruction network, which combines residual learning and feature distillation. It aims to enhance and refine the key features related to coding from both spatial and channel dimensions while suppressing irrelevant features. In addition, we design a weighted guided image filter detail enhancement convolution module, which is specifically used to recover the high-frequency details lost in the denoising process. Finally, we introduce an adaptive discrete cosine transform loss to balance coding efficiency and quality. Experimental results demonstrate that compared with the original codec H.266/VVC, the proposed DCMR can achieve BD-rate (VMAF) and BD-rate (MOS) gains by 21.62% and 12.99% respectively on VVC, UVG, and MCL-JCV datasets.

Multi-feature fusion for efficient inter prediction in versatile video coding

Multi-feature fusion for efficient inter prediction in versatile video coding

OptiRet-Net: An Optimized Low-Light Image Enhancement Technique for CV-based Applications in Resource-Constrained Environments

The illumination of images can significantly impact computer-vision applications such as image classification, multiple object detection, and tracking, leading to a significant decline in detection and tracking accuracy. Recent advancements in deep learning techniques have been applied to low-light image enhancement (LLIE) to combat this issue. Retinex theory-based methods following a decomposition-adjustment pipeline for LLIE have performed well in various aspects. Despite their success, current research on Retinex-based deep learning still needs to improve in terms of optimization techniques and complicated convolution connections, which can be computationally intensive for end-device deployment. We propose an Optimized Retinex-based CNN (OptiRet-Net) deep-learning framework to address these challenges for the low-light image enhancement problem. Our results demonstrate that the proposed method outperforms existing state-of-the-art models in terms of full reference metrics with a PSNR of 21.87, SSIM of 0.80, LPIPS of 0.16, and zero reference metrics with a NIQE of 3.4 and PIQE of 56.6. Additionally, we validate our approach using a comprehensive evaluation comprising five datasets and nine prior methods. Furthermore, we assess the efficacy of our proposed model combining low-light multiple object tracking applications using YOLOX and ByteTrack in versatile video coding (VVC/H.266) across various quantization parameters. Our findings reveal that LLIE-enhanced frames surpass their tracking results with a MOTA of 80.6% and a remarkable precision rate of 96%. Our model also achieves minimal file sizes by effectively compressing the enhanced low-light images while maintaining their quality, making it suitable for resource-constrained environments where storage or bandwidth limitations are a concern.

TRRHA: A two-stream re-parameterized refocusing hybrid attention network for synthesized view quality enhancement

In multi-view video systems, the decoded texture video and its corresponding depth video are utilized to synthesize virtual views from different perspectives using the depth-image-based rendering (DIBR) technology in 3D-high efficiency video coding (3D-HEVC). However, the distortion of the compressed multi-view video and the disocclusion problem in DIBR can easily cause obvious holes and cracks in the synthesized views, degrading the visual quality of the synthesized views. To address this problem, a novel two-stream re-parameterized refocusing hybrid attention (TRRHA) network is proposed to significantly improve the quality of synthesized views. Firstly, a global multi-scale residual information stream is applied to extract the global context information by using refocusing attention module (RAM), and the RAM can detect the contextual feature and adaptively learn channel and spatial attention feature to selectively focus on different areas. Secondly, a local feature pyramid attention information stream is used to fully capture complex local texture details by using re-parameterized refocusing attention module (RRAM). The RRAM can effectively capture multi-scale texture details with different receptive fields, and adaptively adjust channel and spatial weights to adapt to information transformation at different sizes and levels. Finally, an efficient feature fusion module is proposed to effectively fuse the extracted global and local information streams. Extensive experimental results show that the proposed TRRHA achieves significantly better performance than the state-of-the-art methods. The source code will be available at https://github.com/647-bei/TRRHA.

Content-Aware Selective Encryption for H.265/HEVC using Deep Hashing Network and Steganography

Existing selective encryption schemes for High Efficiency Video Coding (HEVC) only focus on the encoding characteristics of syntax elements in entropy coding and lack an understanding of the video content. Consequently, a large amount of unnecessary encryption operations are utilized to protect insensitive video frames, resulting in low encryption efficiency. In this paper, we propose a content-aware selective encryption scheme for H.265/HEVC, which encrypts only the groups of pictures (GOPs) containing sensitive content and thus offers high efficiency. In our scheme, a deep hashing network is first adopted to retrieve video frames to determine the content-sensitive GOPs. Then, multiple prediction and residual syntax elements in sensitive GOPs are encrypted using a keystream sequence generated by the hyperchaotic Lorenz system. In addition, the direct current coefficient of each \(4\times 4\) transform block is exchanged with a pseudo-randomly selected non-zero alternating current coefficient to further offer stronger visual distortion. Finally, the sign bits used for marking each GOP-type are reversibly embedded into the encrypted syntax elements to facilitate the decoder to distinguish the GOPs that need to be decrypted. Experimental results indicate that the proposed content-aware selective encryption scheme can efficiently protect sensitive content and is robust against all common attacks. Furthermore, it outperforms other state-of-the-art HEVC selective encryption algorithms in terms of security performance.

Deep video steganography using temporal-attention-based frame selection and spatial sparse adversarial attack

With the development of deep learning-based steganalysis, video steganography is facing with great challenges. To address the insufficient security against steganalysis of existing deep video steganography, given that the video has both spatial and temporal dimensions, this paper proposes a deep video steganography method using temporal frame selection and spatial sparse adversarial attack. In temporal dimension, a stego frame selection module based on temporal attention is designed to calculate the weight of each frame and selects frames with high weights for message and sparse perturbation embedding. In spatial dimension, sparse adversarial perturbations are performed in the selected frames to improve the ability of resisting steganalysis. Moreover, to control the adversarial perturbations’ sparsity flexibly, an intra-frame dynamic sparsity threshold mechanism is designed by using percentile. Experimental results demonstrate that the proposed method effectively enhances the visual quality and security against steganalysis of video steganography and has controllable sparsity of adversarial perturbations.

A robust coverless image-synthesized video steganography based on asymmetric structure

Due to the ability of hiding secret information without modifying image content, coverless image stegonagraphy has gained higher level of security and become a research hot spot. However, in existing methods, the issue of image order disruption during network transmission is overlooked. In this paper, the image-synthesized video carrier is proposed for the first time. The selected images which represent secret information are synthesized to a video in order, thus the image order will not be disrupted during transmission and the effective capacity is greatly increased. Additionally, an asymmetric structure is designed to improve the robustness, in which only the receiver utilizes a robust image retrieval algorithm to restore secret information. Specifically, certain images are randomly selected from a public image database to create multiple coverless image datasets (MCIDs), with each image in a CID mapped to hash sequence. Images are indexed based on secret segments and synthesized into videos. After that, the synthesized videos are sent to the receiver. The receiver decodes the video into frames, identifies the corresponding CID of each frame, retrieves original image, and restores the secret information with the same mapping rule. Experimental results indicate that the proposed method outperforms existing methods in terms of capacity, robustness, and security.

MetaIP: Meta-Network-Based Intra Prediction With Customized Parameters for Video Coding

MetaIP: Meta-Network-Based Intra Prediction With Customized Parameters for Video Coding

Lightweight macro-pixel quality enhancement network for light field images compressed by versatile video coding

Lightweight macro-pixel quality enhancement network for light field images compressed by versatile video coding

Fast CU decision method based on texture characteristics and decision tree for depth map intra-coding

As the demand for higher quality 3D videos continues to grow, the 3D extensions of the Efficient Video Coding (3D-HEVC) standard are gradually unable to meet the needs of users. Versatile Video Coding Standard (H. 266/VVC), as an advanced video coding standard, adopts the nested Multi-Type Tree quadtree (QTMT) partitioning structure that current fast depth map coding unit (CU) partitioning methods cannot apply. Therefore, we have designed a fast intra-frame CU partitioning algorithm for VVC 3D video depth maps. Our proposed algorithm in this article consists of two steps, including two sub-algorithms. First, by analyzing the relationship between image entropy and variance and depth map CU division, we establish a bi-criterion decision algorithm to determine whether the texture complexity of the current CU is low enough to terminate its partitioning process. Then, for CUs that have been determined by the first algorithm to need further partitioning, we use a decision tree model based on Light Gradient Boosting Machine (LGBM) to predict which direction of Rate-Distortion Optimization (RDO) calculation can be skipped, which can avoid some unnecessary RDO calculations in a certain direction. The final experiment demonstrated the effect of the proposed algorithm, which can reduce 47.65% complexity of VVC 3D video intra-coding with negligible 0.23% Bjøntegaard Delta Bitrate (BDBR) increase, superior to other advanced methods.

Information Bottleneck Driven Deep Video Compression-IBOpenDVCW.

Video compression remains a challenging task despite significant advancements in end-to-end optimized deep networks for video coding. This study, inspired by information bottleneck (IB) theory, introduces a novel approach that combines IB theory with wavelet transform. We perform a comprehensive analysis of information and mutual information across various mother wavelets and decomposition levels. Additionally, we replace the conventional average pooling layers with a discrete wavelet transform creating more advanced pooling methods to investigate their effects on information and mutual information. Our results demonstrate that the proposed model and training technique outperform existing state-of-the-art video compression methods, delivering competitive rate-distortion performance compared to the AVC/H.264 and HEVC/H.265 codecs.

Impact of Wireless Network Packet Loss on Real-Time Video Streaming Application: A Comparative Study of H.265 and H.266 Codecs

The transmission of real-time videos over wireless networks is prone to the negative consequences of packet loss and delay, which can have a potential effect on the video quality during streaming. These impairments can lead to interruptions, buffering, and degradation of visual and auditory elements, resulting in an unsatisfactory user experience. In this paper, we aim to address the challenges associated with packet loss and delay parameters in wireless networks, and propose an approach to alleviate their impact on real-time video transmission. The proposed approach involves utilizing the H.265/H.266 video coding standards. For Versatile Video Coding (VVC), a patch support for VVdeC and VVenC to FFmpeg (Fast Forward Moving Picture Expert Group) is added. As a result, FFmpeg is used to encode, stream and decode all videos. Raw videos of 2K qualities are encoded based on the adaptive quantization (QP) for the above-mentioned codecs. By selecting optimal transmission data based on various network conditions, this approach enhances the Quality of Experience (QoE) for end-users while minimizing resource usage in the wireless network. Furthermore, the proposed approach selects the codec standards according to their bitrates and frame rates. Simulation results indicate that the proposed approach has a significant improvement for real-time video streaming over wireless networks to satisfy the end user experience. The approach also outperforms other related work by gaining a PSNR of +12 dB for H.265 and +13 dB for H.266 when the network packet loss is 1%.

Advanced fine-tuning procedures to enhance DNN robustness in visual coding for machines

Video Coding for Machines (VCM) is gaining momentum in applications like autonomous driving, industry manufacturing, and surveillance, where the robustness of machine learning algorithms against coding artifacts is one of the key success factors. This work complements the MPEG/JVET standardization efforts in improving the resilience of deep neural network (DNN)-based machine models against such coding artifacts by proposing the following three advanced fine-tuning procedures for their training: (1) the progressive increase of the distortion strength as the training proceeds; (2) the incorporation of a regularization term in the original loss function to minimize the distance between predictions on compressed and original content; and (3) a joint training procedure that combines the proposed two approaches. These proposals were evaluated against a conventional fine-tuning anchor on two different machine tasks and datasets: image classification on ImageNet and semantic segmentation on Cityscapes. Our joint training procedure is shown to reduce the training time in both cases and still obtain a 2.4% coding gain in image classification and 7.4% in semantic segmentation, whereas a slight increase in training time can bring up to 9.4% better coding efficiency for the segmentation. All these coding gains are obtained without any additional inference or encoding time. As these advanced fine-tuning procedures are standard-compliant, they offer the potential to have a significant impact on visual coding for machine applications.

Multi-Type Self-Attention-Based Convolutional-Neural-Network Post-Filtering for AV1 Codec

Over the past few years, there has been substantial interest and research activity surrounding the application of Convolutional Neural Networks (CNNs) for post-filtering in video coding. Most current research efforts have focused on using CNNs with various kernel sizes for post-filtering, primarily concentrating on High-Efficiency Video Coding/H.265 (HEVC) and Versatile Video Coding/H.266 (VVC). This narrow focus has limited the exploration and application of these techniques to other video coding standards such as AV1, developed by the Alliance for Open Media, which offers excellent compression efficiency, reducing bandwidth usage and improving video quality, making it highly attractive for modern streaming and media applications. This paper introduces a novel approach that extends beyond traditional CNN methods by integrating three different self-attention layers into the CNN framework. Applied to the AV1 codec, the proposed method significantly improves video quality by incorporating these distinct self-attention layers. This enhancement demonstrates the potential of self-attention mechanisms to revolutionize post-filtering techniques in video coding beyond the limitations of convolution-based methods. The experimental results show that the proposed network achieves an average BD-rate reduction of 10.40% for the Luma component and 19.22% and 16.52% for the Chroma components compared to the AV1 anchor. Visual quality assessments further validated the effectiveness of our approach, showcasing substantial artifact reduction and detail enhancement in videos.

Behavioral video coding analysis of chronic morphine administration in rats.

The present study assessed the behavior of morphine-addicted rats using behavioral video coding technology, to evaluate effective methods for identifying morphine addiction. Rats were divided into a control group (n=15) and a morphine addiction group (n=15). The morphine addiction model was established with a 14-day increasing dose scheme, confirmed using a conditional place preference (CPP) experiment. After successful modeling, the rats' behavior was recorded for 12 h, then coded and analyzed using Observer XT behavior analysis software. Compared with the control group, morphine-addicted rats showed increased heat pain tolerance time (P=0.039) and spent more time in the white box during the CPP experiment (P<0.001). Video coding analysis revealed significant behavioral changes in morphine-addicted rats compared to controls. In addition to being lighter, morphine-addicted rats showed decreased water intake, reduced licking of forelimbs and hind limbs, and altered sleeping posture (sleeping curled up) during the day (all P<0.05). In conclusion, chronic morphine administration in rats leads to distinctive behavioral changes, including decreased licking frequency, reduced water intake and altered sleep posture. Video coding analysis, as a safe and non-invasive method, may provide a convenient and efficient approach for studying morphine addiction in rats.

Type and Distribution of Gross Motor Activity During Physical Therapy in Young Children With Cerebral Palsy.

Physical therapists routinely deliver and prescribe motor practice to improve function. The ability to select optimal practice regimens is limited by a current lack of detail in the measurement of motor practice. The objective of this study was to quantify the type, amount, and timing of gross motor practice during physical therapist sessions. A secondary video coding analysis of physical therapist sessions from the iMOVE clinical trial (NCT02340026) in young children with cerebral palsy (CP) was conducted. The 37 children who completed the treatment phase were included (mean age = 22.1months). Children could initiate pulling to stand but were unable to walk. Videos of randomly selected therapy sessions were coded for gross motor activity (422 videos total). The 10 gross motor activity codes included lying, sitting, four point, crawling, kneeling, knee walking, standing, walking, transitions between floor postures, and transitions to/from an upright posture. Twenty percent of each video was double coded for reliability. Time per session, number of bouts, and median time per bout were calculated for each gross motor activity and for 2 aggregate measures: movement time and upright time. Participants spent more than half of therapy time in sitting and standing combined (60.3%). Transitions occurred more frequently than any other motor activity (49.3 total transitions per session). Movement time accounted for 16.3% of therapy time. Upright time accounted for 53.3% of therapy time. Critical practice time to gain motor skill is not equivalent to chronological time or time spent in therapy. Toddlers with CP spent a small amount of therapy time moving. Future work should explore the relations between motor practice and rehabilitation outcomes. Physical therapists are ideally suited to detail the content of motor practice and ultimately to prescribe optimal patterns of motor practice. We report the characteristics of gross motor practice during therapy in children with CP.

Cluster-based cooperative fog caching for scalable coded videos of multiple content providers

Efficient content caching in video streaming services is important for improving user experience as well as reducing network bandwidth consumption. Fog caching combined with scalable video coding (SVC) has the potential to significantly improve caching efficiency. However, challenges such as the limited storage capacity of fog nodes and determining the optimal number of SVC layers must be overcome for their effective adoption. This becomes more complicated with multiple content providers requiring shared cache resources. To the best of our knowledge, no existing research has simultaneously tackled all these aspects. In this paper, we present Cluster-based Cooperative Fog Caching (CCo-Fog), a holistic caching strategy that enables multiple content providers to share the scarce storage of fog nodes in a multi-tier fog network to judiciously cache SVC videos in a cooperative manner. In particular, CCo-Fog consists of a cluster-based storage partitioning method and tier-wise cooperative content placement policies. The partitioning method distributes the storage of each fog node to multiple content providers for users clustered based on their population density and their proximity to fog nodes. The content placement policies determine the optimal number of SVC layers of each video for different tiers of the fog network by solving a latency-aware content placement optimization problem. Our evaluations on a real-world dataset and various configurations demonstrate the efficacy of CCo-Fog, showing a reduction in latency by about 60% and an increase in fog hit ratio by about 20% on average, compared to state-of-the-art caching strategies.

Exploiting Bidirectional Quality Impulse for Reference Picture Resampled Gaming Video Coding

Exploiting Bidirectional Quality Impulse for Reference Picture Resampled Gaming Video Coding