Video Compression Research Articles

A Coding Framework and Benchmark Towards Low-Bitrate Video Understanding.

Video compression is indispensable to most video analysis systems. Despite saving the transportation bandwidth, it also deteriorates downstream video understanding tasks, especially at low-bitrate settings. To systematically investigate this problem, we first thoroughly review the previous methods, revealing that three principles, i.e., task-decoupled, label-free, and data-emerged semantic prior, are critical to a machine-friendly coding framework but are not fully satisfied so far. In this paper, we propose a traditional-neural mixed coding framework that simultaneously fulfills all these principles, by taking advantage of both traditional codecs and neural networks (NNs). On one hand, the traditional codecs can efficiently encode the pixel signal of videos but may distort the semantic information. On the other hand, highly non-linear NNs are proficient in condensing video semantics into a compact representation. The framework is optimized by ensuring that a transportation-efficient semantic representation of the video is preserved w.r.t. the coding procedure, which is spontaneously learned from unlabeled data in a self-supervised manner. The videos collaboratively decoded from two streams (codec and NN) are of rich semantics, as well as visually photo-realistic, empirically boosting several mainstream downstream video analysis task performances without any post-adaptation procedure. Furthermore, by introducing the attention mechanism and adaptive modeling scheme, the video semantic modeling ability of our approach is further enhanced. Fianlly, we build a low-bitrate video understanding benchmark with three downstream tasks on eight datasets, demonstrating the notable superiority of our approach. All codes, data, and models will be open-sourced for facilitating future research.

Spatio-temporal feature learning for enhancing video quality based on screen content characteristics

With the rising demands for remote desktops and online meetings, screen content videos have drawn significant attention. Different from natural videos, screen content videos often exhibit scene switches where the content abruptly changes from one frame to the next. These scene switches result in obvious distortions in compressed videos. Besides, frame freezing, where the content remains unchanged for a certain duration, is also very common in screen content videos. Existing alignment-based models struggle to effectively enhance scene switch frames and lack efficiency when dealing with frame freezing situations. Therefore, we propose a novel alignment-free method that effectively handles both scene switches and frame freezing. In our approach, we develop a spatial and temporal feature extraction module that compresses and extracts spatio-temporal information from three groups of frame inputs. This enables efficient handling of scene switches. In addition, an edge aware block is proposed for extracting edge information, which guides the model to focus on restoring the high-frequency components in frame freezing situations. The fusion module is then designed to adaptively fuse the features from three groups, considering different positions of video frames, to enhance frames during scene switch and frame freezing scenarios. Experimental results demonstrate the significant advancements achieved by the proposed edge aware with spatio-temporal information fusion network (EAST) in enhancing the quality of compressed videos, surpassing the current state-of-the-art methods.

Application of Interactive Object Model in Sports Teaching

Traditional physical education is just repeated training without quantitative teaching and training data results. Multimedia information analysis and retrieval technology can make repeated research on the content of competition or training, and increase the quantitative analysis of training indicators. Therefore, this paper proposes an interactive compressed domain key video object model to realize the design and production of physical education video, so as to quantify the effect of physical education. This paper discusses the interactive design and application of PE teaching video from three aspects: static, dynamic, and semantic. The interactive object model proposed in this paper can formulate corresponding strategies for different key sports video objects. The distortion rate of the complete key video object is 2.3%, which is 0.09% lower than the distortion rate of the Kim2002 scheme. From a usage perspective, through the scheme proposed in this article, coaches can obtain teaching videos with simple initial extreme values on an interactive interface, and the distortion rate of the video object is lower than that of traditional schemes.

Video compression and optimization technologies - review

Video compression and optimization technologies - review

A fast and efficient data reuse scheme for HEVC Integer Motion Estimation hardware architecture

ABSTRACT High-Efficiency Video Coding (HEVC) is a video compression standard designed to improve video compression efficiency compared to its predecessor, Advanced Video Coding (AVC). HEVC provides approximately twice the compression efficiency compared to the AVC. In the HEVC system, the most computational complexity module is the Motion Estimation (ME). ME module helps reduce redundancy by compensating for motion during video compression. However, the computational complexity makes it a bottleneck in the design of high-resolution video encoders. This paper proposes an efficient architecture for the Integer ME (IME) module of the HEVC encoder. The proposed architecture introduces an efficient memory usage scheme to support the Full Search motion search algorithm. The full pipeline architecture includes 4096 Processing Elements (PE) and an adder tree to compute the Sum of Absolute Difference (SAD) for every Prediction Unit (PU) partition. The proposed architecture was designed and implemented on Xilinx Virtex-7 XC7VX550T FPGA. Our design achieved up to 275 FPS and approximately 10 FPS at 4 K video for the Search Regions of 32 × 16 and 128 × 128 pixels, respectively.

Face forgery video detection based on expression key sequences

In order to minimize additional computational costs in detecting forged videos, and enhance detection accuracy, this paper employs dynamic facial expression sequences as key sequences, replacing original video sequences as inputs for the detection model. A spatio-temporal dual-branch detection network is designed based on the visual Transformer architecture. Specifically, this process involves three steps. Firstly, dynamic facial expression sequences are localized as key sequences using optical flow difference algorithms. Subsequently, the spatial branch network employs the focal self-attention mechanism to focus on dynamic features of expression-relevant regions and uses Factorization Machines to facilitate feature interaction among multiple key sequences. Meanwhile, the temporal branch network concentrates on learning the temporal inconsistency of optical flow differences between adjacent frames. Finally, a binary classification linear SVM combines the Softmax values from the two branch networks to provide the ultimate detection outcome. Experimental results on the Faceforensics++ dataset demonstrate: (a) replacing whole video sequences with facial expression key sequences effectively reduces training and detection time by nearly 80% and 90%, respectively; (b) compared to state-of-the-art methods involving random sequence/frame extraction and key frame extraction based on video compression techniques, the proposed approach in this paper presents a more competitive detection accuracy.

ColorVideoVDP: A visual difference predictor for image, video and display distortions

ColorVideoVDP is a video and image quality metric that models spatial and temporal aspects of vision for both luminance and color. The metric is built on novel psychophysical models of chromatic spatiotemporal contrast sensitivity and cross-channel contrast masking. It accounts for the viewing conditions, geometric, and photometric characteristics of the display. It was trained to predict common video-streaming distortions (e.g., video compression, rescaling, and transmission errors) and also 8 new distortion types related to AR/VR displays (e.g., light source and waveguide non-uniformities). To address the latter application, we collected our novel XR-Display-Artifact-Video quality dataset (XR-DAVID), comprised of 336 distorted videos. Extensive testing on XR-DAVID, as well as several datasets from the literature, indicate a significant gain in prediction performance compared to existing metrics. ColorVideoVDP opens the doors to many novel applications that require the joint automated spatiotemporal assessment of luminance and color distortions, including video streaming, display specification, and design, visual comparison of results, and perceptually-guided quality optimization. The code for the metric can be found at https://github.com/gfxdisp/ColorVideoVDP.

Efficient Key Frame Extraction from Videos Using Convolutional Neural Networks and Clustering Techniques

One of the most reliable information sources is video, and in recent years, online and offline video consumption has increased to an unprecedented degree. One of the main difficulties in extracting information from videos is that unlike images, where information can be gleaned from a single frame, a viewer must watch the entire video in order to comprehend the context. In this work, we try to use various algorithmic techniques, such as deep neural networks and local features, in conjunction with a variety of clustering techniques, to find an efficient method of extracting interesting key frames from videos to summarize them. Video summarization plays a major role in video indexing, browsing, compression, analysis, and many other domains. One of the fundamental elements of video structure analysis is key frame extraction, which pulls significant frames out of the movie. An important frame from a video that may be used to summarize videos is called a key frame. We provide a technique that leverages convolutional neural networks in our suggested model, static video summarization, and key frame extraction from movies.

Multi-domain awareness for compressed deepfake videos detection over social networks guided by common mechanisms between artifacts

The viral spread of massive deepfake videos over social networks has caused serious security problems. Despite the remarkable advancements achieved by existing deepfake detection algorithms, deepfake videos over social networks are inevitably influenced by compression factors. This causes deepfake detection performance to be limited by the following challenging issues: (a) interfering with compression artifacts, (b) loss of feature information, and (c) aliasing of feature distributions. In this paper, we analyze the common mechanism between compression artifacts and deepfake artifacts, revealing the structural similarity between them and providing a reliable theoretical basis for enhancing the robustness of deepfake detection models against compression. Firstly, based on the common mechanism between artifacts, we design a frequency domain adaptive notch filter to eliminate the interference of compression artifacts on specific frequency bands. Secondly, to reduce the sensitivity of deepfake detection models to unknown noise, we propose a spatial residual denoising strategy. Thirdly, to exploit the intrinsic correlation between feature vectors in the frequency domain branch and the spatial domain branch, we enhance deepfake features using an attention-based feature fusion method. Finally, we adopt a multi-task decision approach to enhance the discriminative power of the latent space representation of deepfakes, achieving deepfake detection with robustness against compression. Extensive experiments show that compared with the baseline methods, the detection performance of the proposed algorithm on compressed deepfake videos has been significantly improved. In particular, our model is resistant to various types of noise disturbances and can be easily combined with baseline detection models to improve their robustness.

Spatial Decomposition and Temporal Fusion Based Inter Prediction for Learned Video Compression

Spatial Decomposition and Temporal Fusion Based Inter Prediction for Learned Video Compression

Playful Trauma: TikTok Creators and the Use of the Platformed Body in Times of War

Amid the Russian invasion of Ukraine, TikTok has emerged as a pivotal platform, where creators utilize its compressed video formats to mediate the harsh realities of war zones. In this article, we examine 97 videos produced by 12 Ukrainian and Russian TikTok creators in response to the 2022 war in Ukraine. We focus on the playful embodiment of trauma using digital ethnography, analyzing creators’ practices and their interconnected memetic communication on TikTok. We identified the use of play through three dynamic practices where platforms, bodies, and trauma converge: (1) the utilization of POV (point-of-view) aesthetic in shareable templates to convey the realities of war from engaging first-person perspectives; (2) the incorporation of dance as a means of embodied creative expression and amplification of trauma; and (3) the harnessing of platform features to facilitate whimsical dialogues with followers about life under war. We argue that playfulness is entrenched in and enacted through platform vernacular modes, driving creators to forge communal ties during adversities and shape the ongoing representation of trauma and its accelerated visibility in digital spaces. We present the concept of playful trauma as a framework for understanding the structural dissonance that arises when creators utilize their bodies alongside platform-specific humorous, ironic, or subversive dialects to perform and amplify the gravity of trauma. This tension provides a unique space for creators to convert their shared experiences of grief and resilience into participatory coping mechanisms. In doing so, they subject and harness their playful platformed body as both the medium and the message, documenting injustices, bearing witness, and galvanizing crowds into action during times of war.

On Alpha-Expansion-Based Graph-Cut Optimization for Decoder-Side Depth Estimation

In order to achieve high realism an acceptable level of user experience in immersive videos, it is crucial to provide both the best possible quality of depth maps and minimize computational time. In this paper, we propose a new approach to the decoder-side depth estimation (DSDE) problem, which uses the hierarchical alpha-expansion algorithm with additional improvements for the estimation designed to be more robust to compressed video artifacts and limited computational resources. As shown by the experimental results, the proposal simultaneously results in reduction of computation time of the estimation process (by almost 40%) and an improvement of quality of estimated depth maps. The increased quality is demonstrated by more than 6% Bjøntegaard delta gain compared to the Moving Picture Experts Group (MPEG) immersive video state-of-the-art DSDE method.

N-DEPTH: Neural Depth Encoding for Compression-Resilient 3D Video Streaming

Recent advancements in 3D data capture have enabled the real-time acquisition of high-resolution 3D range data, even in mobile devices. However, this type of high bit-depth data remains difficult to efficiently transmit over a standard broadband connection. The most successful techniques for tackling this data problem thus far have been image-based depth encoding schemes that leverage modern image and video codecs. To our knowledge, no published work has directly optimized the end-to-end losses of a depth encoding scheme sandwiched around a lossy image compression codec. We present N-DEPTH, a compression-resilient neural depth encoding method that leverages deep learning to efficiently encode depth maps into 24-bit RGB representations that minimize end-to-end depth reconstruction errors when compressed with JPEG. N-DEPTH’s learned robustness to lossy compression expands to video codecs as well. Compared to an existing state-of-the-art encoding method, N-DEPTH achieves smaller file sizes and lower errors across a large range of compression qualities, in both image (JPEG) and video (H.264) formats. For example, reconstructions from N-DEPTH encodings stored with JPEG had dramatically lower error while still offering 29.8%-smaller file sizes. When H.264 video was used to target a 10 Mbps bit rate, N-DEPTH reconstructions had 85.1%-lower root mean square error (RMSE) and 15.3%-lower mean absolute error (MAE). Overall, our method offers an efficient and robust solution for emerging 3D streaming and 3D telepresence applications, enabling high-quality 3D depth data storage and transmission.

Video Multi-Scale-Based End-to-End Rate Control in Deep Contextual Video Compression

In recent years, video data have increased in size, which results in enormous transmission pressure. Rate control plays an important role in stabilizing video stream transmissions by balancing the rate and distortion of video compression. To achieve high-quality videos through low-bandwidth transmission, video multi-scale-based end-to-end rate control is proposed. First, to reduce video data, the original video is processed using multi-scale bicubic downsampling as the input. Then, the end-to-end rate control model is implemented. By fully using the temporal coding correlation, a two-branch residual-based network and a two-branch regression-based network are designed to obtain the optimal bit rate ratio and Lagrange multiplier λ for rate control. For restoring high-resolution videos, a hybrid efficient distillation SISR network (HEDS-Net) is designed to build low-resolution and high-resolution feature dependencies, in which a multi-branch distillation network, a lightweight attention LCA block, and an upsampling network are used to transmit deep extracted frame features, enhance feature expression, and improve image detail restoration abilities, respectively. The experimental results show that the PSNR and SSIM BD rates of the proposed multi-scale-based end-to-end rate control are −1.24% and −0.50%, respectively, with 1.82% rate control accuracy.

JND-based multi-module cooperative perceptual optimization for HEVC

Since the just noticeable distortion (JND) reflects the tolerance limit of human visual system (HVS) to coding distortion directly, JND-based perceptual video coding (PVC) plays an increasingly significant role in video compression with the developing explosion of video data. In this paper, we focus on the coupling effect among coding modules and provide a JND-based multi-module cooperative perceptual optimization (JMCPO) scheme for HEVC. The main contribution of the proposed JMCPO scheme includes the following three aspects. (1) Based on quantization distortion estimation, an adaptive perceptual quantization scheme is proposed using the binary search approach on the premise of that the quantization distortion is infinitely close to the estimated JND threshold. (2) The coupling effect among coding modules is first analyzed and a novel perceptual residual filtering scheme is presented based on statistic analysis of coupling strength. (3) The JMCPO scheme is finally developed through collaborative optimization of residual filtering, quantization and rate–distortion optimization. Experimental results show that the proposed JMCPO scheme saves more bitrate with better subjective and objective qualities.

A Comprehensive Literature Review on Image and Video Compression: Trends, Algorithms, and Techniques

A Comprehensive Literature Review on Image and Video Compression: Trends, Algorithms, and Techniques

Learned Video Compression with Adaptive Temporal Prior and Decoded Motion-aided Quality Enhancement

Learned video compression has drawn great attention and shown promising compression performance recently. In this article, we focus on the two components in the learned video compression framework, the conditional entropy model and quality enhancement module, to improve compression performance. Specifically, we propose an adaptive spatial-temporal entropy model for image, motion, and residual compression, which introduces a temporal prior to reduce temporal redundancy of latents and an additional modulated mask to evaluate the similarity and perform refinement. In addition, a quality enhancement module is proposed for predicted frame and reconstructed frame to improve frame quality and reduce the bitrate cost of residual coding. The module reuses decoded optical flow as a motion prior and utilizes deformable convolution to mine high-quality information from the reference frame in a bit-free manner. The two proposed coding tools are integrated into a pixel-domain residual coding–based compression framework to evaluate their effectiveness. Experimental results demonstrate that our framework achieves competitive compression performance in the low-delay scenario compared with recent learning-based methods and traditional H.265/HEVC in terms of Peak Signal-to-Noise Ratio (PSNR) and Multi-Scale Structural Similarity Index (MS-SSIM). The code is available at OpenLVC.

High Efficiency Deep-learning Based Video Compression

Although deep learning technique has achieved significant improvement on image compression, but its advantages are not fully explored in video compression, which leads to the performance of deep-learning-based video compression (DLVC) is obviously inferior to that of hybrid video coding framework. In this article, we proposed a novel network to improve the performance of DLVC from its most important modules, including Motion Process (MP), Residual Compression (RC), and Frame Reconstruction (FR). In MP, we design a split second-order attention and multi-scale feature extraction module to fully remove the warping artifacts from multi-scale feature space and pixel space, which can help reduce the distortion in the following process. In RC, we propose a channel selection mechanism to gradually drop redundant information while preserving informative channels for a better rate-distortion performance. Finally, in FR, we introduce a residual multi-scale recurrent network to improve the quality of the current reconstructed frame by progressively exploiting temporal context information between it and its several previous reconstructed frames. Extensive experiments are conducted on the three widely used video compression datasets (HEVC, UVG, and MCL-JVC), and the performance demonstrates the superiority of our proposed approach over the state-of-the-art methods.

Violence detection in compressed video

Violence detection in compressed video

Tensor Golub–Kahan method based on Einstein product

The Singular Value Decomposition (SVD) of matrices is a widely used tool in scientific computing. In many applications of machine learning, data analysis, signal and image processing, the large datasets are structured into tensors, for which generalizations of SVD have already been introduced, for various types of tensor–tensor products. In this paper, we present innovative methods for approximating this generalization of SVD to tensors in the framework of the Einstein tensor product. These singular elements are called singular values and singular tensors respectively. The proposed method uses the tensor Lanczos bidiagonalization applied to the Einstein product. In most applications, as in the matrix case, the extremal singular values are of special interest. To enhance the approximation of the largest or the smallest singular triplets (singular values and left and right singular tensors), a restarted method based on Ritz augmentation is proposed. Numerical results are proposed to illustrate the effectiveness of the presented method. In addition, applications to video compression and facial recognition are presented.