Video Encoder Research Articles

Towards GPU-enabled serverless cloud edge platforms for accelerating HEVC video coding

Multimedia streaming has become integral to modern living, reshaping entertainment consumption, information access, and global engagement. The ascent of cloud computing, particularly serverless architectures, plays a pivotal role in this transformation, offering dynamic resource allocation, parallel execution, and automatic scaling-especially beneficial in HTTP Adaptive Streaming (HAS) applications. This study presents an event-driven serverless cloud edge platform with graphics processing units (GPUs), managed by Knative, tailored for video encoding. Two implementations of the High Efficiency Video Coding (HEVC) codec have been encapsulated in the functions: HEVC NVENC (Nvidia Encoder), that uses GPU acceleration, and x265 that only uses CPUs. Experiments focused on measuring the impact of replica requested resources on cold start, scalability and resource consumption with different allocated resources on slim and fat virtual machines (VMs). The best results are obtained when four slim replicas of the functions are deployed on a fat VM with a 8.4% reduction in encoding time for x265 and a 15.2% improvement for HEVC NVENC compared with other deployment scenarios. Comparatively, HEVC NVENC encoding is 8.3 times faster than x265. In multiresolution scenarios, GPU encoding drastically reduces segment encoding time by a factor of 12.4 between non-GPU and GPU-accelerated. These findings are important for live streaming applications where low latency is critical at all stages of the streaming process.

Energy Consumption and Carbon Emissions of Modern Software Video Encoders

Energy Consumption and Carbon Emissions of Modern Software Video Encoders

TeachText: CrossModal text-video retrieval through generalized distillation

In recent years, considerable progress on the task of text-video retrieval has been achieved by leveraging large-scale pretraining on visual and audio datasets to construct powerful video encoders. By contrast, despite the natural symmetry, the design of effective algorithms for exploiting large-scale language pretraining remains under-explored. In this work, we investigate the design of such algorithms and propose a novel generalized distillation method, TeachText, which leverages complementary cues from multiple text encoders to provide an enhanced supervisory signal to the retrieval model. TeachText yields significant gains on a number of video retrieval benchmarks without incurring additional computational overhead during inference and was used to produce the winning entry in the Condensed Movie Challenge at ICCV 2021. We show how TeachText can be extended to include multiple video modalities, reducing computational cost at inference without compromising performance. Finally, we demonstrate the application of our method to the task of removing noisy descriptions from the training partitions of retrieval datasets to improve performance. Code and data can be found at https://www.robots.ox.ac.uk/~vgg/research/teachtext/.

Preprocessing for Multi-Dimensional Enhancement and Reconstruction in Neural Video Compression

The surge in ultra-high-definition video content has intensified the demand for advanced video compression techniques. Video encoding preprocessing can improve coding efficiency while ensuring a high degree of compatibility with existing codecs. Existing video encoding preprocessing methods are limited in their ability to fully exploit redundant features in video data and recover high-frequency details, and their network architectures often lack compatibility with neural video encoders. To addressing these challenges, we propose a Multi-Dimensional Enhancement and Reconstruction (MDER) preprocessing method to improve the efficiency of deep learning-based neural video encoders. Firstly, our approach integrates a degradation compensation module to mitigate encoding noise and boost feature extraction efficiency. Secondly, a lightweight fully convolutional neural network is employed, which utilizes residual learning and knowledge distillation to refine and suppress irrelevant features across spatial and channel dimensions. Furthermore, to maximize the use of redundant information, we incorporate Dense Blocks, which can enhance and reconstruct important features in the video data during preprocessing. Finally, the preprocessed frames are then mapped from pixel space to feature space through the Dense Feature-Enhanced Video Compression (DFVC) module, which improves motion estimation and compensation accuracy. The experimental results demonstrate that, compared to neural video encoders, the MDER method can reduce bits per pixel (Bpp) by 0.0714 and 0.0536 under equivalent PSNR and MS-SSIM conditions, respectively. These results demonstrate significant improvements in compression efficiency and reconstruction quality, highlighting the effectiveness of the MDER preprocessing method and its compatibility with neural video codec workflows.

Fast encryption of color medical videos for Internet of Medical Things

Fast encryption of color medical videos for Internet of Medical Things

Deep learning-guided video compression for machine vision tasks

In the video compression industry, video compression tailored to machine vision tasks has recently emerged as a critical area of focus. Given the unique characteristics of machine vision, the current practice of directly employing conventional codecs reveals inefficiency, which requires compressing unnecessary regions. In this paper, we propose a framework that more aptly encodes video regions distinguished by machine vision to enhance coding efficiency. For that, the proposed framework consists of deep learning-based adaptive switch networks that guide the efficient coding tool for video encoding. Through the experiments, it is demonstrated that the proposed framework has superiority over the latest standardization project, video coding for machine benchmark, which achieves a Bjontegaard delta (BD)-rate gain of 5.91% on average and reaches up to a 19.51% BD-rate gain.

Zero-shot prompt-based video encoder for surgical gesture recognition.

In order to produce a surgical gesture recognition system that can support a wide variety of procedures, either a very large annotated dataset must be acquired, or fitted models must generalize to new labels (so-called zero-shot capability). In this paper we investigate the feasibility of latter option. Leveraging the bridge-prompt framework, we prompt-tune a pre-trained vision-text model (CLIP) for gesture recognition in surgical videos. This can utilize extensive outside video data such as text, but also make use of label meta-data and weakly supervised contrastive losses. Our experiments show that prompt-based video encoder outperforms standard encoders in surgical gesture recognition tasks. Notably, it displays strong performance in zero-shot scenarios, where gestures/tasks that were not provided during the encoder training phase are included in the prediction phase. Additionally, we measure the benefit of inclusion text descriptions in the feature extractor training schema. Bridge-prompt and similar pre-trained+prompt-tuned video encoder models present significant visual representation for surgical robotics, especially in gesture recognition tasks. Given the diverse range of surgical tasks (gestures), the ability of these models to zero-shot transfer without the need for any task (gesture) specific retraining makes them invaluable.

Performance Analysis of symmetric and asymmetric Encryption Algorithms Based on File, Image and Video

The rapid evolution of digital technology has exponentially amplified the generation and sharing of diverse digital data, notably files, images, and videos, over the internet, intensifying the critical need for enhanced security measures to safeguard these prevalent data types. This research presents a comprehensive analysis of various encryption algorithms applied to different data types - files, images, and videos. The study categorizes encryption algorithms into symmetric and asymmetric types, with examples including AES, DES, Triple DES, RSA, Diffie-Hellman, and ECC. The paper further explores specific algorithms used for file, image, and video encryption. The objective is to identify the most efficient encryption algorithm for each data type, thereby enhancing data security in the digital age. The paper emphasizes that while encryption is a crucial tool for data security, it should be used in conjunction with other security measures for comprehensive protection.

Learning to enhance areal video captioning with visual question answering

ABSTRACT The utilization of Unmanned Aerial Vehicles (UAV) in remote sensing (RS) has witnessed a significant surge, offering valuable insights into Earth dynamics and human activities. However, this has led to a substantial increase in the volume of video data, rendering manual screening and analysis impractical. Consequently, there is a pressing need for the development of automated interpretation models for these aerial videos. In this paper, we propose a novel approach that leverages visual dialogue to enhance aerial video captioning. Our model adopts an encoder-decoder architecture, integrating a Visual Question Answering (VQA) task before the captioning task. The VQA task aims to enrich the captioning process by soliciting additional information about the image content. Specifically, our video encoder utilizes ViT-L/16, while the decoder employs Generative Pre-trained Transformer-2 (Distill-GPT-2). To validate our model, we introduce a novel benchmark dataset named CapERA-VQA, comprising videos accompanied by sets of questions, answers, and captions. Through experimental validation, we demonstrate the effectiveness of our proposed approach in enhancing the automated captioning of aerial videos.

Dynamic analysis of a novel hyperchaotic system based on STM32 and application in image encryption

This paper presents a novel 4D hyperchaotic system derived from a modified 3D Lorenz chaotic system. A key aspect of this system is the presence of a single equilibrium point, and its stability is carefully analyzed. The dynamic properties, including the Lyapunov exponent spectrum, bifurcation diagram, and chaotic attractors, are investigated using MATLAB simulations. The results reveal that the system displays hyperchaotic behavior across a wide range of the parameter d\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$d$$\\end{document}, with its chaotic attractor transitioning through four states: hyperchaotic, chaotic, periodic, and quasi-periodic, showcasing the system's complex dynamics. Experimental validation using STM32 embedded hardware successfully reproduces these four types of chaotic attractors, confirming the theoretical predictions. The proposed hyperchaotic system is then applied to image encryption, introducing a novel encryption method. The hyperchaotic key sequence generated by this system meets 15 tests of the NIST SP800-22 standard, and further experimental validation with STM32 hardware demonstrates the algorithm's effectiveness, simplicity, non-linearity, and high security. The encrypted images and sequences are rigorously tested key space analysis, histogram similarity analysis, information entropy analysis, statistical attack analysis, differential attack analysis, key sensitivity analysis, and correlation analysis, highlighting the robustness and reliability of the proposed system. This method is versatile and can be extended to various fields, including audio and video encryption, text encryption, IoT security, financial transaction security, and medical data protection.

Chaos Based Novel Video Encryption Scheme to Secure Video/Image Information from Any AI-Generated Attacks

Chaos Based Novel Video Encryption Scheme to Secure Video/Image Information from Any AI-Generated Attacks

Chaos based image encryption scheme to secure sensitive multimedia content in cloud storage

Cloud computing offers a variety of on-demand services to users and has gained significant prominence in the contemporary era. The security of information stored in cloud data centers has become a central concern, especially for sensitive data like medical images, videos, and multimedia content that require heightened protection when stored in data centers. Cloud users have a responsibility to ensure the security of their data through strategic measures. Focusing on maintaining the privacy of images stored in the cloud, this research introduces an innovative image encryption technique that is based on a modified Skew Tent chaotic map. The suggested modification to the chaotic map includes combining the Skew Tent map with both the sine trigonometric function and perturbation technology. This results in enhanced randomness, more intricate dynamical behavior, a broader chaotic range, and increased sensitivity to initial values in contrast to alternative chaotic maps. The incorporation of this adapted map into a stepwise procedure, involving two rounds of permutation followed by diffusion, effectively accomplishes image data encryption for cloud storage systems. These consecutive operations collectively enhance the encryption method’s randomness and robustness. Through simulations conducted using Cloudsim, the cipher-image exhibits a uniform distribution and achieves a commendable information entropy score of 7.996749. The encryption algorithm significantly reduces correlation coefficients from 1 in the original image to 0 in the encrypted image, while maintaining NPCR (Number of Pixel Change Rate) and UACI (Unified Average Changing Intensity) values within the critical range. Additionally, both theoretical analysis and practical evaluations confirm the algorithm’s resilience against exhaustive, occlusion, and classical attacks. Moreover, extending this encryption framework to video data, a novel video encryption approach is proposed.

Complexity analysis of the VVenC versus VVC encoder

The joint video experts team (JVET) has recently finalized a next-generation open-source video codec, called versatile video coding (VVC). The new standard presents a higher gain in the run time and rate compressions. Based on the reference software (VTM, VVC Test Model) of VVC, an optimized encoder was developed these last years resulting in a fast and efficient encoder, called Fraunhofer versatile video encoder (VVenC). Based on the Bjontegaard methodology and the GPROF profiling tool, this paper presents a technical complexity analysis and comparison of both VVC and VVenC. The appropriate comparisons cover the percentage taken by each block in terms of processing time and the resulting whole encoding time. The peak signal-to-noise ratio (PSNR) and bit rate between VVC and VVenC encoders based on the common test conditions manual are also analyzed and compared. The profiling results show that the VVenC encoder presents a maximum gain of runtime and bit rate of 90% and 20% respectively in classes A and D test sequences, compared to the VVC encoder.

A fast and efficient data reuse scheme for HEVC Integer Motion Estimation hardware architecture

ABSTRACT High-Efficiency Video Coding (HEVC) is a video compression standard designed to improve video compression efficiency compared to its predecessor, Advanced Video Coding (AVC). HEVC provides approximately twice the compression efficiency compared to the AVC. In the HEVC system, the most computational complexity module is the Motion Estimation (ME). ME module helps reduce redundancy by compensating for motion during video compression. However, the computational complexity makes it a bottleneck in the design of high-resolution video encoders. This paper proposes an efficient architecture for the Integer ME (IME) module of the HEVC encoder. The proposed architecture introduces an efficient memory usage scheme to support the Full Search motion search algorithm. The full pipeline architecture includes 4096 Processing Elements (PE) and an adder tree to compute the Sum of Absolute Difference (SAD) for every Prediction Unit (PU) partition. The proposed architecture was designed and implemented on Xilinx Virtex-7 XC7VX550T FPGA. Our design achieved up to 275 FPS and approximately 10 FPS at 4 K video for the Search Regions of 32 × 16 and 128 × 128 pixels, respectively.

A Visually Meaningful Color-Video Encryption Scheme That Combines Frame Channel Fusion and a Chaotic System

Compared to text and images, video can show information more vividly and intuitively via a moving picture; therefore, video is widely used in all walks of life. However, videos uploaded or stored in various video applications have not been treated with any protection, and these videos contain a lot of sensitive information that is more likely to be compromised. To solve this problem, video encryption schemes have been proposed. However, the main concern with existing video encryption schemes is that the private information in the encrypted video should be effectively protected, and, thus, the pixel distribution of the original video can be greatly damaged in the process of encryption, resulting in no or poor visual usability of the encrypted video. To this end, a novel color-video encryption scheme is proposed, which can effectively protect video privacy information while retaining certain visual information, thus enhancing the usability of encrypted videos. Firstly, the R, G, and B channels of the original color video are viewed as a whole for splitting. The dimensions of the blocks are three-dimensional, and permutation encryption is performed in three-dimensional blocks, which eliminates the redundancy of information between the video frame channels. Secondly, after permutation encryption, the channels of the video frame are separated, and then each channel is divided into blocks. The shape of the blocks is a square, and substitution encryption and permutation encryption operations are performed in turn. The whole encryption process is combined with the 2D-LSM chaotic system to improve the security of the scheme, as well as to reduce the time. Extensive experiments have been carried out, and the results show that the proposed scheme allows the encrypted video to retain rough visual information and, at the same time, effectively protects privacy, achieving the goal of encrypted video security and usability.

Design, Hardware Implementation, and Application in Video Encryption of the 2-D Memristive Cubic Map

Design, Hardware Implementation, and Application in Video Encryption of the 2-D Memristive Cubic Map

Development of a video encryption algorithm for critical areas using 2D extended Schaffer function map and neural networks

This paper proposes an encryption algorithm for crucial areas of a video based on chaos and a neural network, which SVEA (Selective Video Encryption Algorithm). The critical areas of each frame in a video are extracted by deep learning to the encryption system. A one-step encryption algorithm is used to encrypt these critical areas based on chaos, where scrambling and diffusion are simultaneously performed. A new chaotic system 2D extended Schaffer function map (2D-ESFM) is utilized in the encryption system, inspired by the Schaffer function. The system has demonstrated excellent performance through Lyapunov exponents (LEs), permutation entropy (PE), the 0-1 test, and other methods. Additionally, to resist chosen plaintext attacks, the secret key is generated by a neural network, with the critical areas of the video as inputs to the neural network. The chaotic system generates the biases and weights for the neural network. We evaluate SVEA on our dataset (Gymnastics at the Olympic Games) and public datasets. SVEA exhibits strong security characteristics compared to state-of-the-art algorithms and reduces time complexity by approximately 51.3%.

Quality and recognizability estimation video encryption database

In literature about selective encryption of image and video content, image quality indices are usually used to gauge the degree of encryption. These methods have frequently been shown not to work well for the evaluation of encryption, mainly due to them being trained on predominantly high quality contents. The problem for creating a proper recognition index or visual encryption strength index is the lack of data to train on. In this paper we present the first database of encrypted video content, ranging from high quality to completely unrecognizable, together with human observer scores for quality and recognizability. We also provide a basic evaluation of visual quality indices on this database, directly and in different combination by fusion, to showcase that currently image and video quality indices are ill fit for the purpose of estimating video encryption strength/recognizably. This inability of quality indices to perform also showcases that this database fills the required blind spot of currently available data.

Video Summarization Generation Network Based on Dynamic Graph Contrastive Learning and Feature Fusion

Video summarization aims to analyze the structure and content of videos and extract key segments to construct summarization that can accurately summarize the main content, allowing users to quickly access the core information without browsing the full video. However, existing methods have difficulties in capturing long-term dependencies when dealing with long videos. On the other hand, there is a large amount of noise in graph structures, which may lead to the influence of redundant information and is not conducive to the effective learning of video features. To solve the above problems, we propose a video summarization generation network based on dynamic graph contrastive learning and feature fusion, which mainly consists of three modules: feature extraction, video encoder, and feature fusion. Firstly, we compute the shot features and construct a dynamic graph by using the shot features as nodes of the graph and the similarity between the shot features as the weights of the edges. In the video encoder, we extract the temporal and structural features in the video using stacked L-G Blocks, where the L-G Block consists of a bidirectional long short-term memory network and a graph convolutional network. Then, the shallow-level features are obtained after processing by L-G Blocks. In order to remove the redundant information in the graph, graph contrastive learning is used to obtain the optimized deep-level features. Finally, to fully exploit the feature information of the video, a feature fusion gate using the gating mechanism is designed to fully fuse the shallow-level features with the deep-level features. Extensive experiments are conducted on two benchmark datasets, TVSum and SumMe, and the experimental results show that our proposed method outperforms most of the current state-of-the-art video summarization methods.

Temporal action segmentation for video encryption

Videos contain temporal information, enabling them to capture the dynamic changes of actions and provide richer visual effects. Traditional video encryption methods involve decomposing videos into frames and encrypting them frame by frame, which results in significant resource consumption. This paper proposes a video encryption method based on temporal action segmentation. This methodology involves the identification and extraction of pivotal frames from a video dataset, followed by the encryption of these significant key frames. This approach serves to enhance the efficacy of the video encryption algorithm. The method consists of three modules. The first module uses temporal action segmentation to classify video frames and extract important frames for the second module’s input. The second module encrypts the extracted key frames using a chaos-based encryption algorithm, thereby reducing the time cost of video encryption. The third module outputs the encrypted video. During the encryption process, a large amount of key stream is required. To address this, the paper introduces a new pseudo-random sequence generation method called two-dimensional Gramacy&Lee map (2D-GLM). Comprehensive comparative analysis clearly demonstrates that compared to other systems, 2D-GLM exhibits superior performance and can generate a large number of high-performance pseudo-random sequences. The proposed algorithm is tested on GTEA, and the simulation results demonstrate that it can accomplish video encryption tasks with high security.