Efficient Video Research Articles

Efficient video summarization through MobileNetSSD: a robust deep learning-based framework for efficient video summarization focused on objects of interest

Efficient video summarization through MobileNetSSD: a robust deep learning-based framework for efficient video summarization focused on objects of interest

Content-Adaptive Bitrate Ladder Estimation in High-Efficiency Video Coding Utilizing Spatiotemporal Resolutions

The constant increase in multimedia Internet traffic in the form of video streaming requires new solutions for efficient video coding to save bandwidth and network resources. HTTP adaptive streaming (HAS), the most widely used solution for video streaming, allows the client to adaptively select the bitrate according to the transmission conditions. For this purpose, multiple presentations of the same video content are generated on the video server, which contains video sequences encoded at different bitrates with resolution adjustment to achieve the best Quality of Experience (QoE). This set of bitrate–resolution pairs is called a bitrate ladder. In addition to the traditional one-size-fits-all scheme for the bitrate ladder, context-aware solutions have recently been proposed that enable optimum bitrate–resolution pairs for video sequences of different complexity. However, these solutions use only spatial resolution for optimization, while the selection of the optimal combination of spatial and temporal resolution for a given bitrate has not been sufficiently investigated. This paper proposes bit-ladder optimization considering spatiotemporal features of video sequences and usage of optimal spatial and temporal resolution related to video content complexity. Optimization along two dimensions of resolution significantly increases the complexity of the problem and the approach of intensive encoding for all spatial and temporal resolutions in a wide range of bitrates, for each video sequence, is not feasible in real time. In order to reduce the level of complexity, we propose a data augmentation using a neural network (NN)-based model. To train the NN model, we used seven video sequences of different content complexity, encoded with the HEVC encoder at five different spatial resolutions (SR) up to 4K. Also, all video sequences were encoded using four frame rates up to 120 fps, presenting different temporal resolutions (TR). The Structural Similarity Index Measure (SSIM) is used as an objective video quality metric. After data augmentation, we propose NN models that estimate optimal TR and bitrate values as switching points to a higher SR. These results can be further used as input parameters for the bitrate ladder construction for video sequences of a certain complexity.

Enhanced neural video compression for cloud gaming videos with aligned frame generation

The burgeoning popularity of cloud gaming makes it critical for efficient video compression to relieve the growing bandwidth pressure. While existing neural video coding approaches have demonstrated strong compression potential on natural videos, there is an absence of efficient neural codecs dedicated to gaming videos. To bridge this gap, in this paper, we propose an end-to-end neural video compression method designed specifically for cloud gaming videos. By effectively utilizing the unique camera motion information inherent to cloud gaming, the previous reconstructed frame is maximally aligned to the current frame through a learning-based module with multiple losses, which then replaces the previous reconstructed frame for optical flow estimation. By significantly reducing the displacement between two consecutive frames caused by camera motion, the motion estimation accuracy is enhanced, effectively handling the large and abrupt motion scenarios frequently present in gaming videos. Furthermore, the aligned tensor obtained in the previous step is used to enhance the latent prior of the entropy model, providing a superior temporal prior for coding. Extensive experimental results demonstrate the superior performance of our proposed method compared to one of the previous state-of-the-art approaches, DCVC-HEM, providing significant progress in end-to-end neural compression in cloud gaming videos.

Fast CU decision method based on texture characteristics and decision tree for depth map intra-coding

As the demand for higher quality 3D videos continues to grow, the 3D extensions of the Efficient Video Coding (3D-HEVC) standard are gradually unable to meet the needs of users. Versatile Video Coding Standard (H. 266/VVC), as an advanced video coding standard, adopts the nested Multi-Type Tree quadtree (QTMT) partitioning structure that current fast depth map coding unit (CU) partitioning methods cannot apply. Therefore, we have designed a fast intra-frame CU partitioning algorithm for VVC 3D video depth maps. Our proposed algorithm in this article consists of two steps, including two sub-algorithms. First, by analyzing the relationship between image entropy and variance and depth map CU division, we establish a bi-criterion decision algorithm to determine whether the texture complexity of the current CU is low enough to terminate its partitioning process. Then, for CUs that have been determined by the first algorithm to need further partitioning, we use a decision tree model based on Light Gradient Boosting Machine (LGBM) to predict which direction of Rate-Distortion Optimization (RDO) calculation can be skipped, which can avoid some unnecessary RDO calculations in a certain direction. The final experiment demonstrated the effect of the proposed algorithm, which can reduce 47.65% complexity of VVC 3D video intra-coding with negligible 0.23% Bjøntegaard Delta Bitrate (BDBR) increase, superior to other advanced methods.

Unleashing the Power of Contrastive Learning for Zero-Shot Video Summarization.

Video summarization aims to select the most informative subset of frames in a video to facilitate efficient video browsing. Past efforts have invariantly involved training summarization models with annotated summaries or heuristic objectives. In this work, we reveal that features pre-trained on image-level tasks contain rich semantic information that can be readily leveraged to quantify frame-level importance for zero-shot video summarization. Leveraging pre-trained features and contrastive learning, we propose three metrics featuring a desirable keyframe: local dissimilarity, global consistency, and uniqueness. We show that the metrics can well-capture the diversity and representativeness of frames commonly used for the unsupervised generation of video summaries, demonstrating competitive or better performance compared to past methods when no training is needed. We further propose a contrastive learning-based pre-training strategy on unlabeled videos to enhance the quality of the proposed metrics and, thus, improve the evaluated performance on the public benchmarks TVSum and SumMe.

Robust Mixed-Rate Region-of-Interest-Aware Video Compressive Sensing for Transmission Line Surveillance Video

Classic video compression methods usually suffer from long encode time and requires large memories, making it hard to deploy on edge devices; thus, video compressive sensing, which requires less resources during encoding, is receiving more attention. We propose a robust mixed-rate ROI-aware video compressive sensing algorithm for transmission line surveillance video compression. The proposed method compresses foreground targets and background frames separately and uses reversible neural network to reconstruct original frames. The result on transmission line surveillance video data shows that the proposed compressive sensing method can achieve 26.47, 34.71 PSNR and 0.6839, 0.9320 SSIM higher than existing methods on 1.5% and 15% measurement rates, and the proposed ROI extraction net can precisely retrieve regions under high noise levels. This research not only demonstrates the potential for a more efficient video compression technique in resource-constrained environments, but also lays a foundation for future advancements in video compressive sensing techniques and their applications in various real-time surveillance systems.

5G‐based video summarization: An analysis from a methodological point of view

AbstractSurveillance is one of the fast‐growing applications used for monitoring and watching people, objects, or the environment to collect information and provide security. The surveillance data is in video form, and analyzing large video is challenging because it is essential to do efficient video streaming online. Video summarization comprises selecting, extracting, and aggregating keyframes for creating a synopsis, which is challenging. Though several methods have been proposed for video summarization, most are inconsistent, poor in processing and delivering video content, and do not focus on solving the root problems interlinked with efficient streaming. Thus, video streaming applications require an efficient video summarization model that can overcome existing issues and challenges and improve the overall quality of service integrated with the advanced technology of 5G. This paper has aimed to discuss various methods, approaches, and technologies used for video summarization to design a better model. It also presents various learning models and a taxonomy of available methods and provides a detailed review. The summary of the model used evaluates its outcome and the existing methods for potential future research works. The proposed approach is compared with existing ones to prove the model's efficiency. The result shows that the proposed model achieved a 62.3 and 52.3 F1 score summarizing the TVSum and SumMe datasets, respectively.

Learning-based joint recommendation, caching, and transmission optimization for cooperative edge video caching in Internet of Vehicles

In an era dominated by multimedia information, achieving efficient video transmission in the Internet of Vehicles (IoV) is crucial because of the inherent bandwidth constraints and network volatility within vehicular environments. In this paper, we propose a cooperative edge video caching framework designed to enhance video delivery efficiency in IoV by integrating joint recommendation, caching, and transmission optimization. Leveraging deep reinforcement learning with the discrete soft actor–critic algorithm, our methodology dynamically adapts to fluctuating network conditions and diverse user preferences, aiming to optimize content delivery efficiency and quality of experience. The proposed approach combines recommendation and caching strategies with transmission optimization to provide a comprehensive solution for high-performance video services. Extensive simulation results demonstrate that our framework significantly outperforms traditional baseline methods, achieving superior outcomes in terms of service utility, delivery rate, and delay reduction. These results highlight the robust potential of our solution to facilitate seamless and high-quality video experiences in the complex and dynamic landscape of vehicular networks, advancing the capabilities of IoV content delivery.

Learning-based CU partition prediction for fast panoramic video intra coding

In recent years, Virtual Reality (VR) technology has been applied in various aspects of life. As an indispensable component of VR, panoramic video offers viewers immersive experiences, thereby attracting widespread attention. The high resolution of panoramic video poses significant challenges for its storage and transmission, requiring efficient video compression. The Versatile Video Coding (VVC) exhibits superior coding efficiency, albeit at the expense of a several dozen-fold increase in encoding complexity. In this paper, we propose a fast Coding Unit (CU) partition algorithm for intra-coding based on Support Vector Machine (SVM), which selects appropriate features for online training of multiple classifiers based on frame texture characteristics. Additionally, in response to distortion issues in panoramic video coding, we divide video frames into different regions and adjust the Quantization Parameter (QP) values of CUs within each region to improve the encoding efficiency. Experimental results show that the proposed method achieves an average encoding time reduction of 46.21%, with an average BDBR increase of only 1.01%, achieving a good trade-off between time-saving and encoding efficiency.

Global–local spatio-temporal graph convolutional networks for video summarization

Video summarization aims to create concise and accurate summary to enable users to quickly grasp the key content of the original video for facilitating efficient video browsing. Most existing video summarization methods mainly employ recurrent neural networks to capture long-term dependencies in videos, yielding remarkable results. Nevertheless, these methods overlook the potential spatial features inside the video when modeling the video. To tackle this issue, we introduce a global–local spatio-temporal graph convolutional networks for video summarization (GL-STGCN). Initially inspired by the concept of 3-D convolution, we segment the video into non-overlapping segments to capture localized spatial features of sequential frames. Subsequently, a spatial graph is constructed for each segment, with a fixed time interval between neighboring spatial graphs. Then we use the pooling method to randomly delete the redundant nodes in the graph. We then leverage a temporal gating convolutional network to extract the global temporal relationships within the video. Employing the spatial features, a spatial graph convolutional network is utilized to capture the spatial connections among frames. As the graph node information evolves, the node features furnish a more precise depiction of the video content. Consequently, we employ the temporal gating convolutional network once more to refine the global temporal relations within the video. Extensive experiments on two public datasets are conducted in this paper, showing that our proposed method outperforms most state-of-the-art video summarization methods in terms of performance. Experimental results demonstrate the effectiveness of integrating global temporal and local spatial relationships.

Exploiting Optical Flow Guidance for Transformer-Based Video Inpainting.

Transformers have been widely used for video processing owing to the multi-head self attention (MHSA) mechanism. However, the MHSA mechanism encounters an intrinsic difficulty for video inpainting, since the features associated with the corrupted regions are degraded and incur inaccurate self attention. This problem, termed query degradation, may be mitigated by first completing optical flows and then using the flows to guide the self attention, which was verified in our previous work - flow-guided transformer (FGT). We further exploit the flow guidance and propose FGT++ to pursue more effective and efficient video inpainting. First, we design a lightweight flow completion network by using local aggregation and edge loss. Second, to address the query degradation, we propose a flow guidance feature integration module, which uses the motion discrepancy to enhance the features, together with a flow-guided feature propagation module that warps the features according to the flows. Third, we decouple the transformer along the temporal and spatial dimensions, where flows are used to select the tokens through a temporally deformable MHSA mechanism, and global tokens are combined with the inner-window local tokens through a dual-perspective MHSA mechanism. FGT++ is experimentally evaluated to be outperforming the existing video inpainting networks qualitatively and quantitatively.

Optimizing Video Queries with Declarative Clues

Video Database Management Systems (VDBMS) leverage advancements in computer vision and deep learning for efficient video data analysis and retrieval. This paper introduces the concept of user-specified Clues, allowing users to incorporate domain-specific knowledge, referred to as Clues, into query optimization. Clues are expressed as Clue types, each associated with optimization rules, and applied to queries through Clue instances. The extensible ClueVQS system we present to incorporate these ideas, optimizes queries automatically, utilizing Clues to improve processing efficiency. We also introduce algorithms to optimize queries using Clues allowing for trade-offs between speed and query accuracy. Our proposals and system address challenges such as data-dependent Clue effectiveness, limiting search space, and accuracy-efficiency trade-offs. Detailed experimental results demonstrate query speedups of up to two orders of magnitude compared to other applicable approaches, and a reduction of the query optimizer time by up to 95% while respecting user-specified accuracy constraints, showcasing the effectiveness of the proposed framework.

Ensemble Random Forest-based Gradient Optimization based Energy Efficient Video Processing System for Smart Traffic Surveillance System

Deep learning solutions in big data applications can benefit cloud centres and can also lead to network communication overhead. Typically, data collected from traffic are sent to the traffic management centre for analysis. However, this process can worsen the network route to the traffic management centre. A two-tier mechanism has been developed to address this issue, which performs vehicle speed estimation and traffic congestion detection for efficient traffic management. The real-time traffic video data are captured and the video frames are initially processed through a foreground extraction process, which extracts the temporarily stopped vehicles on the road by removing background pixels from the frames. The video frames are then wrapped in an up-down view to remove the influence of the observation angle. The traffic congestion is then detected accurately based on the traffic characteristics using the proposed Ensemble Random Forest-based Gradient Optimization (ERF-GO) algorithm. The generalization error occurs when learning complex features on frames is minimized using a gradient-based optimization (GO) algorithm. Finally, the learned information on traffic conditions is forwarded to the cloud and edge computing environments based on network connection speed. The efficiency of the proposed ERF-GO is investigated in terms of performance metrics, namely root mean square error, speed detection error, execution time, computational cost, accuracy, latency, workload balance, precision, recall, f-measure, and congestion detection error rate. The analytic result displays that the proposed ERF-GO algorithm attains a greater accuracy rate of about 98.65% in detecting traffic congestion which is comparably higher than state-of-the-art methods.

A Video Action Recognition Method via Dual-Stream Feature Fusion Neural Network with Attention

Video action recognition is a technique for automatically determining the category of a video action. It is necessary to design an efficient video action recognition algorithm to predict video labels. This work proposes a video action recognition model based on dual-stream information fusion with attention mechanisms (DSIFAM), which consists of three different sub-modules. First, this proposes an improved keyframe extraction method (IKFE). Based on K-means clustering, this uses convolutional features to calculate the similarity between video frames instead of pixel points. After obtaining preliminary clustering results, the method performs secondary optimization to obtain more representative keyframes. Second, this proposes a video action recognition model based on dual-stream information fusion (DSIF). The method introduces ConvLSTM in the spatial stream and uses P3D instead of the original convolutional network in the temporal stream, which can better extract spatial-temporal information and improve the classification performance. Third, this designs a multi-scale attention mechanism (MSAM) to enhance the feature extraction stage and obtain higher quality classification features. The resulting features are more prominent and have stronger representation capabilities. Finally, this work conducts systematic experiments on different datasets, the results verify the superiority of DSIFAM for video action recognition.

Efficient Task-driven Video Data Privacy Protection for Smart Camera Surveillance System

As one of the most commonly used AIoT sensors, smart cameras and their supporting services, namely cloud video surveillance (CVS) systems, have brought great convenience to people’s lives. Recent CVS providers use different machine learning techniques to improve their services (regarded as tasks) based on the uploaded video. However, uploading data to the CVS providers may cause severe privacy issues. Existing works that remove privacy information could not achieve a high tradeoff between data usability and privacy, because the importance of information varies with the task. In addition, it is challenging to design a real-time privacy protection mechanism, especially in resource-constrained smart cameras. In this work, we design a task-driven and efficient video privacy protection mechanism for a better tradeoff between privacy and data usability. We use Class Activation Mapping to protect privacy while preserving data usability. To improve the efficiency, we utilize the motion vector and residual matrix produced during video codec. Our work outperforms the region of interest–based methods in data protection while preserving data usability. The attack accuracy drops 70%, while the task accuracy is comparable to those without protection (within ± 4%). The average protection frame rate of the High Definition video can exceed 16 fps+ even on a CPU.

EFFICIENT VIDEO COMPRESSION USING HEVC AND NON-LINEAR CONVOLUTIONAL MOBILEVNET BASED RATE-DISTORTION OPTIMIZATION

With the surge in demand for high-quality video content over various platforms, efficient video compression techniques have become indispensable. High-Efficiency Video Coding (HEVC) has been a cornerstone, yet further enhancements are essential for optimal compression. Despite HEVC’s advancements, achieving optimal compression while maintaining video quality remains challenging. Additionally, existing methods often overlook the computational complexity, hindering real-time applications. We propose a novel approach integrating HEVC with Non-Linear Convolutional MobileNet (NLCM) for enhanced compression efficiency. Our method employs a rate-distortion optimization framework, leveraging the capabilities of both HEVC and NLCM to achieve superior compression performance. NLCM provides adaptive filtering, enhancing spatial and temporal correlations, while HEVC ensures high compression efficiency. Through experimentation on standard video datasets, our method demonstrates significant improvements over existing techniques. Compared to HEVC alone, our approach achieves up to 30% reduction in bitrate at equivalent perceptual quality levels. Moreover, computational complexity is reduced by 15%, enabling real- time applications without compromising performance. The proposed method exhibits competitive results across various resolutions and frame rates, making it versatile for diverse video compression scenarios.

A Framework for Video Summarization using Visual Attention Technique

Objectives: To develop an efficient Video Summarization technique that aims to utilize the saliency map for mimicking the human way of selecting the important events in the given video. Methods: This paper proposes Histogram based Weighted Fusion (HWF) algorithm that uses spatial and temporal saliency maps to act as guidance in creating the summary of the video. The spatial saliency score and temporal saliency score obtained from the corresponding saliency maps are fused using the proposed HWF algorithm to obtain the frame level importance score. It tries to depict the visual attention of the human brain when watching a particular video. Findings: The experimental results show that the proposed HWF algorithm performs better than the state-of-the-art methods. Novelty: The use of Histogram intersection and the incorporation of the exponential function as the weight for the combined feature enhance the summarization ability of the proposed model. Keywords: Video Summarization, Saliency Map, Histogram intersection, Contrast sensitivity function, Attention curves

Supporting efficient video file streaming in P2P cloud storage

Supporting efficient video file streaming in P2P cloud storage

Exploring Temporal Feature Correlation for Efficient and Stable Video Semantic Segmentation

This paper tackles the problem of efficient and stable video semantic segmentation. While stability has been under-explored, prevalent work in efficient video semantic segmentation uses the keyframe paradigm. They efficiently process videos by only recomputing the low-level features and reusing high-level features computed at selected keyframes. In addition, the reused features stabilize the predictions across frames, thereby improving video consistency. However, dynamic scenes in the video can easily lead to misalignments between reused and recomputed features, which hampers performance. Moreover, relying on feature reuse to improve prediction consistency is brittle; an erroneous alignment of the features can easily lead to unstable predictions. Therefore, the keyframe paradigm exhibits a dilemma between stability and performance. We address this efficiency and stability challenge using a novel yet simple Temporal Feature Correlation (TFC) module. It uses the cosine similarity between two frames’ low-level features to inform the semantic label’s consistency across frames. Specifically, we selectively reuse label-consistent features across frames through linear interpolation and update others through sparse multi-scale deformable attention. As a result, we no longer directly reuse features to improve stability and thus effectively solve feature misalignment. This work provides a significant step towards efficient and stable video semantic segmentation. On the VSPW dataset, our method significantly improves the prediction consistency of image-based methods while being as fast and accurate.

An investigation of machine learning-based video compression techniques

As video technology continues to seamlessly weave itself into the fabric of daily life, there is a growing need for enhanced storage and efficient video transmission. This surge in demand has led to heightened expectations and standards for video compression technology. Machine learning as an up-and-coming technology can play its advantages in the field of video compression. This article reviews the current state of research on combining video compression techniques with machine learning. The article provides an overview of various research avenues for enhancement, spanning from conventional video compression algorithms to the fusion of traditional compression frameworks with machine learning methodologies, and even the development of novel end-to-end compression algorithms. In additional, the article explores the possible various application scenarios of machine learning-based video compression algorithms based on the characteristics of such non-standard and arithmetic demanding algorithms. At the end, the article speculates on the future of video compression algorithms based on the content of the various studies reviewed in the article.