Robust Video Research Articles

Novel Robust Video Watermarking Scheme Based on Concentric Ring Subband and Visual Cryptography With Piecewise Linear Chaotic Mapping

Novel Robust Video Watermarking Scheme Based on Concentric Ring Subband and Visual Cryptography With Piecewise Linear Chaotic Mapping

Robust video question answering via contrastive cross-modality representation learning

Robust video question answering via contrastive cross-modality representation learning

A DNN robust video watermarking method in dual-tree complex wavelet transform domain

Deep learning is increasingly being applied in the field of robust watermarking. However, the existing deep learning-based video watermarking methods only uses spatial domain information as the input and the robustness against attacks such as H.264/AVC compression is still not strong. Therefore, this paper proposes a deep learning-based robust video watermarking method in dual-tree complex wavelet transform (DT-CWT) domain. The video frames are transformed into the DT-CWT domain and the suitable high-pass subbands are selected as candidate embedding positions. Then, the 2D and 3D convolutions are combined to extract both intra-frame spatial features and inter-frame temporal features for finding the stable and imperceptible coefficients for watermark embedding in the candidate positions. The convolutional attention module (CBAM) is used to further adjust the embedding coefficients and strengths. In addition, the attack layer, where a differentiable proxy is specially designed in this paper for the simulation of non-differentiable H.264/AVC compression, is introduced to generate distorted watermarked videos for improving the robustness against different attacks. Experimental results show that our method is superior to both the existing deep learning-based methods and traditional methods in the robustness against both spatial and temporal attacks while preserving high video quality. The source code is available at https://github.com/imagecbj/A-DNN-Robust-Video-Watermarking-Method-in-DT-CWT-Domain.

A Video-based Automated Tracking and Analysis System of Plaque Burden in Carotid Artery Using Deep Learning: A Comparison with Senior Sonographers.

The incidence of stroke is rising, and it is the second major cause of mortality and the third leading cause of disability around the globe. The goal of this study was to rapidly and accurately identify carotid plaques and automatically quantify plaque burden using our automated tracking and segmentation US-video system. We collected 88 common carotid artery transection videos (11048 frames) with a history of atherosclerosis or risk factors for atherosclerosis, which were randomly divided into training, test, and validation sets using a 6:3:1 ratio. We first trained different segmentation models to segment the carotid intima and adventitia, and calculate the maximum plaque burden automatically. Finally, we statistically analyzed the plaque burden calculated automatically by the best model and the results of manual labeling by senior sonographers. Of the three Artificial Intelligence (AI) models, the Robust Video Matting (RVM) segmentation model's carotid intima and adventitia Dice Coefficients (DC) were the highest, reaching 0.93 and 0.95, respectively. Moreover, the RVM model has shown the strongest correlation coefficient (0.61±0.28) with senior sonographers, and the diagnostic effectiveness between the RVM model and experts was comparable with paired-t test and Bland-Altman analysis [P= 0.632 and ICC 0.01 (95% CI: -0.24~0.27), respectively]. Our findings have indicated that the RVM model can be used in ultrasound carotid video. The RVM model can automatically segment and quantify atherosclerotic plaque burden at the same diagnostic level as senior sonographers. The application of AI to carotid videos offers more precise and effective methods to evaluate carotid atherosclerosis in clinical practice.

Robust video hashing with canonical polyadic decomposition and Hahn moments

Robust video hashing with canonical polyadic decomposition and Hahn moments

Robust video watermarking with complex conjugate symmetric sequency transform using SURF feature matching

Robust video watermarking with complex conjugate symmetric sequency transform using SURF feature matching

Enhancing multimedia management: cloud-based movie type recognition with hybrid deep learning architecture

Film and movie genres play a pivotal role in captivating relevant audiences across interactive multimedia platforms. With a focus on entertainment, streaming providers are increasingly prioritizing the automatic generation of movie genres within cloud-based media services. In service management, the integration of a hybrid convolutional network proves to be instrumental in effectively distinguishing between a diverse array of video genres. This classification process not only facilitates more refined recommendations and content filtering but also enables targeted advertising. Furthermore, given the frequent amalgamation of components from various genres in cinema, there arises a need for social media networks to incorporate real-time video classification mechanisms for accurate genre identification. In this study, we propose a novel architecture leveraging deep learning techniques for the detection and classification of genres in video films. Our approach entails the utilization of a bidirectional long- and short-term memory (BiLSTM) network, augmented with video descriptors extracted from EfficientNet-B7, an ImageNet pre-trained convolutional neural network (CNN) model. By employing BiLSTM, the network acquires robust video representations and proficiently categorizes movies into multiple genres. Evaluation on the LMTD dataset demonstrates the substantial improvement in the performance of the movie genre classifier system achieved by our proposed architecture. Notably, our approach achieves both computational efficiency and precision, outperforming even the most sophisticated models. Experimental results reveal that EfficientNet-BiLSTM achieves a precision rate of 93.5%. Furthermore, our proposed architecture attains state-of-the-art performance, as evidenced by its F1 score of 0.9012.

Recognizing online video genres using ensemble deep convolutional learning for digital media service management

It's evident that streaming services increasingly seek to automate the generation of film genres, a factor profoundly shaping a film's structure and target audience. Integrating a hybrid convolutional network into service management emerges as a valuable technique for discerning various video formats. This innovative approach not only categorizes video content but also facilitates personalized recommendations, content filtering, and targeted advertising. Given the tendency of films to blend elements from multiple genres, there is a growing demand for a real-time video classification system integrated with social media networks. Leveraging deep learning, we introduce a novel architecture for identifying and categorizing video film genres. Our approach utilizes an ensemble gated recurrent unit (ensGRU) neural network, effectively analyzing motion, spatial information, and temporal relationships. Additionally,w we present a sophisticated deep neural network incorporating the recommended GRU for video genre classification. The adoption of a dual-model strategy allows the network to capture robust video representations, leading to exceptional performance in multi-class movie classification. Evaluations conducted on well-known datasets, such as the LMTD dataset, consistently demonstrate the high performance of the proposed GRU model. This integrated model effectively extracts and learns features related to motion, spatial location, and temporal dynamics. Furthermore, the effectiveness of the proposed technique is validated using an engine block assembly dataset. Following the implementation of the enhanced architecture, the movie genre categorization system exhibits substantial improvements on the LMTD dataset, outperforming advanced models while requiring less computing power. With an impressive F1 score of 0.9102 and an accuracy rate of 94.4%, the recommended model consistently delivers outstanding results. Comparative evaluations underscore the accuracy and effectiveness of our proposed model in accurately identifying and classifying video genres, effectively extracting contextual information from video descriptors. Additionally, by integrating edge processing capabilities, our system achieves optimal real-time video processing and analysis, further enhancing its performance and relevance in dynamic media environments.

Video Moment Retrieval With Noisy Labels.

Video moment retrieval (VMR) aims to localize the target moment in an untrimmed video according to the given nature language query. The existing algorithms typically rely on clean annotations to train their models. However, making annotations by human labors may introduce much noise. Thus, the video moment retrieval models will not be well trained in practice. In this article, we present a simple yet effective video moment retrieval framework via bottom-up schema, which is in end-to-end manners and robust to noisy label training. Specifically, we extract the multimodal features by syntactic graph convolutional networks and multihead attention layers, which are fused by the cross gates and the bilinear approach. Then, the feature pyramid networks are constructed to encode plentiful scene relationships and capture high semantics. Furthermore, to mitigate the effects of noisy annotations, we devise the multilevel losses characterized by two levels: a frame-level loss that improves noise tolerance and an instance-level loss that reduces adverse effects of negative instances. For the frame level, we adopt the Gaussian smoothing to regard noisy labels as soft labels through the partial fitting. For the instance level, we exploit a pair of structurally identical models to let them teach each other during iterations. This leads to our proposed robust video moment retrieval model, which experimentally and significantly outperforms the state-of-the-art approaches on standard public datasets ActivityCaption and textually annotated cooking scene (TACoS). We also evaluate the proposed approach on the different manual annotation noises to further demonstrate the effectiveness of our model.

VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection

The recent contrastive language-image pre-training (CLIP) model has shown great success in a wide range of image-level tasks, revealing remarkable ability for learning powerful visual representations with rich semantics. An open and worthwhile problem is efficiently adapting such a strong model to the video domain and designing a robust video anomaly detector. In this work, we propose VadCLIP, a new paradigm for weakly supervised video anomaly detection (WSVAD) by leveraging the frozen CLIP model directly without any pre-training and fine-tuning process. Unlike current works that directly feed extracted features into the weakly supervised classifier for frame-level binary classification, VadCLIP makes full use of fine-grained associations between vision and language on the strength of CLIP and involves dual branch. One branch simply utilizes visual features for coarse-grained binary classification, while the other fully leverages the fine-grained language-image alignment. With the benefit of dual branch, VadCLIP achieves both coarse-grained and fine-grained video anomaly detection by transferring pre-trained knowledge from CLIP to WSVAD task. We conduct extensive experiments on two commonly-used benchmarks, demonstrating that VadCLIP achieves the best performance on both coarse-grained and fine-grained WSVAD, surpassing the state-of-the-art methods by a large margin. Specifically, VadCLIP achieves 84.51% AP and 88.02% AUC on XD-Violence and UCF-Crime, respectively. Code and features are released at https://github.com/nwpu-zxr/VadCLIP.

EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer

Video Motion Magnification (VMM) aims to break the resolution limit of human visual perception capability and reveal the imperceptible minor motion that contains valuable information in the macroscopic domain. However, challenges arise in this task due to photon noise inevitably introduced by photographic devices and spatial inconsistency in amplification, leading to flickering artifacts in static fields and motion blur and distortion in dynamic fields in the video. Existing methods focus on explicit motion modeling without emphasizing prioritized denoising during the motion magnification process. This paper proposes a novel dynamic filtering strategy to achieve static-dynamic field adaptive denoising. Specifically, based on Eulerian theory, we separate texture and shape to extract motion representation through inter-frame shape differences, expecting to leverage these subdivided features to solve this task finely. Then, we introduce a novel dynamic filter that eliminates noise cues and preserves critical features in the motion magnification and amplification generation phases. Overall, our unified framework, EulerMormer, is a pioneering effort to first equip with Transformer in learning-based VMM. The core of the dynamic filter lies in a global dynamic sparse cross-covariance attention mechanism that explicitly removes noise while preserving vital information, coupled with a multi-scale dual-path gating mechanism that selectively regulates the dependence on different frequency features to reduce spatial attenuation and complement motion boundaries. We demonstrate extensive experiments that EulerMormer achieves more robust video motion magnification from the Eulerian perspective, significantly outperforming state-of-the-art methods. The source code is available at https://github.com/VUT-HFUT/EulerMormer.

Robust video hashing based on deep feature and quaternion generic Fourier descriptor for copy detection

Robust video hashing based on deep feature and quaternion generic Fourier descriptor for copy detection

Traffic spills the beans: A robust video identification attack against YouTube

Traffic-based video identification attack is a side-channel attack that allows an attacker to identify transmitted online videos from encrypted network traffic. Existing attacks face two types of challenges: 1) YouTube's improved streaming technology renders them invalid or impacted; 2) real-world diverse traffic makes them underperform. In this paper, we provide the first study of YouTube's technical improvements and reveal its three key features: i) the mechanism that replaces the MPD file, ii) variable combined segments, and iii) various buffering processes. Based on the above analysis, we propose a Robust and easily scalable Video Identification Attack against YouTube called RoVIA. RoVIA extracts video fingerprints from video file headers and reference sequences from video traffic. RoVIA uses the reference sequence to generate candidate sequences in the pre-built video fingerprint library and achieves video identification by measuring sequence similarity. Experimental results show that RoVIA achieves open-world accuracy of 0.967, 0.924, and 0.863 in open-world, fixed-quality, and poor network experiments. Furthermore, RoVIA's scalability allows the target classes to be adjusted by updating the fingerprint library without collecting new samples and retraining parameters. This study aims to demonstrate the vulnerabilities of YouTube streaming technology and the potential consequences of its malicious use. We also propose two YouTube-friendly countermeasures to help developers build effective defenses.

Enhancing robustness in video recognition models: Sparse adversarial attacks and beyond

Recent years have witnessed increasing interest in adversarial attacks on images, while adversarial video attacks have seldom been explored. In this paper, we propose a sparse adversarial attack strategy on videos (DeepSAVA). Our model aims to add a small human-imperceptible perturbation to the key frame of the input video to fool the classifiers. To carry out an effective attack that mirrors real-world scenarios, our algorithm integrates spatial transformation perturbations into the frame. Instead of using the lp norm to gauge the disparity between the perturbed frame and the original frame, we employ the structural similarity index (SSIM), which has been established as a more suitable metric for quantifying image alterations resulting from spatial perturbations. We employ a unified optimisation framework to combine spatial transformation with additive perturbation, thereby attaining a more potent attack. We design an effective and novel optimisation scheme that alternatively utilises Bayesian Optimisation (BO) to identify the most critical frame in a video and stochastic gradient descent (SGD) based optimisation to produce both additive and spatial-transformed perturbations. Doing so enables DeepSAVA to perform a very sparse attack on videos for maintaining human imperceptibility while still achieving state-of-the-art performance in terms of both attack success rate and adversarial transferability. Furthermore, built upon the strong perturbations produced by DeepSAVA, we design a novel adversarial training framework to improve the robustness of video classification models. Our intensive experiments on various types of deep neural networks and video datasets confirm the superiority of DeepSAVA in terms of attacking performance and efficiency. When compared to the baseline techniques, DeepSAVA exhibits the highest level of performance in generating adversarial videos for three distinct video classifiers. Remarkably, it achieves an impressive fooling rate ranging from 99.5% to 100% for the I3D model, with the perturbation of just a single frame. Additionally, DeepSAVA demonstrates favourable transferability across various time series models. The proposed adversarial training strategy is also empirically demonstrated with better performance on training robust video classifiers compared with the state-of-the-art adversarial training with projected gradient descent (PGD) adversary.

Create and implement a new method for robust video face recognition using convolutional neural network algorithm

Biometric applications have massive demand in today's era. The areas of applications are mostly linked with the security of the system. Biometric features are regarded as the primary resource for security purposes due to their own distinctiveness and non-volatile essence. System authentication using biometrics is considered to be a sophisticated technology. Noise effect inducts variation in the biometric subject that causes an adverse impact on establishing the recognition. The proposed model supported the development of an effective method for performing facial biometric feature recognition. The model's goal is to reduce the number of false approvals and refusals. The proposed algorithm has been applied over a video dataset containing surveillance video frames that captures facial subjects dynamically. The first step is pre-processing of the video frames that have been carried out in the proposed model. Then, the Viola-Jones algorithm was applied to detect the facial subjects in the video frames. Feature extraction from the facial subject has been accomplished by applying a deep reinforcement learning algorithm. Further, the proposed model applied a convolutional neural network (CNN) algorithm to perform feature recognition of facial identity accurately. The proposed technique aims to maintain a huge recognition rate of dynamic facial subjects under various unprecedented noise variations. In the classification algorithm, the recognition accuracy is found to be 98.85%.

Exploring Frame Difference to Enhance Robustness for Video Steganography on Social Networks

The proliferation of video sharing on social networks has created a novel avenue for covert communication. Since most social networking channels are lossy, researchers have devoted efforts to robust video steganography to establish covert communication on social networks. Although there are various methods available, they often overlook the differences between frames in a video and are designed for a single frame. In this paper, we explore the general process of video recompression and present the frame quantization step (FQS) and interframe mutual information (IFMI) to measure the frame difference in the process of video recompression. Based on the two proposed metrics, we design a heuristic frame selection strategy and then propose a new robust video steganographic method in the DWT-SVD domain. Extensive experiments demonstrate that heuristic frame selection can effectively improve the robustness and reduce the computational complexity of video steganography. Our proposed method provides better robustness and higher efficiency than existing methods for building reliable covert communication on social networks, such as YouTube and Vimeo.

Dual-attention Network for View-invariant Action Recognition

View-invariant action recognition has been widely researched in various applications, such as visual surveillance and human–robot interaction. However, view-invariant human action recognition is challenging due to the action occlusions and information loss caused by view changes. Modeling spatiotemporal dynamics of body joints and minimizing representation discrepancy between different views could be a valuable solution for view-invariant human action recognition. Therefore, we propose a Dual-Attention Network (DANet) aims to learn robust video representation for view-invariant action recognition. The DANet is composed of relation-aware spatiotemporal self-attention and spatiotemporal cross-attention modules. The relation-aware spatiotemporal self-attention module learns representative and discriminative action features. This module captures local and global long-range dependencies, as well as pairwise relations among human body parts and joints in the spatial and temporal domains. The cross-attention module learns view-invariant attention maps and generates discriminative features for semantic representations of actions in different views. We exhaustively evaluate our proposed approach on the NTU-60, NTU-120, and UESTC large-scale challenging datasets with multi-type evaluation metrics including Cross-Subject, Cross-View, Cross-Set, and Arbitrary-view. The experimental results demonstrate that our proposed approach significantly outperforms state-of-the-art approaches in view-invariant action recognition.

A robust video stabilization method for camera shoot in mobile devices using GMM-based motion estimator

The application of video stabilization methods to mobile devices with low computing power is an important issue. In this study, we aim to provide a robust video stabilization solution for such mobile devices with limited computing resources. We first propose a fast feature extraction method based on a block-wise gradient. Then, we extract motion information through feature point pairs and rebuild the camera path. Finally, the data were smoothed and the corresponding compensation matrix was calculated to obtain a stable video. However, large foreground objects may lead to incorrect motion estimation. Thus, we propose a fast foreground segmentation method based on GMM for jitter scenes to eliminate foreground features and obtain a more accurate camera trajectory. The experimental results show that the proposed method is robust against large foreground objects and exhibits good real-time performance.

Robust Video Portrait Reenactment via Personalized Representation Quantization

While progress has been made in the field of portrait reenactment, the problem of how to produce high-fidelity and robust videos remains. Recent studies normally find it challenging to handle rarely seen target poses due to the limitation of source data. This paper proposes the Video Portrait via Non-local Quantization Modeling (VPNQ) framework, which produces pose- and disturbance-robust reenactable video portraits. Our key insight is to learn position-invariant quantized local patch representations and build a mapping between simple driving signals and local textures with non-local spatial-temporal modeling. Specifically, instead of learning a universal quantized codebook, we identify that a personalized one can be trained to preserve desired position-invariant local details better. Then, a simple representation of projected landmarks can be used as sufficient driving signals to avoid 3D rendering. Following, we employ a carefully designed Spatio-Temporal Transformer to predict reasonable and temporally consistent quantized tokens from the driving signal. The predicted codes can be decoded back to robust and high-quality videos. Comprehensive experiments have been conducted to validate the effectiveness of our approach.

Audio-Visual Contrastive Learning with Temporal Self-Supervision

We propose a self-supervised learning approach for videos that learns representations of both the RGB frames and the accompanying audio without human supervision. In contrast to images that capture the static scene appearance, videos also contain sound and temporal scene dynamics. To leverage the temporal and aural dimension inherent to videos, our method extends temporal self-supervision to the audio-visual setting and integrates it with multi-modal contrastive objectives. As temporal self-supervision, we pose playback speed and direction recognition in both modalities and propose intra- and inter-modal temporal ordering tasks. Furthermore, we design a novel contrastive objective in which the usual pairs are supplemented with additional sample-dependent positives and negatives sampled from the evolving feature space. In our model, we apply such losses among video clips and between videos and their temporally corresponding audio clips. We verify our model design in extensive ablation experiments and evaluate the video and audio representations in transfer experiments to action recognition and retrieval on UCF101 and HMBD51, audio classification on ESC50, and robust video fingerprinting on VGG-Sound, with state-of-the-art results.