High-dimensional Video Research Articles

Efficient Violence Recognition in Video Streams using ResDLCNN-GRU Attention Network

Detecting Violence in video streams is essential for public safety and security due to the rising frequency of violent incidents. Despite the extensive deployment of CCTV for surveillance, the available human monitoring resources still need to catch up with the need for vigilant supervision. This research presents a new lightweight model to address this gap by accurately identifying and categorizing violent behaviors in various scenarios, including CCTV footage. The proposed method leverages optical flow and RGB data to capture spatiotemporal features in the Violence data. Built on a Residual DLCNN architecture integrated with the Attention mechanism and GRU components, the model effectively handles high-dimensional video data, enhancing accuracy by prioritizing crucial frames containing violent and nonviolent instances. The proposed model's performance was validated on the Hockey Fights (HF), Movie Fights, and SCVD datasets, achieving impressive accuracies of 98.38%, 99.62%, and 90.57%, respectively. Here, we developed the Extended Automatic Violence Detection Dataset (EAVDD), featuring 1530 videos of violent scenes in movies, public spaces, social media, and sports. Testing the model with top fight scenes in rated movies yielded outstanding results. This research supports surveillance systems and advances short video analysis and understanding with applications in public safety, social media, sports, and law enforcement.

Conv3D-Based Video Violence Detection Network Using Optical Flow and RGB Data.

Detecting violent behavior in videos to ensure public safety and security poses a significant challenge. Precisely identifying and categorizing instances of violence in real-life closed-circuit television, which vary across specifications and locations, requires comprehensive understanding and processing of the sequential information embedded in these videos. This study aims to introduce a model that adeptly grasps the spatiotemporal context of videos within diverse settings and specifications of violent scenarios. We propose a method to accurately capture spatiotemporal features linked to violent behaviors using optical flow and RGB data. The approach leverages a Conv3D-based ResNet-3D model as the foundational network, capable of handling high-dimensional video data. The efficiency and accuracy of violence detection are enhanced by integrating an attention mechanism, which assigns greater weight to the most crucial frames within the RGB and optical-flow sequences during instances of violence. Our model was evaluated on the UBI-Fight, Hockey, Crowd, and Movie-Fights datasets; the proposed method outperformed existing state-of-the-art techniques, achieving area under the curve scores of 95.4, 98.1, 94.5, and 100.0 on the respective datasets. Moreover, this research not only has the potential to be applied in real-time surveillance systems but also promises to contribute to a broader spectrum of research in video analysis and understanding.

Disentangling Stochastic PDE Dynamics for Unsupervised Video Prediction.

Unsupervised video prediction aims to predict future outcomes based on the observed video frames, thus removing the need for supervisory annotations. This research task has been argued as a key component of intelligent decision-making systems, as it presents the potential capacities of modeling the underlying patterns of videos. Essentially, the challenge of video prediction is to effectively model the complex spatiotemporal and often uncertain dynamics of high-dimensional video data. In this context, an appealing way of modeling spatiotemporal dynamics is to explore prior physical knowledge, such as partial differential equations (PDEs). In this article, considering real-world video data as a partly observed stochastic environment, we introduce a new stochastic PDE predictor (SPDE-predictor), which models the spatiotemporal dynamics by approximating a generalized form of PDEs while dealing with the stochasticity. A second contribution is that we disentangle the high-dimensional video prediction into low-level dimensional factors of variations: time-varying stochastic PDE dynamics and time-invariant content factors. Extensive experiments on four various video datasets show that SPDE video prediction model (SPDE-VP) outperforms both deterministic and stochastic state-of-the-art methods. Ablation studies highlight our superiority driven by both PDE dynamics modeling and disentangled representation learning and their relevance in long-term video prediction.

A recursive inference method based on invertible neural network for multi-level model updating using video monitoring data

Likelihood-free inference methods have been widely adopted, but they face significant challenges in updating multi-level computational models that have hierarchically embedded sub-models. This difficulty arises from the lack of direct observations of the quantities of interest of the sub-models. In addition, recent advancements in sensing and image processing technologies allow for the collection of a substantial amount of video monitoring data through non-contact sensing techniques. The implicit and very complicated relationship between the uncertain model parameters and video monitoring data adds an additional layer of challenge to the updating of multi-level models. This paper overcomes these challenges by proposing an innovative Recursive Inference method based on Invertible Neural Networks (RINN) for multi-level models. The proposed RINN framework first compresses the high-dimensional video monitoring data into low-dimension latent-space data using a convolutional autoencoder. Using synthetic video monitoring data generated by considering various uncertainty sources in the computational simulation models, a likelihood-free inference model is then trained through a conditional invertible neural network (cINN) and a summary neural network. This model efficiently approximates the posterior distributions of uncertain model parameters without evaluating the computationally intractable likelihood function, given any latent-space data obtained from the autoencoder. To facilitate continuous monitoring and model updating over an extended monitoring period, this paper further proposes a recursive model updating strategy that integrates the cINN-based likelihood-free inference with the particle filtering method. The updating of a degradation model of a miter gate application is employed as an example throughout this paper to explain and demonstrate the efficacy of the proposed RINN framework. The results of the case study show that the RINN is able to effectively reduce uncertainty in the degradation model parameters through strain video monitoring data of a miter gate, and thereby increase the confidence in the remaining useful life (RUL) estimation using the updated degradation model.

Deep Learning Assists Surveillance Experts: Toward Video Data Prioritization

Video summarization (VS) suppresses high-dimensional (HD) video data by only extracting only the important information. However, prior research has not focused on the need for surveillance VS, that is used for many applications to assist video surveillance experts, including video retrieval and data storage. In addition, mainstream techniques commonly use 2D deep models for VS, ignoring event occurrences. Accordingly, we present a two-fold 3D deep learning-assisted VS framework. First, we employ an inflated 3D ConvNet model to extract temporal features; these features are optimized using a proposed encoder mechanism. The input video is temporally segmented using a feature comparison technique for selecting a single frame from each video segment. The segmented shots are evaluated using our novel shot segmentation evaluation scheme and are input into a saliency computation mechanism for keyframe selection in a second fold. Qualitative and quantitative analyses over VS benchmarks and surveillance videos demonstrate the superior performance of our framework, with 0.3- and 4.2-unit increases in the F1 scores for YouTube and TVSum datasets, respectively. Along with accurate VS, a key contribution of our study is the novel shot segmentation criterion prior to VS, which can be used as a benchmark in future research to effectively prioritize HD visual data.

EdgeLeague: Camera Network Configuration With Dynamic Edge Grouping for Industrial Surveillance

Object detection is crucial for surveillance in edge-enabled industrial Internet-of-Things (IIoT). Massive high-dimensional video streams without considering priority differences connect to edges via narrow and time-varying uplink channels, which should be analyzed efficiently for accurate and fast surveillance responses. However, time-varying network environments and constrained edge resources degrade surveillance's accuracy and real-time performance. This paper proposes EdgeLeague for multiple video streams with different quality of service, which maintains high surveillance performance under edge resource limitations and uplink bandwidth dynamics by edge collaboration and camera network configuration. The EdgeLeague scheme is formulated by an NP-hard integer non-linear problem to dynamically configure camera network resolutions and detection models on cooperative edges. To accelerate configuration responses, the formulated problem is decomposed into edge league grouping, video-league matching, and video configuration, solved by low-complexity algorithms. Theoretical analysis is provided for optimal video-league matching. Simulations show EdgeLeague achieves 0.312 s latency and 86.3% surveillance accuracy.

Weakly supervised inference of personalized heart meshes based on echocardiography videos.

Echocardiography provides recordings of the heart chamber size and function and is a central tool for non-invasive diagnosis of heart diseases. It produces high-dimensional video data with substantial stochasticity in the measurements, which frequently prove difficult to interpret. To address this challenge, we propose an automated framework to enable the inference of a high resolution personalized 4D (3D plus time) surface mesh of the cardiac structures from 2D echocardiography video data. Inferring such shape models arises as a key step towards accurate personalized simulation that enables an automated assessment of the cardiac chamber morphology and function. The proposed method is trained using only unpaired echocardiography and heart mesh videos to find a mapping between these distinct visual domains in a self-supervised manner. The resulting model produces personalized 4D heart meshes, which exhibit a high correspondence with the input echocardiography videos. Furthermore, the 4D heart meshes enable the automatic extraction of echocardiographic variables, such as ejection fraction, myocardial muscle mass, and volumetric changes of chamber volumes over time with high temporal resolution.

Modeling spatial point processes in video-imaging via Ripley’s K-function: an application to spatter analysis in additive manufacturing

For an increasing number of applications, the quality and the stability of manufacturing processes can be determined via image and video-image data analysis and new techniques are required to extract and synthesize the relevant information content enclosed in big sensor data to draw conclusions about the process and the final part quality. This paper focuses on video image data where the phenomena under study is captured by a point process whose spatial signature is of interest. A novel approach is proposed which combines spatial data modeling via Ripley’s K-function with Functional Analysis of Variance (FANOVA), i.e., Analysis of Variance on Functional data. The K-function allows to synthesize the spatial pattern information in a function while preserving the capability to capture changes in the process behavior. The method is applicable to quantities and phenomena that can be represented as clusters, or clouds, of spatial points evolving over time. In our case, the motivating case study regards the analysis of spatter ejections caused by the laser-material interaction in Additive Manufacturing via Laser Powder Bed Fusion (L-PBF). The spatial spread of spatters, captured in the form of point particles through in-situ high speed machine vision, can be used as a proxy to select the best conditions to avoid defects (pores) in the manufactured part. The proposed approach is shown to be not only an efficient way to translate the high-dimensional video image data into a lower dimensional format (the K-function curves), but also more effective than benchmark methods in detecting departures from a stable and in-control state.

RockGPT: reconstructing three-dimensional digital rocks from single two-dimensional slice with deep learning

Random reconstruction of three-dimensional (3D) digital rocks from two-dimensional (2D) slices is crucial for elucidating the microstructure of rocks and its effects on pore-scale flow in terms of numerical modeling, since massive samples are usually required to handle intrinsic uncertainties. Despite remarkable advances achieved by traditional process-based methods, statistical approaches and recently famous deep learning-based models, few works have focused on producing several kinds of rocks with one trained model and allowing the reconstructed samples to approximately satisfy certain given properties, such as porosity. To fill this gap, we propose a new framework with deep learning, named RockGPT, which is composed of VQ-VAE and conditional GPT, to synthesize 3D samples based on a single 2D slice from the perspective of video generation. The VQ-VAE is utilized to compress high-dimensional input video, i.e., the sequence of continuous rock slices, to discrete latent codes and reconstruct them. In order to obtain diverse reconstructions, the discrete latent codes are modeled using conditional GPT in an autoregressive manner, while incorporating conditional information from a given slice, rock type, and porosity. We conduct two experiments on five kinds of rocks, and the results demonstrate that RockGPT can produce different kinds of rocks with a single model, and the porosities of reconstructed samples can distribute around specified targets with a narrow range. In a broader sense, through leveraging the proposed conditioning scheme, RockGPT constitutes an effective way to build a general model to produce multiple kinds of rocks simultaneously that also satisfy user-defined properties.

A Reinforcement-Learning-Based Energy-Efficient Framework for Multi-Task Video Analytics Pipeline

Deep-learning-based video processing has yielded transformative results in recent years. However, the video analytics pipeline is energy-intensive due to high data rates and reliance on complex inference algorithms, which limits its adoption in energy-constrained applications. Motivated by the observation of high and variable spatial redundancy and temporal dynamics in video data streams, we design and evaluate an adaptive-resolution optimization framework to minimize the energy use of multi-task video analytics pipelines. Instead of heuristically tuning the input data resolution of individual tasks, our framework utilizes deep reinforcement learning to dynamically govern the input resolution and computation of the entire video analytics pipeline. By monitoring the impact of varying resolution on the quality of high-dimensional video analytics features, hence the accuracy of video analytics results, the proposed end-to-end optimization framework learns the best non-myopic policy for dynamically controlling the resolution of input video streams to globally optimize energy efficiency. Governed by reinforcement learning, optical flow is incorporated into the framework to minimize unnecessary spatio-temporal redundancy that leads to re-computation, while preserving accuracy. The proposed framework is applied to video instance segmentation which is one of the most challenging computer vision tasks, and achieves better energy efficiency than all baseline methods of similar accuracy on the YouTube-VIS dataset.

SnappySonic: An Ultrasound Acquisition Replay Simulator.

SnappySonic provides an ultrasound acquisition replay simulator designed for public engagement and training. It provides a simple interface to allow users to experience ultrasound acquisition without the need for specialist hardware or acoustically compatible phantoms. The software is implemented in Python, built on top of a set of open source Python modules targeted at surgical innovation. The library has high potential for reuse, most obviously for those who want to simulate ultrasound acquisition, but it could also be used as a user interface for displaying high dimensional images or video data.

Video Classification Using Deep Learning

Video order has been comprehensively investigated in PC vision in view of its wide spread applications. In any case, it remains a surprising task by virtue of the mind boggling troubles in fruitful segment extraction and successful arrangement with high-dimensional video depictions. Video groupings present uncommon irregularity as a result of monster scope changes, point of view assortment, and camera development, which pose fantastic challenges for both video depictions and characterization. With the phenomenal accomplishment of significant learning, convolutional neural frameworks (CNNs) and their 3-D varieties have been considered in the video territory for an immense grouping of order assignments. Video depictions have accepted a basically noteworthy activity in video examination, which authentically impact a complete execution of video characterization. Both the spatial and brief information should be gotten and encoded for extensive and educational depiction of video progressions. Significant Learning feature level blend plans have exceptional capacity of video depictions for improving the introduction of video grouping. We have driven wide assessment on four going after for video arrangement including human movement affirmation and dynamic scene grouping.

3-D Deconvolutional Networks for the Unsupervised Representation Learning of Human Motions.

Data representation learning is one of the most important problems in machine learning. Unsupervised representation learning becomes meritorious as it has no necessity of label information with observed data. Due to the highly time-consuming learning of deep-learning models, there are many machine-learning models directly adapting well-trained deep models that are obtained in a supervised and end-to-end manner as feature abstractors to distinct problems. However, it is obvious that different machine-learning tasks require disparate representation of original input data. Taking human action recognition as an example, it is well known that human actions in a video sequence are 3-D signals containing both visual appearance and motion dynamics of humans and objects. Therefore, the data representation approaches with the capabilities to capture both spatial and temporal correlations in videos are meaningful. Most of the existing human motion recognition models build classifiers based on deep-learning structures such as deep convolutional networks. These models require a large quantity of training videos with annotations. Meanwhile, these supervised models cannot recognize samples from the distinct dataset without retraining. In this article, we propose a new 3-D deconvolutional network (3DDN) for representation learning of high-dimensional video data, in which the high-level features are obtained through the optimization approach. The proposed 3DDN decomposes the video frames into spatiotemporal features under a sparse constraint in an unsupervised way. In addition, it also can be regarded as a building block to develop deep architectures by stacking. The high-level representation of input sequential data can be used in multiple downstream machine-learning tasks, we evaluate the proposed 3DDN and its deep models in human action recognition. The experimental results from three datasets: 1) KTH data; 2) HMDB-51; and 3) UCF-101, demonstrate that the proposed 3DDN is an alternative approach to feedforward convolutional neural networks (CNNs), that attains comparable results.

A Light Weight Background Subtraction Algorithm for Motion Detection in Fog Computing

A novel light weight background subtraction algorithm proposed in this letter addresses the issue of executing real time motion detection in fog computing, which is a resource constrained environment. The proposed algorithm compresses the original high dimensional video frame into low dimensional space using random projection (RP) and the motion detection is identified using the structural similarity (SSIM) index. The objective of the proposed algorithm is to select the frames with motion and discard the remaining frame in order to reduce the storage and computational cost for the cloud users. The proposed algorithm is evaluated using CDNET 2014 video data and the results are verified in terms of selected frames per video(SFPV), percentage of reduction (POR) and running time. From the experiment, it is evident that the proposed algorithm reduces nearly 67 percent of storage cost with trifling computational expense.

Structure-Constrained Motion Sequence Generation

Video generation is a challenging task due to the extremely high-dimensional distribution of the solution space. Good constraints in the solution domain would thus reduce the difficulty of approximating optimal solutions. In this paper, instead of directly generating high-dimensional video data, we propose using object landmarks as explicit structure constraints to address this issue. Specifically, we propose a two-stage framework for an action-conditioned video generation task. In our framework, the first stage aims to generate landmark sequences according to predefined motion types, and a recurrent model (RNN/LSTM) is adopted for this purpose. The landmark sequence can be regarded as a low-dimensional structure embedding of high-dimensional video data, and generating landmark sequences is much easier than generating videos. The second stage is inspired by a conditional generative adversarial network (CGAN), and we take the generated landmark sequence as a structure condition to learn a landmark-to-image translation network. Such a one-to-one translation framework avoids the difficulty of generating videos and instead transfers the video generation task to image generation, which is resolvable due to the maturity of current GAN-based models. The experimental results demonstrate that our model not only achieves promising results on rigid/nonrigid motion generation tasks but also can be extended to multiobject motion situations.

A spatial-frequency-temporal domain based saliency model for low contrast video sequences

The last few decades have witnessed rapid development of visual saliency detection, as it can detect object-of-interest from clutter environments to substantially facilitate a wide range of applications. However, traditional visual saliency detection models primarily rely on image features, which may face great challenges in low contrast video stream captured from low lighting scenarios. This paper proposes a dynamic multimodal fusion based visual saliency detection model towards low contrast videos, which combines saliency information from spatial, frequency, and temporal domains. In spatial domain, superpixel covariance is utilized to compute the region dissimilarity under low lighting scenarios; in frequency domain, the amplitude spectrum tuned method is used to suppress the background noise; in temporal domain, the incremental learning is employed to efficiently update background model from high dimensional video streams. Extensive experiments have been conducted to validate the effectiveness of the proposed model.

Deep Ensemble Machine for Video Classification.

Video classification has been extensively researched in computer vision due to its wide spread applications. However, it remains an outstanding task because of the great challenges in effective spatial-temporal feature extraction and efficient classification with high-dimensional video representations. To address these challenges, in this paper, we propose an end-to-end learning framework called deep ensemble machine (DEM) for video classification. Specifically, to establish effective spatio-temporal features, we propose using two deep convolutional neural networks (CNNs), i.e., vision and graphics group and C3-D to extract heterogeneous spatial and temporal features for complementary representations. To achieve efficient classification, we propose ensemble learning based on random projections aiming to transform high-dimensional features into a set of lower dimensional compact features in subspaces; an ensemble of classifiers is trained on the subspaces and combined with a weighting layer during the backpropagation. To further enhance the performance, we introduce rectified linear encoding (RLE) inspired from error-correcting output coding to encode the initial outputs of classifiers, followed by a softmax layer to produce the final classification results. DEM combines the strengths of deep CNNs and ensemble learning, which establishes a new end-to-end learning architecture for more accurate and efficient video classification. We show the great effectiveness of DEM by extensive experiments on four data sets for diverse video classification tasks including action recognition and dynamic scene classification. Results have shown that DEM achieves high performance on all tasks with an improvement of up to 13% on CIFAR10 data set over the baseline model.

A Clustering Algorithm for Key Frame Extraction Based on Density Peak

Aiming at the problem of video key frame extraction, a density peak clustering algorithm is proposed, which uses the HSV histogram to transform high-dimensional abstract video image data into quantifiable low-dimensional data, and reduces the computational complexity while capturing image features. On this basis, the density peak clustering algorithm is used to cluster these low-dimensional data and find the cluster centers. Combining the clustering results, the final key frames are obtained. A large number of key frame extraction experiments for different types of videos show that the algorithm can extract different number of key frames by combining video content, overcome the shortcoming of traditional key frame extraction algorithm which can only extract a fixed number of key frames, and the extracted key frames can represent the main content of video accurately.

Fast action retrieval from videos via feature disaggregation

Learning based hashing methods, which aim at learning similarity-preserving binary codes for efficient nearest neighbor search, have been actively studied recently. A majority of the approaches address hashing problems for image collections. However, due to the extra temporal information, videos are usually represented by much higher dimensional (thousands or even more) features compared with images, causing high computational complexity for conventional hashing schemes. In this paper, we propose a simple and efficient hashing scheme for high-dimensional video data. This method, called Disaggregation Hashing (DH), exploits the correlations among different feature dimensions. An intuitive feature disaggregation method is first proposed, followed by a novel hashing algorithm based on different feature clusters. Additionally, a kernelized version of DH is proposed for better performance. We demonstrate the efficiency and effectiveness of our method by theoretical analysis and exploring its application on action retrieval from video databases. Extensive experiments show the superiority of our binary coding scheme over state-of-the-art hashing methods.

A new indexing method for complex similarity queries in immersive multimedia systems

This paper proposes a novel indexing method for complex similarity queries in high-dimensional image and video systems. In order to provide the indexing method with the flexibility in dealing with multiple features and multiple query objects, we treat every dimension independently. The efficiency of our method is realized by a specialized bitmap indexing that represents all objects in a database as a set of bitmaps. The percentage of data accessed in our indexing method is inversely proportional to the overall dimensionality, and thus the performance deterioration with the increasing dimensionality does not occur. To demonstrate the efficacy of our method we conducted extensive experiments and compared the performance with the VA-file-based index and the linear scan by using real image and video datasets, and obtained a remarkable speed-up over them.