Large Amount Of Video Data Research Articles

COVID-19 Guidelines: A Multimodal Video Analysis of Student Behavioral Compliance during Senior Secondary Certificate Examination

In order to address the economic pressure and the negative impact of school closure on students, Nigerian government approved the reopening of secondary schools for graduating classes starting from 4th August, 2020, with comprehensive guidelines and protocols. Regrettably, many students were reported to have tested positive to COVID-19, questioning their compliance towards the guidelines. This study therefore investigates students’ behavioral compliance towards COVID-19 guidelines, and factors predicting such behavior. Participants involved 205 graduating secondary school students whose behavioral compliance towards the guidelines was observed over three examination days. Although coding large amount of video data and identifying social actions from the data had become a huge challenge for many researchers, our study show that observable behavior can be catalogued using multimodal approach to identify and characterize behavioral frames in rich video data. Three state Hidden Markov Models was estimated in R package on three observed and categorized behavior: hand-washing, use of face mask, and social distancing. Based on probabilities of occurrence of these behaviors, three behavioral frames emerged: cautious, reluctant and defiant attitudes. Results show that defiant attitude was the most prevalent among the behavioral frames, with some level of alternation between reluctant and cautious frame. Our follow-up OLS model indicates that perceived health threat, perceived clarity of guidelines, obligation to obey rule, moral alignment, emotional state, and impulsivity significantly predict students’ behavioral compliance towards COVID-19 guidelines. We recommend stringent measures and intensive awareness campaign to mitigate students’ offending behavior.

Navigating in Numerous Video Data: User Interface Design for an On-Camera Video Analytics Engine

Video analytics powered by artificial intelligence shows high promise in making our society smarter. Harnessing large amounts of video data, however, requires the development of processing systems demonstrating high performance and high efficiency. To this end, this work has contributed to a video analytics system powered by artificial intelligence for object detection and recognition. Rather than streaming all the video frames to the cloud, the system analyzes images on-camera and only returns those of interest to the cloud. This edge analytics research-grade software is available, but it lacks a simple web interface for general use by scientists, engineers, and other experts. To make the system versatile and user-friendly, a web interface was developed for receiving user queries (including parameters such as camera geolocations, video time span, and object class of interest) and presenting the query results. This web interface will make our video analytics system more accessible to domain experts (those in law enforcement, health care, environmental monitoring, etc.).

A Distributed Automatic Video Annotation Platform

In the era of digital devices and the Internet, thousands of videos are taken and share through the Internet. Similarly, CCTV cameras in the digital city produce a large amount of video data that carry essential information. To handle the increased video data and generate knowledge, there is an increasing demand for distributed video annotation. Therefore, in this paper, we propose a novel distributed video annotation platform that explores the spatial information and temporal information. Afterward, we provide higher-level semantic information. The proposed framework is divided into two parts: spatial annotation and spatiotemporal annotation. Therefore, we propose a spatiotemporal descriptor, namely, volume local directional ternary pattern-three orthogonal planes (VLDTP–TOP) in a distributed manner using Spark. Moreover, we developed several state-of-the-art appearance-based and spatiotemporal-based feature descriptors on top of Spark. We also provide the distributed video annotation services for the end-users so that they can easily use the video annotation and APIs for development to produce new video annotation algorithms. Due to the lack of a spatiotemporal video annotation dataset that provides ground truth for both spatial and temporal information, we introduce a video annotation dataset, namely, STAD which provides ground truth for spatial and temporal information. An extensive experimental analysis was performed in order to validate the performance and scalability of the proposed feature descriptors, which proved the excellence of our proposed approach.

Towards Clustering Hand Grasps of Individuals with Spinal Cord Injury in Egocentric Video.

Cervical spinal cord injury (cSCI) can cause paralysis and impair hand function. Existing assessments in clinical settings do not reflect an individual's performance in their daily environment. Videos from wearable cameras (egocentric video) provide a novel avenue to analyze hand function in non-clinical settings. Due to the large amounts of video data generated by this approach, automated analysis methods are necessary. We propose to employ an unsupervised learning process to produce a summary of the grasping strategies used in an egocentric video. To this end, an approach was developed consisting of hand detection, pose estimation, and clustering algorithms. The performance of the method was examined with external evaluation indicators and internal evaluation indicators for an uninjured and injured participant, respectively. The results demonstrated that a Gaussian mixture model obtained the highest accuracy in terms of the maximum match, 0.63, and the Rand index, 0.26, for the uninjured participant, and a silhouette score of 0.13 for the injured participant.

Person anomaly detection-based videos surveillance system in urban integrated pipe gallery

ABSTRACT The integrated pipe gallery, also known as urban lifeline, is a significant content of the smart city. While the video surveillance system is a crucial part of the integrated pipe gallery, which provides a basis for the construction of smart city. Due to the large amount of video data, manual monitoring is a time-consuming and laborious task. To address the above problems, we propose a neural network-based method that incorporates the concept of area under curve (AUC) with the multiple-instance learning (MIL) approach. We formulate the multiple-instance AUC (MIAUC) model that predicts high anomaly scores for anomalous segments. Furthermore, sparsity and temporal smoothness constraints are utilized in the loss function to better detect anomaly. To verify the effectiveness of our proposed method, a new database is established based on the video surveillance system, which consists of 110 real-world surveillance videos with a total length of 24 h. The experimental results on the real-world database show that our method achieves better performance as compared to the baselines methods. Moreover, we design a MIAUC-based video surveillance system and the practical effect reveals the prospect of utilizing the MIL method for person anomaly detection in the integrated pipe gallery.

RETRACTED ARTICLE: A kernel support vector machine based anomaly detection using spatio-temporal motion pattern models in extremely crowded scenes

Millions of security cameras were placed in public spaces, generating large quantities of video data. There is a need to develop smart techniques to identify and classify objects tracking instantly. Most of them concentrate on spatial information, resulting in exposure to noise and background movement. In addition, monitoring individuals in overcrowded scenes is a difficult task, due to the variation of movement and appearance created by the large amount of people in the scene. In this paper, initially, utilizing threshold value, the video is split into frames. Then segmentation of moving objects using Extended Kalman Filters (EKF) to improve the accuracy of the classification. Instead, to distinguish between the foreground object and the background object, the texture features is removed. The artifacts are then labelled using improved Learning Vector Quantization (LVQ) for efficient identification of anomalies. Also an effective classification of Kernel Support Vector Machine (KSVM) predicated on anomaly detection has been suggested utilizing spatio-temporal movement pattern models in overcrowded scenes to solve these problems. Hence, KSVM is more advantageous in accuracy which is used to monitor the object. The result shows the performance of the proposed KSVM obtained high performance compared with SVM and Hidden Markov Model (HMM).

Lossy and Lossless Video Frame Compression: A Novel Approach for High-Temporal Video Data Analytics

The smart city concept has attracted high research attention in recent years within diverse application domains, such as crime suspect identification, border security, transportation, aerospace, and so on. Specific focus has been on increased automation using data driven approaches, while leveraging remote sensing and real-time streaming of heterogenous data from various resources, including unmanned aerial vehicles, surveillance cameras, and low-earth-orbit satellites. One of the core challenges in exploitation of such high temporal data streams, specifically videos, is the trade-off between the quality of video streaming and limited transmission bandwidth. An optimal compromise is needed between video quality and subsequently, recognition and understanding and efficient processing of large amounts of video data. This research proposes a novel unified approach to lossy and lossless video frame compression, which is beneficial for the autonomous processing and enhanced representation of high-resolution video data in various domains. The proposed fast block matching motion estimation technique, namely mean predictive block matching, is based on the principle that general motion in any video frame is usually coherent. This coherent nature of the video frames dictates a high probability of a macroblock having the same direction of motion as the macroblocks surrounding it. The technique employs the partial distortion elimination algorithm to condense the exploration time, where partial summation of the matching distortion between the current macroblock and its contender ones will be used, when the matching distortion surpasses the current lowest error. Experimental results demonstrate the superiority of the proposed approach over state-of-the-art techniques, including the four step search, three step search, diamond search, and new three step search.

Video Big Data Analytics in the Cloud: A Reference Architecture, Survey, Opportunities, and Open Research Issues

The proliferation of multimedia devices over the Internet of Things (IoT) generates an unprecedented amount of data. Consequently, the world has stepped into the era of big data. Recently, on the rise of distributed computing technologies, video big data analytics in the cloud has attracted the attention of researchers and practitioners. The current technology and market trends demand an efficient framework for video big data analytics. However, the current work is too limited to provide a complete survey of recent research work on video big data analytics in the cloud, including the management and analysis of a large amount of video data, the challenges, opportunities, and promising research directions. To serve this purpose, we present this study, which conducts a broad overview of the state-of-the-art literature on video big data analytics in the cloud. It also aims to bridge the gap among large-scale video analytics challenges, big data solutions, and cloud computing. In this study, we clarify the basic nomenclatures that govern the video analytics domain and the characteristics of video big data while establishing its relationship with cloud computing. We propose a service-oriented layered reference architecture for intelligent video big data analytics in the cloud. Then, a comprehensive and keen review has been conducted to examine cutting-edge research trends in video big data analytics. Finally, we identify and articulate several open research issues and challenges, which have been raised by the deployment of big data technologies in the cloud for video big data analytics. To the best of our knowledge, this is the first study that presents the generalized view of the video big data analytics in the cloud. This paper provides the research studies and technologies advancing video analyses in the era of big data and cloud computing.

Start Code-Based Encryption and Decryption Framework for HEVC

In this article, we propose a new selective encryption and decryption framework based on the start code for high efficiency video coding. There is a growing need to encrypt video information to protect video content from privacy invasion and intellectual property infringement caused by information leakage. Although encrypting an entire video is a straightforward approach, the cost to encrypting a large amount of video data is substantial, considering the resulting computational complexity. Consequently, selective encryption algorithms have been actively researched in recent years and have contributed to reducing the computational complexity. However, existing selective encryption algorithms have certain drawbacks. For instance, it is difficult to separate the video encryption algorithm from the video compression algorithm because the encryption framework is based on the syntax elements. Further, a partial reconstruction of the encrypted video bitstream is often unavoidable. To solve these problems, the proposed method encrypts the bitstream based on the start code rather than on the syntax elements. Encrypting the bitstream partially, based on the start code, makes it easy to separate the video encryption algorithm from the video compression algorithm. Furthermore, encrypting the part adjacent to the start code protects the video content, as video reconstruction using a video decoder is impossible, unless the correct start code is returned to the bitstream. The experimental results show that the proposed method reduced the encryption and decryption times by approximately 97% and 98%, respectively, compared to the encryption and decryption of the entire video bitstream.

FedVision: Federated Video Analytics With Edge Computing

Widely deployed smart cameras are generating a large amount of video data and capable of processing frames on devices. Empowered by edge computing, the video data can also be offloaded to edge servers for processing. By leveraging the on-device processing and computation offloading, we propose a federated video analytics system named FedVision to efficiently provision video analytics across devices and servers. The challenge of designing FedVision is to optimally use the computing and networking resources for video analytics. Since there is no closed-form expression of the system performance, black-box optimization is employed to optimize the system performance. However, using black-box optimization directly incurs excessive system queries that lead to very poor system performance. To solve this problem, we design a new optimization method that integrates black-box optimization with Neural Processes (NPs) as a system performance approximator. This method allows black-box optimizer to query NPs instead of the real system. We validate the performance of FedVision and the new optimization method using both numerical results and experiments with a testbed.

SIAT: A Distributed Video Analytics Framework for Intelligent Video Surveillance

In recent years, the amount of intelligent CCTV cameras installed in public places for surveillance has increased enormously and as a result, a large amount of video data is produced every moment. Due to this situation, there is an increasing request for the distributed processing of large-scale video data. In an intelligent video analytics platform, a submitted unstructured video undergoes through several multidisciplinary algorithms with the aim of extracting insights and making them searchable and understandable for both human and machine. Video analytics have applications ranging from surveillance to video content management. In this context, various industrial and scholarly solutions exist. However, most of the existing solutions rely on a traditional client/server framework to perform face and object recognition while lacking the support for more complex application scenarios. Furthermore, these frameworks are rarely handled in a scalable manner using distributed computing. Besides, existing works do not provide any support for low-level distributed video processing APIs (Application Programming Interfaces). They also failed to address a complete service-oriented ecosystem to meet the growing demands of consumers, researchers and developers. In order to overcome these issues, in this paper, we propose a distributed video analytics framework for intelligent video surveillance known as SIAT. The proposed framework is able to process both the real-time video streams and batch video analytics. Each real-time stream also corresponds to batch processing data. Hence, this work correlates with the symmetry concept. Furthermore, we introduce a distributed video processing library on top of Spark. SIAT exploits state-of-the-art distributed computing technologies with the aim to ensure scalability, effectiveness and fault-tolerance. Lastly, we implant and evaluate our proposed framework with the goal to authenticate our claims.

Variational Autoencoder-Based Multiple Image Captioning Using a Caption Attention Map

Image captioning is a promising research topic that is applicable to services that search for desired content in a large amount of video data and a situation explanation service for visually impaired people. Previous research on image captioning has been focused on generating one caption per image. However, to increase usability in applications, it is necessary to generate several different captions that contain various representations for an image. We propose a method to generate multiple captions using a variational autoencoder, which is one of the generative models. Because an image feature plays an important role when generating captions, a method to extract a Caption Attention Map (CAM) of the image is proposed, and CAMs are projected to a latent distribution. In addition, methods for the evaluation of multiple image captioning tasks are proposed that have not yet been actively researched. The proposed model outperforms in the aspect of diversity compared with the base model when the accuracy is comparable. Moreover, it is verified that the model using CAM generates detailed captions describing various content in the image.

A Low-Complexity Mosaicing Algorithm for Stock Assessment of Seabed-Burrowing Species

This paper proposes an algorithm for mosaicing videos generated during stock assessment of seabed-burrowing species. In these surveys, video transects of the seabed are captured and the population is estimated by counting the number of burrows in the video. The mosaicing algorithm is designed to process a large amount of video data and summarize the relevant features for the survey in a single image. Hence, the algorithm is designed to be computationally inexpensive while maintaining a high degree of robustness. We adopt a registration algorithm that employs a simple translational motion model and generates a mapping to the mosaic coordinate system using a concatenation of frame-by-frame homographies. A temporal smoothness prior is used in a maximum a posteriori homography estimation algorithm to reduce noise in the motion parameters in images with small amounts of texture detail. A multiband blending scheme renders the mosaic and is optimized for the application requirements. Tests on a large data set show that the algorithm is robust enough to allow the use of mosaics as a medium for burrow counting. This will increase the verifiability of the stock assessments as well as generate a ground truth data set for the learning of an automated burrow counting algorithm.

Forensics Usage of Video-Surveillance and Associated (Meta) Data

Forensic video analysis is the offline analysis of video aimed at finding video or digital evidence during an investigation. In order to find video evidence during an investigation (terrorism, kidnapping, homicide, ...), investigators need to analyze and watch large amounts of video data (equivalent to a high number of video hours). Despite the powerful existing techniques and algorithms for video processing, investigators are still facing many problems including the large volume of video data and time consuming. The correct interpreta-tion of videos requires a set of accurate miscellaneous data called the metadata. In this paper, we focus on the contribution that can come from the correlation between video surveillance, metadata and context infor-mation. Such a contribution can be demonstrated at the end of the projects we are currently working on in collaboration with the police services.

Reduction of bubble-like frames using a RSS filter in wireless capsule endoscopy video

Wireless capsule endoscopy (WCE) is an advanced technique for inner visualization of the digestive tract. However, its large amount of video data collected continuously for several hours by wireless image sensors usually results in tiresome and tedious reading for physicians. To develop this technique in practice, the automatic reduction of large amounts of video data arises as a helpful alternative, which is partly implemented by detecting the non-useful frames, such as those filled with intestinal juices accompanying large bubble-like regions. Based on the bubbles’ morphology properties, a novel automatic approach for detecting the bubble-like frames in capsule video endoscopy that is based on a new ring shape selective (RSS) filter is presented. According to the shape of the bubbles’ bright ring boundary, a group of the shape probability functions and the response intensity functions produced by Hessian matrix eigenvalues are used to construct the RSS filters to enhance the boundary areas of the bubbles. Then, the bubble-like frames can be distinguished by in the bright areas after binary and morphological processing in the filtered images. The experimental results show that the uninformative bubble-like frames can be detected effectively by this approach even under non-uniform illumination conditions.

DC programming for solving a sparse modeling problem of video key frame extraction

Extracting key frames from a video can reduce redundancies in continuous scenes and pithy represent the entire video. This technique copes with the issue of how to efficiently manage a large amount of video data and has many applications such as video indexing, querying, and browsing. Current key frame extraction methods often utilize sparse modeling, which assumes that each video frame can be expressed as a linear combination of a few representative key frames. Although these methods are successful, they are less aware of the fact that videos have spatio-temporal structure and used ℓ1 norm based sparsity measure. In this paper, we propose a key frame extraction method that considers spatio-temporal information of videos. We employ a sparsity measure that is based on the determinant of the Gram matrix computed from entire video frames. The determinant sparsity measure is computed for the entire matrix, not for single frames, and thus it reflects the spatial or joint sparseness of the entire video. With this measure, the solutions are sparser than conventional sparseness measures but the cost function is non-convex and optimization becomes harder. Then, utilizing the fact that the proposed cost function is the difference of two convex functions, we employ an efficient algorithm based on the difference-of-convex (DC) programming, which often finds the global solution of the non-convex cost function. Experiments show that the proposed algorithm generates high-quality, high-compression key frames, compared with the existing key frame extraction method with the ℓ1 norm. The determinant sparsity measure was recently proposed and can be utilized for more video processing applications such as video saliency detection.

Robust video summarization using collaborative representation of adjacent frames

With the ever increasing volume of video content, efficient and effective video summarization (VS) techniques are urgently demanded to manage a large amount of video data. Recent developments on sparse representation based approaches have demonstrated promising results for VS. However, these existing approaches treat each frame independently, so the performance can be greatly influenced by each individual frame. In this paper, we formulate the VS problem with a collaborative representation model to take the visual similarity of adjacent frames into consideration. To be specific, during the procedure of reconstruction, both each individual frame and their adjacent frames are reconstructed collaboratively, so the impact of an individual frame can be weakened. In addition, a greedy iterative algorithm is designed for model optimization, where the sparsity and the average percentage of reconstruction (APOR) are adopted to control the iteration. Experimental results on two benchmark datasets with various types of videos demonstrate that the proposed method not only outperforms the state of the art, but also improves the robustness to transitional frames and “outlier” frames.

Depth Super-Resolution on RGB-D Video Sequences with Large Displacement 3D Motion.

To enhance the resolution and accuracy of depth data, some video-based depth super-resolution methods have been proposed which utilizes its neighboring depth images in the temporal domain. They often consist of two main stages: motion compensation of temporally neighboring depth images and fusion of compensated depth images. However, large displacement 3D motion often leads to compensation error, and the compensation error is further introduced into the fusion. A video-based depth super-resolution method with novel motion compensation and fusion approaches is proposed in this paper. We claim that, 3D Nearest Neighboring Field (NNF) is a better choice than using positions with true motion displacement for depth enhancements. To handle large displacement 3D motion, the compensation stage utilized 3D NNF instead of true motion used in previous methods. Next, the fusion approach is modeled as a regression problem to predict the super-resolution result efficiently for each depth image by using its compensated depth images. A new deep convolutional neural network architecture is designed for fusion, which is able to employ a large amount of video data for learning the complicated regression function. We comprehensively evaluate our method on various RGB-D video sequences to show its superior performance.

Multi-marker tracking for large-scale X-ray stereo video data

Analyzing large amounts of video data is one of the key challenges in the trend towards big data. In the field of medical research, for example, to analyze infected cardiac movements, stereo X-ray sequences of beating animal hearts implanted with radiopaque markers are recorded. As manual annotation of exact marker positions in large-scale recordings is time-consuming and infeasible, research on automatic tracking of multiple markers is a crucial task. We propose an efficient two-stage graph-based data association approach to tackle this problem. Difficulties of the sequences like 2D occlusions, low contrast, inhomogeneous movement, and inaccurate detections, are considered in the framework. Reconstructed 3D observations are modeled and connected using a weighted directed acyclic graph to obtain tracklets with high confidence via shortest path extraction. Afterwards, tracklets are linked into longer tracks by a tracklet graph in a similar manner while global features are adopted. The approach is validated on eight X-ray cardiac datasets of beating sheep hearts with various diseases. Outperforming standard tracking approaches, e.g. particle filter, the experimental results show a high accuracy comparable to human experts and efficiency in the meantime. The proposed approach is generic and can be directly applied to other video data as well.

Automatic key frame extraction in continuous videos from construction monitoring by using color, texture, and gradient features

Abstract On-site video recording systems are increasingly used for monitoring construction activities. The recorded videos contain rich and useful jobsite information that can be used for a variety of purposes. The large amount of video data generated by continuous monitoring, however, creates tremendous challenges on data storage and retrieval. Due to the relatively slow pace of construction activities, a significant portion of the recorded data is redundant. Therefore, archiving raw construction videos into a concise and structured set of key frames would facilitate data storage, retrieval and analysis. Three key issues in automatic key frame extraction from construction videos are studied, including the selection of proper video features, scene segmentation, and key frame extraction. New image features and methods are developed to address the three issues. A validation experiment indicates that the developed features and methods can effectively and efficiently extract representative key frames from the complex and dynamic construction videos. The developed techniques can be used to develop a construction video summary system that serves the purpose of effectively archiving construction jobsite videos.