Massive Amounts Of Video Data Research Articles

Adaptive spatial down-sampling method based on object occupancy distribution for video coding for machines

As the performance of machine vision continues to improve, it is being used in various industrial fields to analyze and generate massive amounts of video data. Although the demand for and consumption of video data by machines has increased significantly, video coding for machines needs to be improved. It is therefore necessary to consider a new codec that differs from conventional codecs based on the human visual system (HVS). Spatial down-sampling plays a critical role in video coding for machines because it reduces the volume of the video data to be processed while maintaining the shape of the data’s features that are important for the machine to reference when processing the video. An effective method of determining the intensity of spatial down-sampling as an efficient coding tool for machines is still in the early stages. Here, we propose a method of determining an optimal scale factor for spatial down-sampling by collecting and analyzing information on the number of objects and the ratio of the area occupied by the object within a picture. We compare the data reduction ratio to the machine accuracy error ratio (DRAER) to evaluate the performance of the proposed method. By applying the proposed method, the DRAER was found to be a maximum of 21.40 dB and a minimum of 11.94 dB. This shows that video coding gain for the machines could be achieved through the proposed method while maintaining the accuracy of machine vision tasks.

Improved RNN Model for Real-Time Human Activity Recognition

This study focuses on the automatic detection of human actions in video streams. The requirement to detect what human activities happen in videos is recognition of human action due to significant differences in people's visual and motion appearance and actions, camera perspective shifts, moving background, occlusions, noise, and a massive amount of video data. The human activity recognition challenge involves identifying physical activities carried out by individuals or groups based on traces of movements, including gestures, actions, interactions, and group activities. The detection of concepts usually requires additional annotations for the training dataset. In this paper, useful methods for categorizing human action recognition are discussed. The current models are an accurate deep learning method that is based on models that have been changed to be more useful. The large disparities that result from the backdrop and the size of the objects have prevented the identification of activities in videos from being fully and effectively addressed. The main objective is to achieve better accuracy for the Long Short-Term Memory (LSTM) method, which was used to improve the Recurrent Neural Networks (RNN) model. In this paper, LSTM is used to come up with models for different action recognition tasks. The model was made better by making the LSTM have four layers and putting 128 units, 64 units, 32 units, and 16 units in each layer, respectively. In addition, the performance evaluation of deep learning-based approaches has been compared to other related works. Therefore, an improved approach to RNN is proposed to recognize human actions. To classify the videos, a multilayer RNN with a specific type of LSTM is used to extract features from video sequences. The UCF-101 and UCF Sports human action recognition datasets are utilized in this study for both training and assessment. Test findings demonstrate that the suggested strategy achieved increased accuracy. Finally, the enhanced RNN model's total model accuracy in the UCF-101 dataset is 93.78% and 95.70% for the UCF Sport dataset.

Fast and accurate visual vibration measurement via derivative-enhanced phase-based optical flow

Visual vibration measurement using a digital camera is an emerging non-contact modality capable of enabling flexibility, high spatial resolution, and full field. However, the massive amounts of video data with vibration information (thousands of time-lapse images per measurement) to be handled lead to an actual bottleneck. This prevents the system from extracting the vibration information with high speed and fidelity. Here, a straightforward phase-based optical flow approach is proposed by introducing a derivative operation to perform vibration measurements in a more accurate and efficient manner, in contrast to the state-of-the-art phase-based visual method. This approach generates analytical signals by considering each raw gray image as the real part and its first-order derivative as the imaginary part. As a result, the physical vibration can be extracted from the phase variations of the analytical signals from one to the next. Numerical simulations and experiments are conducted to validate the effectiveness and accuracy of this simple but powerful approach. The results indicate that the proposed approach provides a significantly higher speed than current methods and a correlation coefficient of 99.2 % with the identification results.

Edge Intelligence-Assisted Asymmetrical Network Control and Video Decoding in the Industrial IoT with Speculative Parallelization

Industrial Internet of Things (IIoTs) has drawn significant attention in the industry. Among its rich applications, the field’s video surveillance deserves particular interest due to its advantage in better understanding network control. However, existing decoding methods are limited by the video coding order, which cannot be decoded in parallel, resulting in low decoding efficiency and the inability to process the massive amount of video data in real time. In this work, a parallel decoding framework based on the speculative technique is proposed. In particular, the video is first speculatively decomposed into data blocks, and then a verification method is designed to ensure the correctness of the decomposition. After verification, the data blocks having passed the validation can be decoded concurrently in the parallel computing platform. Finally, the concurrent decoding results are concatenated in line with the original encoding order to form the output. Experiments show that compared with traditional serial decoding ones, the proposed method can improve the performance by 9 times on average in the parallel computing environment with NVIDIA Tegra 4 chips, thus significantly enhancing the real-time video data’s decoding efficiency with guaranteed accuracy. Furthermore, proposed and traditional serial methods obtain almost the same peak signal-to-noise ratio (PSNR) and mean square error (MSE) metrics at different bit rates and resolutions, showing that the introduction of the speculative technique does not degrade the decoding accuracy.

DeepVQL: Deep Video Queries on PostgreSQL

The recent development of mobile and camera devices has led to the generation, sharing, and usage of massive amounts of video data. As a result, deep learning technology has gained attention as an alternative for video recognition and situation judgment. Recently, new systems supporting SQL-like declarative query languages have emerged, focusing on developing their own systems to support new queries combined with deep learning that are not supported by existing systems. The proposed DeepVQL system in this paper is implemented by expanding the PostgreSQL system. DeepVQL supports video database functions and provides various user-defined functions for object detection, object tracking, and video analytics queries. The advantage of this system is its ability to utilize queries with specific spatial regions or temporal durations as conditions for analyzing moving objects in traffic videos.

Robust Video-Text Retrieval Via Noisy Pair Calibration

Video-text retrieval is a fundamental task in managing the emerging massive amounts of video data. The main challenge focuses on learning a common representation space for videos and queries where the similarity measurement can reflect the semantic closeness. However, existing video-text retrieval models may suffer from the following noise in the common space learning procedure: First, the video-text correspondences in positive pairs may not be exact matches. The crowdsourcing annotation for existing datasets leads to inevitable tagging noise for non-expert annotators. Second, the learning of video-text representation is based on the negative samples randomly sampled. Instances that are semantically similar to the query may be incorrectly categorized as negative samples. To alleviate the adverse impact of these noisy pairs, we propose a novel robust video-text retrieval method that protects the model from noisy positive and negative pairs by identifying and calibrating noisy pairs with their uncertainty score. In particular, we propose a noisy pair identifier, which divides the training dataset into noisy and clean subsets based on the estimated uncertainty of each pair. Then, with the help of uncertainties, we calibrate the two types of noisy pairs with an adaptive margin triplet loss and a weighted triplet loss function, respectively. To verify the effectiveness of our methods, we conduct extensive experiments on three widely used datasets. Experimental results show that the proposed robust video-text retrieval methods successfully identify and calibrate the noisy pairs and improve retrieval performance.

An NLP-guided ontology development and refinement approach to represent and query visual information

The ubiquitous presence of surveillance systems generates massive amounts of video data. Storage and analysis of this data in real-time is a substantial challenge. There is huge potential in representing data in machine-readable and machine-interpretable format due to the presence of hidden semantics in images and videos. However, such representation requires ontology, which calls for expert domain knowledge. In this paper, a novel NLP-guided approach to generate an ontology for multimedia representation and information retrieval is proposed. A semi-automatic NLP-guided framework, which extracts all possible relations among objects is presented. This framework leverages the textual data of the domain to generate possible descriptions and actions within the domain. Relations among objects get embedded as object properties, whereas the category of an object as a class. Features and attributes of objects encode the data properties of the ontology. The proposed ontology is compared with existing multimedia ontologies and evaluated with regard to its capability to represent relations occurring in benchmark datasets, demonstrating the completeness and thorough coverage of the domain concepts. Spatial reasoning rules are established using Semantic Web Rule Language (SWRL) rules, and information retrieval is demonstrated using Description Logic (DL) and SPARQL queries. The proposed NLP-guided ontology generation approach is general enough to help in the development of ontologies for other domains as well, by providing video and textual data of the domain of interest, with limited human involvement.

A Video Key Frame Extraction Method Based on Multiview Fusion

A massive amount of video data is stored in the real-time road monitoring system, especially in high-speed scenes. Traditional methods of video key frame extraction have the problems of large computation and long-time consumption. Thus, it is imperative to decrease the massive video data generated by monitoring and help researchers to study key frames. Aiming at the above problems, we propose an efficient key frame extraction method based on multiview fusion, where the autoencoder is used to compress the video data. Specifically, all the video frames of the video data are subjected to feature dimensionality reduction, and the features after dimensionality reduction are subjected to multiview fusion. Finally, dynamic programming and clustering are used to extract key frames. The experimental results show that the proposed method has lower computational complexity in extracting key frames, while the mutual information in the extracted key frames is large. It illustrates the reliability and efficiency of the proposed method, which provides technical support for subsequent video research.

Automated Quantification of Brittle Stars in Seabed Imagery Using Computer Vision Techniques.

Underwater video surveys play a significant role in marine benthic research. Usually, surveys are filmed in transects, which are stitched into 2D mosaic maps for further analysis. Due to the massive amount of video data and time-consuming analysis, the need for automatic image segmentation and quantitative evaluation arises. This paper investigates such techniques on annotated mosaic maps containing hundreds of instances of brittle stars. By harnessing a deep convolutional neural network with pre-trained weights and post-processing results with a common blob detection technique, we investigate the effectiveness and potential of such segment-and-count approach by assessing the segmentation and counting success. Discs could be recommended instead of full shape masks for brittle stars due to faster annotation among marker variants tested. Underwater image enhancement techniques could not improve segmentation results noticeably, but some might be useful for augmentation purposes.

When Crowd Meets Big Video Data: Cloud-Edge Collaborative Transcoding for Personal Livecast

Deep penetration of personal computing devices and high-speed Internet has enabled everyone to be a broadcaster. In this crowdsourced live streaming service, numerous amateur broadcasters lively stream their video contents to viewers around the world. Consequently, these broadcasters generate a massive amount of video data. The set of video sources and recipients are big as well, so are demand for the storage and computational resources. Transcoding becomes a must to better service these viewers with different network and device configurations. However, the massive amount of video data contributed by countless channels even makes cloud significantly expensive for providing transcoding services to the whole community. In this paper, inspired by the paradigm of Edge Computing, we propose a Cloud-edge collaborative system which combines the idle end-viewers' resources with the cloud to transcode the massive amount of videos at scale. Specifically, we put forward tailored viewer selection algorithms after empirically analyses the viewer behavior data. In the meantime, we propose auction-based payment schemes to motivate these viewers participating in the transcoding. Large-scale trace-driven simulations demonstrate the superiority of our approach in cost reduction and service stability. We further implement a prototype in PlanetLab to prove the feasibility of our design.

Cloud-Assisted Multiview Video Summarization Using CNN and Bidirectional LSTM

The massive amount of video data produced by surveillance networks in industries instigate various challenges in exploring these videos for many applications, such as video summarization (VS), analysis, indexing, and retrieval. The task of multiview video summarization (MVS) is very challenging due to the gigantic size of data, redundancy, overlapping in views, light variations, and interview correlations. To address these challenges, various low-level features and clustering-based soft computing techniques are proposed that cannot fully exploit MVS. In this article, we achieve MVS by integrating deep neural network based soft computing techniques in a two-tier framework. The first online tier performs target-appearance-based shots segmentation and stores them in a lookup table that is transmitted to cloud for further processing. The second tier extracts deep features from each frame of a sequence in the lookup table and pass them to deep bidirectional long short-term memory (DB-LSTM) to acquire probabilities of informativeness and generates a summary. Experimental evaluation on benchmark dataset and industrial surveillance data from YouTube confirms the better performance of our system compared to the state-of-the-art MVS methods.

A New Intelligent Video Surveillance Architecture

Aimed at the old mode of traditional video surveillance for manual operation and forensics, and the fact that massive amounts of video data occupy a large amount of storage and transmission resources, an intelligent video surveillance architecture based on network and big data is proposed. The architecture includes three core modules: face real-time detecting, face intelligent coding, and face fast retrieval, which can automatically detect human faces, and then intelligently encode them and compare them with databases quickly. It greatly reduces the space occupied by video data storage and transmission, improves the intelligent level of video surveillance, and transforms “after-the-fact analysis” to “in-the-fact analysis” to meet modern criminal investigation requirements.

A Fast-Iterative Data Association Technique for Multiple Object Tracking

A massive amount of video data is recorded daily for forensic post analysis and computer vision applications. The analyses of this data often require multiple object tracking (MOT). Advancements in image analysis algorithms and global optimization techniques have improved the accuracy of MOT, often at the cost of slow processing speed which limits its applications only to small video datasets. With the focus on speed, a fast-iterative data association technique (FIDA) for MOT that uses a tracking-by-detection paradigm and finds a locally optimal solution with a low computational overhead is introduced. The performance analyses conducted on a set of benchmark video datasets show that the proposed technique is significantly faster (50–600 times) than the existing state-of-the-art techniques that produce a comparable tracking accuracy.

DPPDL: A Dynamic Partial-Parallel Data Layout for Green Video Surveillance Storage

Video surveillance requires storing massive amounts of video data, which results in the rapid increasing of storage energy consumption. With the popularization of video surveillance, green storage for video surveillance is very attractive. The existing energy-saving methods for massive storage mostly concentrate on the data centers, mainly with random access, whereas the storage of video surveillance has inherent workload characteristics and access pattern, which can be fully exploited to save more energy. A dynamic partial-parallel data layout (DPPDL) is proposed for green video surveillance storage. It adopts a dynamic partial-parallel strategy, which dynamically allocates the storage space with an appropriate degree of partial parallelism according to performance requirement. Partial parallelism benefits energy conservation by scheduling only partial disks to work; a dynamic degree of parallelism can provide appropriate performances for various intensity workloads. DPPDL is evaluated by a simulated video surveillance consisting of 60–300 cameras with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1920 \times 1080$ </tex-math></inline-formula> pixels. The experiment shows that DPPDL is most energy efficient, while tolerating single disk failure and providing more than 20% performance margin. On average, it saves 7%, 19%, 31%, 36%, 56%, and 59% more energy than a CacheRAID, Semi-RAID, Hibernator, MAID, eRAID5, and PARAID, respectively.

A NEW COMBINATION METHOD FOR ENCRYPTION OF MOVING OBJECTS DETECTION IN VIDEO

Nowadays surveillance systems have been widely deployed in various places and generate massive amount of video data every day. This raises threats of unauthorized access and potential privacy leakage as the recorded videos usually contain rich identifiable information such as facial biometrics. In order to mitigate the threats, many existing methods perform symmetric encryption on the entire frames in the videos. Unfortunately, these methods could introduce additional computation cost and storage. Moreover, as surveillance systems could be a part of distributed system, the key management is critical and challenging. In this paper, we propose a novel method which incorporates background subtraction technique and RSA encryption algorithm. Rather than encrypting the entire frames of the videos, the proposed detect the regions around moving objects in the frames of video and then perform RSA encryption on the detected regions. And RSA encryption technique has its advantages of key distribution and management. Our experimental results show that the proposed method only involve moderate computation cost and storage.

Understanding the YouTube partners and their data: Measurement and analysis

User generated content, e.g., from YouTube, the most popular online video sharing site, is one of the major sources of today's big data and it is crucial to understand their inherent characteristics. Recently, YouTube has started working with content providers (known as YouTube partners) to promote the users' watching and sharing activities. The substantial benefit is to further augment its service and monetize more videos, which is crucial to both YouTube and its partners, as well as to other providers of relevant services. In this paper, our main contribution is to analyze the massive amounts of video data from a YouTube partner's view. We make effective use of Insight, a new analytics service of YouTube that offers simple data analysis for partners. To provide the practical guidance from the raw Insight data, we enable more complex investigations for the inherent features that affect the popularity of the videos. Our findings facilitate YouTube partners to re-design current video publishing strategies, having more opportunities to attract more views.

Real-Time Fish Observation and Fish Category Database Construction

This paper proposes a distributed real-time video stream system for underwater fish observation in the real world. The system, based on a three-tier architecture, includes capture devices unit, stream processor unit, and display devices unit. It supports variety of capture source devices, such as HDV, DV, WebCam, TV Card, Capture Card, and video compression formats, such as WMV, FLV/SWF, MJPEG, MPEG-2/4. The system has been demonstrated in Taiwan for long-term underwater fish observation. CCTV cameras and high-definition cameras are deployed on our system. Video compression methods and image processing methods are implemented to reduce network transfer flow and data storage space. Marine ecologists and end users can browse these real-time video streams via the Internet to understand the ecological changes immediately. These video data is preserved to form a resource base for marine ecologists. Based on the video data, fish detection is implemented. However, it is complicated in the unconstrained underwater environment, due to the water flow causes the water plants sway severely. In this paper, a bounding-surrounding boxes method is proposed to overcome the problem. It efficiently classifies moving fish as the foreground objects and the swaying water plants as the background objects. It enables to remove the irrelevant information (without fish) to reduce the massive amount of video data. Moreover, fish tracking is implemented to acquire multiple species of fish images with varied angles, sizes, shapes, and illumination to construct a fish category database.