Raw Video Data Research Articles

The project for objective measures using computational psychiatry technology (PROMPT): Rationale, design, and methodology

IntroductionDepressive and neurocognitive disorders are debilitating conditions that account for the leading causes of years lived with disability worldwide. However, there are no biomarkers that are objective or easy-to-obtain in daily clinical practice, which leads to difficulties in assessing treatment response and developing new drugs. New technology allows quantification of features that clinicians perceive as reflective of disorder severity, such as facial expressions, phonic/speech information, body motion, daily activity, and sleep. MethodsMajor depressive disorder, bipolar disorder, and major and minor neurocognitive disorders as well as healthy controls are recruited for the study. A psychiatrist/psychologist conducts conversational 10-min interviews with participants ≤10 times within up to five years of follow-up. Interviews are recorded using RGB and infrared cameras, and an array microphone. As an option, participants are asked to wear wrist-band type devices during the observational period. Various software is used to process the raw video, voice, infrared, and wearable device data. A machine learning approach is used to predict the presence of symptoms, severity, and the improvement/deterioration of symptoms. DiscussionThe overall goal of this proposed study, the Project for Objective Measures Using Computational Psychiatry Technology (PROMPT), is to develop objective, noninvasive, and easy-to-use biomarkers for assessing the severity of depressive and neurocognitive disorders in the hopes of guiding decision-making in clinical settings as well as reducing the risk of clinical trial failure. Challenges may include the large variability of samples, which makes it difficult to extract the features that commonly reflect disorder severity. Trial RegistrationUMIN000021396, University Hospital Medical Information Network (UMIN).

Emergency Clinical Procedure Detection With Deep Learning.

Information about a patient's state is critical for hospitals to provide timely care and treatment. Prior work on improving the information flow from emergency medical services (EMS) to hospitals demonstrated the potential of using automated algorithms to detect clinical procedures. However, prior work has not made effective use of video sources that might be available during patient care. In this paper we explore the use convolutional neural networks (CNNs) on raw video data to determine how well video data alone can automatically identify clinical procedures. We apply multiple deep learning models to this problem, with significant variation in results. Our findings indicate performance improvements compared to prior work, but also indicate a need for more training data to reach clinically deployable levels of success.

Real-Time and Deep Learning Based Vehicle Detection and Classification Using Pixel-Wise Code Exposure Measurements

One key advantage of compressive sensing is that only a small amount of the raw video data is transmitted or saved. This is extremely important in bandwidth constrained applications. Moreover, in some scenarios, the local processing device may not have enough processing power to handle object detection and classification and hence the heavy duty processing tasks need to be done at a remote location. Conventional compressive sensing schemes require the compressed data to be reconstructed first before any subsequent processing can begin. This is not only time consuming but also may lose important information in the process. In this paper, we present a real-time framework for processing compressive measurements directly without any image reconstruction. A special type of compressive measurement known as pixel-wise coded exposure (PCE) is adopted in our framework. PCE condenses multiple frames into a single frame. Individual pixels can also have different exposure times to allow high dynamic ranges. A deep learning tool known as You Only Look Once (YOLO) has been used in our real-time system for object detection and classification. Extensive experiments showed that the proposed real-time framework is feasible and can achieve decent detection and classification performance.

Semisupervised Distributed Learning With Non-IID Data for AIoT Service Platform

Thanks to the advances in wireless communication and machine learning technologies, we can envision a novel AIoT (AI + IoT) service platform that collects video data from the individuals’ edge devices. Then, it transforms the video data into useful information, providing services to IoT or smart city applications. However, collecting raw video data directly to the cloud server is merely possible due to network bandwidth limitations and data privacy concerns. One possible solution is to adopt federated learning, which enables edge devices to collaboratively train a shared model without sending the raw data to the cloud. Unfortunately, this scheme cannot directly be applied to the targeted scenario since it assumes labeled data for training, and only at the cloud, we have the human power and time to label the video data. Thus, to tackle those issues, we propose an edge learning system based on semisupervised learning and federated learning technologies. The system trains AI models at edge devices using an improved semisupervised learning scheme and periodically uploads the training results to the cloud server to form a single model by adapting the federated learning technology. Then, we observe that in the real world, the data on the end devices are nonindependent and identically distributed (non-IID) such that it may cause weight divergence during training and result in a considerable decrease in the model performance. Therefore, we propose a new operation called federated swapping (FedSwap) to replace partial federated learning operations based on a few shared data during federated training to alleviate the adverse impact of weight divergence. We evaluate our system on both image classification using the state-of-the-art benchmark data and object detection using real-world video data. The experimental results show that the proposed system can have up to 5.9% higher accuracy of object detection for the video analysis applications by fully utilizing unlabeled data, compared with the situation that only labeled data are used. Moreover, the proposed FedSwap can improve the accuracy of image classification by 3.8% and the object detection task by 1.1%.

DEEPEYE

Video object detection and action recognition typically require deep neural networks (DNNs) with huge number of parameters. It is thereby challenging to develop a DNN video comprehension unit in resource-constrained terminal devices. In this article, we introduce a deeply tensor-compressed video comprehension neural network, called DEEPEYE, for inference on terminal devices. Instead of building a Long Short-Term Memory (LSTM) network directly from high-dimensional raw video data input, we construct an LSTM-based spatio-temporal model from structured, tensorized time-series features for object detection and action recognition. A deep compression is achieved by tensor decomposition and trained quantization of the time-series feature-based LSTM network. We have implemented DEEPEYE on an ARM-core-based IOT board with 31 FPS consuming only 2.4W power. Using the video datasets MOMENTS, UCF11 and HMDB51 as benchmarks, DEEPEYE achieves a 228.1× model compression with only 0.47% mAP reduction; as well as 15 k × parameter reduction with up to 8.01% accuracy improvement over other competing approaches.

Efficient In-loop Filtering Based on Enhanced Deep Convolutional Neural Networks for HEVC.

The raw video data can be compressed much by the latest video coding standard, high efficiency video coding (HEVC). However, the block-based hybrid coding used in HEVC will incur lots of artifacts in compressed videos, the video quality will be severely influenced. To settle this problem, the in-loop filtering is used in HEVC to eliminate artifacts. Inspired by the success of deep learning, we propose an efficient in-loop filtering algorithm based on the enhanced deep convolutional neural networks (EDCNN) for significantly improving the performance of in-loop filtering in HEVC. Firstly, the problems of traditional convolutional neural networks models, including the normalization method, network learning ability, and loss function, are analyzed. Then, based on the statistical analyses, the EDCNN is proposed for efficiently eliminating the artifacts, which adopts three solutions, including a weighted normalization method, a feature information fusion block, and a precise loss function. Finally, the PSNR enhancement, PSNR smoothness, RD performance, subjective test, and computational complexity/GPU memory consumption are employed as the evaluation criteria, and experimental results show that when compared with the filter in HM16.9, the proposed in-loop filtering algorithm achieves an average of 6.45% BDBR reduction and 0.238 dB BDPSNR gains.

A new framework for video data retrieval using hierarchical clustering technique

The amount of information produced every year is rapidly growing due to many factor among all media, video is a particular media embedding visual, motion, audio and textual information. Given this huge amount of information we need general framework for video data mining to be applied to the raw videos (surveillance videos, news reading, Person reading books in library etc.).We introduce new techniques which are essential to process the video files. The first step of our frame work for mining raw video data in grouping input frames to a set of basic units which are relevant to the structure of the video. The second step is charactering the unit to cluster into similar groups, to detect interesting patterns. To do this we extract some features (object, colors etc.)From the unit. A histogram based color descriptors also introduced to reliably capture and represent the color properties of multiple images. The preliminary experimental studies indicate that the proposed framework is promising.

Learning Object-Action Relations from Bimanual Human Demonstration Using Graph Networks

Recognizing human actions is a vital task for a humanoid robot, especially in domains like programming by demonstration. Previous approaches on action recognition primarily focused on the overall prevalent action being executed, but we argue that bimanual human motion cannot always be described sufficiently with a single action label. We present a system for framewise action classification and segmentation in bimanual human demonstrations. The system extracts symbolic spatial object relations from raw RGB-D video data captured from the robot's point of view in order to build graph-based scene representations. To learn object-action relations, a graph network classifier is trained using these representations together with ground truth action labels to predict the action executed by each hand. We evaluated the proposed classifier on a new RGB-D video dataset showing daily action sequences focusing on bimanual manipulation actions. It consists of 6 subjects performing 9 tasks with 10 repetitions each, which leads to 540 video recordings with 2 hours and 18 minutes total playtime and per-hand ground truth action labels for each frame. We show that the classifier is able to reliably identify (action classification macro F1-score of 0.86) the true executed action of each hand within its top 3 predictions on a frame-by-frame basis without prior temporal action segmentation.

Classifying ASD children with LSTM based on raw videos

Autism spectrum disorder (ASD) is a serious neurodevelopmental disorder that impairs a child's ability to communicate and interact with others. Usually, recognizing a child with ASD needs the diagnosis by professional doctors. However, it is not only expensive and time-consuming, but also the results are influenced by subjective factors, such as the experience of a doctor. Recently, some methods which identify ASD based on biomarkers have been developed, but there are rarely works specific to raw video data. This paper is the first attempt to help diagnose the children with ASD in raw video data using a deep learning technique. Firstly, in order to investigate different gaze patterns between ASD children and typically developing (TD) children, we track the eye movement in each video by the tracking-learning-detection method. Secondly, we divide these tracking trajectories into two components: (i) the length; and (ii) the angle. Afterwards, we calculate an accumulative histogram for each component. Finally, we adopt three-layer Long Short-Term Memory (LSTM) network for classification. Experimental results on our extended dataset (Ext-Dataset) containing 272 videos captured from 136 ASD children and 136 TD children show the LSTM network outperforms the traditional machine learning methods, e.g., Support Vector Machine, with the improvement of accuracy by 6.2% (from 86.4% to 92.6%). Especially, for ASD, we obtain the sensitivity (the true positive rate, TPR) of 91.9% and the specificity (the true negative rate, TNR) of 93.4%, which demonstrates the effectiveness of our method.

Robust Anomaly Detection in Videos Using Multilevel Representations

Detecting anomalies in surveillance videos has long been an important but unsolved problem. In particular, many existing solutions are overly sensitive to (often ephemeral) visual artifacts in the raw video data, resulting in false positives and fragmented detection regions. To overcome such sensitivity and to capture true anomalies with semantic significance, one natural idea is to seek validation from abstract representations of the videos. This paper introduces a framework of robust anomaly detection using multilevel representations of both intensity and motion data. The framework consists of three main components: 1) representation learning using Denoising Autoencoders, 2) level-wise representation generation using Conditional Generative Adversarial Networks, and 3) consolidating anomalous regions detected at each representation level. Our proposed multilevel detector shows a significant improvement in pixel-level Equal Error Rate, namely 11.35%, 12.32% and 4.31% improvement in UCSD Ped 1, UCSD Ped 2 and Avenue datasets respectively. In addition, the model allowed us to detect mislabeled anomalies in the UCDS Ped 1.

Video Analytics for Railroad Safety Research: An Artificial Intelligence Approach

The volume of video data in the railroad industry has increased significantly in recent years. Surveillance cameras are situated on nearly every part of the railroad system, such as inside the cab, along the track, at grade crossings, and in stations. These camera systems are manually monitored, either live or subsequently reviewed in an archive, which requires an immense amount of human resources. To make the video analysis much less labor-intensive, this paper develops a framework for utilizing artificial intelligence (AI) technologies for the extraction of useful information from these big video datasets. This framework has been implemented based on the video data from one grade crossing in New Jersey. The AI algorithm can automatically detect unsafe trespassing of railroad tracks (called near-miss events in this paper). To date, the AI algorithm has analyzed hours of video data and correctly detected all near-misses. This pilot study indicates the promise of using AI for automated analysis of railroad video big data, thereby supporting data-driven railroad safety research. For practical use, our AI algorithm has been packaged into a computer-aided decision support tool (named AI-Grade) that outputs near-miss video clips based on user-provided raw video data. This paper and its sequent studies aim to provide the railroad industry with next-generation big data analysis methods and tools for quickly and reliably processing large volumes of video data in order to better understand human factors in railroad safety research.

Content-Aware Proactive Caching for Backhaul Offloading in Cellular Network

Proactive caching is considered a cost-effective method to address the backhaul bottleneck problem in cellular network. In this paper, we propose a novel popularity predicting–caching procedure that takes raw video data as an input to determine an optimal cache placement policy, which deals with both published and unpublished videos. To anticipate the popularity of unpublished videos of which the statistical information is not available, we apply the content-based approach by extracting and condensing video features into a high-dimensional vector. Subsequently, we form $G$ clusters of features representing the potential video categories (VCs) and map the feature vector into a $G$ -dimensional space, where each element indicates the percentage to which the video contains the features of the corresponding VC. Finally, we train a prediction model to foresee the popularity, where the set of published videos is used as training data. Last, the prediction with expert advice method is used to update the training set, and to gain insight into how the predictor output will deviate from the best expert prediction, we address the concept of expected cumulative loss and derive the analytical expression for its upper bound. Extensive simulation results are shown to gain insight into our proposed system subject to different factors, such as network size, cache capacity, and user’s preference profile. In summary, we show that applying intelligence-based content-aware proactive caching is an efficient approach to significantly improving the operation of cellular networks in the future.

Exploiting the potential of unlabeled endoscopic video data with self-supervised learning.

Surgical data science is a new research field that aims to observe all aspects of the patient treatment process in order to provide the right assistance at the right time. Due to the breakthrough successes of deep learning-based solutions for automatic image annotation, the availability of reference annotations for algorithm training is becoming a major bottleneck in the field. The purpose of this paper was to investigate the concept of self-supervised learning to address this issue. Our approach is guided by the hypothesis that unlabeled video data can be used to learn a representation of the target domain that boosts the performance of state-of-the-art machine learning algorithms when used for pre-training. Core of the method is an auxiliary task based on raw endoscopic video data of the target domain that is used to initialize the convolutional neural network (CNN) for the target task. In this paper, we propose the re-colorization of medical images with a conditional generative adversarial network (cGAN)-based architecture as auxiliary task. A variant of the method involves a second pre-training step based on labeled data for the target task from a related domain. We validate both variants using medical instrument segmentation as target task. The proposed approach can be used to radically reduce the manual annotation effort involved in training CNNs. Compared to the baseline approach of generating annotated data from scratch, our method decreases exploratively the number of labeled images by up to 75% without sacrificing performance. Our method also outperforms alternative methods for CNN pre-training, such as pre-training on publicly available non-medical (COCO) or medical data (MICCAI EndoVis2017 challenge) using the target task (in this instance: segmentation). As it makes efficient use of available (non-)public and (un-)labeled data, the approach has the potential to become a valuable tool for CNN (pre-)training.

Learning to Recognize Actions From Limited Training Examples Using a Recurrent Spiking Neural Model.

A fundamental challenge in machine learning today is to build a model that can learn from few examples. Here, we describe a reservoir based spiking neural model for learning to recognize actions with a limited number of labeled videos. First, we propose a novel encoding, inspired by how microsaccades influence visual perception, to extract spike information from raw video data while preserving the temporal correlation across different frames. Using this encoding, we show that the reservoir generalizes its rich dynamical activity toward signature action/movements enabling it to learn from few training examples. We evaluate our approach on the UCF-101 dataset. Our experiments demonstrate that our proposed reservoir achieves 81.3/87% Top-1/Top-5 accuracy, respectively, on the 101-class data while requiring just 8 video examples per class for training. Our results establish a new benchmark for action recognition from limited video examples for spiking neural models while yielding competitive accuracy with respect to state-of-the-art non-spiking neural models.

A deep-learning based feature hybrid framework for spatiotemporal saliency detection inside videos

Abstract Although research on detection of saliency and visual attention has been active over recent years, most of the existing work focuses on still image rather than video based saliency. In this paper, a deep learning based hybrid spatiotemporal saliency feature extraction framework is proposed for saliency detection from video footages. The deep learning model is used for the extraction of high-level features from raw video data, and they are then integrated with other high-level features. The deep learning network has been found extremely effective for extracting hidden features than that of conventional handcrafted methodology. The effectiveness for using hybrid high-level features for saliency detection in video is demonstrated in this work. Rather than using only one static image, the proposed deep learning model take several consecutive frames as input and both the spatial and temporal characteristics are considered when computing saliency maps. The efficacy of the proposed hybrid feature framework is evaluated by five databases with human gaze complex scenes. Experimental results show that the proposed model outperforms five other state-of-the-art video saliency detection approaches. In addition, the proposed framework is found useful for other video content based applications such as video highlights. As a result, a large movie clip dataset together with labeled video highlights is generated.

GRAPHITE: A Graphical Environment for Scalable in situ Video Tracking of Moving Insects.

1. Methods for measuring animal movement are critical for understanding numerous ecological and evolutionary processes. However, few methods are available for small organisms, and even fewer methods offer consistent individual-level resolution while remaining affordable, scalable and operable in the field. 2. We describe a low-cost animal movement tracking method with a user-friendly graphical interface, called GRAPHITE. Our automated software can quantify motions of insects by offline video analysis of inexpensive and lightweight human-readable tags attached to individual insects. The integrated graphical editor provides a full-featured environment for users to review the generated tracking data and make individual- or group-level edits. 3. GRAPHITE is a novel video analysis and graphical editing software (MATLAB v.9.0.0+) that identifies tags in image frames with a minimal false negative rate, links sequences of corresponding tags into "tracks" for each individual insect, infers the tag identifier, and provides a user-friendly graphical environment for editing tracking data. Users can either batch process raw video data using the full analysis pipeline or execute GRAPHITE modules independently for a tailored analysis. 4. We demonstrate the efficacy of the developed software with a specific application to the movement of honey bees at the entrance of hives. However, this system can be easily modified to track individually marked insects of 3 mm and larger. A notable advantage of this method is its ability to provide easy access to individual-level tracking data using human-readable tags.

DeepPredict: A deep predictive intelligence platform for patient monitoring.

A novel platform, DeepPredict, for predicting hospital bed exit events from video camera systems is proposed. DeepPredict processes video data with a deep convolutional neural network consisting of five main layers: a 1 × 1 3D convolutional layer used for generating feature maps from raw video data, a context-aware pooling layer used for rectifying data from different camera angles, two fully connected layers used for applying pre-trained deep features, and an output layer used to provide a likelihood of a bed exit event. Results for a model trained on 180 hours of data demonstrate accuracy, sensitivity, and specificity of 86.47%, 78.87%, and 94.07%, respectively, when predicting a bed exit event up to seven seconds in advance.

Genuine reversible data hiding technology using compensation for H.264 bitstreams

Reversible data hiding technologies have been considered largely impractical because those are, in most cases, applicable to raw video data rather than prevailing compressed data. Even though, many algorithms have been recently developed in the compressed video domain, most of them cannot guarantee the reversibility of cover video due to the lossy characteristics of video compression standards. We suggest completely practical data hiding scheme for H.264 baseline bitstream by achieving genuine reversibility for both I and P frames. Regardless of the data hiding algorithm, the proposed scheme can increase embedding payload by 66.9% and reduce computational complexity by 93%. Also, a novel compensation based difference expansion method with clever coefficient pairing strategy is proposed as a data hiding algorithm and achieved superior embedding payload vs. image quality performance. The proposed algorithm improves payload by 48.9% on average at almost the same video quality distortion.

Predicting bull behavior events in a multiple-sire pasture with video analysis, accelerometers, and classification algorithms

Parentage data from beef calves has shown that in multiple-sire pastures a disproportionate number of calves are born from a single bull. Investigating and accurately quantifying bull behavior within multiple-sire pastures will begin to determine reason(s) for the variability in the number of calves sired. The study objective was to assess accelerometer data and various classification algorithms to accurately predict bull behavior events in a multiple-sire pasture. Behavior events of interest in this study included lying, standing, walking, and mounting.Two bulls and ten estrous synchronized cows were used. True behavior events were determined during daylight hours with video analysis, and matched with accelerometer data. Accelerometers were attached to both ears, withers, and neck of both bulls. Accelerometer data were recorded for every second over 3days. Accelerometer data were used to generate algorithms and accuracy was evaluated compared to known video behavioral data.The prevalence based on the raw video data for lying was 32.6%, standing was 59.4%, walking was 7.4%, and mounting was 0.6%. The random forest classifier had the highest accuracy compared to other classifiers (random tree and decision tree) for each tag location and behavior of interest. The accuracies from the random forest algorithms ranged from 92 to 99% for lying, 85 to 90% for standing, 73 to 77% for walking, and 74% to 80% for mounting. The classification algorithm was able to accurately predict a lying and standing event, and predict a walking and mounting event with a lower accuracy. Further research is needed to determine how behaviors between bulls affects overall parentage data.

Efficient PCIe transmission for Multi-Channel video using dynamic splicing and conditional prefetching

PCI Express (PCIe) interface has been extensively used in high-speed digital systems for multimedia communication. With the migration of the video processing algorithms from host to embedded hardware, multi-channel video capturing systems will produce not only different channels of raw video data but also different types of auxiliary data, such as analyzed data and compressed stream. In order to display multi-channel video in real-time and explore the auxiliary data, conventional transmission strategies are no longer applicable, due to the fact that heterogeneous data will cause frequent interactions and lead to the waste of PCIe bandwidth. In this paper, an efficient PCIe transmission method for multi-channel video is presented. Firstly, for the transmission of multi-type video data, a dynamic splicing mechanism is proposed to combine the video analyzed data and the compressed stream with the raw video to avoid the individual transmission of the auxiliary data. Secondly, as the spliced data are from different channels, a conditional prefetching mechanism is employed to determine whether there exists any entire video frame in other channel buffers, so that multi-channel video data can be transmitted possibly at one time. Finally, in the host-side driver, direct kernel buffer access technique is used to improve the application I/O request packet (IRP) performance. And to ensure the transmission efficiency of the conditional prefetching, DMA circular queue buffer and timer self-feedback monitor techniques are designed to avoid the possible visit bursts and abnormal interruptions. Experimental results demonstrate that compared with the conventional methods, the proposed method reduces the interrupt interactions by 60%, increases the transmission channel number by 94%, and also increases the application IRP number by 54%. The peak transmission speed of PCIe is up to 155 MB/s, which can meet 7 channels 704 × 576 YUV raw video and its auxiliary data transmission requirements using one 1-lane PCIe endpoint.