Video Event Detection Research Articles

Detection of Lane-Change Events in Naturalistic Driving Videos.

Lane changes are important behaviors to study in driving research. Automated detection of lane-change events is required to address the need for data reduction of a vast amount of naturalistic driving videos. This paper presents a method to deal with weak lane-marker patterns as small as a couple of pixels wide. The proposed method is novel in its approach to detecting lane-change events by accumulating lane-marker candidates over time. Since the proposed method tracks lane markers in temporal domain, it is robust to low resolution and many different kinds of interferences. The proposed technique was tested using 490 h of naturalistic driving videos collected from 63 drivers. The lane-change events in a 10-h video set were first manually coded and compared with the outcome of the automated method. The method's sensitivity was 94.8% and the data reduction rate was 93.6%. The automated procedure was further evaluated using the remaining 480-h driving videos. The data reduction rate was 97.4%. All 4971 detected events were manually reviewed and classified as either true or false lane-change events. Bootstrapping showed that the false discovery rate from the larger data set was not significantly different from that of the 10-h manually coded data set. This study demonstrated that the temporal processing of lane markers is an effcient strategy for detecting lane-change events involving weak lane-marker patterns in naturalistic driving.

Learning an event-oriented and discriminative dictionary based on an adaptive label-consistent K-SVD method for event detection in soccer videos

In this paper, we formulate the soccer video event detection task as a sparse representation problem by learning a supervised, discriminative and event-oriented dictionary based on learned weighted local features. To this end, we present a novel framework based on two ideas: First, we propose an approach for computing the representativeness of each video frame for each soccer event. Second, we propose an Adaptive Label-Consistent K-SVD (ALC-KSVD) algorithm to learn an event-oriented and discriminative dictionary based on the computed representativeness of frames to transfer video frames to a sparse space. To improve discrimination among frames of different events, we proposed a weighting method to identify local features that are more representative in each event category. Next, the representativeness score of each frame is calculated by aggregating the weighted local features within each frame. The calculated representativeness score of each frame indicates its belonging degree to each event. The representativeness score matrix, being a discriminative term, is combined with the reconstruction error to form an objective function to improve the discrimination ability in the sparse representation during the dictionary learning process. The obtained objective function is efficiently and optimally solved by the K-SVD algorithm. The representativeness score matrix, which is automatically calculated based on the training samples, defines an adaptive correspondence between the dictionary atoms and the labels of the frames. We demonstrate the effectiveness of the proposed framework on the detection and classification of several soccer events based on an extensive experimental investigation that was conducted using a large collection of video data. The experimental results indicate that our approach maintains good classification performance and outperforms the state-of-the-art methods.

A comprehensive solution for detecting events in complex surveillance videos

Event detection have long been a fundamental problem in computer vision society. Various datasets for recognizing human events and activities have been proposed to help developing better models and methods, such as UCF101, HMDB51, etc. These datasets all share the same properties that either predefined scripts are provided or the images are almost actor-oriented with little background noise. These properties, however, are completely different from that of surveillance event detection, making the effective solutions on these datasets totally not suitable. Event detection in complex surveillance video is a much more difficult task with several challenges: heavy occlusions between pedestrians, low image resolution and uncontrolled scene condition. TRECVID-SED evaluation, aiming at detecting events in highly crowded airport, is well-known for its great difficulties. To deal with event detection in realistic scene, such as TRECVID-SED, we introduce a comprehensive solution framework based on pedestrian detection, deep key-pose detection and trajectory analysis. Explicitly, instead of detecting whole body of one person, we detect the head-shoulder of pedestrian, addressing the issue of heavy occlusion of pedestrians in complex scene. We also propose a trajectory-based event detection method so as to better focus on the key actors of events. For those events with discriminative poses, we model the event detection as key pose detection by taking advantages of Faster R-CNN. The presented framework achieves the best result in TRECVID-SED 2016 evaluation.

Extracting Key Segments of Videos for Event Detection by Learning From Web Sources

In this paper, we present a novel approach of extracting the key segments for event detection in unconstrained videos. The key segments are automatically extracted by transferring the knowledge learned from Web images and Web videos to consumer videos. We propose an adaptive latent structural support vector machine model, where the locations of key segments in videos are regarded as latent variables due to the unavailability of the ground truth of key-segment locations in training data. In order to alleviate the time-consuming and labor-expensive manual annotation of huge amounts of training videos, a large number of loosely labeled Web images as well as videos are collected from the Web sources. Additionally, a limited number of labeled consumer videos are utilized to guarantee the precision of the model. Considering the semantic diversity of key segments, we learn a set of concepts as the semantic description of key segments and explore the temporal information of concepts to capture the sequential relations between the segments. The concepts are automatically discovered by using Web images and videos with their associated tags and description sentences. Comprehensive experiments on the Columbia's consumer video and the TRECVID 2014 Multimedia Event Detection datasets demonstrate that our method outperforms the state-of-the-art methods.

Learning Sparse Representation With Variational Auto-Encoder for Anomaly Detection

Anomaly detection has a wide range of applications in security area such as network monitoring and smart city/campus construction. It has become an active research issue of great concern in recent years. However, most algorithms of the existing studies are powerless for large-scale and high-dimensional data, and the intermediate data extracted by some methods that can handle high-dimensional data will consume lots of storage space. In this paper, we propose a novel sparse representation framework that learns dictionaries based on the latent space of variational auto-encoder. For large-scale data sets, it can play the role of dimensionality reduction to obtain hidden information, and extract more high-level features than hand-crafted features. At the same time, for the storage of normal information, the space cost can be greatly reduced. To verify the versatility and performance of the proposed learning algorithm, we have experimented on different types of anomaly detection tasks, including KDD-CUP data set for network intrusion detection, Mnist data set for image anomaly detection, and UCSD pedestrian’s data set for abnormal event detection in surveillance videos. The experimental results demonstrate that the proposed algorithm outperforms competing algorithms in all kinds of anomaly detection tasks.

LSTM-based multi-label video event detection

Since large-scale surveillance videos always contain complex visual events, how to generate video descriptions effectively and efficiently without human supervision has become mandatory. To address this problem, we propose a novel architecture for jointly recognizing multiple events in a given surveillance video, motivated by the sequence to sequence network. The proposed architecture can predict what happens in a video directly without the preprocessing of object detection and tracking. We evaluate several variants of the proposed architecture with different visual features on a novel dataset perpared by our group. Moreover, we compute a wide range of quantitative metrics to evaluate this architecture. We further compare it to the popular Support Vector Machine-based visual event detection method. The comparison results suggest that the proposal method can outperform the traditional computer vision pipelines for visual event detection.

Event Detection in Continuous Video: An Inference in Point Process Approach.

We propose a novel approach toward event detection in real-world continuous video sequences. The method: 1) is able to model arbitrary-order non-Markovian dependences in videos to mitigate local visual ambiguities; 2) conducts simultaneous event segmentation and labeling; and 3) is time-window free. The idea is to represent a video as an event stream of both high-level semantic events and low-level video observations. In training, we learn a point process model called a piecewise-constant conditional intensity model (PCIM) that is able to capture complex non-Markovian dependences in the event streams. In testing, event detection can be modeled as the inference of high-level semantic events, given low-level image observations. We develop the first inference algorithm for PCIM and show it samples exactly from the posterior distribution. We then evaluate the video event detection task on real-world video sequences. Our model not only provides competitive results on the video event segmentation and labeling task, but also provides benefits, including being interpretable and efficient.

SVAS: Surveillance Video Analysis System

Abstract This paper introduces a Surveillance Video Analysis System, called SVAS, for surveillance domain, in which the semantic rules and the definition of event models can be learned or defined by the user for automatic detection and inference of complex video events. In the scope of SVAS, an event model method named Interval-Based Spatio-Temporal Model (IBSTM) is proposed. SVAS can learn action models and event models without any predefined threshold values and generates understandable and manageable IBSTM event models. Hybrid machine learning methods are proposed and used. A set of feature models named Threshold Model, which reflects the spatio-temporal motion analysis of an event, is kept as the first model. As the second model, Bag of Actions (BoA) model is used in order to reduce the search space in the detection phase. Markov Logic Network (MLN) model, which provides understandable and manageable logic predicates for users, is kept as the third model. SVAS has high performance event detection capability due to its interval-based hierarchical manner. It determines related candidate intervals for each main model of IBSTM and uses the related main model when needed rather than using all models as a whole. The main contribution of this study is to fill the semantic gap between humans and video computer systems such that, on the one hand it decreases human intervention through its learning capabilities, but on the other hand it also enables human intervention when necessary through its manageable event model method. The study achieves all of them in the most efficient way through its machine learning methods. The proposed system is applied to different event datasets from CAVIAR, BEHAVE and our synthetic datasets. The experimental results show that our approach improves the event recognition performance and precision as compared to the current state-of-the-art approaches.

Captioning Videos Using Large-Scale Image Corpus

Video captioning is the task of assigning complex high-level semantic descriptions (e.g., sentences or paragraphs) to video data. Different from previous video analysis techniques such as video annotation, video event detection and action recognition, video captioning is much closer to human cognition with smaller semantic gap. However, the scarcity of captioned video data severely limits the development of video captioning. In this paper, we propose a novel video captioning approach to describe videos by leveraging freely-available image corpus with abundant literal knowledge. There are two key aspects of our approach: 1) effective integration strategy bridging videos and images, and 2) high efficiency in handling ever-increasing training data. To achieve these goals, we adopt sophisticated visual hashing techniques to efficiently index and search large-scale images for relevant captions, which is of high extensibility to evolving data and the corresponding semantics. Extensive experimental results on various real-world visual datasets show the effectiveness of our approach with different hashing techniques, e.g., LSH (locality-sensitive hashing), PCA-ITQ (principle component analysis iterative quantization) and supervised discrete hashing, as compared with the state-of-the-art methods. It is worth noting that the empirical computational cost of our approach is much lower than that of an existing method, i.e., it takes 1/256 of the memory requirement and 1/64 of the time cost of the method of Devlin et al.

OPML: A one-pass closed-form solution for online metric learning

To achieve a low computational cost when performing online metric learning for large-scale data, we present a one-pass closed-form solution namely OPML in this paper. Typically, the proposed OPML first adopts a one-pass triplet construction strategy, which aims to use only a very small number of triplets to approximate the representation ability of whole original triplets obtained by batch-manner methods. Then, OPML employs a closed-form solution to update the metric for new coming samples, which leads to a low space (i.e., O(d)) and time (i.e., O(d2)) complexity, where d is the feature dimensionality. In addition, an extension of OPML (namely COPML) is further proposed to enhance the robustness when in real case the first several samples come from the same class (i.e., cold start problem). In the experiments, we have systematically evaluated our methods (OPML and COPML) on three typical tasks, including UCI data classification, face verification, and abnormal event detection in videos, which aims to fully evaluate the proposed methods on different sample number, different feature dimensionalities and different feature extraction ways (i.e., hand-crafted and deeply-learned). The results show that OPML and COPML can obtain the promising performance with a very low computational cost. Also, the effectiveness of COPML under the cold start setting is experimentally verified.

Hierarchical Latent Concept Discovery for Video Event Detection.

Semantic information is important for video event detection. How to automatically discover, model, and utilize semantic information to facilitate video event detection has been a challenging problem. In this paper, we propose a novel hierarchical video event detection model, which deliberately unifies the processes of underlying semantics discovery and event modeling from video data. Specially, different from most of the approaches based on manually pre-defined concepts, we devise an effective model to automatically uncover video semantics by hierarchically capturing latent static-visual concepts in frame-level and latent activity concepts (i.e., temporal sequence relationships of static-visual concepts) in segment-level. The unified model not only enables a discriminative and descriptive representation for videos, but also alleviates error propagation problem from video representation to event modeling existing in previous methods. A max-margin framework is employed to learn the model. Extensive experiments on four challenging video event datasets, i.e., MED11, CCV, UQE50, and FCVID, have been conducted to demonstrate the effectiveness of the proposed method.

DECK: Discovering Event Composition Knowledge from Web Images for Zero-Shot Event Detection and Recounting in Videos

We address the problem of zero-shot event recognition in consumer videos. An event usually consists of multiple human-human and human-object interactions over a relative long period of time. A common approach proceeds by representing videos with banks of object and action concepts, but requires additional user inputs to specify the desired concepts per event. In this paper, we provide a fully automatic algorithm to select representative and reliable concepts for event queries. This is achieved by discovering event composition knowledge (DECK) from web images. To evaluate our proposed method, we use the standard zero-shot event detection protocol (ZeroMED), but also introduce a novel zero-shot event recounting (ZeroMER) problem to select supporting evidence of the events. Our ZeroMER formulation aims to select video snippets that are relevant and diverse. Evaluation on the challenging TRECVID MED dataset show that our proposed method achieves promising results on both tasks.

Goal!! Event detection in sports video

Goal!! Event detection in sports video

Detection of complex video events through visual rhythm

The recognition of complex events in videos has currently several important applications, particularly due to the wide availability of digital cameras in environments such as airports, train and bus stations, shopping centers, stadiums, hospitals, schools, buildings, roads, among others. Advances in digital technology have enhanced the capabilities for detection of video events through the development of devices with high resolution, small physical size, and high sampling rates. This work presents and evaluates the use of feature descriptors extracted from visual rhythms of video sequences in three computer vision problems: abnormal event detection, human action classification, and gesture recognition. Experiments conducted on well-known public datasets demonstrate that the method produces promising results.

Early and Late Level Fusion of Deep Convolutional Neural Networks for Visual Concept Recognition

Visual concept recognition is an active research field in the last decade. Related to this attention, deep learning architectures are showing great promise in various computer vision domains including image classification, object detection, event detection and action recognition in videos. In this study, we investigate various aspects of convolutional neural networks for visual concept recognition. We analyze recent studies and different network architectures both in terms of running time and accuracy. In our proposed visual concept recognition system, we first discuss various important properties of popular convolutional network architecture under consideration. Then we describe our method for feature extraction at different levels of abstraction. We present extensive empirical information along with best practices for big data practitioners. Using these best practices we propose efficient fusion mechanisms both for single and multiple network models. We present state-of-the-art results on benchmark datasets while keeping computational costs at low level. Our results show that these state-of-the-art results can be reached without using extensive data augmentation techniques.

TagBook: A Semantic Video Representation Without Supervision for Event Detection

We consider the problem of event detection in video for scenarios where only a few, or even zero, examples are available for training. For this challenging setting, the prevailing solutions in the literature rely on a semantic video representation obtained from thousands of pretrained concept detectors. Different from existing work, we propose a new semantic video representation that is based on freely available social tagged videos only, without the need for training any intermediate concept detectors. We introduce a simple algorithm that propagates tags from a video's nearest neighbors, similar in spirit to the ones used for image retrieval, but redesign it for video event detection by including video source set refinement and varying the video tag assignment. We call our approach TagBook and study its construction, descriptiveness, and detection performance on the TRECVID 2013 and 2014 multimedia event detection datasets and the Columbia Consumer Video dataset. Despite its simple nature, the proposed TagBook video representation is remarkably effective for few-example and zero-example event detection, even outperforming very recent state-of-the-art alternatives building on supervised representations.

Robust spatial-temporal deep model for multimedia event detection

The task of multimedia event detection (MED) aims at training a set of models that can automatically detect the most event-relevant videos from large datasets. In this paper, we attempt to build a robust spatial-temporal deep neural network for large-scale video event detection. In our setting, each video follows a multiple instance assumption, where its visual segments contain both spatial and temporal properties of events. Regarding these properties, we try to implement the MED system by a two-step training phase: unsupervised recurrent video reconstruction and supervised fine-tuning. We conduct extensive experiments on the challenging TRECVID MED14 dataset, which indicate that with the consideration of both spatial and temporal information, the detection performance can be further boosted compared with the state-of-the-art MED models.

Combining multi-representation for multimedia event detection using co-training

In recent years, multimedia event detection has been attracting extensive research attention because of the exponential increase in volume of web video data. Traditional approaches usually utilize single visual representation, which may suffer from the problem of insufficient descriptive power. How to jointly employ multiple types of visual representation to facilitate multimedia event detection (MED) in videos remains an open problem. In this work, we propose a novel system for event detection based on combination of multi-view representations and co-training algorithm. Specifically, given several types of low-level visual features (i.e., Convolutional Neural Networks (CNNs) and Fisher vector), we first train an initial classifier for each type of visual feature. Then, we use these classifiers to separately predict labels of unlabeled videos, and those with consistent prediction are merged into the training set. We alternatively repeat the processes of training the classifiers and enlarging the training set until convergence. To investigate the relationship among different types of visual features, the prediction scores of the two classifiers are fused by a linear weighted fusion method. We evaluate our MED system on the TRECVID MED11 data set, and the experimental results have demonstrated the outstanding performance of the proposed approach as compared to several other state-of-the-art algorithms.

Pattern Recognition, Part 1 [Guest editors' introduction

This special issue reports the advances in pattern recognition theory and applications, particularly, the basic issues of pattern classification and image analysis and their applications. The selected articles address image feature representation, contextual pattern analysis, compact low-rank sparse representation for abnormal event detection in video, customer churn prediction using ensemble learning, online text-independent writer identification using deep convolutional neural network, image depth ordering reasoning, and brain MR image tumor segmentation, respectively.

Abnormal event detection and localisation in traffic videos based on group sparse topical coding

In visual surveillance, detecting and localising abnormal events are of great interest. In this study, an unsupervised method is proposed to automatically discover abnormal events occurring in traffic videos. For learning typical motion patterns occurring in such videos, a group sparse topical coding (GSTC) framework and an improved version of it are applied to optical flow features extracted from video clips. Then a very simple and efficient algorithm is proposed for GSTC. It is shown that discovered motion patterns can be employed directly in detecting abnormal events. A variety of abnormality metrics based on the resulting sparse codes for detection of abnormality are investigated. Experiments show that the result of the approach in detection and localisation of abnormal events is promising. In comparison with other usual methods (probabilistic latent semantic analysis, latent Dirichlet allocation, sparse topical coding (STC) and improved STC), according to the values of area under ROC, the proposed method achieves at least 14% improvement in abnormal event detection.