Video Event Detection Research Articles

Scalable multimedia content analysis on parallel platforms using python

In this new era dominated by consumer-produced media there is a high demand for web-scalable solutions to multimedia content analysis. A compelling approach to making applications scalable is to explicitly map their computation onto parallel platforms. However, developing efficient parallel implementations and fully utilizing the available resources remains a challenge due to the increased code complexity, limited portability and required low-level knowledge of the underlying hardware. In this article, we present PyCASP, a Python-based framework that automatically maps computation onto parallel platforms from Python application code to a variety of parallel platforms. PyCASP is designed using a systematic, pattern-oriented approach to offer a single software development environment for multimedia content analysis applications. Using PyCASP, applications can be prototyped in a couple hundred lines of Python code and automatically scale to modern parallel processors. Applications written with PyCASP are portable to a variety of parallel platforms and efficiently scale from a single desktop Graphics Processing Unit (GPU) to an entire cluster with a small change to application code. To illustrate our approach, we present three multimedia content analysis applications that use our framework: a state-of-the-art speaker diarization application, a content-based music recommendation system based on the Million Song Dataset, and a video event detection system for consumer-produced videos. We show that across this wide range of applications, our approach achieves the goal of automatic portability and scalability while at the same time allowing easy prototyping in a high-level language and efficient performance of low-level optimized code.

Video event detection for fault monitoring in assembly automation

A major goal of many manufacturers is to minimise production downtime caused by machine faults and equipment breakdowns. This goal is typically achieved using sensor-based systems that can quickly detect and diagnose machine faults of various types. This paper proposes the use of a video event detection method based on spatiotemporal volumes STVs in a fault monitoring application to complement and improve upon existing systems. To detect faults, images are captured using a single camera from several different regions of an assembly machine testbed. The motion is segmented in each image creating binary frames which are stacked to build a STV. Normal operation of the machine is modelled by building a STV from several training sequences. New STVs are compared to the model and classified as either normal or faulty behaviour based on a calculated similarity measure. Test results show that the system is very effective on the data sets collected.

Real-time video event detection in crowded scenes using MPEG derived features: A multiple instance learning approach

This paper presents an investigation into event detection in crowded scenes, where the event of interest co-occurs with other activities and only binary labels at the clip level are available. The proposed approach incorporates a fast feature descriptor from the MPEG domain, and a novel multiple instance learning (MIL) algorithm using sparse approximation and random sensing. MPEG motion vectors are used to build particle trajectories that represent the motion of objects in uniform video clips, and the MPEG DCT coefficients are used to compute a foreground map to remove background particles. Trajectories are transformed into the Fourier domain, and the Fourier representations are quantized into visual words using the K-Means algorithm. The proposed MIL algorithm models the scene as a linear combination of independent events, where each event is a distribution of visual words. Experimental results show that the proposed approaches achieve promising results for event detection compared to the state-of-the-art.

Multimedia Event Detection Using A Classifier-Specific Intermediate Representation

Multimedia event detection (MED) plays an important role in many applications such as video indexing and retrieval. Current event detection works mainly focus on sports and news event detection or abnormality detection in surveillance videos. Differently, our research aims to detect more complicated and generic events within a longer video sequence. In the past, researchers have proposed using intermediate concept classifiers with concept lexica to help understand the videos. Yet it is difficult to judge how many and what concepts would be sufficient for the particular video analysis task. Additionally, obtaining robust semantic concept classifiers requires a large number of positive training examples, which in turn has high human annotation cost. In this paper, we propose an approach that exploits the external concepts-based videos and event-based videos simultaneously to learn an intermediate representation from video features. Our algorithm integrates the classifier inference and latent intermediate representation into a joint framework. The joint optimization of the intermediate representation and the classifier makes them mutually beneficial and reciprocal. Effectively, the intermediate representation and the classifier are tightly correlated. The classifier dependent intermediate representation not only accurately reflects the task semantics but is also more suitable for the specific classifier. Thus we have created a discriminative semantic analysis framework based on a tightly coupled intermediate representation. Extensive experiments on multimedia event detection using real-world videos demonstrate the effectiveness of the proposed approach.

Discovering joint audio–visual codewords for video event detection

Detecting complex events in videos is intrinsically a multimodal problem since both audio and visual channels provide important clues. While conventional methods fuse both modalities at a superficial level, in this paper we propose a new representation--called bi-modal words--to explore representative joint audio---visual patterns. We first build a bipartite graph to model relation across the quantized words extracted from the visual and audio modalities. Partitioning over the bipartite graph is then applied to produce the bi-modal words that reveal the joint patterns across modalities. Different pooling strategies are then employed to re-quantize the visual and audio words into the bi-modal words and form bi-modal Bag-of-Words representations. Since it is difficult to predict the suitable number of bi-modal words, we generate bi-modal words at different levels (i.e., codebooks with different sizes), and use multiple kernel learning to combine the resulting multiple representations during event classifier learning. Experimental results on three popular datasets show that the proposed method achieves statistically significant performance gains over methods using individual visual and audio feature alone and existing popular multi-modal fusion methods. We also find that average pooling is particularly suitable for bi-modal representation, and using multiple kernel learning to combine multi-modal representations at various granularities is helpful.

Video Anomaly Search in Crowded Scenes via Spatio-Temporal Motion Context

Video anomaly detection plays a critical role for intelligent video surveillance. We present an abnormal video event detection system that considers both spatial and temporal contexts. To characterize the video, we first perform the spatio-temporal video segmentation and then propose a new region-based descriptor called “Motion Context,” to describe both motion and appearance information of the spatio-temporal segment. For anomaly measurements, we formulate the abnormal event detection as a matching problem, which is more robust than statistic model-based methods, especially when the training dataset is of limited size. For each testing spatio-temporal segment, we search for its best match in the training dataset, and determine how normal it is using a dynamic threshold. To speed up the search process, compact random projections are also adopted. Experiments on the benchmark dataset and comparisons with the state-of-the-art methods validate the advantages of our algorithm.

Event detection in consumer videos using GMM supervectors and SVMs

Abstract In large-scale multimedia event detection, complex target events are extracted from a large set of consumer-generated web videos taken in unconstrained environments. We devised a multimedia event detection method based on Gaussian mixture model (GMM) supervectors and support vector machines. A GMM supervector consists of the parameters of a GMM for the distribution of low-level features extracted from a video clip. A GMM is regarded as an extension of the bag-of-words framework to a probabilistic framework, and thus, it can be expected to be robust against the data insufficiency problem. We also propose a camera motion cancelled feature, which is a spatio-temporal feature robust against camera motions found in consumer-generated web videos. By combining these methods with the existing features, we aim to construct a high-performance event detection system. The effectiveness of our method is evaluated using TRECVID MED task benchmark.

Semantic detection of video events based on probabilistic hyper-graph

The categorization of video events is blur and uncertain.Expanding the hyper-graph's point-set matrix into the probabilistic form of soft hyper-graph,and analyzing the correlation of events and semantic,the precision and recall rate of the multi-event retrieval system will be improved.This paper proposed a video event semantic detection algorithm based on probabilistic hyper-graph model(PHVESD).Firstly,the method integrated the low-level visual features,color,gray-level co-occurrence matrix,Tchebichef moment and Local Binary Pattern(LBP);then defined the intimacy function of video segmentation and used the intimacy information construct probabilistic hyper-graph model,in which each hyper-edge corresponded to an event semantics;adopted Markov random walk model to predict the probability of video segment belonging to each hyper-edge;finally,classified the unknown video event with MAP.The authors used this method on traffic emergent multi-semantic events detection and compared the proposed method to another.The experimental results show that,compared to Multi Label Random Walk(MLRW) algorithm based on the hyper-graph model,the proposed method makes the multi-semantic event detection precision improve by 10%,the recall rate increase by 8%.

Hidden conditional random field-based soccer video events detection

Detect highlight event is an important step for semantic-based video retrieval. Hidden conditional random field (HCRF) is a discriminative model, which is effective in fusing observations for event inference. Mid-level semantics and their refinements are more robust than low-level visual features in event detection for learning models. To make full use of the contextual information, two aspects are taken into account during soccer video event detection. The first is parsing video sequences into event clips. The second is fusing the temporal transitions of the mid-level semantics of an event clip to determine the event type. In this study, HCRF is utilised to model the observations of mid-level semantics of an event clip for event detection. Comparisons are made with the dynamic Bayesian networks, hidden Markov model (HMM), enhanced HMM and conditional random field-based event detection approaches. Experimental results show the effectiveness of the proposed method.

Video Event Detection Technology and its Application in Intelligent Transportation

Video event detection technology has become a hot issue in the Intelligent Transportation Systems(ITS) research. It mainly uses in highways, tunnels, urban roads and other video surveillance systems. This paper makes a brief overview on the development of video detection technology at home and abroad. It describes the working principles and key technologies of the two kinds of incident detection technology based on virtual detection loop and video vehicle tracking, compare above two technology to sum up their advantages and disadvantages. At last it introduces the typical products of the domestic and foreign video event detection technology, their applications and performance index.

Classification-oriented structure learning in Bayesian networks for multimodal event detection in videos

We investigate the use of structure learning in Bayesian networks for a complex multimodal task of action detection in soccer videos. We illustrate that classical score-oriented structure learning algorithms, such as the K2 one whose usefulness has been demonstrated on simple tasks, fail in providing a good network structure for classification tasks where many correlated observed variables are necessary to make a decision. We then compare several structure learning objective functions, which aim at finding out the structure that yields the best classification results, extending existing solutions in the literature. Experimental results on a comprehensive data set of 7 videos show that a discriminative objective function based on conditional likelihood yields the best results, while augmented approaches offer a good compromise between learning speed and classification accuracy.

VideoSkip: event detection in social web videos with an implicit user heuristic

In this paper, we present a user-based event detection method for social web videos. Previous research in event detection has focused on content-based techniques, such as pattern recognition algorithms that attempt to understand the contents of a video. There are few user-centric approaches that have considered either search keywords, or external data such as comments, tags, and annotations. Moreover, some of the user-centric approaches imposed an extra effort to the users in order to capture required information. In this research, we are describing a method for the analysis of implicit users' interactions with a web video player, such as pause, play, and thirty-seconds skip or rewind. The results of our experiments indicated that even the simple user heuristic of local maxima might effectively detect the same video-events, as indicated manually. Notably, the proposed technique was more accurate in the detection of events that have a short duration, because those events motivated increased user interaction in video hot-spots. The findings of this research provide evidence that we might be able to infer semantics about a piece of unstructured data just from the way people actually use it.

Semantic Model Vectors for Complex Video Event Recognition

We propose semantic model vectors, an intermediate level semantic representation, as a basis for modeling and detecting complex events in unconstrained real-world videos, such as those from YouTube. The semantic model vectors are extracted using a set of discriminative semantic classifiers, each being an ensemble of SVM models trained from thousands of labeled web images, for a total of 280 generic concepts. Our study reveals that the proposed semantic model vectors representation outperforms-and is complementary to-other low-level visual descriptors for video event modeling. We hence present an end-to-end video event detection system, which combines semantic model vectors with other static or dynamic visual descriptors, extracted at the frame, segment, or full clip level. We perform a comprehensive empirical study on the 2010 TRECVID Multimedia Event Detection task (http://www.nist.gov/itl/iad/mig/med10.cfm), which validates the semantic model vectors representation not only as the best individual descriptor, outperforming state-of-the-art global and local static features as well as spatio-temporal HOG and HOF descriptors, but also as the most compact. We also study early and late feature fusion across the various approaches, leading to a 15% performance boost and an overall system performance of 0.46 mean average precision. In order to promote further research in this direction, we made our semantic model vectors for the TRECVID MED 2010 set publicly available for the community to use (http://www1.cs.columbia.edu/~mmerler/SMV.html).

Semantic trajectory-based event detection and event pattern mining

Video event detection is an effective way to automatically understand the semantic content of the video. However, due to the mismatch between low-level visual features and high-level semantics, the research of video event detection encounters a number of challenges, such as how to extract the suitable information from video, how to represent the event, how to build up reasoning mechanism to infer the event according to video information. In this paper, we propose a novel event detection method. The method detects the video event based on the semantic trajectory, which is a high-level semantic description of the moving object’s trajectory in the video. The proposed method consists of three phases to transform low-level visual features to middle-level raw trajectory information and then to high-level semantic trajectory information. Event reasoning is then carried out with the assistance of semantic trajectory information and background knowledge. Additionally, to release the users’ burden in manual event definition, a method is further proposed to automatically discover the event-related semantic trajectory pattern from the sample semantic trajectories. Furthermore, in order to effectively use the discovered semantic trajectory patterns, the associative classification-based event detection framework is adopted to discover the possibly occurred event. Empirical studies show our methods can effectively and efficiently detect video events.

Evolutionary optimization of video event classification

Detection and classification of semantic events in video documents have been major tasks in the domain of automatic video analysis. The multimodality of video data brings about challenging issues and the effectiveness of its automatic semantic processing has been hampered by the video semantic gap, that is, the gap between low-level visual and spatio-temporal features used as a digital representation of video documents and their semantic interpretation. Traditionally, these low-level features are concatenated in high-dimensional spaces and classified as high-level semantic events through computational learning methods. Support vector machines (SVMs), widely used in the literature, have often shown to outperform other popular learning techniques for this task. However, concatenation of several features with different intrinsic properties and wide dynamical ranges may result in curse of dimensionality and redundancy issues. Among the factors impacting the effectiveness of classification are (i) feature subset selection, (ii) tuning of classifiers’ parameters and (iii) selection of proper training instances. In this paper, we address these factors and propose a technique (denoted as GAoptSVM) for an optimal SVM-based video event classification through the use of an evolutionary optimization technique. Extensive experiments on the 50 h video data set of TRECVid 2008 event detection task and large quantities of video data collected from Youtube and CMU Graphics Lab Motion Capture Database demonstrate that our approach outperforms the traditional SVM and effectively classify video events with noticeable accuracy.

A novel framework for soccer goal detection based on semantic rule

Focusing on the problem of goal event detection in soccer videos, a novel method based on Hidden Markov Model (HMM) and the semantic rule is proposed. Firstly, a HMM for a goal event is constructed. Then a Normalized Semantic Weighted Sum (NSWS) rule is established by defining a new feature of shots, semantic observation weight. The test video is detected based on the HMM and the NSWS rule, respectively. Finally, a fusion scheme based on logic distance is proposed and the detection results of the HMM and the NSWS rule are fused by optimal weights in the decision level, obtaining the final result. Experimental results indicate that the proposed method achieves 96.43% precision and 100% recall, which shows the effectiveness of this letter.

HMM based soccer video event detection using enhanced mid-level semantic

Highlight detection is a fundamental step in semantics based video retrieval and personalized sports video browsing. In this paper, an effective hidden Markov models (HMMs) based soccer video event detection method based on a hierarchical video analysis framework is proposed. Soccer video shots are classified into four coarse mid-level semantics: global, median, close-up and audience. Global and local motion information is utilized for the refinement of coarse mid-level semantics. Sequential soccer video is segmented into event clips. Both the temporal transitions of the mid-level semantics and the overall features of an event clip are fused using HMMs to determine the type of event. Highlight detection performance of dynamic Bayesian networks (DBN), conditional random fields (CRF) and the proposed HMM based approach are compared. The average F-score of our highlights (including goal, shoot, foul and placed kick) detection approach is 82.92%, which outperforms that of DBN and CRF by 9.85% and 11.12% respectively. The effects of number of hidden states, overall features, and the refinement of mid-level semantics on the event detection performance are also discussed.

Introduction to the special issue of the multimedia tools and applications journal on events in multimedia

Humans think in terms of events and entities. Events provide a natural abstraction of happenings in the real world. The concept of events has a long history in foundational sciences such as philosophy and linguistics. After first developing object-based and entitybased approaches, computer science research is now addressing the concept of events and building many applications that consider events at least as important as objects. Consequently, we find many different solutions and approaches for modeling, detecting, and processing events. In addition, we find different applications that are based on events and that make use of events. This special issue of Springer’s Multimedia Tools and Applications journal on Events in Multimedia focuses on the detection, modeling, and processing of events and applications that make use of events in the context of multimedia data. The first paper “Ice hockey shooting event modeling with mixture hidden Markov model” presents a framework for analyzing events in ice hockey videos using mixture hidden Markov model (HMM). Ice hockey videos are hard to analyze due to the homogeneity of the video material. The paper shows that mixture HMM is a suitable approach for event detection in ice hockey videos with high accuracy. The second paper, “Event retrieval in video archives using rough set theory and partially supervised learning” presents a query-by-example approach where users provide example event shots in order to retrieve event shots from the TRECVID 2009 video data set. As a single model cannot be used for retrieval, different classifiers are combined into rules using rough set theory. Counter examples, which are necessary for applying rough set theory, do not need to be provided by the users but are compensated by applying a partially supervised learning method. Multimed Tools Appl (2012) 57:127–129 DOI 10.1007/s11042-011-0814-1

A Collecting and Distribution System of Urban Traffic Information Based on Video Events Detection

A collection and distribution scheme of urban intelligent traffic information based on video events detection is presented in this paper. In this scheme, advanced technology of computer vision, digital image processing technology, database technology, communications technology, control technology, image sensor technology and system integration technology are integrated and applied in the transportation field effectively to achieve the collection and processing of various urban traffic information and the release of integrated information, thereby traffic jams and accidents are reduced effectively. Therefore, the scheme is of great significance to promote the rapid development of transportation business and speed up the informatization process of urban transportation. And in the paper, the bus information collection and release system is employed as an example to explain the application of the system.

Detection of dominant flow and abnormal events in surveillance video

We propose an algorithm for abnormal event detection in surveillance video. The proposed algorithm is based on a semi-unsupervised learning method, a kind of feature-based approach so that it does not detect the moving object individually. The proposed algorithm identifies dominant flow without individual object tracking using a latent Dirichlet allocation model in crowded environments. It can also automatically detect and localize an abnormally moving object in real-life video. The performance tests are taken with several real-life databases, and their results show that the proposed algorithm can efficiently detect abnormally moving objects in real time. The proposed algorithm can be applied to any situation in which abnormal directions or abnormal speeds are detected regardless of direction.