Event Recognition Task Research Articles

Event recognition method based on feature synthesizing for a zero-shot intelligent distributed optical fiber sensor

Phase-sensitive optical time domain reflectometer (Φ-OTDR) is an emergent distributed optical sensing system with the advantages of high localization accuracy and high sensitivity. It has been widely used for intrusion identification, pipeline monitoring, under-ground tunnel monitoring, etc. Deep learning-based classification methods work well for Φ-OTDR event recognition tasks with sufficient samples. However, the lack of training data samples is sometimes a serious problem for these data-driven algorithms. This paper proposes a novel feature synthesizing approach to solve this problem. A mixed class approach and a reinforcement learning-based guided training method are proposed to realize high-quality feature synthesis. Experiment results in the task of eight event classifications, including one unknown class, show that the proposed method can achieve an average classification accuracy of 42% for the unknown class and obtain its event type, meanwhile achieving a 74% average overall classification accuracy. This is 29% and 7% higher, respectively, than those of the ordinary instance synthesizing method. Moreover, this is the first time that the Φ-OTDR system can recognize a specific event and tell its event type without collecting its data sample in advance.

Dynamic Spatio-Temporal Graph Reasoning for VideoQA With Self-Supervised Event Recognition.

Video question answering (VideoQA) requires the ability of comprehensively understanding visual contents in videos. Existing VideoQA models mainly focus on scenarios involving a single event with simple object interactions and leave event-centric scenarios involving multiple events with dynamically complex object interactions largely unexplored. These conventional VideoQA models are usually based on features extracted from the global visual signals, making it difficult to capture the object-level and event-level semantics. Although there exists a recent work utilizing a static spatio-temporal graph to explicitly model object interactions in videos, it ignores the dynamic impact of questions for graph construction and fails to exploit the implicit event-level semantic clues in questions. To overcome these limitations, we propose a Self-supervised Dynamic Graph Reasoning (SDGraphR) model for video question answering (VideoQA). Our SDGraphR model learns a question-guided spatio-temporal graph that dynamically encodes intra-frame spatial correlations and inter-frame correspondences between objects in the videos. Furthermore, the proposed SDGraphR model discovers event-level cues from questions to conduct self-supervised learning with an auxiliary event recognition task, which in turn helps to improve its VideoQA performances without using any extra annotations. We carry out extensive experiments to validate the substantial improvements of our proposed SDGraphR model over existing baselines.

ViGAT: Bottom-Up Event Recognition and Explanation in Video Using Factorized Graph Attention Network

In this paper a pure-attention bottom-up approach, called ViGAT, that utilizes an object detector together with a Vision Transformer (ViT) backbone network to derive object and frame features, and a head network to process these features for the task of event recognition and explanation in video, is proposed. The ViGAT head consists of graph attention network (GAT) blocks factorized along the spatial and temporal dimensions in order to capture effectively both local and long-term dependencies between objects or frames. Moreover, using the weighted in-degrees (WiDs) derived from the adjacency matrices at the various GAT blocks, we show that the proposed architecture can identify the most salient objects and frames that explain the decision of the network. A comprehensive evaluation study is performed, demonstrating that the proposed approach provides state-of-the-art results on three large, publicly available video datasets (FCVID, Mini-Kinetics, ActivityNet).

Multi-view representation for sound event recognition

The sound event recognition (SER) task is gaining lot of importance in emerging applications such as machine audition, audio surveillance, and environmental audio scene recognition. The recognition of sound events with noisy conditions in real-time surveillance applications is a difficult task. In this paper, we focus on learning patterns using multiple forms (views) of the given sound events. We propose two variants of the Multi-View Representation (MVR)-based approach for the SER task. The first variant combines the auditory image-based features and the cepstral features from sound signal. The second variant combines the statistical features extracted from the auditory images and the cepstral features of sound signal. In addition to these variants, Constant Q-transform and Variable Q-transform image-based features are also explored to study the other effective forms of multi-view representations. A discriminative model-based classifier is then used to recognize these representations as environmental sound events. The performance of the proposed MVR approaches is evaluated on three benchmark sound event datasets namely ESC-50, DCASE2016 Task 2, and DCASE2018 Task 2 for the SER task. The recognition accuracy of the proposed MVR approach is significantly better than the other approaches proposed in the recent literature.

Environmental Audio Scene and Sound Event Recognition for Autonomous Surveillance

Monitoring of human and social activities is becoming increasingly pervasive in our living environment for public security and safety applications. The recognition of suspicious events is important in both indoor and outdoor environments, such as child-care centers, smart-homes, old-age homes, residential areas, office environments, elevators, and smart cities. Environmental audio scene and sound event recognition are the fundamental tasks involved in many audio surveillance applications. Although numerous approaches have been proposed, robust environmental audio surveillance remains a huge challenge due to various reasons, such as various types of overlapping audio sounds, background noises, and lack of universal and multi-modal datasets. The goal of this article is to review various features of representing audio scenes and sound events and provide appropriate machine learning algorithms for audio surveillance tasks. Benchmark datasets are categorized based on the real-world scenarios of audio surveillance applications. To have a quantitative understanding, some of the state-of-the-art approaches are evaluated based on two benchmark datasets for audio scenes and sound event recognition tasks. Finally, we outline the possible future directions for improving the recognition of environmental audio scenes and sound events.

How Deep Features Have Improved Event Recognition in Multimedia

Event recognition is one of the areas in multimedia that is attracting great attention of researchers. Being applicable in a wide range of applications, from personal to collective events, a number of interesting solutions for event recognition using multimedia information sources have been proposed. On the other hand, following their immense success in classification, object recognition, and detection, deep learning has been shown to perform well in event recognition tasks also. Thus, a large portion of the literature on event analysis relies nowadays on deep learning architectures. In this article, we provide an extensive overview of the existing literature in this field, analyzing how deep features and deep learning architectures have changed the performance of event recognition frameworks. The literature on event-based analysis of multimedia contents can be categorized into four groups, namely (i) event recognition in single images; (ii) event recognition in personal photo collections; (iii) event recognition in videos; and (iv) event recognition in audio recordings. In this article, we extensively review different deep-learning-based frameworks for event recognition in these four domains. Furthermore, we also review some benchmark datasets made available to the scientific community to validate novel event recognition pipelines. In the final part of the manuscript, we also provide a detailed discussion on basic insights gathered from the literature review, and identify future trends and challenges.

Semi-supervised online structure learning for composite event recognition

Online structure learning approaches, such as those stemming from statistical relational learning, enable the discovery of complex relations in noisy data streams. However, these methods assume the existence of fully-labelled training data, which is unrealistic for most real-world applications. We present a novel approach for completing the supervision of a semi-supervised structure learning task. We incorporate graph-cut minimisation, a technique that derives labels for unlabelled data, based on their distance to their labelled counterparts. In order to adapt graph-cut minimisation to first order logic, we employ a suitable structural distance for measuring the distance between sets of logical atoms. The labelling process is achieved online (single-pass) by means of a caching mechanism and the Hoeffding bound, a statistical tool to approximate globally-optimal decisions from locally-optimal ones. We evaluate our approach on the task of composite event recognition by using a benchmark dataset for human activity recognition, as well as a real dataset for maritime monitoring. The evaluation suggests that our approach can effectively complete the missing labels and eventually, improve the accuracy of the underlying structure learning system.

Web Video Event Recognition by Semantic Analysis From Ubiquitous Documents.

In recent years, the task of event recognition from videos has attracted increasing interest in multimedia area. While most of the existing research was mainly focused on exploring visual cues to handle relatively small-granular events, it is difficult to directly analyze video content without any prior knowledge. Therefore, synthesizing both the visual and semantic analysis is a natural way for video event understanding. In this paper, we study the problem of Web video event recognition, where Web videos often describe large-granular events and carry limited textual information. Key challenges include how to accurately represent event semantics from incomplete textual information and how to effectively explore the correlation between visual and textual cues for video event understanding. We propose a novel framework to perform complex event recognition from Web videos. In order to compensate the insufficient expressive power of visual cues, we construct an event knowledge base by deeply mining semantic information from ubiquitous Web documents. This event knowledge base is capable of describing each event with comprehensive semantics. By utilizing this base, the textual cues for a video can be significantly enriched. Furthermore, we introduce a two-view adaptive regression model, which explores the intrinsic correlation between the visual and textual cues of the videos to learn reliable classifiers. Extensive experiments on two real-world video data sets show the effectiveness of our proposed framework and prove that the event knowledge base indeed helps improve the performance of Web video event recognition.

Deep Fusion of Multiple Semantic Cues for Complex Event Recognition.

We present a deep learning strategy to fuse multiple semantic cues for complex event recognition. In particular, we tackle the recognition task by answering how to jointly analyze human actions (who is doing what), objects (what), and scenes (where). First, each type of semantic features (e.g., human action trajectories) is fed into a corresponding multi-layer feature abstraction pathway, followed by a fusion layer connecting all the different pathways. Second, the correlations of how the semantic cues interacting with each other are learned in an unsupervised cross-modality autoencoder fashion. Finally, by fine-tuning a large-margin objective deployed on this deep architecture, we are able to answer the question on how the semantic cues of who, what, and where compose a complex event. As compared with the traditional feature fusion methods (e.g., various early or late strategies), our method jointly learns the essential higher level features that are most effective for fusion and recognition. We perform extensive experiments on two real-world complex event video benchmarks, MED'11 and CCV, and demonstrate that our method outperforms the best published results by 21% and 11%, respectively, on an event recognition task.

Weakly-Supervised Cross-Domain Dictionary Learning for Visual Recognition

We address the visual categorization problem and present a method that utilizes weakly labeled data from other visual domains as the auxiliary source data for enhancing the original learning system. The proposed method aims to expand the intra-class diversity of original training data through the collaboration with the source data. In order to bring the original target domain data and the auxiliary source domain data into the same feature space, we introduce a weakly-supervised cross-domain dictionary learning method, which learns a reconstructive, discriminative and domain-adaptive dictionary pair and the corresponding classifier parameters without using any prior information. Such a method operates at a high level, and it can be applied to different cross-domain applications. To build up the auxiliary domain data, we manually collect images from Web pages, and select human actions of specific categories from a different dataset. The proposed method is evaluated for human action recognition, image classification and event recognition tasks on the UCF YouTube dataset, the Caltech101/256 datasets and the Kodak dataset, respectively, achieving outstanding results.

Selection of time-variant features for earthquake classification at the Nevado-del-Ruiz volcano

Seismic event recognition is an important task for hazard assessment, eruption prediction and risk mitigation, since it can be used to determine the state of a volcano. Usually, expert technicians read features extracted from the seismogram, such as, cepstral derived coefficients, energy centroids, instant frequency, instant envelop, among others. However, there are few studies about the selection of important features for classifying several types of seismic events, i.e., taking into account the temporal contribution of each considered feature. This paper presents a feature selection strategy based on a relevance measure of time-variant features for seismic event classification. In this research, features are selected as those with the maximal information preserved within the time analysis. Since features selection stage is performed by incremental training, a simple k-nearest neighbor classification rule is used to properly determine the dimension of the final feature set. The employed feature extraction and feature selection methodologies are tested on an isolated event recognition task. The database used to test the methodology is composed of the following classes: volcano-tectonic, long period earthquakes, tremors and hybrid events. Data was recorded at the seismic monitoring stations located at the Nevado-del-Ruiz volcano, Colombia. Using a classifier based on hidden Markov models, accomplished results exhibit a performance improvement from 78% to 88% using the proposed methodology in comparison to the state-of-the-art feature sets.

One-Class SVMs Challenges in Audio Detection and Classification Applications

Support vector machines (SVMs) have gained great attention and have been used extensively and successfully in the field of sounds (events) recognition. However, the extension of SVMs to real-world signal processing applications is still an ongoing research topic. Our work consists of illustrating the potential of SVMs on recognizing impulsive audio signals belonging to a complex real-world dataset. We propose to apply optimized one-class support vector machines (1-SVMs) to tackle both sound detection and classification tasks in the sound recognition process. First, we propose an efficient and accurate approach for detecting events in a continuous audio stream. The proposed unsupervised sound detection method which does not require any pretrained models is based on the use of the exponential family model and 1-SVMs to approximate the generalized likelihood ratio. Then, we apply novel discriminative algorithms based on 1-SVMs with new dissimilarity measure in order to address a supervised sound-classification task. We compare the novel sound detection and classification methods with other popular approaches. The remarkable sound recognition results achieved in our experiments illustrate the potential of these methods and indicate that 1-SVMs are well suited for event-recognition tasks.

Story Processing in Patients with Damage to the Prefrontal Cortex

The prefrontal cortex is known to be involved in performing complex cognitive tasks requiring reasoning, planning and decision-making. Neuropsychological evidence also supports the idea that the prefrontal cortex is generally involved in encoding and retrieving complex events, such as action and narrative knowledge. Patients with frontal lobe damage are reported to have difficulty in processing different aspects of narrative representations, such as the figurative moral meaning, syntactic features, and inference generation. In the present study, we examined story processing in 17 patients with frontal lobe lesions and compared their performance to 7 amnesic patients and 17 normal controls. Two stories were presented by using two slightly different processing demands in order to assess the subject's ability to draw inferences on-line during the course of comprehension or later retrieval. Although all patients had impaired story memory, patients with frontal lobe lesions showed a pattern of deficit at an early stage of story comprehension that specifically involved the ability to reconstruct the sequential links among events and to extract inferential knowledge from the text during encoding. Amnesic patients were severely impaired in recalling story semantic units, including single events and larger narrative constituents, as well as in the event recognition task. Consequently, they were unable to establish inferential relations among the events and, thus, the global sequential structure of the stories during retrieval. In contrast, they had no difficulty in extracting inferential knowledge during story comprehension on the basis of readily available information. This study shows that damage to different cortical regions may induce impairments at various levels of story processing.

Interactive Seismic Event Recognition and its Applications

Many methods for automatic seismic event picking have been proposed over the years. All fully automatic methods lackrobustness. Given that present-day machines can at best mimichuman picking procedures, it is hardly surprising that they will failwhen things get tough. Attenuation, sensor coupling variations, random scattering in the overburden and the competition ofmultiple arrivals for a given time slot all complicate the task ofdesigning a robust picking machine. In this paper, we set out thearchitecture of a semi-automatic seismic event recognition system, in which robustness is achieved by allowing an observer to train themachine interactively both to accomplish a given event recognitiontask and to cope with variations in the nature of that task. Theevent-recognition task itself is accomplished by moving fromconventional envelope threshold detection to phase sensitivewindowed cross-correlation. The art of enhancing the precision oftime picking lies in supervising the transition between the two asrising signal-to-noise levels permit. The scheme has been used ona standard PC to extract arrival times from full waveform soniclogs, reflection and tomographic in-mine borehole radar surveys, and 3D seismic reflection surveys for coal mining.