Large Video Dataset Research Articles

Automatic Classification of Cichlid Behaviors Using 3D Convolutional Residual Networks.

SummaryMany behaviors that are critical for survival and reproduction are expressed over extended time periods. The ability to inexpensively record and store large volumes of video data creates new opportunities to understand the biological basis of these behaviors and simultaneously creates a need for tools that can automatically quantify behaviors from large video datasets. Here, we demonstrate that 3D Residual Networks can be used to classify an array of complex behaviors in Lake Malawi cichlid fishes. We first apply pixel-based hidden Markov modeling combined with density-based spatiotemporal clustering to identify sand disturbance events. After this, a 3D ResNet, trained on 11,000 manually annotated video clips, accurately (>76%) classifies the sand disturbance events into 10 fish behavior categories, distinguishing between spitting, scooping, fin swipes, and spawning. Furthermore, animal intent can be determined from these clips, as spits and scoops performed during bower construction are classified independently from those during feeding.

Learning multiple instance deep quality representation for robust object tracking

Robustly tracking various objects within a video stream with complex objects and backgrounds is a useful technique in next generation computer vision systems. However, in practice, it is difficult to design a successful video-based object tracking system due to the varied light conditions, possible occlusions, and fast-moving objects. In this work, a novel weakly-supervised and quality-guided visual object tracking model is proposed, wherein the key is a bidirectional long short-term memory recurrent neural network (BLSTM-RNN) that captures the feature sequence and predicts the quality score of each candidate window. More specifically, given a rich set of training videos annotated with the target objects, a weakly-supervised learning algorithm is first used to project all the candidate window features onto the semantic space. Next, we propose a two-stage algorithm to select the key frames from the video sequences, where both the shallow and deep filtering operations are conducted. Subsequently, the so-called BLSTM-RNN is proposed to characterize the feature sequence temporally, based on which the maximally possible object window can be calculated and finally output. In our experiment, a large video dataset containing 2045 NBA regular seasons and playoff basketball games was compiled. Based on this, a comparative study is conducted between the proposed algorithm and state-of-the-art video tracking methods. Extensive visualization results and comparative tracking precisions show the competitiveness of the proposed method.

Automated laparoscopic colorectal surgery workflow recognition using artificial intelligence: Experimental research.

Identifying laparoscopic surgical videos using artificial intelligence (AI) facilitates the automation of several currently time-consuming manual processes, including video analysis, indexing, and video-based skill assessment. This study aimed to construct a large annotated dataset comprising laparoscopic colorectal surgery (LCRS) videos from multiple institutions and evaluate the accuracy of automatic recognition for surgical phase, action, and tool by combining this dataset with AI. A total of 300 intraoperative videos were collected from 19 high-volume centers. A series of surgical workflows were classified into 9 phases and 3 actions, and the area of 5 tools were assigned by painting. More than 82 million frames were annotated for a phase and action classification task, and 4000 frames were annotated for a tool segmentation task. Of these frames, 80% were used for the training dataset and 20% for the test dataset. A convolutional neural network (CNN) was used to analyze the videos. Intersection over union (IoU) was used as the evaluation metric for tool recognition. The overall accuracies for the automatic surgical phase and action classification task were 81.0% and 83.2%, respectively. The mean IoU for the automatic tool segmentation task for 5 tools was 51.2%. A large annotated dataset of LCRS videos was constructed, and the phase, action, and tool were recognized with high accuracy using AI. Our dataset has potential uses in medical applications such as automatic video indexing and surgical skill assessments. Open research will assist in improving CNN models by making our dataset available in the field of computer vision.

Boosting Temporal Binary Coding for Large-Scale Video Search

In recent years, there has been an explosive increase in the amount of existing visual data. Hashing techniques have been successfully applied to deal with the large-scale nearest neighbor search problem among data on this massive scale. However, existing hashing methods usually learn a single hash code for each data point, and only by taking the content correlations among them into account. In practice, however, when handling complex visual data such as video, strong temporal relations exist among the successive frames. Moreover, if the preferred performance for large-scale video search is to be delivered, multiple hash codes are required for each data point in order to build multiple hash table indices. To address these problems, in this paper, we first study the multi-table learning problem for video search and attempt to learn binary codes by capturing the intrinsic video similarities from both the visual and the temporal aspects. By regarding the search over multiple tables as an ensemble prediction, the whole multi-table learning problem can be solved in a boosting learning manner to complementarily cover the nearest neighbors. For each table, a temporal binary coding solution is devised that thinks over the intrinsic relations among the visual content and the temporal consistency among the successive frames simultaneously. More specifically, we approximate the intrinsic visual similarities using a low-rank matrix based on sparse, non-negative feature expression. Furthermore, to essentially preserve the temporal consistency, we introduce a subspace rotation to model the variation among the successive frames. Under the boosting learning framework, the binary codes, hash functions and temporal variation of each table can be efficiently and jointly optimized. Extensive experiments on three large video datasets demonstrate that the proposed approach significantly outperforms a number of state-of-the-art hashing methods.

Improved Accuracy of Suspicious Activity Detection in Surveillance Video

Suspicious activity detection from surveillance video is the main objective of the work presented in this paper. The method developed consist of various stages of suspicious frame detection, and verifying the frame for suspicious activity related analysis of human movements within obtained set of suspicious frames. The method consist of GLCM feature extraction which constitutes the features such as energy, prominence, contrast, entropy, homogeneity type of features and matching using Euclidian distance along with descriptor features obtained by using Harris corner features and cosine similarity index estimation. The successful suspicious activity detection rate is analyzed which shows better performance and time saving method while analyzing large surveillance video dataset.

Motion Vector based Robust Video Hash

A novel robust video hashing scheme is proposed in this paper. Unlike most existing robust video hashing algorithms, the proposed video hash is generated based on the motion vectors instead of the image textures in the video stream. Therefore, neither full decoding of the video stream nor complex computation of pixel values is required. Based on analysis of motion vector properties regarding their suitability for robust hashing, an improved feature extraction mechanism is proposed and several optimization mechanisms are introduced in order to achieve better robustness and discriminability. The proposed hashing scheme is evaluated by a large and modern video data set and the experimental results demonstrate the excellent performance of the proposed hashing algorithm, which is comparable or even better than the complicated texture-based approaches.

Topological reconstruction of sub-cellular motion with Ensemble Kalman velocimetry

Microscopy imaging of plant cells allows the elaborate analysis of sub-cellular motions of organelles. The large video data set can be efficiently analyzed by automated algorithms. We develop a novel, data-oriented algorithm, which can track organelle movements and reconstruct their trajectories on stacks of image data. Our method proceeds with three steps: (ⅰ) identification, (ⅱ) localization, and (ⅲ) linking. This method combines topological data analysis and Ensemble Kalman Filtering, and does not assume a specific motion model. Application of this method on simulated data sets shows an agreement with ground truth. We also successfully test our method on real microscopy data.

Single Modality-Based Event Detection Framework for Complex Videos

Event detection of rare and complex events in large video datasets or in unconstrained user-uploaded videos on internet is a challenging task. The presence of irregular camera movement, viewpoint changes, illumination variations and significant changes in the background make extremely difficult to capture underlying motion in videos. In addition, extraction of features using different modalities (single streams) may offer computational complexities and cause abstraction of confusing and irrelevant spatial and semantic features. To address this problem, we present a single stream (RGB only) based on feature of spatial and semantic features extracted by modified 3D Residual Convulsion Network. We combine the spatial and semantic features based on this assumption that difference between both types of features can discover the accurate and relevant features. Moreover, introduction of temporal encoding builds the relationship in consecutive video frames to explore discriminative long-term motion patterns. We conduct extensive experiments on prominent publically available datasets. The obtained results demonstrate the great power of our proposed model and improved accuracy compared with existing state-of-the-art methods.

Animal behavior analysis at scale

Research Methods Video data provides insights into the social dynamics of wild animals, but processing large video datasets can be expensive and labor intensive. Using a deep convolutional neural network, Schofield et al. developed an automated pipeline to detect, track, and recognize individual animals from video in the wild. They demonstrate and evaluate this approach using a multiyear chimpanzee dataset. This system allowed researchers to monitor individual animals and automatically extract behavioral and social dynamics of individuals and groups directly from video. Sci. Adv. 10.1126/sciadv.aaw0736 (2019).

Occam’s Razor for Big Data? On Detecting Quality in Large Unstructured Datasets

Detecting quality in large unstructured datasets requires capacities far beyond the limits of human perception and communicability and, as a result, there is an emerging trend towards increasingly complex analytic solutions in data science to cope with this problem. This new trend towards analytic complexity represents a severe challenge for the principle of parsimony (Occam’s razor) in science. This review article combines insight from various domains such as physics, computational science, data engineering, and cognitive science to review the specific properties of big data. Problems for detecting data quality without losing the principle of parsimony are then highlighted on the basis of specific examples. Computational building block approaches for data clustering can help to deal with large unstructured datasets in minimized computation time, and meaning can be extracted rapidly from large sets of unstructured image or video data parsimoniously through relatively simple unsupervised machine learning algorithms. Why we still massively lack in expertise for exploiting big data wisely to extract relevant information for specific tasks, recognize patterns and generate new information, or simply store and further process large amounts of sensor data is then reviewed, and examples illustrating why we need subjective views and pragmatic methods to analyze big data contents are brought forward. The review concludes on how cultural differences between East and West are likely to affect the course of big data analytics, and the development of increasingly autonomous artificial intelligence (AI) aimed at coping with the big data deluge in the near future.

A new fire detection method using a multi-expert system based on color dispersion, similarity and centroid motion in indoor environment

In this paper, a video fire detection method is proposed, which demonstrated good performance in indoor environment. Three main novel ideas have been introduced. Firstly, a flame color model in RGB and HIS color space is used to extract pre-detected regions instead of traditional motion differential method, as it is more suitable for fire detection in indoor environment. Secondly, according to the flicker characteristic of the flame, similarity and two main values of centroid motion are proposed. At the same time, a simple but effective method for tracking the same regions in consecutive frames is established. Thirdly, a multi-expert system consisting of color component dispersion, similarity and centroid motion is established to identify flames. The proposed method has been tested on a very large dataset of fire videos acquired both in real indoor environment tests and from the Internet. The experimental results show that the proposed approach achieved a balance between the false positive rate and the false negative rate, and demonstrated a better performance in terms of overall accuracy and F standard with respect to other similar fire detection methods in indoor environment.

Egocentric Visitors Localization in Cultural Sites

We consider the problem of localizing visitors in a cultural site from egocentric (first-person) images. Localization information can be useful both to assist the user during his visit (e.g., by suggesting where to go and what to see next) and to provide behavioral information to the manager of the cultural site (e.g., how much time has been spent by visitors at a given location? What has been liked most?). To tackle the problem, we collected a large dataset of egocentric videos using two cameras: a head-mounted HoloLens device and a chest-mounted GoPro. Each frame has been labeled according to the location of the visitor and to what he was looking at. The dataset is freely available in order to encourage research in this domain. The dataset is complemented with baseline experiments performed considering a state-of-the-art method for location-based temporal segmentation of egocentric videos. Experiments show that compelling results can be achieved to extract useful information for both the visitor and the site-manager.

CARF-Net: CNN attention and RNN fusion network for video-based person reidentification

Video-based person reidentification is a challenging and important task in surveillance-based applications. Toward this, several shallow and deep networks have been proposed. However, the performance of existing shallow networks does not generalize well on large datasets. To improve the generalization ability, we propose a shallow end-to-end network which incorporates two stream convolutional neural networks, discriminative visual attention and recurrent neural network with triplet and softmax loss to learn the spatiotemporal fusion features. To effectively use both spatial and temporal information, we apply spatial, temporal, and spatiotemporal pooling. In addition, we contribute a large dataset of airborne videos for person reidentification, named DJI01. It includes various challenging conditions, such as occlusion, illuminationchanges, people with similar clothes, and the same people on different days. We perform elaborate qualitative and quantitative analyses to demonstrate the robust performance of the proposed model.

Object and motion cues based collaborative approach for human activity localization and recognition in unconstrained videos

This paper addresses the problem of activity localization and recognition in large scale video datasets by the collaborative use of holistic and motion based information (called motion cues). The concept of salient objects is used to obtain the holistic information while the motion cues are obtained by affine motion model and optical flow. The motion cues compensate the camera motion and localize the object of interest in a set of object proposals. Furthermore, the holistic information and motion cues are fused to get a reliable object of interest. In recognition phase, the holistic and motion based features are extracted from the object of interest for the training and testing of classifier. The extreme learning machine is adopted as a classifier to reduce the training and testing time and increase the classification accuracy. The effectiveness of the proposed approach is tested on UCF sports dataset. The detailed experimentation reveals that the proposed approach performs better than state-of-the-art action localization and recognition approaches.

A Visual Tracking Scheme for Accurate Object Retrieval in Low Frame Rate Videos

Visual tracking in low frame rate (LFR) videos has many inherent difficulties for achieving accurate target recovery, such as occlusions, abrupt motions and rapid pose changes. Thus, conventional tracking methods cannot be applied reliably. In this paper, we offer a new scheme for tracking objects in low frame rate videos. We present a method of integrating multiple metrics for template matching, as an extension for the particle filter. By inspecting a large data set of videos for tracking, we show that our method not only outperforms other related benchmarks in the field, but it also achieves better results both visually and quantitatively, once compared to actual ground truth data.

Scalable Semi-Automatic Annotation for Multi-Camera Person Tracking.

This paper proposes a generic methodology for the semi-automatic generation of reliable position annotations for evaluating multi-camera people-trackers on large video data sets. Most of the annotation data are automatically computed, by estimating a consensus tracking result from multiple existing trackers and people detectors and classifying it as either reliable or not. A small subset of the data, composed of tracks with insufficient reliability, is verified by a human using a simple binary decision task, a process faster than marking the correct person position. The proposed framework is generic and can handle additional trackers. We present results on a data set of $sim 6$ h captured by 4 cameras, featuring a person in a holiday flat, performing activities such as walking, cooking, eating, cleaning, and watching TV. When aiming for a tracking accuracy of 60 cm, 80% of all video frames are automatically annotated. The annotations for the remaining 20% of the frames were added after human verification of an automatically selected subset of data. This involved $sim 2.4$ h of manual labor. According to a subsequent comprehensive visual inspection to judge the annotation procedure, we found 99% of the automatically annotated frames to be correct. We provide guidelines on how to apply the proposed methodology to new data sets. We also provide an exploratory study for the multi-target case, applied on the existing and new benchmark video sequences.

Efficient processing of video containment queries by using composite ordinal features

Video containment queries find the videos that have similar sequence of frames to query video clips. Applying sequence matching to all possible subsequences for video containment queries is computationally expensive for large volumes of video data. In this paper, we propose an efficient candidate segment selection scheme, which selects only a small set of subsequences to be matched to the query sequence, by using a cluster of similar frames, called a frame cluster. We also propose a new type of the ordinal feature, called a composite ordinal feature that allows multiple ranks to certain cells. In experiments with large scale video data sets, we show our method improves the query response time by efficiently selecting a set of subsequences for sequence matching.

Real-Time Fire Detection for Video-Surveillance Applications Using a Combination of Experts Based on Color, Shape, and Motion

In this paper, we propose a method that is able to detect fires by analyzing videos acquired by surveillance cameras. Two main novelties have been introduced. First, complementary information, based on color, shape variation, and motion analysis, is combined by a multiexpert system. The main advantage deriving from this approach lies in the fact that the overall performance of the system significantly increases with a relatively small effort made by the designer. Second, a novel descriptor based on a bag-of-words approach has been proposed for representing motion. The proposed method has been tested on a very large dataset of fire videos acquired both in real environments and from the web. The obtained results confirm a consistent reduction in the number of false positives, without paying in terms of accuracy or renouncing the possibility to run the system on embedded platforms.

A data-driven approach for tag refinement and localization in web videos

Tagging of visual content is becoming more and more widespread as web-based services and social networks have popularized tagging functionalities among their users. These user-generated tags are used to ease browsing and exploration of media collections, e.g. using tag clouds, or to retrieve multimedia content. However, not all media are equally tagged by users. Using the current systems is easy to tag a single photo, and even tagging a part of a photo, like a face, has become common in sites like Flickr and Facebook. On the other hand, tagging a video sequence is more complicated and time consuming, so that users just tag the overall content of a video. In this paper we present a method for automatic video annotation that increases the number of tags originally provided by users, and localizes them temporally, associating tags to keyframes. Our approach exploits collective knowledge embedded in user-generated tags and web sources, and visual similarity of keyframes and images uploaded to social sites like YouTube and Flickr, as well as web sources like Google and Bing. Given a keyframe, our method is able to select “on the fly” from these visual sources the training exemplars that should be the most relevant for this test sample, and proceeds to transfer labels across similar images. Compared to existing video tagging approaches that require training classifiers for each tag, our system has few parameters, is easy to implement and can deal with an open vocabulary scenario. We demonstrate the approach on tag refinement and localization on DUT-WEBV, a large dataset of web videos, and show state-of-the-art results.

Learning Relational Event Models from Video

Event models obtained automatically from video can be used in applications ranging from abnormal event detection to content based video retrieval. When multiple agents are involved in the events, characterizing events naturally suggests encoding interactions as relations. Learning event models from this kind of relational spatio-temporal data using relational learning techniques such as Inductive Logic Programming (ILP) hold promise, but have not been successfully applied to very large datasets which result from video data. In this paper, we present a novel framework REMIND (Relational Event Model INDuction) for supervised relational learning of event models from large video datasets using ILP. Efficiency is achieved through the learning from interpretations setting and using a typing system that exploits the type hierarchy of objects in a domain. The use of types also helps prevent over generalization. Furthermore, we also present a type-refining operator and prove that it is optimal. The learned models can be used for recognizing events from previously unseen videos. We also present an extension to the framework by integrating an abduction step that improves the learning performance when there is noise in the input data. The experimental results on several hours of video data from two challenging real world domains (an airport domain and a physical action verbs domain) suggest that the techniques are suitable to real world scenarios.