Large-scale Video Dataset Research Articles

Detecting clinical medication errors with AI enabled wearable cameras

Drug-related errors are a leading cause of preventable patient harm in the clinical setting. We present the first wearable camera system to automatically detect potential errors, prior to medication delivery. We demonstrate that using deep learning algorithms, our system can detect and classify drug labels on syringes and vials in drug preparation events recorded in real-world operating rooms. We created a first-of-its-kind large-scale video dataset from head-mounted cameras comprising 4K footage across 13 anesthesiology providers, 2 hospitals and 17 operating rooms over 55 days. The system was evaluated on 418 drug draw events in routine patient care and a controlled environment and achieved 99.6% sensitivity and 98.8% specificity at detecting vial swap errors. These results suggest that our wearable camera system has the potential to provide a secondary check when a medication is selected for a patient, and a chance to intervene before a potential medical error.

Tiny Object Tracking: A Large-Scale Dataset and a Baseline.

Tiny objects, frequently appearing in practical applications, have weak appearance and features, and receive increasing interests in many vision tasks, such as object detection and segmentation. To promote the research and development of tiny object tracking, we create a large-scale video dataset, which contains 434 sequences with a total of more than 217K frames. Each frame is carefully annotated with a high-quality bounding box. In data creation, we take 12 challenge attributes into account to cover a broad range of viewpoints and scene complexities, and annotate these attributes for facilitating the attribute-based performance analysis. To provide a strong baseline in tiny object tracking, we propose a novel multilevel knowledge distillation network (MKDNet), which pursues three-level knowledge distillations in a unified framework to effectively enhance the feature representation, discrimination, and localization abilities in tracking tiny objects. Extensive experiments are performed on the proposed dataset, and the results prove the superiority and effectiveness of MKDNet compared with state-of-the-art methods. The dataset, the algorithm code, and the evaluation code are available at https://github.com/mmic-lcl/Datasets-and-benchmark-code.

GMMFormer: Gaussian-Mixture-Model Based Transformer for Efficient Partially Relevant Video Retrieval

Given a text query, partially relevant video retrieval (PRVR) seeks to find untrimmed videos containing pertinent moments in a database. For PRVR, clip modeling is essential to capture the partial relationship between texts and videos. Current PRVR methods adopt scanning-based clip construction to achieve explicit clip modeling, which is information-redundant and requires a large storage overhead. To solve the efficiency problem of PRVR methods, this paper proposes GMMFormer, a Gaussian-Mixture-Model based Transformer which models clip representations implicitly. During frame interactions, we incorporate Gaussian-Mixture-Model constraints to focus each frame on its adjacent frames instead of the whole video. Then generated representations will contain multi-scale clip information, achieving implicit clip modeling. In addition, PRVR methods ignore semantic differences between text queries relevant to the same video, leading to a sparse embedding space. We propose a query diverse loss to distinguish these text queries, making the embedding space more intensive and contain more semantic information. Extensive experiments on three large-scale video datasets (i.e., TVR, ActivityNet Captions, and Charades-STA) demonstrate the superiority and efficiency of GMMFormer.

Data-Driven Detection of Video Forgery for Secured Microcontrollers: Dataset and Method

The microcontrollers in a camera often capture videos at a low frame rate due to limited processing capability. To satisfy the requirement of high quality of service, low-frame-rate videos are often forged as the high-frame-rate ones by the Frame Rate Up-Conversion (FRUC) operation. Therefore, detecting the existence of FRUC has become a necessary job for secured microcontrollers. In this paper, we propose a data-driven detection to identify whether a video is forged by FRUC. The core of detection is the creation of a large-scale video dataset VifFRUC (Videos forged by FRUC). Various types of forged videos can continue to be added into VifFRUC, making the detection more universal and robust. To match with VifFRUC, we have also designed a neural network, which trains a number of Long Short Term Memory (LSTM) units in parallel to learn the data-driven detection. The parallel LSTM structure of network can continually adapt to the newly added FRUC methods in VifFRUC. Extensive experiments on VifFRUC demonstrate the effectiveness of data-driven detection for FRUC, resulting in the security improvement of microcontrollers.

Multi-temporal dependency handling in video smoke recognition: A holistic approach spanning spatial, short-term, and long-term perspectives

Accurately recognizing video-based smoke is still a profoundly challenging task due to the special characteristics of smoke, such as non-rigid morphology, semi-transparent representations, serious interferences (namely non-smoky white steams and clouds etc.), and so on. In this study, we present the Spatio-Temporal Local-Enhanced Network (STENet), which is meticulously designed to handle the spatio-temporal dependencies of smoke video in multi-temporal views, effectively synthesizing spatial, long-term, and short-term smoke features. To account for the semi-translucent nature of smoke, we introduce a short-term temporal branch within STENet that models the transient temporal dependencies between adjacent frames, thereby distinguishing smoke from visually similar interferences such as clouds and water steam. Furthermore, the representation of spatial and short-term temporal features, which capture the instantaneous spatio-temporal characteristics of a single smoke plume, can be fine-grained aligned and mutually guided, which is accomplished through our Collaborative Alignment(CoAM). Moreover, to address the unique drifting characteristics and dynamic variations of non-rigid objects across different temporal scales, we propose a new Prioritized Local-enhanced Transformer (PLT) that selectively focuses on key frame subsets for capturing long-term temporal dependencies. Experiments conducted on benchmarks demonstrate that the proposed STENet achieves state-of-the-art results on large-scale video dataset for recognizing industrial smoke emissions (RISE) and forest fire surveillance video (FSmoke). In particular, our STENet achieves 0.893 F-score on RISE dataset with 6.9M parameters and 3.86G FLOPs, being 4×-18× smaller and using 5×-51× fewer FLOPs than previous baseline models.

Sparsely-Supervised Object Tracking.

Recent years have witnessed the incredible performance boost of data-driven deep visual object trackers. Despite the success, these trackers require millions of sequential manual labels on videos for supervised training, implying the heavy burden of human annotating. This raises a crucial question: how to train a powerful tracker from abundant videos using limited manual annotations? In this paper, we challenge the conventional belief that frame-by-frame labeling is indispensable, and show that providing a small number of annotated bounding boxes in each video is sufficient for training a strong tracker. To facilitate that, we design a novel SParsely-supervised Object Tracking (SPOT) framework. It regards the sparsely annotated boxes as anchors and progressively explores in the temporal span to discover unlabeled target snapshots. Under the teacher-student paradigm, SPOT leverages the unique transitive consistency inherent in the tracking task as supervision, extracting knowledge from both anchor snapshots and unlabeled target snapshots. We also utilize several effective training strategies, i.e., IoU filtering, asymmetric augmentation, and temporal calibration to further improve the training robustness of SPOT. The experimental results demonstrate that, given less than 5 labels for each video, trackers trained via SPOT perform on par with their fully-supervised counterparts. Moreover, our SPOT exhibits two desirable properties: 1) SPOT enables us to fully exploit large-scale video datasets by efficiently allocating sparse labels to more videos even under a limited labeling budget; 2) when equipped with a target discovery module, SPOT can even learn from purely unlabeled videos for performance gain. We hope this work could inspire the community to rethink the current annotation principles and make a step towards practical label-efficient deep tracking.

3D mouse pose from single-view video and a new dataset

We present a method to infer the 3D pose of mice, including the limbs and feet, from monocular videos. Many human clinical conditions and their corresponding animal models result in abnormal motion, and accurately measuring 3D motion at scale offers insights into health. The 3D poses improve classification of health-related attributes over 2D representations. The inferred poses are accurate enough to estimate stride length even when the feet are mostly occluded. This method could be applied as part of a continuous monitoring system to non-invasively measure animal health, as demonstrated by its use in successfully classifying animals based on age and genotype. We introduce the Mouse Pose Analysis Dataset, the first large scale video dataset of lab mice in their home cage with ground truth keypoint and behavior labels. The dataset also contains high resolution mouse CT scans, which we use to build the shape models for 3D pose reconstruction.

Contrastive Masked Autoencoders for Self-Supervised Video Hashing

Self-Supervised Video Hashing (SSVH) models learn to generate short binary representations for videos without ground-truth supervision, facilitating large-scale video retrieval efficiency and attracting increasing research attention. The success of SSVH lies in the understanding of video content and the ability to capture the semantic relation among unlabeled videos. Typically, state-of-the-art SSVH methods consider these two points in a two-stage training pipeline, where they firstly train an auxiliary network by instance-wise mask-and-predict tasks and secondly train a hashing model to preserve the pseudo-neighborhood structure transferred from the auxiliary network. This consecutive training strategy is inflexible and also unnecessary. In this paper, we propose a simple yet effective one-stage SSVH method called ConMH, which incorporates video semantic information and video similarity relationship understanding in a single stage. To capture video semantic information for better hashing learning, we adopt an encoder-decoder structure to reconstruct the video from its temporal-masked frames. Particularly, we find that a higher masking ratio helps video understanding. Besides, we fully exploit the similarity relationship between videos by maximizing agreement between two augmented views of a video, which contributes to more discriminative and robust hash codes. Extensive experiments on three large-scale video datasets (i.e., FCVID, ActivityNet and YFCC) indicate that ConMH achieves state-of-the-art results. Code is available at https://github.com/huangmozhi9527/ConMH.

Application of visual communication in digital animation advertising design using convolutional neural networks and big data.

In the age of big data, visual communication has emerged as a critical means of engaging with customers. Among multiple modes of visual communication, digital animation advertising is an exceptionally potent tool. Advertisers can create lively and compelling ads by harnessing the power of digital animation technology. This article proposes a multimodal visual communication system (MVCS) model based on multimodal video emotion analysis. This model automatically adjusts video content and playback mode according to users' emotions and interests, achieving more personalized video communication. The MVCS model analyses videos from multiple dimensions, such as vision, sound, and text, by training on a large-scale video dataset. We employ convolutional neural networks to extract the visual features of videos, while the audio and text features are extracted and analyzed for emotions using recurrent neural networks. By integrating feature information, the MVCS model can dynamically adjust the video's playback mode based on users' emotions and interaction behaviours, which increases its playback volume. We conducted a satisfaction survey on 106 digitally corrected ads created using the MVCS method to evaluate our approach's effectiveness. Results showed that 92.6% of users expressed satisfaction with the adjusted ads, indicating the MVCS model's efficacy in enhancing digital ad design effectiveness.

Detecting anomalies from liquid transfer videos in automated laboratory setting.

In this work, we address the problem of detecting anomalies in a certain laboratory automation setting. At first, we collect video images of liquid transfer in automated laboratory experiments. We mimic the real-world challenges of developing an anomaly detection model by considering two points. First, the size of the collected dataset is set to be relatively small compared to large-scale video datasets. Second, the dataset has a class imbalance problem where the majority of the collected videos are from abnormal events. Consequently, the existing learning-based video anomaly detection methods do not perform well. To this end, we develop a practical human-engineered feature extraction method to detect anomalies from the liquid transfer video images. Our simple yet effective method outperforms state-of-the-art anomaly detection methods with a notable margin. In particular, the proposed method provides 19% and 76% average improvement in AUC and Equal Error Rate, respectively. Our method also quantifies the anomalies and provides significant benefits for deployment in the real-world experimental setting.

Bullet Train Motion Video-Based Noise-Barrier Defects Inspection Method

Vision-based automatic noise-barrier inspection of high-speed railway, instead of manual patrol, remains a great challenge. Even though many supervised learning-based methods have been developed, massive redundant video frames and scarce defective samples are the main obstacles to leverage the performance of the noise-barrier inspection task. To tackle the problems, we present a novel Vision-based Noise-barrier Inspection System (VNIS), which is deployed on the bullet train to inspect the noise-barrier defects by using motion video. VNIS uses the proposed panorama generation model based on motion video to obtain panoramic images from massive redundant video sequences. Then, we employ a self-supervised learning deep network to solve the problem of the scarce defective samples. Comprehensive experiments are conducted on a large-scale video dataset of bullet train. VNIS yields competitive performance on noise-barrier defects inspection. Specifically, an average accuracy of 99.14% is achieved for noise-barrier defects inspection.

EAN: Event Adaptive Network for Enhanced Action Recognition

Efficiently modeling spatial–temporal information in videos is crucial for action recognition. To achieve this goal, state-of-the-art methods typically employ the convolution operator and the dense interaction modules such as non-local blocks. However, these methods cannot accurately fit the diverse events in videos. On the one hand, the adopted convolutions are with fixed scales, thus struggling with events of various scales. On the other hand, the dense interaction modeling paradigm only achieves sub-optimal performance as action-irrelevant parts bring additional noises for the final prediction. In this paper, we propose a unified action recognition framework to investigate the dynamic nature of video content by introducing the following designs. First, when extracting local cues, we generate the spatial–temporal kernels of dynamic-scale to adaptively fit the diverse events. Second, to accurately aggregate these cues into a global video representation, we propose to mine the interactions only among a few selected foreground objects by a Transformer, which yields a sparse paradigm. We call the proposed framework as Event Adaptive Network because both key designs are adaptive to the input video content. To exploit the short-term motions within local segments, we propose a novel and efficient Latent Motion Code module, further improving the performance of the framework. Extensive experiments on several large-scale video datasets, e.g., Something-to-Something V1 &V2, Kinetics, and Diving48, verify that our models achieve state-of-the-art or competitive performances at low FLOPs. Codes are available at: https://github.com/tianyuan168326/EAN-Pytorch.

Knowledge Transfer from Macro-world to Micro-world: Enhancing 3D Cryo-ET Classification through Fine-Tuning Video-based Deep Models.

Deep learning models have achieved remarkable success in a wide range of natural-world tasks, such as vision, language, and speech recognition. These accomplishments are largely attributed to the availability of open-source large-scale datasets. More importantly, pre-trained foundational modellearnings exhibit a surprising degree of transferability to downstream tasks, enabling efficient learning even with limited training examples. However, the application of such natural-domain models to the domain of tiny Cryo-Electron Tomography (Cryo-ET) images has been a relatively unexplored frontier. This research is motivated by the intuition that 3D Cryo-ET voxel data can be conceptually viewed as a sequence of progressively evolving video frames. Leveraging the above insight, we propose a novel approach that involves the utilization of 3D models pre-trained on large-scale video datasets to enhance Cryo-ET subtomogram classification. Our experiments, conducted on both simulated and real Cryo-ET datasets, reveal compelling results. The use of video initialization not only demonstrates improvements in classification accuracy but also substantially reduces training costs. Further analyses provide additional evidence of the value of video initialization in enhancing subtomogram feature extraction. Additionally, we observe that video initialization yields similar positive effects when applied to medical 3D classification tasks, underscoring the potential of cross-domain knowledge transfer from video-based models to advance the state-of-the-art in a wide range of biological and medical data types. https://github.com/xulabs/aitom.

Video Visual Relation Detection via 3D Convolutional Neural Network

Video visual relation detection, which aims to detect the visual relations between objects in the form of relation triplet &#x003C;subject, predicate, object&#x003E; (<i>e.g.</i>, &#x201C;person-ride-bike&#x201D;, &#x201C;dog-toward-car&#x201D;, <i>etc.</i>), is a significant and fundamental task in computer vision. However, most of the existing works about visual relation instances are focused on static images. Modeling the non-static relationships in videos has drawn little attention due to lacking large-scale video dataset support. In our work, we propose a video dataset named Video Predicate Detection and Reasoning (VidPDR) for dynamic video visual relation detection, which consists of 1,000 videos with dense manually dynamic labeled annotations on 21 object classes and 37 predicates classes. Moreover, we propose a novel spatio-temporal feature extraction framework with 3D Convolutional Neural Networks (ST3DCNN), which includes three modules <b>1)</b> object trajectory, <b>2)</b> short-term relation prediction, and <b>3)</b> greedy relational association. We conducted appropriate experiments on public datasets and our own dataset (VidPDR). Results demonstrate that our proposed method has a great improvement in comparison to the state-of-the-art baselines.

Concept2Robot: Learning manipulation concepts from instructions and human demonstrations

We aim to endow a robot with the ability to learn manipulation concepts that link natural language instructions to motor skills. Our goal is to learn a single multi-task policy that takes as input a natural language instruction and an image of the initial scene and outputs a robot motion trajectory to achieve the specified task. This policy has to generalize over different instructions and environments. Our insight is that we can approach this problem through learning from demonstration by leveraging large-scale video datasets of humans performing manipulation actions. Thereby, we avoid more time-consuming processes such as teleoperation or kinesthetic teaching. We also avoid having to manually design task-specific rewards. We propose a two-stage learning process where we first learn single-task policies through reinforcement learning. The reward is provided by scoring how well the robot visually appears to perform the task. This score is given by a video-based action classifier trained on a large-scale human activity dataset. In the second stage, we train a multi-task policy through imitation learning to imitate all the single-task policies. In extensive simulation experiments, we show that the multi-task policy learns to perform a large percentage of the 78 different manipulation tasks on which it was trained. The tasks are of greater variety and complexity than previously considered robot manipulation tasks. We show that the policy generalizes over variations of the environment. We also show examples of successful generalization over novel but similar instructions.

A COMPARATIVE STUDY FOR VIDEO CLASSIFICATION TECHNIQUES USING DIRECT FEATURES MATCHING, MACHINE LEARNING, AND DEEP LEARNING

Videos are considered the new era communication language between internet users due to the explosion of smart-phones usage and increase in internet bandwidth and storage space. This has fueled the need to develop robust video analysis techniques. Specifically, video classification presents a unique task for field researchers, as it has numerous critical applications, such as video indexing, searching, annotation and surveillance. Videos inherently embody static and dynamic information that is encoded in frames. The task is further prioritized due to the gigantic amounts of available videos in the digital world, which requires a robust way to organize these videos. Throughout literature, researchers have generally adopted three main techniques to classify videos, i.e., direct features matching, machine learning-based methods, and deep learning-based methods. Each of these methods is suitable for a specific application type. This paper is designed to assess which of the three common working approaches are better for video classification. Furthermore, the paper aims to examine whether and how these methods affect/improve video classification performance and key factors to constructing a robust video classification system. This novel research paper covers an important research gap by introducing a rigorous comparative analysis of the three methods highlighting their advantages and disadvantages and guiding field researchers. A comprehensive analysis brings the paper findings together using a benchmark group of challenging large-scale video datasets (~29k videos). This would provide field researchers with the necessary information to choose the best method for their video classification research work.

Project RISE: Recognizing Industrial Smoke Emissions

Industrial smoke emissions pose a significant concern to human health. Prior works have shown that using Computer Vision (CV) techniques to identify smoke as visual evidence can influence the attitude of regulators and empower citizens to pursue environmental justice. However, existing datasets are not of sufficient quality nor quantity to train the robust CV models needed to support air quality advocacy. We introduce RISE, the first large-scale video dataset for Recognizing Industrial Smoke Emissions. We adopted a citizen science approach to collaborate with local community members to annotate whether a video clip has smoke emissions. Our dataset contains 12,567 clips from 19 distinct views from cameras that monitored three industrial facilities. These daytime clips span 30 days over two years, including all four seasons. We ran experiments using deep neural networks to establish a strong performance baseline and reveal smoke recognition challenges. Our survey study discussed community feedback, and our data analysis displayed opportunities for integrating citizen scientists and crowd workers into the application of Artificial Intelligence for Social Impact.

Classification-enhancement deep hashing for large-scale video retrieval

With the explosive growth of video data on the Internet, retrieving and detecting similar video contents effectively has become a challenging problem. Whereas hashing is a mature technique for dealing with this problem, especially in image retrieval, when hashing techniques are applied to videos, a large gap in performance is always encountered compared with that of image hashing methods because of the complicated structures of videos. Generally, existing video hashing methods just directly apply image hashing approaches into video frames without considering temporal structure, leading to low performance in video retrieval. In this study, we proposed a video hashing method, called classification-enhancement deep hashing (CEDH), for large-scale video searches. The proposed CEDH first fuses the spatial–temporal information of videos in a deep end-to-end hashing network, and then leverages both neighborhood structure of semantics and triple similarity information to learn video hash codes. Subsequently, to enhance the precision of the hash codes during hash learning, a classification module is added after the fully connected layer of the deep network. We also use an additional code constraint to make the hash codes more suitable for containing sufficient information. Extensive experiments on three real-world large-scale video datasets show that our proposed method significantly outperforms state-of-the-art algorithms.

Saliency Prediction in the Deep Learning Era: Successes and Limitations.

Visual saliency models have enjoyed a big leap in performance in recent years, thanks to advances in deep learning and large scale annotated data. Despite enormous effort and huge breakthroughs, however, models still fall short in reaching human-level accuracy. In this work, I explore the landscape of the field emphasizing on new deep saliency models, benchmarks, and datasets. A large number of image and video saliency models are reviewed and compared over two image benchmarks and two large scale video datasets. Further, I identify factors that contribute to the gap between models and humans and discuss the remaining issues that need to be addressed to build the next generation of more powerful saliency models. Some specific questions that are addressed include: in what ways current models fail, how to remedy them, what can be learned from cognitive studies of attention, how explicit saliency judgments relate to fixations, how to conduct fair model comparison, and what are the emerging applications of saliency models.

Metric-Based Attention Feature Learning for Video Action Recognition

Conventional approaches for video action recognition were designed to learn feature maps using 3D convolutional neural networks (CNNs). For better action recognition, they trained the large-scale video datasets with the representation power of 3D CNN. However, action recognition is still a challenging task. Since the previous methods rarely distinguish human body from environment, they often overfit background scenes. Note that separating human body from background allows to learn distinct representations of human action. This paper proposes a novel attention module aiming at only action part(s), while neglecting non-action part(s) such as background. First, the attention module employs triplet loss to differentiate active features from non-active or less active features. Second, two attention modules based on spatial and channel domains are proposed to enhance the feature representation ability for action recognition. The spatial attention module is to learn spatial correlation of features, and the channel attention module is to learn channel correlation. Experimental results show that the proposed method achieves state-of-the-art performance of 41.41% and 55.21% on Diving48 and Something-V1 datasets, respectively. In addition, the proposed method provides competitive performance even on UCF101 and HMDB-51 datasets, i.e., 95.83% on UCF-101 and 74.33% on HMDB-51.