Video Event Research Articles

VID-WIN: Fast Video Event Matching With Query-Aware Windowing at the Edge for the Internet of Multimedia Things

Efficient video processing is a critical component in many IoMT applications to detect events of interest. Presently, many window optimization techniques have been proposed in event processing with an underlying assumption that the incoming stream has a structured data model. Videos are highly complex due to the lack of any underlying structured data model. Video stream sources such as CCTV cameras and smartphones are resource-constrained edge nodes. At the same time, video content extraction is expensive and requires computationally intensive Deep Neural Network (DNN) models that are primarily deployed at high-end (or cloud) nodes. This paper presents VID-WIN, an adaptive 2-stage allied windowing approach to accelerate video event analytics in an edge-cloud paradigm. VID-WIN runs parallelly across edge and cloud nodes and performs the query and resource-aware optimization for state-based complex event matching. VID-WIN exploits the video content and DNN input knobs to accelerate the video inference process across nodes. The paper proposes a novel content-driven micro-batch resizing, queryaware caching and micro-batch based utility filtering strategy of video frames under resource-constrained edge nodes to improve the overall system throughput, latency, and network usage. Extensive evaluations are performed over five real-world datasets. The experimental results show that VID-WIN video event matching achieves ~2.3X higher throughput with minimal latency and ~99% bandwidth reduction compared to other baselines while maintaining query-level accuracy and resource bounds.

VSumVis: Interactive Visual Understanding and Diagnosis of Video Summarization Model

With the rapid development of mobile Internet, the popularity of video capture devices has brought a surge in multimedia video resources. Utilizing machine learning methods combined with well-designed features, we could automatically obtain video summarization to relax video resource consumption and retrieval issues. However, there always exists a gap between the summarization obtained by the model and the ones annotated by users. How to help users understand the difference, provide insights in improving the model, and enhance the trust in the model remains challenging in the current study. To address these challenges, we propose VSumVis under a user-centered design methodology, a visual analysis system with multi-feature examination and multi-level exploration, which could help users explore and analyze video content, as well as the intrinsic relationship that existed in our video summarization model. The system contains multiple coordinated views, i.e., video view, projection view, detail view, and sequential frames view. A multi-level analysis process to integrate video events and frames are presented with clusters and nodes visualization in our system. Temporal patterns concerning the difference between the manual annotation score and the saliency score produced by our model are further investigated and distinguished with sequential frames view. Moreover, we propose a set of rich user interactions that enable an in-depth, multi-faceted analysis of the features in our video summarization model. We conduct case studies and interviews with domain experts to provide anecdotal evidence about the effectiveness of our approach. Quantitative feedback from a user study confirms the usefulness of our visual system for exploring the video summarization model.

Using 3D Convolutional Neural Networks for Real-time Detection of Soccer Events

Developing systems for the automatic detection of events in video is a task which has gained attention in many areas including sports. More specifically, event detection for soccer videos has been studied widely in the literature. However, there are still a number of shortcomings in the state-of-the-art such as high latency, making it challenging to operate at the live edge. In this paper, we present an algorithm to detect events in soccer videos in real time, using 3D convolutional neural networks. We test our algorithm on three different datasets from SoccerNet, the Swedish Allsvenskan, and the Norwegian Eliteserien. Overall, the results show that we can detect events with high recall, low latency, and accurate time estimation. The trade-off is a slightly lower precision compared to the current state-of-the-art, which has higher latency and performs better when a less accurate time estimation can be accepted. In addition to the presented algorithm, we perform an extensive ablation study on how the different parts of the training pipeline affect the final results.

Video event detection, classification and retrieval using ensemble feature selection

Video event detection, classification and retrieval using ensemble feature selection

Prediction and Description of Near-Future Activities in Video

Most of the existing works on human activity analysis focus on recognition or early recognition of the activity labels from complete or partial observations. Similarly, almost all of the existing video captioning approaches focus on the observed events in videos. Predicting the labels and the captions of future activities where no frames of the predicted activities have been observed is a challenging problem, with important applications that require anticipatory response. In this work, we propose a system that can infer the labels and the captions of a sequence of future activities. Our proposed network for label prediction of a future activity sequence has three branches where the first branch takes visual features from the objects present in the scene, the second branch takes observed sequential activity features, and the third branch captures the last observed activity features. The predicted labels and the observed scene context are then mapped to meaningful captions using a sequence-to-sequence learning-based method. Experiments on four challenging activity analysis datasets and a video description dataset demonstrate that our label prediction approach achieves comparable performance with the state-of-the-arts and our captioning framework outperform the state-of-the-arts.

Learning Precise Temporal Point Event Detection with Misaligned Labels

This work addresses the problem of robustly learning precise temporal point event detection despite only having access to poorly aligned labels for training. While standard (cross entropy-based) methods work well in noise-free setting, they often fail when labels are unreliable since they attempt to strictly fit the annotations. A common solution to this drawback is to transform the point prediction problem into a distribution prediction problem. However, we show that this approach raises several issues that negatively affect the robust learning of temporal localization. Thus, in an attempt to overcome these shortcomings, we introduce a simple and versatile training paradigm combining soft localization learning with counting-based sparsity regularization. In fact, unlike its counterparts, our approach allows to directly infer clear-cut point predictions in an end-to-end fashion while relaxing the reliance of the training on the exact position of labels. We achieve state-of-the-art performance against standard benchmarks in a number of challenging experiments (e.g., detection of instantaneous events in videos and music transcription) by simply replacing the original loss function with our novel alternative---without any additional fine-tuning.

Machine Learning based KPI Monitoring of Video Streaming Traffic for QoE Estimation

Quality of Experience (QoE) monitoring of video streaming traffic is crucial task for service providers. Nowadays it is challenging due to the increased usage of end-to-end encryption. In order to overcome this issue, machine learning (ML) approaches for QoE monitoring have gained popularity in the recent years. This work proposes a framework which includes a machine learning pipeline that can be used for detecting key QoE related events such as buffering events and video resolution changes for ongoing YouTube video streaming sessions in real-time. For this purpose, a ML model has been trained using YouTube streaming traffic collected from Android devices. Later on, the trained ML model is deployed in the framework's pipeline to make online predictions. The ML model uses statistical traffic information observed from the network-layer for learning and predicting the video QoE related events. It reaches 88% overall testing accuracy for predicting the video events. Although our work is yet at an early stage, the application of the ML model for online detection and prediction of video events yields quite promising results.

Key frame extraction based on global motion statistics for team-sport videos

Key frame extraction is an important manner of video summarization. It can be used to interpret video content quickly. Existing approaches first partition the entire video into video clips by shot boundary detection, and then, extract key frames by frame clustering. However, in most team-sport videos, a video clip usually includes many events, and it is difficult to extract the key frames related to all of these events accurately, because different events of a game shot can have features of similar appearance. As is well known, most events in team-sport videos are attack and defense conversions, which are related to global translation. Therefore, by using fine-grained partition based on the global motion, a shot could be further partitioned into more video clips, from which more key frames could be extracted and they are related to the events. In this study, global horizontal motion is introduced to further partition video clips into fine-grained video clips. Furthermore, global motion statistics are utilized to extract candidate key frames. Finally, the representative key frames are extracted based on the spatial–temporal consistence and hierarchical clustering, and the redundant frames are removed. A dataset called SportKF is built, which includes 25 videos of 197,878 frames in 112 min and 764 key frames from four types of sports (basketball, football, American football and field hockey). The experimental results demonstrate that the proposed scheme achieves state-of-the-art performance by introducing global motion statistics.

Video Abnormal Event Detection Based on Optical Flow and 3DCNN

With the advancement of technology, the detection of abnormal events in videos has extensive theoretical and practical significance. Most events in videos are normal events, but a small number of abnormal events contain a lot of information. How to identify individual abnormal events from massive videos is the focus of concern. The paper made full use of the spatio-temporal continuity of video images. Firstly, a space-time cube was constructed by integrating optical flow information; secondly, a multi-layer three-dimensional convolutional neural network model was constructed; finally, the entire process of video abnormal event detection was realized based on this model. The experiment on the UCSD dataset showed that the method in this paper can effectively detect abnormal events in videos.

WITHDRAWN: Video event classification using KNN classifier with hybrid features

Event detection from videos has become a demanding and challenging subject with the creation of huge video data. Detection of complex interesting events automatically in user generated videos is a difficult job because of its features that differ from video broadcasting. We present an effective technique for detecting actions or events in UCF 101 video large scale data. Histogram equalization is initially used on UCF101 database for contrast enhancement in the pre processing stage. In the later stage gradient local ternary pattern, histogram of oriented gradient and tamuras features are extracted from the enhanced images to accomplish feature extraction. Finally k nearest neighbor (KNN) classifier is used to classify the events. In this work we compared Random Forest classifier with KNN classifier in terms of accuracy, specificity, sensitivity, false discovery rate, false omission rate and error rate. KNN gave 90.25% accuracy where as Random forest classifier yielded 77.18% accuracy. The result exhibits that KNN classifier gives better performance than RF classifier.

Accelerated masked transformer for dense video captioning

Dense video captioning aims to generate dense descriptions for all possible events in an untrimmed video. The task is challenging that it requires accurately localizing events in the video and simultaneously describe each event with a sentence. Current approaches usually decompose this task into two independent stages—the proposal localization stage and the caption generation stage, resulting in a suboptimal solution. Masked Transformer (MT) model [30] has been proposed to integrate the two stages and optimize them in an end-to-end philosophy. Despite the superior performance that the MT has achieved, its runtime efficiency is unsatisfactory which severely limits its applicability in real-world scenarios. In this paper, we devise an improved Accelerated Masked Transformer (AMT) model that enjoys the dual-benefit of effectiveness and efficiency. Taking MT as our reference model, we respectively introduce accelerating strategies to the two stages: 1) in the proposal localization stage, we introduce a lightweight anchor-free proposal in company with a local attention mechanism; and 2) in the caption generation stage, we introduce the single-shot feature masking strategy along with an average attention mechanism. Extensive experiments on two benchmark datasets ActivityNet-Caption and YouCookII demonstrate that AMT achieves competitive performance on both datasets with significant speed improvement. On the ActivityNet-Caption dataset, AMT reduces up to 2× running time with comparable performance when compared to the reference MT model.

Abnormal Behavior Detection and Warning Based on Deep Intelligent Video Analysis for Geriatric Patients

With the rapid growth of the aging population, abnormal detection has become a key issue in the medical and health field. Accurately detecting abnormal events in the monitoring video and giving feedback in real time can effectively reduce injuries even deaths caused by falls in the elderly. A novel Deepgait-based event detection method is proposed in this study for the elderly monitoring application. The classes of this problem contains abnormal behaviors, unusual events and normal daily life activities. To solve the above recognition problem, a novel deep convolutional features combined with joint Bayesian is proposed to model view variance. To evaluate the performance of the proposed model, two open behavioral data sets: Multiple cameras fall dataset and UR Fall Detection Dataset are adopted, where the mAP value of the improved Deepgait algorithm is 2.23% higher than that of Deepgait algorithm, and 5.4% higher than that of energy model (SSD) approach. The results suggest that our proposed monitoring system is an effective method for the elderly monitoring application.

A Computer Vision Platform to Automatically Locate Critical Events in Surgical Videos: Documenting Safety in Laparoscopic Cholecystectomy.

The aim of this study was to develop a computer vision platform to automatically locate critical events in surgical videos and provide short video clips documenting the critical view of safety (CVS) in laparoscopic cholecystectomy (LC). Intraoperative events are typically documented through operator-dictated reports that do not always translate the operative reality. Surgical videos provide complete information on surgical procedures, but the burden associated with storing and manually analyzing full-length videos has so far limited their effective use. A computer vision platform named EndoDigest was developed and used to analyze LC videos. The mean absolute error (MAE) of the platform in automatically locating the manually annotated time of the cystic duct division in full-length videos was assessed. The relevance of the automatically extracted short video clips was evaluated by calculating the percentage of video clips in which the CVS was assessable by surgeons. A total of 155 LC videos were analyzed: 55 of these videos were used to develop EndoDigest, whereas the remaining 100 were used to test it. The time of the cystic duct division was automatically located with a MAE of 62.8 ± 130.4 seconds (1.95% of full-length video duration). CVS was assessable in 91% of the 2.5 minutes long video clips automatically extracted from the considered test procedures. Deep learning models for workflow analysis can be used to reliably locate critical events in surgical videos and document CVS in LC. Further studies are needed to assess the clinical impact of surgical data science solutions for safer laparoscopic cholecystectomy.

Automatic Summarization of Cricket Highlights using Audio Processing

The problem of automatic excitement detection in cricket videos is considered and applied for highlight generation. This paper focuses on detecting exciting events in video using complementary information from the audio and video domains. First, a method of audio and video elements separation is proposed. Thereafter, the “level-of-excitement” is measured using features such as amplitude, and spectral center of gravity extracted from the commentators speech’s amplitude to decide the threshold. Our experiments using actual cricket videos show that these features are well correlated with human assessment of excitability. Finally, audio/video information is fused according to time-order scenes which has “excitability” in order to generate highlights of cricket. The techniques described in this paper are generic and applicable to a variety of topic and video/acoustic domains.

Text query based summarized event searching interface system using deep learning over cloud

In the digital era, the growth of multimedia data is increasing at a rapid pace, which demands both effective and efficient summarization techniques. Such advanced techniques are required so that the users can quickly access the video content, recorded by multiple cameras for a certain period. At present, it is very challenging to manage and search a huge amount of multiview video data, which contains the inter-views dependencies, significant illumination changes, and many low-active frames. This work highlights an efficient summarization technique to summarize and then search the events in such multi-view videos over cloud through text query. Deep learning framework is employed to extract the features of moving objects in the frames. The inter-views dependencies among multiple views of the video are captured via local alignment. Parallel Virtual Machines (VMs) in the Cloud environment have been used to process the multiple video clip independently at a time. Object tracking is applied to filter the low-active frames. Experimental Results indicate that the model successfully reduces the video content, while preserving the momentous information in the form of the events. A computing analysis also indicates that it meets the requirement of real-time applications.

Cricket Video Events Recognition using HOG, LBP and Multi-class SVM

The world has witnessed a growth in multimedia data, especially video data over the past few years due to increased internet bandwidth and higher processing power of computers. Having a large number of video data also require techniques to store, summarize, index and information retrieval. More attention has been given in recent years to develop techniques which summarize, index and retrieve sports videos due to its commercial aspects. This paper proposes a framework which classifies a cricket video into one of the four events namely Bowled Out, Caught Behind, Catch Out and LBW Out. The framework uses training videos from each category of event and summarizes the videos into key frames. HOG and LBP features are computed for key frames and fused to form a single feature vector which will be labeled accordingly which represents a single event in video. Feature vector is given to a Multi-Class SVM which classifies the video into one of the four events. The experimental results show that the Precision of our technique is 77.23%, Recall is 77.86%, F-Measure is 77.55% and the Accuracy is 65.62%. The evaluation metrics of our technique are promising, because in the literature there is no other technique present so far for event detection & classification in cricket videos.

Designing devices for global surgery: evaluation of participatory and frugal design methods

Introduction: Most people living in low- and middle-income countries have no access to surgical care. Equipping under-resourced health care contexts with appropriate surgical equipment is thus critical. “Global” technologies must be designed specifically for these contexts. But while models, approaches and methods have been developed for the design of equipment for global surgery, few studies describe their implementation or evaluate their adequacy for this purpose. Methods: A multidisciplinary team applied participatory and frugal design methods to design a surgical device for gasless laparoscopy. The team employed a formal roadmap, devised to guide the development of global surgical equipment, to structure the design process into phases. Phases 0–1 comprised primary research with surgeons working in low-resource settings and forming collaborative partnerships with key stakeholders. These participated in phases 2–3 through design workshops and video events. To conclude, surgical stakeholders (n=13) evaluated a high-fidelity prototype in a cadaveric study. Results: The resulting design, “RAIS” (Retractor for Abdominal Insufflation-less Surgery), received positive feedback from rural surgeons keen to embrace and champion innovation as a result of the close collaboration and participatory design methods employed. The roadmap provided a valuable means to structure the design process but this evaluation highlighted the need for further development to detail specific methodology. The project outcomes were used to develop recommendations for innovators designing global surgical equipment. To inform early phases in the design roadmap, engaging a variety of stakeholders to provide regular input is crucial. Effective communication is vital to elucidate clear functional design requirements and hence reveal opportunities for frugal innovation. Finally, responsible innovation must be embedded within the process of designing devices for global surgery. Conclusion: A community-wide effort is required to formally evaluate and optimize processes for designing global surgical devices and hence accelerate adoption of frugal surgical technologies in low-resource settings.

Development of Basketball Tactics Basic Cooperation Teaching System Based on CNN and BP Neural Network.

The traditional basketball teaching mode cannot meet the needs of students for the basic cooperation of basketball tactics. Therefore, a basic cooperation teaching system of basketball tactics based on artificial neural network is studied and designed. The system has a professional basketball game video tactical learning module. The events in the basketball game video are classified through a convolutional neural network and combined with the explanation of teachers to make the students have an intuitive understanding of the basic cooperation of basketball tactics and then design the basketball game module based on a BP neural network to provide students with an online basketball tactics training platform. Finally, the teacher scores the performance of the actual on-site training students in the basic cooperation of basketball tactics through the tactical scoring module on the system. The results show that after the introduction of global and collective motion patterns, the classification accuracy of the convolutional neural network is improved by 22.48%, which has significant optimization. The average accuracy of basketball game video event classification is 62.35%, and the accuracy of snatch event classification is improved to 95.28%. The recognition rate of the BP neural network combined with momentum gradient descent method is 75%, the number of weight adjustment is less, and the memory is small while ensuring fast running speed. Students who accept the basic basketball tactics cooperation teaching system based on the artificial neural network for basketball teaching have an overall score of 27.99 ± 2.11 points The overall score of exchange defense cooperation was 24.12 ± 2.03, which was higher than that of the control group. The above results show that the basketball tactical basic cooperation teaching system based on the artificial neural network has a good teaching effect in improving students' basketball tactical basic cooperation ability.

Machine learning for video event recognition

In recent years, the spread of video sensor networks both in public and private areas has grown considerably. Smart algorithms for video semantic content understanding are increasingly developed to support human operators in monitoring different activities, by recognizing events that occur in the observed scene. With the term event, we refer to one or more actions performed by one or more subjects (e.g., people or vehicles) acting within the same observed area. When these actions are performed by subjects that do not interact with each other, the events are usually classified as simple. Instead, when any kind of interaction occurs among subjects, the involved events are typically classified as complex. This survey starts by providing the formal definitions of both scene and event, and the logical architecture for a generic event recognition system. Subsequently, it presents two taxonomies based on features and machine learning algorithms, respectively, which are used to describe the different approaches for the recognition of events within a video sequence. This paper also discusses key works of the current state-of-the-art of event recognition, providing the list of datasets used to evaluate the performance of reported methods for video content understanding.

A Minimal Setup for Spontaneous Smile Quantification Applicable for Valence Detection.

Tracking emotional responses as they unfold has been one of the hallmarks of applied neuroscience and related disciplines, but recent studies suggest that automatic tracking of facial expressions have low validation. In this study, we focused on the direct measurement of facial muscles involved in expressions such as smiling. We used single-channel surface electromyography (sEMG) to evaluate the muscular activity from the Zygomaticus Major face muscle while participants watched music videos. Participants were then tasked with rating each video with regard to their thoughts and responses to each of them, including their judgment of emotional tone (“Valence”), personal preference (“Liking”) and rating of whether the video displayed strength and impression (“Dominance”). Using a minimal recording setup, we employed three ways to characterize muscular activity associated with spontaneous smiles. The total time spent smiling (ZygoNum), the average duration of smiles (ZygoLen), and instances of high valence (ZygoTrace). Our results demonstrate that Valence was the emotional dimension that was most related to the Zygomaticus activity. Here, the ZygoNum had higher discriminatory power than ZygoLen for Valence quantification. An additional investigation using fractal properties of sEMG time series confirmed previous studies of the Facial Action Coding System (FACS) documenting a smoother contraction of facial muscles for enjoyment smiles. Further analysis using ZygoTrace responses over time to the video events discerned “high valence” stimuli with a 76% accuracy. Additional validation of this approach came against previous findings on valence detection using features derived from a single channel EEG setup. We discuss these results in light of both the recent replication problems of facial expression measures, and in relation to the need for methods to reliably assess emotional responses in more challenging conditions, such as Virtual Reality, in which facial expressions are often covered by the equipment used.