Event Recognition Research Articles

Dual-branch Cross-scale Feature Interaction for Temporal Action Detection

Temporal Action Detection (TAD) is aim to predict action boundary and category simultaneously. Most existing RGB-based methods model temporal dependency using pyramid-style features without interaction among different scales, which usually result in inaccurate prediction for long-term actions. The reason is that features at different scales involve information with different granularity, which is suitable for either prediction of action boundary or category. In this paper, we present a novel Dual-branch Cross-scale Feature Interaction (DCFI) method that directly exchanges different scale information from both temporal and spatial perspective for TAD. To be specific, in one branch, a cross-scale temporal transformer module is devised to enable both semantic and temporal communications among different scale features with a merge-to-split mechanism. While the other branch designs a cross-scale spatial mixer module to mine the most salient spatial difference between consecutive and long-term frames via a scale-mixer. Benefiting from these two modules, DCFI achieves comprehensive temporal as well as spatial interaction across all feature scales, and thus accurately predicts the boundaries of different time-span action instances. Extensive experiments on two challenging benchmarks, i.e.., THUMOS-14 and ActivityNet-1.3, demonstrate that our DCFI achieves new state-of-the-art performance with only RGB.

Discriminative action tubelet detector for weakly-supervised action detection

We propose a novel framework for spatiotemporal action detection using only video-level class labels as weak supervision. Traditional fully-supervised approaches rely on labor-intensive manual annotation of bounding boxes for each frame. In contrast, collecting video-level class labels is significantly less tedious and more feasible compared to annotating frame-level sequences with bounding boxes. To address this challenge, we propose a discriminative action tubelet detector, called DAT-detector, designed to discern discriminative tubelets from action tubelet proposals (ATPs). Whereas the previous approaches have only focused on tubelet selection among the predefined object proposals, our DAT-detector prioritizes the generation of more precise action tubelets using regression and attention modules. Moreover, we introduce an ATP generation method that enhances the quality of tubelet proposals. Our approach achieves state-of-the-art performance on several benchmarks, and also demonstrates competitive performance even with fully-supervised approaches.

An improved mobile reinforcement learning for wrong actions detection in aerobics training videos

An improved mobile reinforcement learning for wrong actions detection in aerobics training videos

Audio-based event detection in the operating room.

Even though workflow analysis in the operating room has come a long way, current systems are still limited to research. In the quest for a robust, universal setup, hardly any attention has been given to the dimension of audio despite its numerous advantages, such as low costs, location, and sight independence, or little required processing power. We present an approach for audio-based event detection that solely relies on two microphones capturing the sound in the operating room. Therefore, a new data set was created with over 63h of audio recorded and annotated at the University Hospital rechts der Isar. Sound files were labeled, preprocessed, augmented, and subsequently converted to log-mel-spectrograms that served as a visual input for an event classification using pretrained convolutional neural networks. Comparing multiple architectures, we were able to show that even lightweight models, such as MobileNet, can already provide promising results. Data augmentation additionally improved the classification of 11 defined classes, including inter alia different types of coagulation, operating table movements as well as an idle class. With the newly created audio data set, an overall accuracy of 90%, a precision of 91% and a F1-score of 91% were achieved, demonstrating the feasibility of an audio-based event recognition in the operating room. With this first proof of concept, we demonstrated that audio events can serve as a meaningful source of information that goes beyond spoken language and can easily be integrated into future workflow recognition pipelines using computational inexpensive architectures.

Sports Action Detection and Counting Algorithm Based on Pose Estimation and Its Application in Physical Education Teaching

Sports Action Detection and Counting Algorithm Based on Pose Estimation and Its Application in Physical Education Teaching

Unsupervised exceptional human action detection from repetition of human assembling tasks using entropy signal clustering

Unsupervised exceptional human action detection from repetition of human assembling tasks using entropy signal clustering

Development of IOT-based Smart Anti – Theft Latch

In our day to day lives, security is of utmost concern. Everyone wants to be as secured as possible. This paper develops an Internet of Things (IoT)-based smart security system and supports the concept of a "IOT close to life and easy to use”. One of the many useful applications of IoT is home security; among other things, we're leveraging it to build affordable home security systems. The IoT is a network of physically connected objects that can exchange data and communicate with one another without the need for human intervention. The simplicity of setup, low cost, and minimal maintenance are some of this system's key benefits. This system is executed with the help of Raspberry Pi 3B. Face recognition system used determines if the individual is lively enough to receive authorization. If an unauthorized access is detected then the buzzer will be triggered and an alert notification will be sent on an app which is designed for the owner which will also include a photo detected by the camera and with an option to reset the buzzer. This system comprises of dual authentication that is Radio Frequency Identification (RFID) key and a face recognition system. This approach is able to protect the homes from intrusions by continuous detection of the suspicious actions using soft computing, that is, Convolutional Neural Network (CNN) algorithm for face recognition.

Research and Implementation of Robotic Arm Positioning and Grasping Based on Visual Guidance

Abstract This paper proposes a robotic arm vision guidance system based on the ROS platform, which combines the Hikrobot Robotics 2D vision sensor with the Xinsong GCR5 robotic arm to successfully realise vision-guided robotic arm positioning and grasping. Camera calibration, hand-eye calibration, and deep learning techniques are used to establish an accurate transformation relationship between the base coordinate system of the robotic arm and the camera coordinate system to achieve efficient target goal recognition and localisation. Adopting the MoveIt framework and RRT-connect algorithm, the robotic arm can flexibly plan the movement path and successfully complete the target grasping, which provides the theoretical basis and practical experience for robotic arm intelligent grasping.

POTLoc: Pseudo-label Oriented Transformer for point-supervised temporal Action Localization

This paper tackles the challenge of point-supervised temporal action detection, wherein only a single frame is annotated for each action instance in the training set. Most of the current methods, hindered by the sparse nature of annotated points, struggle to effectively represent the continuous structure of actions or the inherent temporal and semantic dependencies within action instances. Consequently, these methods frequently learn merely the most distinctive segments of actions, leading to the creation of incomplete action proposals. This paper proposes POTLoc, a Pseudo-label Oriented Transformer for weakly-supervised Action Localization utilizing only point-level annotation. POTLoc is designed to identify and track continuous action structures via a self-training strategy. The base model begins by generating action proposals solely with point-level supervision. These proposals undergo refinement and regression to enhance the precision of the estimated action boundaries, which subsequently results in the production of ‘pseudo-labels’ to serve as supplementary supervisory signals. The architecture of the model integrates a transformer with a temporal feature pyramid to capture video snippet dependencies and model actions of varying duration. The pseudo-labels, providing information about the coarse locations and boundaries of actions, assist in guiding the transformer for enhanced learning of action dynamics. POTLoc outperforms the state-of-the-art point-supervised methods on THUMOS’14 and ActivityNet-v1.2 datasets.

Self-regulation and conflict goals management capabilities of ecosystem entrepreneurs: a case study of Haier ecosystem.

The inherent dual roles of "follower" and "leader" among ecosystem entrepreneurs inevitably introduce challenges in managing conflicting dependent and independent goals. Ecosystem entrepreneurs' capabilities in conflict goals management directly influence new venture survival and development. This single-case qualitative study explores how ecosystem entrepreneurs develop conflict goals management capabilities through self-regulation, which is not only a unique practical challenge in ecosystem entrepreneurship, but also a cutting-edge topic in current theoretical research. Through research of entrepreneurs in Haier Entrepreneurship Ecosystem, the paper finds: (1) strategic corresponding and mechanism adapting emerge as the two trigger factors enabling ecosystem entrepreneurs to recognize the equilibrium or disequilibrium between conflicting goals; (2) by leveraging self-control, grit, and metacognition, ecosystem entrepreneurs construct decoupling mechanisms for antagonistic goal recognition and coupling mechanisms for synergistic goal recognition; (3) ecosystem entrepreneurs enhance their conflict goals management capabilities by developing both segregative and synergistic management capabilities. Furthermore, this research explores the self-regulation process underlying ecosystem entrepreneurs' conflict goals management behaviors, including environmental interaction perception, conflict goals analysis, and delineation of goal relationships. Findings provide insights for ecosystem entrepreneurs on improving their conflict goals management capabilities through self-assessment and skill development.

“Counting sheep PSG”: EEGLAB-compatible open-source matlab software for signal processing, visualization, event marking and staging of polysomnographic data

BackgroundProgress in advancing sleep research employing polysomnography (PSG) has been negatively impacted by the limited availability of widely available, open-source sleep-specific analysis tools. New methodHere, we introduce Counting Sheep PSG, an EEGLAB-compatible software for signal processing, visualization, event marking and manual sleep stage scoring of PSG data for MATLAB. ResultsKey features include: (1) signal processing tools including bad channel interpolation, down-sampling, re-referencing, filtering, independent component analysis, artifact subspace reconstruction, and power spectral analysis, (2) customizable display of polysomnographic data and hypnogram, (3) event marking mode including manual sleep stage scoring, (4) automatic event detections including movement artifact, sleep spindles, slow waves and eye movements, and (5) export of main descriptive sleep architecture statistics, event statistics and publication-ready hypnogram. Comparison with existing methodsCounting Sheep PSG was built on the foundation created by sleepSMG (https://sleepsmg.sourceforge.net/). The scope and functionalities of the current software have made significant advancements in terms of EEGLAB integration/compatibility, preprocessing, artifact correction, event detection, functionality and ease of use. By comparison, commercial software can be costly and utilize proprietary data formats and algorithms, thereby restricting the ability to distribute and share data and analysis results. ConclusionsThe field of sleep research remains shackled by an industry that resists standardization, prevents interoperability, builds-in planned obsolescence, maintains proprietary black-box data formats and analysis approaches. This presents a major challenge for the field of sleep research. The need for free, open-source software that can read open-format data is essential for scientific advancement to be made in the field.

Exploiting relation of video segments for temporal action detection

With the development of urban intelligence, intelligent identification of human activities has become imminent. Temporal Action Detection (TAD) is designed to identify real human social activities, which is a challenging task in video understanding. The current methods mainly use global features for boundary matching or predefine all possible proposals, while ignoring the interference of background and the causal relevance of the temporal action, resulting in the generation of more redundant proposals and the decline of detection accuracy. To fill this gap, we propose a novel Dilated Convolution Locate and Action Relevant Score model called DCAR. Specifically, DCAR includes a Dilated Location Network (DL-Net) and Action Relevance Calculation (ARC) block. For the DL-Net, we design a Boundary Feature Enhancement (BFE) block, which enhances the boundary feature of actions and fuses the similar features of different channels by pooling and channel squeeze to reduce the interference of the background. Meanwhile, we also design multiple dilated convolutional structures to aggregate long contextual information in time point/interval after boundary enhancement. For the ARC block, we use the hyperbolic space distance and cross attention to calculate the causal correlation of action proposals classification, which removes the misclassification of action proposals. We conduct extensive experiments on Thumos14 and ActivityNet-1.3, which shows our method significantly improves the performance and achieves state-of-the-art results. On Thumos14, at tIoU 0.3, 0.4 and 0.5, it achieves mAP of 63.2%, 57.0% and 48.5%, respectively; On ActivityNet-1.3 it reaches 9.42% at tIoU 0.95.

Robotics Perception: Intention Recognition to Determine the Handball Occurrence during a Football or Soccer Match

In football or soccer, a referee controls the game based on the set rules. The decisions made by the referee are final and can’t be appealed. Some of the decisions, especially after a handball event, whether to award a penalty kick or a yellow/red card can greatly affect the final results of a game. It is therefore necessary that the referee does not make an error. The objective is therefore to create a system that can accurately recognize such events and make the correct decision. This study chose handball, an event that occurs in a football game (Not to be confused with the game of Handball). We define a handball event using object detection and robotic perception and decide whether it is intentional or not. Intention recognition is a robotic perception of emotion recognition. To define handball, we trained a model to detect the hand and ball which are primary objects. We then determined the intention using gaze recognition and finally combined the results to recognize a handball event. On our dataset, the results of the hand and the ball object detection were 96% and 100% respectively. With the gaze recognition at 100%, if all objects were recognized, then the intention and handball event recognition were at 100%.

Uncertainty model-based edge detection technology in badminton

As virtual reality technology develops, the analysis and processing of video content have become hot spots in the field of computer vision. Video Action Detection aims to locate features in network video, and its research spans many fields, such as computer vision and spatial prediction. In view of the problem of low-efficiency classification models and inaccurate localization of small-scale targets in complex scenes, we propose a novel method to generate candidate intervals for action detection. The action recognition model is adopted to generate the action score sequence on the video time series. We also propose the uncertainty model of the descending pose detection algorithm. The pre-reaction phase generates a candidate list in the form of concatenated videos containing exactly the same pose to detect action poses that are not identical and of non-maximum duration. Experiments with traditional target detection and multiple deep learning models show that the proposed Non-Maximum Suppression algorithm has a strong ability to extract neural network features. Furthermore, compared with traditional ATSS and Faster R-CNN methods, the detection quality and performance are improved by more than 15.2% and 7.8%, respectively. Our method can fully utilize perception information to improve the quality of decision planning and plays a connecting role between perception fusion and decision planning.

LGAFormer: transformer with local and global attention for action detection

LGAFormer: transformer with local and global attention for action detection

Heterogeneous Feature Fusion for Improving Performance of Action Detection

We present a novel framework aimed at improving video action detection through the integration of heterogeneous features. Conventional action detection methods which focus on modeling the relationships between person/object instances rely exclusively on video features and do not exploit valuable intra-instance heterogeneous features, such as person pose, positional information or object category, that can support action recognition. Our proposed framework, termed Heterogeneous Feature Fusion (HFF) framework, addresses this limitation by integrating such intra-instance heterogeneous features for person/object instances, and can improve existing action detection methods. To efficiently exploit each heterogeneous feature, which vary in importance depending on actions and/or scenes, we introduce an attention mechanism to dynamically enhance important heterogeneous features within an instance. Experiments on JHMDB and AVA v2.2 datasets show that our HFF significantly enhances the action detection performance of two existing methods.

A Deep Learning Framework for Start-End Frame Pair-Driven Motion Synthesis.

A start-end frame pair and a motion pattern-based motion synthesis scheme can provide more control to the synthesis process and produce content-various motion sequences. However, the data preparation for the motion training is intractable, and concatenating feature spaces of the start-end frame pair and the motion pattern lacks theoretical rationality in previous works. In this article, we propose a deep learning framework that completes automatic data preparation and learns the nonlinear mapping from start-end frame pairs to motion patterns. The proposed model consists of three modules: action detection, motion extraction, and motion synthesis networks. The action detection network extends the deep subspace learning framework to a supervised version, i.e., uses the local self-expression (LSE) of the motion data to supervise feature learning and complement the classification error. A long short-term memory (LSTM)-based network is used to efficiently extract the motion patterns to address the speed deficiency reflected in the previous optimization-based method. A motion synthesis network consists of a group of LSTM-based blocks, where each of them is to learn the nonlinear relation between the start-end frame pairs and the motion patterns of a certain joint. The superior performances in action detection accuracy, motion pattern extraction efficiency, and motion synthesis quality show the effectiveness of each module in the proposed framework.

DITA: DETR with improved queries for end-to-end temporal action detection

The DEtection TRansformer (DETR), with its elegant architecture and set prediction, has revolutionized object detection. However, DETR-like models have yet to achieve comparable success in temporal action detection (TAD). To address this gap, we introduce a series of improvements to the original DETR, proposing a new DETR-based model for TAD that achieves competitive performance relative to conventional TAD methods. Specifically, we adapt advanced techniques from DETR variants used in object detection, including deformable attention, denoising training, and selective query recollection. Furthermore, we propose several new techniques aimed at enhancing detection precision and model convergence speed, such as geographic query grouping and learnable proposals. Leveraging these innovations, we introduce a new model called DETR with Improved queries for Temporal Action Detection (DITA). DITA not only adheres to DETR’s elegant design philosophy but is also competitive to state-of-the-art action detection models. Remarkably, it is the first TAD model to achieve an mAP over 70% on THUMOS14, outperforming the previous best DETR variant by 13.5 percentage points.

Complex event recognition and anomaly detection with event behavior model

Complex event recognition and anomaly detection with event behavior model

Radiopharmaceuticals Adverse Events Management.

Radiopharmaceuticals are radioactive compounds used for diagnostic or therapeutic purposes which are most often administered intravenously. Adverse events that may induce both adverse reactions and drug-to-drug interactions with changes in expected biodistribution, potentially affecting patient safety and diagnostic accuracy. Adverse reactions are relatively rare due to the small doses and under-reporting is the norm. The aim of this study is to increase awareness of the need to report in order to create protocols for the management of such adverse events among professionals in a Nuclear Medicine Department. A reporting system was established a decade ago through an electronic form to enhance adverse event registration. The radiopharmacist collects data for further communication with National Health authorities and develops an annual report with recommendations on the management of these adverse events. A total of 128 reports were collected, including 65 cases of extravasations, 18 adverse reactions, and 45 drug interactions. Over the years, reporting has been increasing, adverse reactions occurred at a higher incidence than reported in the literature, and each anomalous biodistribution was analysed for possible drug interaction. The annual reports have been used to develop a local guideline for the management of adverse reactions and recommendations for discontinuation of treatment to avoid interactions with radiopharmaceuticals. The recognition of adverse events associated with radiopharmaceuticals is increasing, underlining the need for vigilant reporting and improved management strategies. An efficient reporting system promotes awareness of possible interactions between radiopharmaceuticals and other medicines and their potential adverse reactions to enhance patient safety.