Video Segmentation Research Articles

Harnessing Deep Learning for Video Based Weapon Detection

This research addresses the escalating global issue of handgun-related crimes by proposing an innovative Intelligent Video Surveillance System (IVSS) that leverages advanced deep learning (DL) techniques for remote firearm detection and timely threat response. The system employs Convolutional Neural Networks (CNN) and the YOLO v3 model, uniquely integrating Transfer Learning (TL) to enhance adaptability and efficacy. Experimental validation using the Internet Movie Firearms Database (IMFDB) demonstrates the system's versatility in detecting various pistols and guns, achieving promising results that surpass existing systems in accuracy and efficiency. Challenges in real-time weapon recognition, such as the absence of a standardized weapon dataset, occlusion, and small object sizes, are acknowledged. Emphasis is placed on the critical need for reliable data acquisition, precise labeling, and preprocessing tailored to different detection algorithms. The implementation encompasses video collection, preprocessing, model loading, algorithm application, segmentation, and classification, alongside a user-friendly webcam interface for real-time detection. Additionally, the system integrates the pyttsx3 library for voice alerts and the Twilio API for voice call alerts to enhance responsiveness. In summary, this study presents a novel CNN-based model combining Transfer Learning with YOLO v3, achieving superior weapon identification and distinguishing between real and fake firearms, representing a significant advancement in intelligent video surveillance and contributing to the reduction of weapon violence.

AI-ENHANCED TRACKSEGNET AN ADVANCED MACHINE LEARNING TECHNIQUE FOR VIDEO SEGMENTATION AND OBJECT TRACKING

Video segmentation and object tracking are critical tasks in computer vision with applications spanning surveillance, autonomous driving, and interactive media. Traditional methods often struggle with the dynamic nature of video data, where object occlusions, variations in illumination, and complex motion patterns present significant challenges. Existing segmentation and tracking systems frequently suffer from inaccuracies in handling real-time video sequences, particularly in distinguishing and tracking multiple overlapping objects. The limitations of current models in addressing these issues necessitate the development of more advanced techniques that can effectively manage dynamic scenes and improve tracking accuracy. To address these challenges, we propose an advanced machine learning technique, AI-Enhanced TrackSegNet, which integrates deep learning with novel attention mechanisms for improved video segmentation and object tracking. Our method utilizes a combination of Convolutional Neural Networks (CNNs) for feature extraction and Long Short-Term Memory (LSTM) networks for temporal sequence modeling. We introduce an attention-based mechanism to dynamically focus on relevant features, enhancing the model's ability to handle occlusions and varying object appearances. The model was trained on a diverse dataset of video sequences, incorporating both synthetic and real-world footage. The AI-Enhanced TrackSegNet demonstrated significant improvements in performance compared to existing techniques. Our method achieved an average Intersection over Union (IoU) score of 86.7% for segmentation and a tracking precision rate of 91.3% on the MOT17 benchmark dataset. These results represent a 10.2% improvement in IoU and a 7.5% increase in tracking precision compared to state-of-the-art methods. The model also exhibited enhanced robustness in complex scenes, handling occlusions and motion variations with greater accuracy.

Subtle signals: Video-based detection of infant non-nutritive sucking as a neurodevelopmental cue

Non-nutritive sucking (NNS), which refers to the act of sucking on a pacifier, finger, or similar object without nutrient intake, plays a crucial role in assessing healthy early development. In the case of preterm infants, NNS behavior is a key component in determining their readiness for feeding. In older infants, the characteristics of NNS behavior offer valuable insights into neural and motor development. Additionally, NNS activity has been proposed as a potential safeguard against sudden infant death syndrome (SIDS). However, the clinical application of NNS assessment is currently hindered by labor-intensive and subjective finger-in-mouth evaluations. Consequently, researchers often resort to expensive pressure transducers for objective NNS signal measurement. To enhance the accessibility and reliability of NNS signal monitoring for both clinicians and researchers, we introduce a vision-based algorithm designed for non-contact detection of NNS activity using baby monitor footage in natural settings. Our approach involves a comprehensive exploration of optical flow and temporal convolutional networks, enabling the detection and amplification of subtle infant-sucking signals. We successfully classify short video clips of uniform length into NNS and non-NNS periods. Furthermore, we investigate manual and learning-based techniques to piece together local classification results, facilitating the segmentation of longer mixed-activity videos into NNS and non-NNS segments of varying duration. Our research introduces two novel datasets of annotated infant videos, including one sourced from our clinical study featuring 18 infant subjects and 183 h of overnight baby monitor footage. Additionally, we incorporate a second, shorter dataset obtained from publicly available YouTube videos. Our NNS action recognition algorithm achieves an impressive 95.8% accuracy in binary classification, based on 960 2.5-s balanced NNS versus non-NNS clips from our clinical dataset. We also present results for a subset of clips featuring challenging video conditions. Moreover, our NNS action segmentation algorithm achieves an average precision of 93.5% and an average recall of 92.9% across 30 heterogeneous 60-s clips from our clinical dataset.

Enhancing left ventricular segmentation in echocardiography with a modified mixed attention mechanism in SegFormer architecture

Echocardiography is a key tool for the diagnosis of cardiac diseases, and accurate left ventricular (LV) segmentation in echocardiographic videos is crucial for the assessment of cardiac function. However, since semantic segmentation of video needs to take into account the temporal correlation between frames, this makes the task very challenging. This article introduces an innovative method that incorporates a modified mixed attention mechanism into the SegFormer architecture, enabling it to effectively grasp the temporal correlation present in video data. The proposed method processes each time series by encoding the image input into the encoder to obtain the current time feature map. This map, along with the historical time feature map, is then fed into a time-sensitive mixed attention mechanism type of convolution block attention module (TCBAM). Its output can serve as the historical time feature map for the subsequent sequence, and a combination of the current time feature map and historical time feature map for the current sequence. The processed feature map is then input into the Multilayer Perceptron (MLP) and subsequent networks to generate the final segmented image. Through extensive experiments conducted on two different datasets: Hamad Medical Corporation, Tampere University, and Qatar University (HMC-QU), Cardiac Acquisitions for Multi-structure Ultrasound Segmentation (CAMUS) and Sunnybrook Cardiac Data (SCD), achieving a Dice coefficient of 97.92 % on the SCD dataset and an F1 score of 0.9263 on the CAMUS dataset, outperforming all other models. This research provides a promising solution to the temporal modeling challenge in video semantic segmentation tasks using transformer-based models and points out a promising direction for future research in this field.

RegionFilter: Region-aware video filtering mechanism on resource-constrained edge nodes

The video analytics pipeline consumes substantial network and computing resources, and in many scenarios videos are sent from cameras to servers. Due to limited camera-side resources, most existing systems are frame-level filtering approaches that filter out frames that are unchanged or irrelevant to applications before transmitting data. However, video applications usually care about the regions of interest (RoIs) of frames rather than background regions, and frame-level filtering fails to consider the spatial redundancy of intra-frame regions. We argue that video analytics should adopt a region-level filtering approach. Unfortunately, how to perform region-aware filtering on cameras without GPU is a significant challenge. To overcome this, we built RegionFilter, which achieves region awareness on resource-constrained edge nodes through feedback results from server-side DNN models. RegionFilter splits each video segment into multiple sub-segments in spatial distribution through region awareness. Then different configuration parameters (e.g., resolution, quantization parameter (QP)) are selected according to the video application, and the sub-segments are encoded into sub-streamings with different qualities that are uploaded to servers for further processing. Experiments on various video tasks and datasets show that RegionFilter achieves considerable filtering benefits (bandwidth savings of 11.7–41.2% when transmitting HD video) while always meeting the desired accuracy.

Classroom teacher action recognition based on spatio-temporal dual-branch feature fusion

The classroom teaching action recognition task refers to recognizing and understanding teacher action through video temporal and spatial information. Due to complex backgrounds and significant occlusions, recognizing teacher action in the classroom environment poses substantial challenges. In this study, we propose a classroom teacher action recognition approach based on a spatio-temporal dual-branch feature fusion architecture, where the core task involves utilizing continuous human keypoint heatmap information and single-frame image information. Specifically, we fuse features from two modalities to propose a method combining image spatial information with temporal human keypoint heatmap information for teacher action recognition. Our approach ensures recognition accuracy while reducing the model’s parameters and computational complexity. Additionally, we constructed a teacher action dataset (CTA) in a real classroom environment, comprising 12 action categories, 13k+ video segments, and a total duration exceeding 15 h. The experimental results on the CTA dataset validate the effectiveness of our proposed method. Our research explores action recognition tasks in real complex classroom environments, providing a technical framework for classroom teaching intelligent analysis.

Detecting Temporal shape changes with the Euler Characteristic Transform

Abstract Organoids are multi-cellular structures that are cultured in vitro from stem cells to resemble specific organs (e.g., brain, liver) in their three-dimensional composition. Dynamic changes in the shape and composition of these model systems can be used to understand the effect of mutations and treatments in health and disease. In this paper, we propose a new technique in the field of topological data analysis for DEtecting Temporal shape changes with the Euler Characteristic Transform (DETECT). DETECT is a rotationally invariant signature of dynamically changing shapes. We demonstrate our method on a data set of segmented videos of mouse small intestine organoid experiments and show that it outperforms classical shape descriptors. We verify our method on a synthetic organoid data set and illustrate how it generalizes to 3D. We conclude that DETECT offers rigorous quantification of organoids and opens up computationally scalable methods for distinguishing different growth regimes and assessing treatment effects.

VICR: A Novel Software for Unbiased Video and Image Analysis in Scientific Research.

In scientific research, objectivity and unbiased data analysis is crucial for the validity and reproducibility of outcomes. This is particularly important for studies involving video or image categorization. A common approach of decreasing the bias is delegating data analysis to researchers unfamiliar with the experimental settings. However, this requires additional personnel and is prone to cognitive biases. Here we describe the Video & Image Cutter & Randomizer (VICR) software (https://github.com/kkihnphd/VICR), designed for unbiased analysis by segmenting and then randomizing videos or still images. VICR allows a single researcher to conduct and analyze studies in a blinded manner, eliminating the bias in analysis and streamlining the research process. We describe the features of the VICR software and demonstrate its capabilities using zebrafish behavior studies. To our knowledge, VICR is the only software for the randomization of video and image segments capable of eliminating bias in data analysis in a variety of research fields.

Features and quality metrics datasets for video coding in DASH

Nowadays, the majority of media Internet traffic consists of H.264-encoded streaming videos due to its high compatibility. One of the most popular streaming technology used to deliver videos over Internet is Dynamic Adaptive Streaming over HTTP (DASH). It transmit the video as a sequence of independent short video segments tailored to the receiver’s limitations (related to several factors such as available bandwidth or resolution on reception), aiming to enhance the Quality of Experience. In this paper, we present two types of datasets created from 4065 video segments of 2 seconds. The first type consists of extracting features related to color, space, and time from the segments across different vertical resolutions (240, 360, 480, 720, 1080, 1440 and 4K). The second one includes several quality metrics obtained from the same segments when they are encoded with different compression levels in the different resolutions.

Experiences of the change process during Emotion-focused Group Therapy

ABSTRACT Emotion-focused Group Therapy has demonstrated preliminary efficacy for multiple syndromes. The groups involve individual chair-work in the context of the group, followed by interpersonal processing of the aroused emotions. So far there has been limited investigation of the mechanisms of change, including participants’ experiences of the change process. This study aimed to qualitatively examine participants’ experiences of the change process during Emotion-focused Group Therapy. Sixteen participants were interviewed about their experiences of the change process during the therapy group. The interview was structured to move from overarching reflections on the process of change, to specific experiences of their own chair-work and watching others’ chair-work. During the interview, participants were shown salient video segments of the group to prime their recall of moments of high arousal that were deemed likely to contain emotional change processes. Thematic Analysis extracted five main themes: Value of Vulnerability; Connectedness; Vicarious Emotional Processing; Understanding My Critic; and Barriers to Emotional Processing. These findings suggest that there may be both common and unique change processes that occur during Emotion-focused Group Therapy, compared with the individual Emotion-focused Therapy.

Stacked collaborative transformer network with contrastive learning for video moment localization

Video grounding intends to perform temporal localization in multimedia information retrieval. The temporal bounds of the target video span are determined for the given input query. A novel interactive multi-head self-attention (IMSA) transformer is proposed to localize an unseen moment in the untrimmed video for the given image. A new semantic-trained self-supervised approach is considered in this paper to perform cross-domain learning to match the image query – video segment. It normalizes the convolution function enabling efficient correlation and collecting of semantically related video segments across time based on the image query. A double hostile Contrastive learning with Gaussian distribution parameters method is advanced to learn the representations of video. The proposed approach performs dynamically on various video components to achieve exact semantic synchronization and localization among queries and video. In the proposed approach, the IMSA model localizes frames greatly compared to other approaches. Experiments on benchmark datasets show that the proposed model can significantly increase temporal grounding accuracy. The moment occurrence is identified in the video with a start and end boundary ascertains an average recall of 86.45% and a mAP of 59.3%.

Enhanced Video-Level Anomaly Feature Detection for Nuclear Power Plant Component Inspections Using the Latency Mechanism

The conditions of various nuclear power plant facilities are regularly examined through manual inspections. Remote visual inspection is commonly applied and requires engineers to watch lengthy inspection footage and seek anomaly features therein. This is a labor-intensive process as anomaly features of interest usually only appear in very short segments of the original whole video. Therefore, an automated anomaly detection system is preferred to lessen the intensive labor cost in the inspection process. The detection process could also benefit from useful information that could potentially contribute to addressing reasoning traceability. With a well-prepared training data set of the anomaly feature, a convolutional neural network (CNN) can be developed to automatically detect anomaly indications in the inspection video. However, false-positive detections may occur and can be difficult to remove without seeking manual verification. To overcome this problem, we present a new automated video-level anomaly detection framework that utilizes the latency mechanism to effectively lessen false-positive occurrences, and therefore, increase detection accuracy. In this framework, a CNN-based anomaly classifier first performs initial scanning of the anomaly type of interest in every region of the sampled frames. Then our latency mechanism is applied to refine the initial scanning results by flagging up a region as an “anomaly” indication only when “anomaly” is detected by CNN in the current frame and also in a sequence of previous consecutive frames of the same region. We present a case study of crack feature detection in superheater inspection videos to illustrate the performance of the proposed framework. The results show that the latency mechanism can effectively remove the original false-positive detections seen in the initial scanning. In order to provide a primary exploration of suggesting possible formats for addressing reasoning traceability, knowledge graphs of the reasoning process in the video-level detection framework are built to provide a better understanding of why a specific section of the video is flagged as anomaly contents by the video-level detection framework.

Joy: the key to using media to move the needle on climate change

IntroductionReaching the public and engaging them in addressing climate change could be effectively motivated by film and television. Unfortunately, to-date, there is limited understanding regarding the mechanisms by which such media motivates behavior change to address climate. In this research, we sought to investigate how media exposure can motivate climate and environmentally-conscious behaviors by assessing how emotional responses of viewers or demographic characteristics affected subsequent behavior over weeks following exposure.MethodsParticipants (N = 352) were recruited online and randomly assigned to watch one of three video segments. In surveys before and 4 weeks after viewing, they reported on specific environmentally friendly actions they had taken in recent weeks. Using analysis of covariance (ANCOVA) adjusted for actions reported at baseline, we assessed whether the number of actions reported at follow-up differed by assigned video. Two ANCOVA models were pre-specified, one with video only and one with backwards selection on a list of demographic factors and emotions experienced while watching the videos.ResultsIn the multivariable model, a higher level of action was associated with self-reported joy (p < 0.001) and fear (p < 0.01).DiscussionThese findings suggest that climate communication that engages audience members in a joyful emotional state or, to a lesser degree, a frightening experience may be most effective in increasing climate action.

Leg stereotypy syndrome: phenomenological and quantitative analysis.

Leg stereotypy syndrome (LSS) is a very common, yet underrecognized condition. The pathophysiology of the condition is not well understood. To evaluate and describe the visual kinematic characteristics of the repetitive leg movements in individuals with LSS. In this study, we identified and videotaped individuals diagnosed with LSS at the Parkinson's Disease Center and Movement Disorders Clinic, Baylor College of Medicine, Houston, Texas between 2000 and 2023. Only patients with LSS and without any co-morbidities were included in the study. Their medical records were carefully reviewed, and the demographic and clinical data were entered into a database. Video recordings of the repetitive leg movements were then analyzed using TremAn software. We identified 14 individuals with LSS who were videotaped at our center. The videos of the 5 cases were too brief and therefore not suitable for TremAn quantitative analysis. The remaining 9 individuals exhibited regular rhythmic oscillations of the legs. Among these, two individuals displayed rhythmic movements only in video segments where their legs were in crossed positions. The other 7 individuals had regular rhythmic oscillations, always with the toes resting on the floor with the heels raised. Frequency analysis showed values between 4.5 and 6.5Hz, fairly consistent with a variance below 0.5Hz in individual cases. The oscillation frequency changed from 5.7 Hz to 2.7 Hz while standing. In this study, 6 of 9 individuals with LSS showed 4.5-6.5Hz regular rhythmic leg movements. Studies involving a larger LSS population with additional electrophysiological evaluations are needed to obtain further insights into this common movement disorder.

Collaborative Video Caching in the Edge Network using Deep Reinforcement Learning

With the enormous growth in mobile data traffic over the 5G environment, Adaptive BitRate (ABR) video streaming has become a challenging problem. Recent advances in Mobile Edge Computing (MEC) technology make it feasible to use Base Stations (BSs) intelligently by network caching, popularity-based video streaming, and more. Additional computing resources on the edge node offer an opportunity to reduce network traffic on the backhaul links during peak traffic hours. More recently, it has been found in the literature that collaborative caching strategies between neighbouring BSs (i.e., MEC servers) make it more efficient to reduce backhaul traffic and network congestion and thus improve the viewer experience substantially. In this work, we propose a Reinforcement Learning (RL)–based collaborative caching mechanism in which the edge servers cooperate to serve the requested content from the end-users. Specifically, this research aims to improve the overall cache hit rate at the MEC, where the edge servers are clustered based on their geographic locations. This task is modelled as a multi-objective optimization problem and solved using an RL framework. In addition, a novel cache admission and eviction policy is defined by calculating the priority score of video segments in the clustered MEC mesh network.

Pain in Dementia: An Empirical Test of a Common Assumption

Numerous, and often largely overlapping, observational pain assessment tools have been developed specifically to assess pain in older adults with dementia under the assumption that a specialized approach is necessary to evaluate pain in this population. However, this assumption has never been tested empirically. As an empirical test of this implicit assumption, our goal was to compare existing tools for people living with dementia (with respect to psychometric properties), not only against each other, but also against a tool developed for a different population with cognitive impairments. Videos of older adults with severe dementia recorded in long-term care settings were coded for pain behaviors in the laboratory. Trained coders coded pain behaviors in video segments of older adults with dementia during a quiet baseline condition as well as during a physical examination (designed to identify painful areas), using various observational pain assessment tools. An observational measure of agitation was employed to facilitate the assessment of discriminant validity. Consistent with our expectations, all pain tools (including the tool developed for younger people with cognitive impairments) successfully differentiated between painful and nonpainful states, with large effect sizes. This was the first study to compare tools specifically developed to assess pain in people living with dementia to a tool developed for a different population. Given that all tools under study showed satisfactory psychometric properties when tested on persons with dementia, this study suggests that the assumption that different tools are necessary for different populations with cognitive impairments cannot be taken for granted. PerspectiveThis article challenges an implicitly held assumption that specialized tools are needed to assess pain in different populations with cognitive impairments. Given commonalities in pain expression across populations, further research is needed to determine whether population-specific tools are needed.

A surgical activity model of laparoscopic cholecystectomy for co-operation with collaborative robots

BackgroundLaparoscopic cholecystectomy is a very frequent surgical procedure. However, in an ageing society, less surgical staff will need to perform surgery on patients. Collaborative surgical robots (cobots) could address surgical staff shortages and workload. To achieve context-awareness for surgeon-robot collaboration, the intraoperative action workflow recognition is a key challenge.MethodsA surgical process model was developed for intraoperative surgical activities including actor, instrument, action and target in laparoscopic cholecystectomy (excluding camera guidance). These activities, as well as instrument presence and surgical phases were annotated in videos of laparoscopic cholecystectomy performed on human patients (n = 10) and on explanted porcine livers (n = 10). The machine learning algorithm Distilled-Swin was trained on our own annotated dataset and the CholecT45 dataset. The validation of the model was conducted using a fivefold cross-validation approach.ResultsIn total, 22,351 activities were annotated with a cumulative duration of 24.9 h of video segments. The machine learning algorithm trained and validated on our own dataset scored a mean average precision (mAP) of 25.7% and a top K = 5 accuracy of 85.3%. With training and validation on our dataset and CholecT45, the algorithm scored a mAP of 37.9%.ConclusionsAn activity model was developed and applied for the fine-granular annotation of laparoscopic cholecystectomies in two surgical settings. A machine recognition algorithm trained on our own annotated dataset and CholecT45 achieved a higher performance than training only on CholecT45 and can recognize frequently occurring activities well, but not infrequent activities.The analysis of an annotated dataset allowed for the quantification of the potential of collaborative surgical robots to address the workload of surgical staff. If collaborative surgical robots could grasp and hold tissue, up to 83.5% of the assistant’s tissue interacting tasks (i.e. excluding camera guidance) could be performed by robots.

BONES: Near-Optimal Neural-Enhanced Video Streaming

Accessing high-quality video content can be challenging due to insufficient and unstable network bandwidth. Recent advances in neural enhancement have shown promising results in improving the quality of degraded videos through deep learning. Neural-Enhanced Streaming (NES) incorporates this new approach into video streaming, allowing users to download low-quality video segments and then enhance them to obtain high-quality content without violating the playback of the video stream. We introduce BONES, an NES control algorithm that jointly manages the network and computational resources to maximize the quality of experience (QoE) of the user. BONES formulates NES as a Lyapunov optimization problem and solves it in an online manner with near-optimal performance, making it the first NES algorithm to provide a theoretical performance guarantee. Comprehensive experimental results indicate that BONES increases QoE by 5% to 20% over state-of-the-art algorithms with minimal overhead. Our code is available at https://github.com/UMass-LIDS/bones.

SgLFT: Semantic-guided Late Fusion Transformer for video corpus moment retrieval

Video corpus moment retrieval aims to locate video segments that semantically correspond to natural language queries in a large collection. Effective representation learning across the video, subtitle, and language modalities is crucial for success. However, existing methods prematurely combine visual and subtitle features, leading to redundant information that hinders language query matching. Moreover, these methods lack query guidance during visual encoding, overlooking fine-grained semantics. To address these limitations, we propose a simple yet effective approach called Semantic-guided Late Fusion Transformer (SgLFT) for video corpus moment retrieval. Our method leverages a Semantic-guided Locality-aware Transformer to capture fine-grained visual embeddings using distinguishable query semantics, independent of the presence of subtitles. Furthermore, we introduce a late fusion approach that merges subtitle and visual features under a Cross-modality Global-aware Context Fusion unit, enhancing global contextual details. Finally, a Query-aware Feature Learning module aligns the query and video into a unified representation for localization. Our framework effectively models fine-grained semantic interactions between modalities with query guidance, advancing cross-modal representation learning. Extensive experiments on the TVR and DiDeMo benchmarks demonstrate that SgLFT significantly outperforms previous state-of-the-art methods.

Semantic segmentation of oblique UAV video based on ConvLSTM in complex urban area

Semantic segmentation of oblique UAV video based on ConvLSTM in complex urban area