Video Segmentation Research Articles

Continuous Prediction of Wrist Joint Kinematics Using Surface Electromyography from the Perspective of Muscle Anatomy and Muscle Synergy Feature Extraction.

Post-stroke upper limb dysfunction severely impacts patients' daily life quality. Utilizing sEMG signals to predict patients' motion intentions enables more effective rehabilitation by precisely adjusting the assistance level of rehabilitation robots. Employing the muscle synergy (MS) features can establish more accurate and robust mappings between sEMG and motion intentions. However, traditional matrix factorization algorithms based on blind source separation still exhibit certain limitations in extracting MS features. This paper proposes four deep learning models to extract MS features from four distinct perspectives: spatiotemporal convolutional kernels, compression and reconstruction of sEMG, graph topological structure, and the anatomy of target muscles. Among these models, the one based on 3DCNN predicts motion intentions from the muscle anatomy perspective for the first time. It reconstructs 1D sEMG samples collected at each time point into 2D sEMG frames based on the anatomical distribution of target muscles and sEMG electrode placement. These 2D frames are then stacked as video segments and input into 3DCNN for MS feature extraction. Experimental results on both our wrist motion dataset and public Ninapro DB2 dataset demonstrate that the proposed 3DCNN model outperforms other models in terms of prediction accuracy, robustness, training efficiency, and MS feature extraction for continuous prediction of wrist flexion/extension angles. Specifically, the average nRMSE and R2 values of 3DCNN on these two datasets are (0.14/0.93) and (0.04/0.95), respectively. Furthermore, compared to existing studies, the 3DCNN outperforms musculoskeletal models based on direct collocation optimization, physics-informed GANs, and CNN-LSTM-based deep Kalman filter models when evaluated on our dataset.

ПОКРАЩЕННЯ ВІДЕОПОСЛІДОВНОСТІ В СИСТЕМАХ ВІДЕОАНАЛІТИКИ

A video enhancement method based on applying gamma correction is proposed. Its realization implies processing all video frames using a single algorithm. The proposed algorithm, due to the preservation of inter-frame coherence throughout the entire video, significantly reduces the time spent searching for the optimal value of the gamma parameter, which ensures the highest quality of the frame reaching the maximum possible level in the automatic mode in the sense of both visual observation and the determination of key points and the selection of contours of objects in images. The method is characterized by high adaptability to sudden changes in the lighting of the scene, preservation of interframe coherence, and the absence of any side negative artifacts in the enhanced video. A toolkit for automatically determining the optimal value of the gamma parameter for video frames is developed. It significantly increases the efficiency of video analytics systems, image and video segmentation and processing processes due to reducing the negative impact of the scene lighting mode on image quality. Keywords: gamma-correction, video analytics system, video sequence enhancement, histogram, cumulative histogram, video processing, interframe coherence.

STFormer: Spatial-Temporal-Aware Transformer for Video Instance Segmentation.

Video instance segmentation (VIS) is a challenging task, requiring handling object classification, segmentation, and tracking in videos. Existing Transformer-based VIS approaches have shown remarkable success, combining encoded features and instance queries as decoder inputs. However, their decoder inputs are low-resolution due to computational cost, resulting in a loss of fine-grained information, sensitivity to background interference, and poor handling of small objects. Moreover, the queries are randomly initialized without location information, hindering convergence efficiency and accurate object instance localization. To address these issues, we propose a novel VIS approach, STFormer, with a spatial-temporal feature aggregation (STFA) module and spatial-temporal-aware Transformer (STT). Specifically, STFA obtains robust high-resolution masked features efficiently for the decoder, while STT's location-guided instance query (LGIQ) improves initial instance queries. STFormer preserves more fine-grained information, improves convergence efficiency, and localizes object instance features accurately. Extensive experiments on YouTube-VIS 2019, YouTube-VIS 2021, and OVIS datasets show that STFormer outperforms mainstream VIS methods.

Toward Video Anomaly Retrieval From Video Anomaly Detection: New Benchmarks and Model.

Video anomaly detection (VAD) has been paid increasing attention due to its potential applications, its current dominant tasks focus on online detecting anomalies, which can be roughly interpreted as the binary or multiple event classification. However, such a setup that builds relationships between complicated anomalous events and single labels, e.g., "vandalism", is superficial, since single labels are deficient to characterize anomalous events. In reality, users tend to search a specific video rather than a series of approximate videos. Therefore, retrieving anomalous events using detailed descriptions is practical and positive but few researches focus on this. In this context, we propose a novel task called Video Anomaly Retrieval (VAR), which aims to pragmatically retrieve relevant anomalous videos by cross-modalities, e.g., language descriptions and synchronous audios. Unlike the current video retrieval where videos are assumed to be temporally well-trimmed with short duration, VAR is devised to retrieve long untrimmed videos which may be partially relevant to the given query. To achieve this, we present two large-scale VAR benchmarks and design a model called Anomaly-Led Alignment Network (ALAN) for VAR. In ALAN, we propose an anomaly-led sampling to focus on key segments in long untrimmed videos. Then, we introduce an efficient pretext task to enhance semantic associations between video-text fine-grained representations. Besides, we leverage two complementary alignments to further match cross-modal contents. Experimental results on two benchmarks reveal the challenges of VAR task and also demonstrate the advantages of our tailored method. Captions are publicly released at https://github.com/Roc-Ng/VAR.

Pixel-Wise Contrastive Learning for Multi-Class Instrument Segmentation in Endoscopic Robotic Surgery Videos Using Dataset-Wide Sample Queues

Pixel-Wise Contrastive Learning for Multi-Class Instrument Segmentation in Endoscopic Robotic Surgery Videos Using Dataset-Wide Sample Queues

Dynamic-guided Spatiotemporal Attention for Echocardiography Video Segmentation.

Left ventricle (LV) endocardium segmentation in echocardiography video has received much attention as an important step in quantifying LV ejection fraction. Most existing methods are dedicated to exploiting temporal information on top of 2D convolutional networks. In addition to single appearance semantic learning, some research attempted to introduce motion cues through the optical flow estimation (OFE) task to enhance temporal consistency modeling. However, OFE in these methods is tightly coupled to LV endocardium segmentation, resulting in noisy inter-frame flow prediction, and post-optimization based on these flows accumulates errors. To address these drawbacks, we propose dynamic-guided spatiotemporal attention (DSA) for semi-supervised echocardiography video segmentation. We first fine-tune the off-the-shelf OFE network RAFT on echocardiography data to provide dynamic information. Taking inter-frame flows as additional input, we use a dual-encoder structure to extract motion and appearance features separately. Based on the connection between dynamic continuity and semantic consistency, we propose a bilateral feature calibration module to enhance both features. For temporal consistency modeling, the DSA is proposed to aggregate neighboring frame context using deformable attention that is realized by offsets grid attention. Dynamic information is introduced into DSA through a bilateral offset estimation module to effectively combine with appearance semantics and predict attention offsets, thereby guiding semantic-based spatiotemporal attention. We evaluated our method on two popular echocardiography datasets, CAMUS and EchoNet-Dynamic, and achieved state-of-the-art.

Cascaded Inner-Outer Clip Retformer for Ultrasound Video Object Segmentation.

Computer-aided ultrasound (US) imaging is an important prerequisite for early clinical diagnosis and treatment. Due to the harsh ultrasound (US) image quality and the blurry tumor area, recent memory-based video object segmentation models (VOS) achieve frame-level segmentation by performing intensive similarity matching among the past frames which could inevitably result in computational redundancy. Furthermore, the current attention mechanism utilized in recent models only allocates the same attention level among whole spatial-temporal memory features without making distinctions, which may result in accuracy degradation. In this paper, we first build a larger annotated benchmark dataset for breast lesion segmentation in ultrasound videos, then we propose a lightweight clip-level VOS framework for achieving higher segmentation accuracy while maintaining the speed. The Inner-Outer Clip Retformer is proposed to extract spatialtemporal tumor features in parallel. Specifically, the proposed Outer Clip Retformer extracts the tumor movement feature from past video clips to locate the current clip tumor position, while the Inner Clip Retformer detailedly extracts current tumor features that can produce more accurate segmentation results. Then a Clip Contrastive loss function is further proposed to align the extracted tumor features along both the spatial-temporal dimensions to improve the segmentation accuracy. In addition, the Global Retentive Memory is proposed to maintain the complementary tumor features with lower computing resources which can generate coherent temporal movement features. In this way, our model can significantly improve the spatial-temporal perception ability without increasing a large number of parameters, achieving more accurate segmentation results while maintaining a faster segmentation speed. Finally, we conduct extensive experiments to evaluate our proposed model on several video object segmentation datasets, the results show that our framework outperforms state-of-theart segmentation methods.

VICR: A novel software for unbiased video and image analysis in scientific research.

In scientific research, objectivity and unbiased data analysis is crucial for the validity and reproducibility of outcomes. This is particularly important for studies involving video or image categorization. A common approach of decreasing the bias is delegating data analysis to researchers unfamiliar with the experimental settings. However, this requires additional personnel and is prone to cognitive biases. Here we describe the Video & Image Cutter & Randomizer (VICR) software (https://github.com/kkihnphd/VICR), designed for unbiased analysis by segmenting and then randomizing the segmented videos or still images. VICR allows a single researcher to conduct and analyze studies in a blinded manner, eliminating the bias in analysis and streamlining the research process. We describe the features of the VICR software and demonstrate its capabilities using zebrafish behavior studies. To our knowledge, VICR is the only software for the randomization of video and image segments capable of eliminating bias in data analysis in a variety of research fields.

A deep learning-based highlight detection with pairwise segment ranking for effective video summarization

In the modern era, the sheer volume of video content available poses significant challenges for viewers trying to extract critical information swiftly. Addressing this issue, our research introduces an innovative Highlight Detection and Pairwise Ranking Model for video summarization. This paper delves into the design and evaluation of a hybrid model that effectively identifies and ranks video segments, producing concise and coherent video summaries. By integrating deep learning techniques for highlight detection with pairwise ranking, our model assesses not only the individual importance of each segment but also its relevance in relation to others. Through extensive experiments on diverse datasets, we demonstrate that our approach consistently outperforms existing state-of-the-art video summarization techniques in terms of both user satisfaction and objective measures. Our findings indicate a promising direction for the efficient summarization of long-form videos, catering to the evolving needs of contemporary audiences.

Automatic Identification of Facial Tics Using Selfie-Video.

The intrinsic nature of tic disorders, characterized by symptom variability and fluctuation, poses challenges in clinical evaluations. Currently, tic assessments predominantly rely on subjective questionnaires administered periodically during clinical visits, thus lacking continuous quantitative evaluation. This study aims to establish an automatic objective measure of tic expression in natural behavioral settings. A custom-developed smartphone application was used to record selfie-videos of children and adolescents with tic disorders exhibiting facial motor tics. Facial landmarks were utilized to extract tic-related features from video segments labeled as either "tic" or "non-tic". These features were then passed through a tandem of custom deep neural networks to learn spatial and temporal properties for tic classification of these segments according to their labels. The model achieved a mean accuracy of 95% when trained on data across all subjects, and consistently exceeded 90% accuracy in leave-one-session-out and leave-one-subject-out cross validation training schemes. This automatic tic identification measure may provide a valuable tool for clinicians in facilitating diagnosis, patient follow-up, and treatment efficacy evaluation. Combining this measure with standard smartphone technology has the potential to revolutionize large-scale clinical studies, thereby expediting the development and testing of novel interventions.

CSS-Net: A Consistent Segment Selection Network for Audio-Visual Event Localization

Audio-visual event (AVE) localization aims to localize the temporal boundaries of events that contains visual and audio contents, to identify event categories in unconstrained videos. Existing work usually utilizes successive video segments for temporal modeling. However, ambient sounds or irrelevant visual targets in some segments often cause the problem of audio-visual semantics inconsistency, resulting in inaccurate global event modeling. To tackle this issue, we present a consistent segment selection network (CSS-Net) in this paper. First, we propose a novel bidirectional guided co-attention (BGCA) block, containing two distinct attention paths from audio to vision and from vision to audio, to focus on sound-related visual regions and event-related sound segments. Then, we propose a novel context-aware similarity measure (CASM) module to select semantic consistent visual and audio segments. A cross-correlation matrix is constructed using the correlation coefficients between the visual and audio feature pairs in all time steps. By extracting highly correlated segments and discarding low correlated segments, visual and audio features can learn global event semantics in videos. Finally, we propose a novel audio-visual contrastive loss to learn the similar semantics representation for visual and audio global features under the constraints of cosine and L2 similarities. Extensive experiments on public AVE dataset demonstrates the effectiveness of our proposed CSS-Net. The localization accuracies achieve the best performance of 80.5% and 76.8% in both fully- and weakly-supervised settings compared with other state-of-the-art methods.

Neighbor-Guided Pseudo-Label Generation and Refinement for Single-Frame Supervised Temporal Action Localization.

Due to the sparse single-frame annotations, current Single-Frame Temporal Action Localization (SF-TAL) methods generally employ threshold-based pseudo-label generation strategies. However, these approaches suffer from inefficient data utilization, as only parts of unlabeled frames with confidence scores surpassing a predefined threshold are selected for training. Moreover, the variability of single-frame annotations and unreliable model predictions introduce pseudo-label noise. To address these challenges, we propose two strategies by using the relationship of the video segments with their neighbors': 1) temporal neighbor-guided soft pseudo-label generation (TNPG); and 2) semantic neighbor-guided pseudo-label refinement (SNPR). TNPG utilizes a local-global self-attention mechanism in a transformer encoder to capture temporal neighbor information while focusing on the whole video. Then the generated self-attention map is multiplied by the network predictions to propagate information between labeled and unlabeled frames, and produce soft pseudo-label for all segments. Despite this, label noise persists due to unreliable model predictions. To mitigate this, SNPR refines pseudo-labels based on the assumption that predictions should resemble their semantic nearest neighbors'. Specifically, we search for semantic nearest neighbors of each video segment by cosine similarity in the feature space. Then the refined soft pseudo-labels can be obtained by a weight combination of the original pseudo-label and the semantic nearest neighbors'. Finally, the model can be trained with the refined pseudo-labels, and the performance has been greatly improved. Comprehensive experimental results on different benchmarks show that we achieve state-of-the-art performances on THUMOS14, ActivityNet1.2, and ActivityNet1.3 datasets.

Video Enhanced Didactic Instruction for Mass Casualty Triage

Abstract Objectives High-Fidelity training in triage for Mass Casualty Incidents (MCI) is resource intensive and therefore limited in frequency despite knowing that this skill degrades with time. Didactic classroom materials offer a non-resource-intensive introduction to foundational concepts, but the extent to which it can prepare providers for real world scenarios is limited. Virtual reality (VR) is an effective training alternative, but in its infancy of development and accessibility. To bridge the gap between these modalities, we propose a video-enhanced method to didactic training. Methods During Uniformed Service University’s (USU) annual Operation Bushmaster field exercise we produced video footage of individual casualties along with multiple casualty scenes consisting of several 5-15 second videos capturing various elements of a primary survey. Results The videos offer a higher-fidelity alternative to written casualty descriptions allowing the learner to visually process casualty presentation that is more representative of reality. The video segments facilitate thinking through specific points in the decision-making process for triage and life saving interventions. Conclusions Video enhanced didactic training offers a bridge between written didactic training and exercise or VR training for MCI triage skills acquisition and sustainment. Assessment and validation of this training methodology with various MCI responders is recommended.

Optimal Information Hiding: Advanced Bitstream‐Based Watermarking for Secure and Efficient Integration of Image Data in Digital Videos

Video watermarking is a technique used to embed a visible or invisible mark onto a video file in order to claim ownership, protect intellectual property rights, and detect unauthorized users. The issue of video piracy arises due to the advancement of Internet services and the utilization of various storage technologies. This problem becomes particularly significant with the widespread propagation of media through online channels. The constant progress in sharing of data, videos, images, and applications necessitates the implementation of measures focused on security and the protection of intellectual ownership rights. To address this issue, we proposed a novel Bitstream video watermarking technique that integrates cutting‐edge methodologies to seamlessly embed image data within digital videos. This technique stands apart from less sophisticated watermarking methods by capitalizing on spatial and temporal domain concepts in video processing. An experiment is conducted to analyze the time required for the insertion and extraction processes to gain insights into the performance of bitstream watermarking in the video. While enhancing the existing approach, we systematically explored diverse strategies aimed at augmenting watermark capacity, bolstering robustness, and heightening visual imperceptibility. These strategies encompassed the intricate infusion of watermark fragments into the frame area through chaotic mechanisms, the integration of error‐correcting codes to ensure data fidelity, and the selective omission of dynamic video segments during information extraction processes. Rigorous experimentations are undertaken by the constraints of lossy compression scenarios, maintaining equivalent or reduced bitrate parameters. The findings substantiate the favorable outcomes stemming from the implemented enhancement, affirming the tangible suitability of the phase watermarking technique for the protection of video copyrights. In order to assess the quality of the bitstream video watermarking technique, the peak signal‐to‐noise ratio (PSNR), structural similarity index (SSIM), and normalized correlation coefficient (NCC) are computed and examined by the testing data. The suggested model achieved a PSNR value up to 32.52 dB, SSIM value up to 0.97, and NCC value is 1. The dynamic range (DR) and spatial resolution (SR) are measured in the evaluation process. Security is enhanced significantly after implementing the multifaceted approach. The watermarking scheme is more robust and can effectively resist the potential attacks. In summary, this study advances the field of video watermarking by proposing a sophisticated bitstream technique and rigorously evaluating its performance. The outcomes underscore its practical viability for copyright protection, contributing to a more secure digital media landscape.

Recognition and detection of unusual activities in ATM using dual-channel capsule generative adversarial network

Real-time detection of human activity has become essential for monitoring and security of public spaces such as bank ATMs and workplaces, due to daily rise in criminal activities. Currently, monocular CCTV cameras that only record RGB video are used for monitoring such restricted areas. In addition to RGB data, the RGB + D sensor also offers depth information about the scene. To overcome the issue of online recognition of anomalous activities in Bank ATMs, a supervised deep learning method utilizing Dual-Channel Capsule Generative Adversarial Network (DCCGAN) and RGB + D sensor is proposed. The input RGB + D Dataset is given to super pixel motion detection method to find the region of interest (ROI). Motion detection is a significant stage examination of wide scene for background subtraction and foreground detection. After arranging the motion detection, its region is positioned frame by frame. Then, the detected ROI is given to fast discrete curvelet transform with the wrapping (FDCT-WRP) method. FDCT-WRP feature extraction method. These extracting features are supplied to Deep Convolutional Spiking neural network (DCSNN), which realize the object. RGB and depth video segments are used to create motion templates from the RGB + D data online video stream. These templates are trained on DCCGANs to identify distrustful events in current activity and categorized as normal and abnormal. Additionally, a unique RGB + D dataset is employed because there was no existing dataset available for analyzing human activity in ATMs. The proposed DUA-DCCGAN-ATM approach is assessed on qualitative with quantitative statistical evaluation parameters and identify suspect occurrence with 0.932 precision and 98.2 % accuracy. The detailed statistical analysis exemplifies that the proposed technique can identify distrustful events in a real-time online manner prior completing the anomalous activity.

JANet: A joint attention network for balancing accuracy and speed in left ventricular ultrasound video segmentation

Multiple cardiac diseases are closely associated with functional parameters of the left ventricle, but functional parameter quantification still requires manual involvement, a time-consuming and less reproducible task. We develop a joint attention network (JANet) and expand it into two versions (V1 and V2) that can be used to segment the left ventricular region in echocardiograms to assist physicians in diagnosis. V1 is a smaller model with a size of 56.3 MB, and V2 has a higher accuracy. The proposed JANet V1 and V2 achieve a mean dice score (DSC) of 93.59/93.69(V1/V2), respectively, outperforming the state-of-the-art models. We grade 1264 patients with 87.24/87.50 (V1/V2) accuracy when using the 2-level classification criteria and 83.62/84.18 (V1/V2) when using the 5-level classification criteria. The results of the consistency analysis show that the proposed method is comparable to that of clinicians.

Understanding the effect of group emotions on consumer instant order cancellation behavior in livestreaming E-commerce: Empirical evidence from TikTok

Livestreaming e-commerce (LSE) exhibits a noteworthy idiosyncrasy in the form of escalating instant order cancellations, wherein consumers swiftly revoke their purchases during livestreaming before receiving the products. This study leveraged the emotional contagion theory to construct an econometric model and investigate the factors influencing viewers' instant order cancellation behaviors. An extensive dataset was assembled, comprising 4984 product introduction video segments from 768 TikTok livestreaming rooms in China, along with 1.29 million bullet screen comments and 513,551 viewers' information. Employing advanced multimodal data processing technologies, the study extracted anchors' (livestreaming hosts) multimodal emotion features. The empirical findings demonstrate the significant impact of anchors' emotions, encompassing body movements, facial expressions, voices, and conversation contents, on the arousal and valence of group emotions, where the latter exhibit opposing effects on instant order cancellation behavior. This research establishes the paramount roles of anchor and group emotions in viewers' instant order cancellation behaviors, lending credence to the epistemological applicability of emotional contagion theory in the context of LSE. The study's findings offer LSE practitioners practical implications to enhance business decision-making, fostering more engaging and emotionally resonant livestreaming experiences to reduce cancellations and improve customer retention.

Towards Impact of Chunk-Level Characteristics on Mobile Live Streaming Performance

Today, mobile live streaming is gaining a rapid growth in use, which refers to watching the media content recorded and broadcast in real time on mobile devices. In live streaming process, each video segment must go through recording, encoding, uploading, transcoding, publishing, downloading, decoding before playback. The ingest algorithm inside the streamer decides the upload bitrate, while the adaptive bitrate (ABR) algorithm inside the player determines the download bitrate. Thanks to the chunked CMAF standard, each segment is split into smaller chunks that can be independently encoded, transferred, decoded and played. It is of great help to quantify the impact of chunk-level characteristics on mobile live streaming performance. In this paper, we establish a tandem queuing model to describe the whole streaming system. Based on the model, we respectively characterize rebuffering probability, rebuffering count, streaming latency, and average bitrate, analyzing the impact of chunk upload and download rates, upload and download time variances, startup threshold and chunk length on them. From analysis results, we propose insights and recommendations for bitrate adaptation in mobile live streaming and design simple heuristic ingest and ABR algorithms leveraging them. Extensive simulations verify the insights as well as effectiveness of designed algorithms.

TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut.

In this paper, we describe a graph-based algorithm that uses the features obtained by a self-supervised transformer to detect and segment salient objects in images and videos. With this approach, the image patches that compose an image or video are organised into a fully connected graph, in which the edge between each pair of patches is labeled with a similarity score based on the features learned by the transformer. Detection and segmentation of salient objects can then be formulated as a graph-cut problem and solved using the classical Normalized Cut algorithm. Despite the simplicity of this approach, it achieves state-of-the-art results on several common image and video detection and segmentation tasks. For unsupervised object discovery, this approach outperforms the competing approaches by a margin of 6.1%, 5.7%, and 2.6% when tested with the VOC07, VOC12, and COCO20K datasets. For the unsupervised saliency detection task in images, this method improves the score for Intersection over Union (IoU) by 4.4%, 5.6% and 5.2%. When tested with the ECSSD, DUTS, and DUT-OMRON datasets. This method also achieves competitive results for unsupervised video object segmentation tasks with the DAVIS, SegTV2, and FBMS datasets. Our implementation is available at https://www.m-psi.fr/Papers/TokenCut2022/.

Kids Save Lives by Learning through a Serious Game

This study focuses on the development and assessment of a serious game for health (SGH) aimed at educating children about cardiopulmonary resuscitation (CPR). A video game was created using the Berkeley Snap platform, which uses block programming. Eye-tracking technology was utilized to validate the graphic design. To assess the tool’s effectiveness, a pre-post analytical study was conducted with primary education children to measure the knowledge acquired. The study involved 52 participants with a mean age of 9 years. The results from a custom questionnaire used to measure their theoretical CPR knowledge showed significant improvements in CPR knowledge after the use of the videogame, and their emotional responses improved as well. The assessment of the knowledge acquired through the video game obtained an average score of 5.25 out of 6. Ten video segments consisting of 500 frames each (20 s of video per segment) were analyzed. Within these segments, specific areas that captured the most relevant interaction elements were selected to measure the child’s attention during game play. The average number of gaze fixations, indicating the points in which the child’s attention was placed within the area of interest, was 361.5 out of 500. In conclusion, the utilization of SGH may be an effective method for educating kids about CPR, to provide them with fundamental knowledge relevant to their age group.