Video Segmentation Research Articles

A Semi-supervised Four-Chamber Echocardiographic Video Segmentation Algorithm Based on Multilevel Edge Perception and Calibration Fusion

ObjectiveEchocardiographic videos are commonly used for automatic semantic segmentation of endocardium, which is crucial in evaluating cardiac function and assisting doctors to make accurate diagnoses of heart disease. However, this task faces two distinct challenges: one is the edge blurring, which is caused by the presence of speckle noise or excessive de-noising operation, and the other is the lack of an effective feature fusion approach for multilevel features for obtaining accurate endocardium. MethodsIn this study, a deep learning model, based on multilevel edge perception and calibration fusion is proposed to improve the segmentation performance. First, a multilevel edge perception module is proposed to comprehensively extract edge features through both a detail branch and a semantic branch to alleviate the adverse impact of noise. Second, a calibration fusion module is proposed that calibrates and integrates various features, including semantic and detailed information, to maximize segmentation performance. Furthermore, the features obtained from the calibration fusion module are stored by using a memory architecture to achieve semi-supervised segmentation through both labeled and unlabeled data. ResultsOur method is evaluated on two public echocardiography video data sets, achieving average Dice coefficients of 93.05% and 93.93%, respectively. Additionally, we validated our method on a local hospital clinical data set, achieving a Pearson correlation of 0.765 for predicting left ventricular ejection fraction. ConclusionThe proposed model effectively solves the challenges encountered in echocardiography by using semi-supervised networks, thereby improving the segmentation accuracy of the ventricles. This indicates that the proposed model can assist cardiologists in obtaining accurate and effective research and diagnostic results.

Unsupervised video summarization with adversarial graph-based attention network

Video summarization aims to select a subset of video segments that best capture the video storyline. Our study seeks to train an encoder to transform the raw frame features extracted from pre-trained CNN models into representations that embody importance and guide the selection of the video segment. Our main idea is to use graph modeling and attention mechanisms to train the encoder adversarially. The graph representation enables the model to learn the relationship among frames, revealing the intrinsic structure of a video. The attention mechanism allows the model to capture the magnitude of these relationships. In the proposed model, an attention-based encoder is trained using a graph-based generator that reconstructs videos using the encoded features and a discriminator that guides the generator, distinguishing the original and reconstructed video. Thus, by leveraging graph attention and refining mechanisms, the proposed model offers distinct advantages over existing methods, including enhanced summarization accuracy, improved preservation of temporal coherence, and the ability to capture complex semantic linkages within video content. These advancements are substantiated through a comprehensive ablation study, which demonstrates the efficacy of our model using various evaluation metrics — Kendall and Spearman coefficients. The proposed model is evaluated on TVSum and SumMe datasets and achieves results on par with supervised models that used similar encoders and achieved state-of-the-art results compared to other unsupervised models.

Computer-vision-based artificial intelligence for detection and recognition of instruments and organs during radical laparoscopic gastrectomy for gastric cancer: a multicenter study

Objective: To investigate the feasibility and accuracy of computer vision-based artificial intelligence technology in detecting and recognizing instruments and organs in the scenario of radical laparoscopic gastrectomy for gastric cancer. Methods: Eight complete laparoscopic distal radical gastrectomy surgery videos were collected from four large tertiary hospitals in China (First Medical Center of Chinese PLA General Hospital [three cases], Liaoning Cancer Hospital [two cases], Liyang Branch of Jiangsu Province People's Hospital [two cases], and Fudan University Shanghai Cancer Center [one case]). PR software was used to extract frames every 5-10 seconds and convert them into image frames. To ensure quality, deduplication was performed manually to remove obvious duplication and blurred image frames. After conversion and deduplication, there were 3369 frame images with a resolution of 1,920×1,080 PPI. LabelMe was used for instance segmentation of the images into the following 23 categories: veins, arteries, sutures, needle holders, ultrasonic knives, suction devices, bleeding, colon, forceps, gallbladder, small gauze, Hem-o-lok, Hem-o-lok appliers, electrocautery hooks, small intestine, hepatogastric ligaments, liver, omentum, pancreas, spleen, surgical staplers, stomach, and trocars. The frame images were randomly allocated to training and validation sets in a 9:1 ratio. The YOLOv8 deep learning framework was used for model training and validation. Precision, recall, average precision (AP), and mean average precision (mAP) were used to evaluate detection and recognition accuracy. Results: The training set contained 3032 frame images comprising 30 895 instance segmentation counts across 23 categories. The validation set contained 337 frame images comprising 3407 instance segmentation counts. The YOLOv8m model was used for training. The loss curve of the training set showed a smooth gradual decrease in loss value as the number of iteration calculations increased. In the training set, the AP values of all 23 categories were above 0.90, with a mAP of 0.99, whereas in the validation set, the mAP of the 23 categories was 0.82. As to individual categories, the AP values for ultrasonic knives, needle holders, forceps, gallbladders, small pieces of gauze, and surgical staplers were 0.96, 0.94, 0.91, 0.91, 0.91, and 0.91, respectively. The model successfully inferred and applied to a 5-minutes video segment of laparoscopic gastroenterostomy suturing. Conclusion: The primary finding of this multicenter study is that computer vision can efficiently, accurately, and in real-time detect organs and instruments in various scenarios of radical laparoscopic gastrectomy for gastric cancer.

Real-time segmentation of short videos under VR technology in dynamic scenes

Abstract This work addresses the challenges of scene segmentation and low segmentation accuracy in short videos by employing virtual reality (VR) technology alongside a 3D DenseNet model for real-time segmentation in dynamic scenes. First, this work extracted short videos by frame and removed redundant background information. Then, the volume rendering algorithm in VR technology was used to reconstruct short videos in dynamic scenes in 3D. It enriched the detailed information of short videos, and finally used the 3D DenseNet model for real-time segmentation of short videos in dynamic scenes, improving the accuracy of segmentation. The experiment compared the performance of High resolution network, Mask region based convolutional neural network, 3D U-Net, Efficient neural network models on the Densely annotation video segmentation dataset. The experimental results showed that the segmentation accuracy of the 3D DenseNet model has reached 99.03%, which was 15.11% higher than that of the ENet model. The precision rate reached 98.33%, and the average segmentation time reached 0.64 s, improving the segmentation accuracy and precision rate. It can adapt to various scene situations and has strong robustness. The significance of this research lies in its innovative approach in tackling these issues. By integrating VR technology with advanced deep learning models, we can achieve more precise segmentation of dynamic scenes in short videos, enabling real-time processing. This has significant practical implications for fields such as video editing, VR applications, and intelligent surveillance. Furthermore, the outcomes of this research contribute to advancing computer vision in video processing, providing valuable insights for the development of future intelligent video processing systems.

Lightweight and real-time semantic segmentation of UAV traffic videos based on siamese network for keyframe recognition

Lightweight and real-time semantic segmentation of UAV traffic videos based on siamese network for keyframe recognition

BONES: Near-Optimal Neural-Enhanced Video Streaming

Accessing high-quality video content can be challenging due to insufficient and unstable network bandwidth. Recent advances in neural enhancement have shown promising results in improving the quality of degraded videos through deep learning. Neural-Enhanced Streaming (NES) incorporates this new approach into video streaming, allowing users to download low-quality video segments and then enhance them to obtain high-quality content without violating the playback of the video stream. We introduce BONES, an NES control algorithm that jointly manages the network and computational resources to maximize the quality of experience (QoE) of the user. BONES formulates NES as a Lyapunov optimization problem and solves it in an online manner with near-optimal performance, making it the first NES algorithm to provide a theoretical performance guarantee. Comprehensive experimental results indicate that BONES increases QoE by 5% to 20% over state-of-the-art algorithms with minimal overhead. Our code is available at https://github.com/UMass-LIDS/bones.

Edwards curve digital signature algorithm for video integrity verification on blockchain framework

Within the field of forensic science, video integrity is an essential component that ensures the validity and dependability of visual evidence that is essential for court cases. Maintaining the integrity of justice in a time when digital modification tools are easily accessible requires making sure that video recordings are unaltered. Video footage is carefully inspected by forensic analysts who use advanced tools to look for indications of manipulation or change. This paper presents a novel, lightweight approach for verifying the integrity of video data. The proposed method utilizes the blockchain and the Edwards Curve Digital Signature Algorithm, coupled with the BLAKE2b hash function. Signatures are generated for video clips with a predetermined size called video segments and stored in blocks chronologically. The signature of the previous block is stored in the current block to add another layer of security. At the time of validation, these signatures are verified. Experimental results show that the proposed method outperforms state-of-the-art methods in both speed and security. Our method is capable of identifying any type of forgery on any video file, by anyone, at any given moment, with insignificant additional storage requirements. Furthermore, our security analysis shows that our method is resilient against various types of attacks, including collision attacks, key substitution attacks, side-channel attacks, and chosen message attacks. The proposed lightweight video integrity verification method is better suited for use in resource constrained devices.

МЕТОД ДВОКАСКАДНОЇ ІМПЛАНТАЦІЇ ПРИХОВАНОЇ ІНФОРМАЦІЇ НА ОСНОВІ СТЕГАНОКОМПРЕСІЙНИХ ПЕРЕТВОРЕНЬ

In the article, the need for further development of the state is directly related to: solving issues of increasing defense capability and information security; development of information and intelligent systems. The necessity of ensuring the required level of completeness of information, compliance with the requirements of its relevance, achievement and maintenance of the appropriate level of integrity, accessibility and confidentiality is substantiated. The article emphasizes that for this purpose, complex systems of coding and information protection are being built. It is shown that recently, in order to further increase the level of security of information resources, methods of hidden embedding of information have been used. In this field of scientific and applied research, the direction of timely delivery of integral video information in a secure mode is outlined. For protection, the following can be used: meta-messages, which are formed on the basis of intelligent analysis of video frames; separate video segments of aerial photographs containing the most important information for decision-making. This article examines a class of methods of steganographic transformations, which are associated with embedding messages in digital containers, which are formed by a stream of video segments (VS). However, existing steganographic systems are based mainly on the use of the amount of psychovisual (PSV) redundancy available for reduction. Therefore, an increase in steganographic capacity leads to a loss of integrity and efficiency of delivery of video-container information. Steganocompression coding in the stegano-polyadic basis was created on the basis of taking into account the amount of redundancy permissible for reduction. In this case, in the process of compression of the components of the VS, the direct embedding of hidden information is organized. Concealment of information is carried out by technological stages, depending on the compression transformation of the transitional syntax of the quilted sequence. The key stage is the technology of two-stage implementation of the elements of the concealed message added to the stegano sequences to the syntax of the steganocompression representation of the BC components.

Exploiting relation of video segments for temporal action detection

With the development of urban intelligence, intelligent identification of human activities has become imminent. Temporal Action Detection (TAD) is designed to identify real human social activities, which is a challenging task in video understanding. The current methods mainly use global features for boundary matching or predefine all possible proposals, while ignoring the interference of background and the causal relevance of the temporal action, resulting in the generation of more redundant proposals and the decline of detection accuracy. To fill this gap, we propose a novel Dilated Convolution Locate and Action Relevant Score model called DCAR. Specifically, DCAR includes a Dilated Location Network (DL-Net) and Action Relevance Calculation (ARC) block. For the DL-Net, we design a Boundary Feature Enhancement (BFE) block, which enhances the boundary feature of actions and fuses the similar features of different channels by pooling and channel squeeze to reduce the interference of the background. Meanwhile, we also design multiple dilated convolutional structures to aggregate long contextual information in time point/interval after boundary enhancement. For the ARC block, we use the hyperbolic space distance and cross attention to calculate the causal correlation of action proposals classification, which removes the misclassification of action proposals. We conduct extensive experiments on Thumos14 and ActivityNet-1.3, which shows our method significantly improves the performance and achieves state-of-the-art results. On Thumos14, at tIoU 0.3, 0.4 and 0.5, it achieves mAP of 63.2%, 57.0% and 48.5%, respectively; On ActivityNet-1.3 it reaches 9.42% at tIoU 0.95.

Weakly Supervised Video Anomaly Detection via Self-Guided Temporal Discriminative Transformer.

Weakly supervised video anomaly detection is generally formulated as a multiple instance learning (MIL) problem, where an anomaly detector learns to generate frame-level anomaly scores under the supervision of MIL-based video-level classification. However, most previous works suffer from two drawbacks: 1) they lack ability to model temporal relationships between video segments and 2) they cannot extract sufficient discriminative features to separate normal and anomalous snippets. In this article, we develop a weakly supervised temporal discriminative (WSTD) paradigm, that aims to leverage both temporal relation and feature discrimination to mitigate the above drawbacks. To this end, we propose a transformer-styled temporal feature aggregator (TTFA) and a self-guided discriminative feature encoder (SDFE). Specifically, TTFA captures multiple types of temporal relationships between video snippets from different feature subspaces, while SDFE enhances the discriminative powers of features by clustering normal snippets and maximizing the separability between anomalous snippets and normal centers in embedding space. Experimental results on three public benchmarks indicate that WSTD outperforms state-of-the-art unsupervised and weakly supervised methods, which verifies the superiority of the proposed method.

Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding

Grounding temporal video segments described in natural language queries effectively and efficiently is a crucial capability needed in vision-and-language fields. In this paper, we deal with the fast video temporal grounding (FVTG) task, aiming at localizing the target segment with high speed and favorable accuracy. Most existing approaches adopt elaborately designed cross-modal interaction modules to improve the grounding performance, which suffer from the test-time bottleneck. Although several common space-based methods enjoy the high-speed merit during inference, they can hardly capture the comprehensive and explicit relations between visual and textual modalities. In this paper, to tackle the dilemma of speed-accuracy tradeoff, we propose a commonsense-aware cross-modal alignment network (C 2 AN), which incorporates commonsense-guided visual and text representations into a complementary common space for fast video temporal grounding. Specifically, the commonsense concepts are explored and exploited by extracting the structural semantic information from a language corpus. Then, a commonsense-aware interaction module is designed to obtain bridged visual and text features by utilizing the learned commonsense concepts. Finally, to maintain the original semantic information of textual queries, a cross-modal complementary common space is optimized to obtain matching scores for performing FVTG. Extensive results on two challenging benchmarks show that our C 2 AN method performs favorably against state-of-the-arts while running at high speed. Our code is available at https://github.com/ZiyueWu59/CCA.

A hands-on guide to use network video recorders, internet protocol cameras, and deep learning models for dynamic monitoring of trout and salmon in small streams.

This study outlines a method for using surveillance cameras and an algorithm that calls a deep learning model to generate video segments featuring salmon and trout in small streams. This automated process greatly reduces the need for human intervention in video surveillance. Furthermore, a comprehensive guide is provided on setting up and configuring surveillance equipment, along with instructions on training a deep learning model tailored to specific requirements. Access to video data and knowledge about deep learning models makes monitoring of trout and salmon dynamic and hands-on, as the collected data can be used to train and further improve deep learning models. Hopefully, this setup will encourage fisheries managers to conduct more monitoring as the equipment is relatively cheap compared with customized solutions for fish monitoring. To make effective use of the data, natural markings of the camera-captured fish can be used for individual identification. While the automated process greatly reduces the need for human intervention in video surveillance and speeds up the initial sorting and detection of fish, the manual identification of individual fish based on natural markings still requires human effort and involvement. Individual encounter data hold many potential applications, such as capture-recapture and relative abundance models, and for evaluating fish passages in streams with hydropower by spatial recaptures, that is, the same individual identified at different locations. There is much to gain by using this technique as camera captures are the better option for the fish's welfare and are less time-consuming compared with physical captures and tagging.

SDN: Semantic Decoupling Network for Temporal Language Grounding.

Temporal language grounding (TLG) is one of the most challenging cross-modal video understanding tasks, which aims at retrieving the most relevant video segment from an untrimmed video according to a natural language sentence. The existing methods can be separated into two dominant types: 1) proposal-based and 2) proposal-free methods, where the former conduct contextual interactions and the latter localizes timestamps flexibly. However, the constant-scale candidates in proposal-based methods limit the localization precision and bring extra computational costs. In contrast, the proposal-free methods perform well on high-precision metrics-based on the fine-grained features but suffer from a lack of coarse-grained interactions, which cause degeneration when the video becomes complex. In this article, we propose a novel framework termed semantic decoupling network (SDN) that combines the advantages of proposal-based and proposal-free methods and overcomes their defects. It contains three key components: 1) semantic decoupling module (SDM); 2) context modeling block (CMB); and 3) semantic cross-level aggregation module (SCAM). By capturing the video-text contexts in multilevel semantics, the SDM and CMB effectively utilize the benefits of proposal-based methods. Meanwhile, the SCAM maintains the merit of proposal-free methods in that it localizes timestamps precisely. The experiments on three challenge datasets, i.e., Charades-STA, TACoS, and ActivityNet-Caption, show that our proposed SDN method significantly outperforms recent state-of-the-art methods, especially the proposal-free methods. Extensive analyses, as well as the implementation code of the proposed SDN method, are provided at https://github.com/CFM-MSG/Code_SDN.

Image and Video Segmentation Using Yolo-Nas and Segment Anything Model (Sam): Machine Learning

In this article, we evaluate the effectiveness of computer vision and machine learning, which are revolutionizing photography and video studies, ushering in an unprecedented era of creativity and informed consent for study. Despite many advances in the field, object recognition and classification have become technologies with many applications, from advanced analytics to enhancing driving safety for augmented reality. This research uses two advanced models, YOLO-NAS (You Just Look at Neural Architecture Search) and Segment One (SAM), to analyze the evolution of image and video segmentation. In real-time search of products and components, YOLO-NAS and SAM models have established a good relationship and are expected to become a unified system for image and video segmentation in digital transformation. Key Words: YOLO-NAS, Convolutional Neural Networks (Cnns), Object Recognition, Computer Vision, Spatial Attention Module (SAM), Instance Video Segmentation, And Image Segmentation.

VSD: A Novel Method for Video Segmentation and Storage in DNA Using RS Code

As data continue to grow in complexity and size, there is an imperative need for more efficient and robust storage solutions. DNA storage has emerged as a promising avenue to solve this problem, but existing approaches do not perform efficiently enough on video data, particularly for information density and time efficiency. This paper introduces VSD, a pioneering encoding method for video segmentation and storage in DNA, leveraging the Reed–Solomon (RS) error correction code. This method addresses these limitations through an innovative combination of segmentation and encoding, accompanied by RS coding to bolster error resilience. Additionally, the method ensures that the GC-content of the resultant DNA sequences remains around 50%, which further enhances the storage robustness. The experimental results demonstrate the method has commendable encoding efficiency and offers a solution to the prevailing issue of time inefficiency and error correction rates in DNA storage. This groundbreaking approach paves the way for the practical and reliable storage of large-scale video data in DNA, heralding a new era in the domain of information storage.

Multi-Level interaction network for temporal sentence grounding in videos

Temporal sentence grounding in videos (TSGV), which aims to retrieve video segments from an untrimmed videos that semantically match a given query. Most previous methods focused on learning either local or global query features and then performed cross-modal interaction, but ignore the complementarity between local and global features. In this paper, we propose a novel Multi-Level Interaction Network for Temporal Sentence Grounding in Videos. This network explores the semantics of queries at both phrase and sentence levels, interacting phrase-level features with video features to highlight video segments relevant to the query phrase and sentence-level features with video features to learn more about global localization information. A stacked fusion gate module is designed, which effectively captures the temporal relationships and semantic information among video segments. This module also introduces a gating mechanism to enable the model to adaptively regulate the fusion degree of video features and query features, further improving the accuracy of predicting the target segments. Extensive experiments on the ActivityNet Captions and Charades-STA benchmark datasets demonstrate that the proposed method outperforms the state-of-the-art methods.

Research on trajectory control of multi‐degree‐of‐freedom industrial robot based on visual image

AbstractIn order to improve the trajectory control effect of multi‐degree‐of‐freedom industrial robots, this paper combines visual image technology to conduct research on trajectory control of multi‐degree‐of‐freedom industrial robots. Aiming at the problem of video segmentation under sudden illumination changes, this paper uses a Gaussian mixture model based on the global illumination function to adopt a variety of illumination invariant features, and proposes a scene segmentation algorithm suitable for sudden illumination changes. Moreover, this paper compares and verifies the algorithm from the subjective and objective perspectives through experiments, which shows that the algorithm in this paper can segment the scene more accurately even in the environment of sudden changes in illumination. In addition, the results of the accuracy test and the trajectory control test show that the research method of the multi‐degree‐of‐freedom industrial robot trajectory control based on the visual image proposed in this paper can effectively improve the trajectory control effect of the robot.

From Microservice to Monolith: A Multivocal Literature Review

Recently, the phenomenon of switching back from microservice to monolith has increased in frequency, leading to intense debate in the industry. In this paper, we conduct a multivocal literature review to investigate reasoning and key aspects to pay attention to when switching back and analyze other practitioners’ opinions. The results show four cases of switching back from microservice to monolith: Istio control plane, Amazon Prime Video monitoring service, Segment, and InVision. The five main reasons that led to switching back are cost, complexity, scalability, performance, and organization. During the switching back process, six key aspects need to be addressed: (1) stopping the development of more services, (2) consolidating and testing paths, (3) unifying data storage, (4) implementing the message bus principle, (5) giving up diverse techniques, and (6) learning to use modular design principles. As to the practitioners’ opinions, they had mixed views about the switching back phenomenon. However, most thought that switching back required consideration of the actual system situation and principles. These results pave the way for further research and guide researchers and companies through the process of switching back from microservice to monolith.

Learning Reliable Dense Pseudo-Labels for Point-Level Weakly-Supervised Action Localization

Point-level weakly-supervised temporal action localization aims to accurately recognize and localize action segments in untrimmed videos, using only point-level annotations during training. Current methods primarily focus on mining sparse pseudo-labels and generating dense pseudo-labels. However, due to the sparsity of point-level labels and the impact of scene information on action representations, the reliability of dense pseudo-label methods still remains an issue. In this paper, we propose a point-level weakly-supervised temporal action localization method based on local representation enhancement and global temporal optimization. This method comprises two modules that enhance the representation capacity of action features and improve the reliability of class activation sequence classification, thereby enhancing the reliability of dense pseudo-labels and strengthening the model’s capability for completeness learning. Specifically, we first generate representative features of actions using pseudo-label feature and calculate weights based on the feature similarity between representative features of actions and segments features to adjust class activation sequence. Additionally, we maintain the fixed-length queues for annotated segments and design a action contrastive learning framework between videos. The experimental results demonstrate that our modules indeed enhance the model’s capability for comprehensive learning, particularly achieving state-of-the-art results at high IoU thresholds.

Cognitive Impairment Detection Based on Frontal Camera Scene While Performing Handwriting Tasks

Diagnosing cognitive impairment is an ongoing field of research especially in the elderly. Assessing the health status of the elderly can be a complex process that requires both subjective and objective measures. Subjective measures, such as self-reported responses to questions, can provide valuable information about a person’s experiences, feelings, and beliefs. However, from a scientific point of view, objective measures, based on quantifiable data that can be used to assess a person’s physical and cognitive functioning, are more appropriate and rigorous. The proposed system is based on the use of non-invasive instrumentation, which includes video images acquired with a frontal camera while the user performs different handwriting tasks on a Wacom tablet. We have acquired a new multimodal database of 191 elder subjects, which has been classified by human experts into healthy and cognitive impairment users by means of the standard pentagon copying test. The automatic classification was carried out using a video segmentation algorithm through the technique of shot boundary detection, in conjunction with a Transformer neural network. We obtain a multiclass classification accuracy of 77% and two-class accuracy of 83% based on frontal camera images, which basically detects head movements during handwriting tasks. Our automatic system can replicate human classification of handwritten pentagon copying test, opening a new method for cognitive impairment detection based on head movements. We also demonstrate the possibility to identifying the handwritten task performed by the user, based on frontal camera images and a Transformer neural network.