Video Segmentation Research Articles

Real-time segmentation of short videos under VR technology in dynamic scenes

Abstract This work addresses the challenges of scene segmentation and low segmentation accuracy in short videos by employing virtual reality (VR) technology alongside a 3D DenseNet model for real-time segmentation in dynamic scenes. First, this work extracted short videos by frame and removed redundant background information. Then, the volume rendering algorithm in VR technology was used to reconstruct short videos in dynamic scenes in 3D. It enriched the detailed information of short videos, and finally used the 3D DenseNet model for real-time segmentation of short videos in dynamic scenes, improving the accuracy of segmentation. The experiment compared the performance of High resolution network, Mask region based convolutional neural network, 3D U-Net, Efficient neural network models on the Densely annotation video segmentation dataset. The experimental results showed that the segmentation accuracy of the 3D DenseNet model has reached 99.03%, which was 15.11% higher than that of the ENet model. The precision rate reached 98.33%, and the average segmentation time reached 0.64 s, improving the segmentation accuracy and precision rate. It can adapt to various scene situations and has strong robustness. The significance of this research lies in its innovative approach in tackling these issues. By integrating VR technology with advanced deep learning models, we can achieve more precise segmentation of dynamic scenes in short videos, enabling real-time processing. This has significant practical implications for fields such as video editing, VR applications, and intelligent surveillance. Furthermore, the outcomes of this research contribute to advancing computer vision in video processing, providing valuable insights for the development of future intelligent video processing systems.

Lightweight and real-time semantic segmentation of UAV traffic videos based on siamese network for keyframe recognition

Lightweight and real-time semantic segmentation of UAV traffic videos based on siamese network for keyframe recognition

BONES: Near-Optimal Neural-Enhanced Video Streaming

Accessing high-quality video content can be challenging due to insufficient and unstable network bandwidth. Recent advances in neural enhancement have shown promising results in improving the quality of degraded videos through deep learning. Neural-Enhanced Streaming (NES) incorporates this new approach into video streaming, allowing users to download low-quality video segments and then enhance them to obtain high-quality content without violating the playback of the video stream. We introduce BONES, an NES control algorithm that jointly manages the network and computational resources to maximize the quality of experience (QoE) of the user. BONES formulates NES as a Lyapunov optimization problem and solves it in an online manner with near-optimal performance, making it the first NES algorithm to provide a theoretical performance guarantee. Comprehensive experimental results indicate that BONES increases QoE by 5% to 20% over state-of-the-art algorithms with minimal overhead. Our code is available at https://github.com/UMass-LIDS/bones.

Edwards curve digital signature algorithm for video integrity verification on blockchain framework

Within the field of forensic science, video integrity is an essential component that ensures the validity and dependability of visual evidence that is essential for court cases. Maintaining the integrity of justice in a time when digital modification tools are easily accessible requires making sure that video recordings are unaltered. Video footage is carefully inspected by forensic analysts who use advanced tools to look for indications of manipulation or change. This paper presents a novel, lightweight approach for verifying the integrity of video data. The proposed method utilizes the blockchain and the Edwards Curve Digital Signature Algorithm, coupled with the BLAKE2b hash function. Signatures are generated for video clips with a predetermined size called video segments and stored in blocks chronologically. The signature of the previous block is stored in the current block to add another layer of security. At the time of validation, these signatures are verified. Experimental results show that the proposed method outperforms state-of-the-art methods in both speed and security. Our method is capable of identifying any type of forgery on any video file, by anyone, at any given moment, with insignificant additional storage requirements. Furthermore, our security analysis shows that our method is resilient against various types of attacks, including collision attacks, key substitution attacks, side-channel attacks, and chosen message attacks. The proposed lightweight video integrity verification method is better suited for use in resource constrained devices.

МЕТОД ДВОКАСКАДНОЇ ІМПЛАНТАЦІЇ ПРИХОВАНОЇ ІНФОРМАЦІЇ НА ОСНОВІ СТЕГАНОКОМПРЕСІЙНИХ ПЕРЕТВОРЕНЬ

In the article, the need for further development of the state is directly related to: solving issues of increasing defense capability and information security; development of information and intelligent systems. The necessity of ensuring the required level of completeness of information, compliance with the requirements of its relevance, achievement and maintenance of the appropriate level of integrity, accessibility and confidentiality is substantiated. The article emphasizes that for this purpose, complex systems of coding and information protection are being built. It is shown that recently, in order to further increase the level of security of information resources, methods of hidden embedding of information have been used. In this field of scientific and applied research, the direction of timely delivery of integral video information in a secure mode is outlined. For protection, the following can be used: meta-messages, which are formed on the basis of intelligent analysis of video frames; separate video segments of aerial photographs containing the most important information for decision-making. This article examines a class of methods of steganographic transformations, which are associated with embedding messages in digital containers, which are formed by a stream of video segments (VS). However, existing steganographic systems are based mainly on the use of the amount of psychovisual (PSV) redundancy available for reduction. Therefore, an increase in steganographic capacity leads to a loss of integrity and efficiency of delivery of video-container information. Steganocompression coding in the stegano-polyadic basis was created on the basis of taking into account the amount of redundancy permissible for reduction. In this case, in the process of compression of the components of the VS, the direct embedding of hidden information is organized. Concealment of information is carried out by technological stages, depending on the compression transformation of the transitional syntax of the quilted sequence. The key stage is the technology of two-stage implementation of the elements of the concealed message added to the stegano sequences to the syntax of the steganocompression representation of the BC components.

Exploiting relation of video segments for temporal action detection

With the development of urban intelligence, intelligent identification of human activities has become imminent. Temporal Action Detection (TAD) is designed to identify real human social activities, which is a challenging task in video understanding. The current methods mainly use global features for boundary matching or predefine all possible proposals, while ignoring the interference of background and the causal relevance of the temporal action, resulting in the generation of more redundant proposals and the decline of detection accuracy. To fill this gap, we propose a novel Dilated Convolution Locate and Action Relevant Score model called DCAR. Specifically, DCAR includes a Dilated Location Network (DL-Net) and Action Relevance Calculation (ARC) block. For the DL-Net, we design a Boundary Feature Enhancement (BFE) block, which enhances the boundary feature of actions and fuses the similar features of different channels by pooling and channel squeeze to reduce the interference of the background. Meanwhile, we also design multiple dilated convolutional structures to aggregate long contextual information in time point/interval after boundary enhancement. For the ARC block, we use the hyperbolic space distance and cross attention to calculate the causal correlation of action proposals classification, which removes the misclassification of action proposals. We conduct extensive experiments on Thumos14 and ActivityNet-1.3, which shows our method significantly improves the performance and achieves state-of-the-art results. On Thumos14, at tIoU 0.3, 0.4 and 0.5, it achieves mAP of 63.2%, 57.0% and 48.5%, respectively; On ActivityNet-1.3 it reaches 9.42% at tIoU 0.95.

Weakly Supervised Video Anomaly Detection via Self-Guided Temporal Discriminative Transformer.

Weakly supervised video anomaly detection is generally formulated as a multiple instance learning (MIL) problem, where an anomaly detector learns to generate frame-level anomaly scores under the supervision of MIL-based video-level classification. However, most previous works suffer from two drawbacks: 1) they lack ability to model temporal relationships between video segments and 2) they cannot extract sufficient discriminative features to separate normal and anomalous snippets. In this article, we develop a weakly supervised temporal discriminative (WSTD) paradigm, that aims to leverage both temporal relation and feature discrimination to mitigate the above drawbacks. To this end, we propose a transformer-styled temporal feature aggregator (TTFA) and a self-guided discriminative feature encoder (SDFE). Specifically, TTFA captures multiple types of temporal relationships between video snippets from different feature subspaces, while SDFE enhances the discriminative powers of features by clustering normal snippets and maximizing the separability between anomalous snippets and normal centers in embedding space. Experimental results on three public benchmarks indicate that WSTD outperforms state-of-the-art unsupervised and weakly supervised methods, which verifies the superiority of the proposed method.

Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding

Grounding temporal video segments described in natural language queries effectively and efficiently is a crucial capability needed in vision-and-language fields. In this paper, we deal with the fast video temporal grounding (FVTG) task, aiming at localizing the target segment with high speed and favorable accuracy. Most existing approaches adopt elaborately designed cross-modal interaction modules to improve the grounding performance, which suffer from the test-time bottleneck. Although several common space-based methods enjoy the high-speed merit during inference, they can hardly capture the comprehensive and explicit relations between visual and textual modalities. In this paper, to tackle the dilemma of speed-accuracy tradeoff, we propose a commonsense-aware cross-modal alignment network (C 2 AN), which incorporates commonsense-guided visual and text representations into a complementary common space for fast video temporal grounding. Specifically, the commonsense concepts are explored and exploited by extracting the structural semantic information from a language corpus. Then, a commonsense-aware interaction module is designed to obtain bridged visual and text features by utilizing the learned commonsense concepts. Finally, to maintain the original semantic information of textual queries, a cross-modal complementary common space is optimized to obtain matching scores for performing FVTG. Extensive results on two challenging benchmarks show that our C 2 AN method performs favorably against state-of-the-arts while running at high speed. Our code is available at https://github.com/ZiyueWu59/CCA.

A hands-on guide to use network video recorders, internet protocol cameras, and deep learning models for dynamic monitoring of trout and salmon in small streams.

This study outlines a method for using surveillance cameras and an algorithm that calls a deep learning model to generate video segments featuring salmon and trout in small streams. This automated process greatly reduces the need for human intervention in video surveillance. Furthermore, a comprehensive guide is provided on setting up and configuring surveillance equipment, along with instructions on training a deep learning model tailored to specific requirements. Access to video data and knowledge about deep learning models makes monitoring of trout and salmon dynamic and hands-on, as the collected data can be used to train and further improve deep learning models. Hopefully, this setup will encourage fisheries managers to conduct more monitoring as the equipment is relatively cheap compared with customized solutions for fish monitoring. To make effective use of the data, natural markings of the camera-captured fish can be used for individual identification. While the automated process greatly reduces the need for human intervention in video surveillance and speeds up the initial sorting and detection of fish, the manual identification of individual fish based on natural markings still requires human effort and involvement. Individual encounter data hold many potential applications, such as capture-recapture and relative abundance models, and for evaluating fish passages in streams with hydropower by spatial recaptures, that is, the same individual identified at different locations. There is much to gain by using this technique as camera captures are the better option for the fish's welfare and are less time-consuming compared with physical captures and tagging.

SDN: Semantic Decoupling Network for Temporal Language Grounding.

Temporal language grounding (TLG) is one of the most challenging cross-modal video understanding tasks, which aims at retrieving the most relevant video segment from an untrimmed video according to a natural language sentence. The existing methods can be separated into two dominant types: 1) proposal-based and 2) proposal-free methods, where the former conduct contextual interactions and the latter localizes timestamps flexibly. However, the constant-scale candidates in proposal-based methods limit the localization precision and bring extra computational costs. In contrast, the proposal-free methods perform well on high-precision metrics-based on the fine-grained features but suffer from a lack of coarse-grained interactions, which cause degeneration when the video becomes complex. In this article, we propose a novel framework termed semantic decoupling network (SDN) that combines the advantages of proposal-based and proposal-free methods and overcomes their defects. It contains three key components: 1) semantic decoupling module (SDM); 2) context modeling block (CMB); and 3) semantic cross-level aggregation module (SCAM). By capturing the video-text contexts in multilevel semantics, the SDM and CMB effectively utilize the benefits of proposal-based methods. Meanwhile, the SCAM maintains the merit of proposal-free methods in that it localizes timestamps precisely. The experiments on three challenge datasets, i.e., Charades-STA, TACoS, and ActivityNet-Caption, show that our proposed SDN method significantly outperforms recent state-of-the-art methods, especially the proposal-free methods. Extensive analyses, as well as the implementation code of the proposed SDN method, are provided at https://github.com/CFM-MSG/Code_SDN.

Image and Video Segmentation Using Yolo-Nas and Segment Anything Model (Sam): Machine Learning

In this article, we evaluate the effectiveness of computer vision and machine learning, which are revolutionizing photography and video studies, ushering in an unprecedented era of creativity and informed consent for study. Despite many advances in the field, object recognition and classification have become technologies with many applications, from advanced analytics to enhancing driving safety for augmented reality. This research uses two advanced models, YOLO-NAS (You Just Look at Neural Architecture Search) and Segment One (SAM), to analyze the evolution of image and video segmentation. In real-time search of products and components, YOLO-NAS and SAM models have established a good relationship and are expected to become a unified system for image and video segmentation in digital transformation. Key Words: YOLO-NAS, Convolutional Neural Networks (Cnns), Object Recognition, Computer Vision, Spatial Attention Module (SAM), Instance Video Segmentation, And Image Segmentation.

VSD: A Novel Method for Video Segmentation and Storage in DNA Using RS Code

As data continue to grow in complexity and size, there is an imperative need for more efficient and robust storage solutions. DNA storage has emerged as a promising avenue to solve this problem, but existing approaches do not perform efficiently enough on video data, particularly for information density and time efficiency. This paper introduces VSD, a pioneering encoding method for video segmentation and storage in DNA, leveraging the Reed–Solomon (RS) error correction code. This method addresses these limitations through an innovative combination of segmentation and encoding, accompanied by RS coding to bolster error resilience. Additionally, the method ensures that the GC-content of the resultant DNA sequences remains around 50%, which further enhances the storage robustness. The experimental results demonstrate the method has commendable encoding efficiency and offers a solution to the prevailing issue of time inefficiency and error correction rates in DNA storage. This groundbreaking approach paves the way for the practical and reliable storage of large-scale video data in DNA, heralding a new era in the domain of information storage.

Multi-Level interaction network for temporal sentence grounding in videos

Temporal sentence grounding in videos (TSGV), which aims to retrieve video segments from an untrimmed videos that semantically match a given query. Most previous methods focused on learning either local or global query features and then performed cross-modal interaction, but ignore the complementarity between local and global features. In this paper, we propose a novel Multi-Level Interaction Network for Temporal Sentence Grounding in Videos. This network explores the semantics of queries at both phrase and sentence levels, interacting phrase-level features with video features to highlight video segments relevant to the query phrase and sentence-level features with video features to learn more about global localization information. A stacked fusion gate module is designed, which effectively captures the temporal relationships and semantic information among video segments. This module also introduces a gating mechanism to enable the model to adaptively regulate the fusion degree of video features and query features, further improving the accuracy of predicting the target segments. Extensive experiments on the ActivityNet Captions and Charades-STA benchmark datasets demonstrate that the proposed method outperforms the state-of-the-art methods.

Research on trajectory control of multi‐degree‐of‐freedom industrial robot based on visual image

AbstractIn order to improve the trajectory control effect of multi‐degree‐of‐freedom industrial robots, this paper combines visual image technology to conduct research on trajectory control of multi‐degree‐of‐freedom industrial robots. Aiming at the problem of video segmentation under sudden illumination changes, this paper uses a Gaussian mixture model based on the global illumination function to adopt a variety of illumination invariant features, and proposes a scene segmentation algorithm suitable for sudden illumination changes. Moreover, this paper compares and verifies the algorithm from the subjective and objective perspectives through experiments, which shows that the algorithm in this paper can segment the scene more accurately even in the environment of sudden changes in illumination. In addition, the results of the accuracy test and the trajectory control test show that the research method of the multi‐degree‐of‐freedom industrial robot trajectory control based on the visual image proposed in this paper can effectively improve the trajectory control effect of the robot.

From Microservice to Monolith: A Multivocal Literature Review

Recently, the phenomenon of switching back from microservice to monolith has increased in frequency, leading to intense debate in the industry. In this paper, we conduct a multivocal literature review to investigate reasoning and key aspects to pay attention to when switching back and analyze other practitioners’ opinions. The results show four cases of switching back from microservice to monolith: Istio control plane, Amazon Prime Video monitoring service, Segment, and InVision. The five main reasons that led to switching back are cost, complexity, scalability, performance, and organization. During the switching back process, six key aspects need to be addressed: (1) stopping the development of more services, (2) consolidating and testing paths, (3) unifying data storage, (4) implementing the message bus principle, (5) giving up diverse techniques, and (6) learning to use modular design principles. As to the practitioners’ opinions, they had mixed views about the switching back phenomenon. However, most thought that switching back required consideration of the actual system situation and principles. These results pave the way for further research and guide researchers and companies through the process of switching back from microservice to monolith.

Learning Reliable Dense Pseudo-Labels for Point-Level Weakly-Supervised Action Localization

Point-level weakly-supervised temporal action localization aims to accurately recognize and localize action segments in untrimmed videos, using only point-level annotations during training. Current methods primarily focus on mining sparse pseudo-labels and generating dense pseudo-labels. However, due to the sparsity of point-level labels and the impact of scene information on action representations, the reliability of dense pseudo-label methods still remains an issue. In this paper, we propose a point-level weakly-supervised temporal action localization method based on local representation enhancement and global temporal optimization. This method comprises two modules that enhance the representation capacity of action features and improve the reliability of class activation sequence classification, thereby enhancing the reliability of dense pseudo-labels and strengthening the model’s capability for completeness learning. Specifically, we first generate representative features of actions using pseudo-label feature and calculate weights based on the feature similarity between representative features of actions and segments features to adjust class activation sequence. Additionally, we maintain the fixed-length queues for annotated segments and design a action contrastive learning framework between videos. The experimental results demonstrate that our modules indeed enhance the model’s capability for comprehensive learning, particularly achieving state-of-the-art results at high IoU thresholds.

Cognitive Impairment Detection Based on Frontal Camera Scene While Performing Handwriting Tasks

Diagnosing cognitive impairment is an ongoing field of research especially in the elderly. Assessing the health status of the elderly can be a complex process that requires both subjective and objective measures. Subjective measures, such as self-reported responses to questions, can provide valuable information about a person’s experiences, feelings, and beliefs. However, from a scientific point of view, objective measures, based on quantifiable data that can be used to assess a person’s physical and cognitive functioning, are more appropriate and rigorous. The proposed system is based on the use of non-invasive instrumentation, which includes video images acquired with a frontal camera while the user performs different handwriting tasks on a Wacom tablet. We have acquired a new multimodal database of 191 elder subjects, which has been classified by human experts into healthy and cognitive impairment users by means of the standard pentagon copying test. The automatic classification was carried out using a video segmentation algorithm through the technique of shot boundary detection, in conjunction with a Transformer neural network. We obtain a multiclass classification accuracy of 77% and two-class accuracy of 83% based on frontal camera images, which basically detects head movements during handwriting tasks. Our automatic system can replicate human classification of handwritten pentagon copying test, opening a new method for cognitive impairment detection based on head movements. We also demonstrate the possibility to identifying the handwritten task performed by the user, based on frontal camera images and a Transformer neural network.

MEDUSA: A Dynamic Codec Switching Approach in HTTP Adaptive Streaming

HTTP Adaptive Streaming (HAS) solutions utilize various Adaptive BitRate (ABR) algorithms to dynamically select appropriate video representations, aiming to adapt to fluctuations in network bandwidth. However, current ABR implementations have a limitation in that they are designed to function with one set of video representations, i.e. , the bitrate ladder, which differ in bitrate and resolution, but are encoded with the same video codec. When multiple codecs are available, current ABR algorithms select one of them prior to the streaming session and stick to it throughout the entire streaming session. Although newer codecs are generally preferred over older ones, their compression efficiencies differ depending on the content’s complexity, which varies over time. Therefore, it is necessary to select the appropriate codec for each video segment to reduce the requested data while delivering the highest possible quality. In this paper, we first provide a practical example where we compare compression efficiencies of different codecs on a set of video sequences. Based on this analysis, we formulate the optimization problem of selecting the appropriate codec for each user and video segment (on a per-segment basis in the outmost case), refining the selection of the ABR algorithms by exploiting key metrics, such as the perceived segment quality and size. Subsequently, to address the scalability issues of this centralized model, we introduce a novel distributed plug-in ABR algorithm for Video on Demand (VoD) applications called MEDUSA to be deployed on top of existing ABR algorithms. MEDUSA enhances the user’s Quality of Experience (QoE) by utilizing a multi-objective function that considers the quality and size of video segments when selecting the next representation. Using quality information and segment size from the modified Media Presentation Description (MPD) , MEDUSA utilizes buffer occupancy to prioritize quality or size by assigning specific weights in the objective function. To show the impact of MEDUSA, we compare the proposed plug-in approach on top of state-of-the-art techniques with their original implementations and analyze the results for different network traces, video content, and buffer capacities. According to the experimental findings, MEDUSA shows the ability to improve QoE for various test videos and scenarios. The results reveal an impressive improvement in the QoE score of up to 42% according to the ITU-T P.1203 model (mode 0). Additionally, MEDUSA can reduce the transmitted data volume by up to more than 40% achieving a QoE similar to the techniques compared, reducing the burden on streaming service providers for delivery costs.

Basketball Tactics Analysis Based on Improved Openpose Algorithm and Its Application

The use of information technology has currently resulted in a change in people's production and lifestyles as well as a catalyst for the sports industry's evolution. Currently, there is a growing use of digitization in the sports industry's basketball realm. Using 2D posture estimation, a novel lightweight deep learning architecture (LDLA) is built to accomplish the automatic analysis of basketball game footage. We offer a real-time method to identify numerous people's 2D poses in a video. Additionally, it examines how 2D pose estimation might be used to analyse basketball shooting videos. First, group and global motion features are extracted to represent semantic events. A full basketball game video is broken down into three stages: clip-based segmentation, classification of semantics occurrences utilizing audio and video, and temporal sequence characteristics employing GRU CNN. Video is first processed to create some basic video categorization and segmentation using the visual, movement, and auditory data. Further use of domain knowledge is made to find important events in the basketball video. Shot and image threshold recognition algorithms for videos have used both optical and kinematic prediction information; scene classification comes next. The placements of probable semantic events, such as "fouling" and "shooting at the basket," are then discovered by comparing the multidimensional data with supplementary domain expertise. Experimental results demonstrate the proposed LDLA method achieves 99.6% of accuracy, 93.2% of precision, 93.5% of recall and 89.5% of F1-score.

IMF-MF: Interactive moment localization with adaptive multimodal fusion and self-attention

Precise video moment retrieval is crucial for enabling users to locate specific moments within a large video corpus. This paper presents Interactive Moment Localization with Multimodal Fusion (IMF-MF), a novel interactive moment localization with multimodal fusion model that leverages the power of self-attention to achieve state-of-the-art performance. IMF-MF effectively integrates query context and multimodal features, including visual and audio information, to accurately localize moments of interest. The model operates in two distinct phases: feature fusion and joint representation learning. The first phase dynamically calculates fusion weights for adapting the combination of multimodal video content, ensuring that the most relevant features are prioritized. The second phase employs bi-directional attention to tightly couple video and query features into a unified joint representation for moment localization. This joint representation captures long-range dependencies and complex patterns, enabling the model to effectively distinguish between relevant and irrelevant video segments. The effectiveness of IMF-MF is demonstrated through comprehensive evaluations on three benchmark datasets: TVR for closed-world TV episodes and Charades for open-world user-generated videos, DiDeMo dataset, Open-world, diverse video moment retrieval dataset. The empirical results indicate that the proposed approach surpasses existing state-of-the-art methods in terms of retrieval accuracy, as evaluated by metrics like Recall (R1, R5, R10, and R100) and Intersection-of-Union (IoU). The results consistently demonstrate IMF-MF’s superior performance compared to existing state-of-the-art methods, highlighting the benefits of its innovative interactive moment localization approach and the use of self-attention for feature representation and attention modeling.