Video Object Research Articles

Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation

Recently, video object segmentation (VOS) referred by multi-modal signals, e.g., language and audio, has evoked increasing attention in both industry and academia. It is challenging for exploring the semantic alignment within modalities and the visual correspondence across frames. However, existing methods adopt separate network architectures for different modalities, and neglect the inter-frame temporal interaction with references. In this paper, we propose MUTR, a Multi-modal Unified Temporal transformer for Referring video object segmentation. With a unified framework for the first time, MUTR adopts a DETR-style transformer and is capable of segmenting video objects designated by either text or audio reference. Specifically, we introduce two strategies to fully explore the temporal relations between videos and multi-modal signals. Firstly, for low-level temporal aggregation before the transformer, we enable the multi-modal references to capture multi-scale visual cues from consecutive video frames. This effectively endows the text or audio signals with temporal knowledge and boosts the semantic alignment between modalities. Secondly, for high-level temporal interaction after the transformer, we conduct inter-frame feature communication for different object embeddings, contributing to better object-wise correspondence for tracking along the video. On Ref-YouTube-VOS and AVSBench datasets with respective text and audio references, MUTR achieves +4.2% and +8.7% J&F improvements to state-of-the-art methods, demonstrating our significance for unified multi-modal VOS. Code is released at https://github.com/OpenGVLab/MUTR.

Context Enhanced Transformer for Single Image Object Detection in Video Data

With the increasing importance of video data in real-world applications, there is a rising need for efficient object detection methods that utilize temporal information. While existing video object detection (VOD) techniques employ various strategies to address this challenge, they typically depend on locally adjacent frames or randomly sampled images within a clip. Although recent Transformer-based VOD methods have shown promising results, their reliance on multiple inputs and additional network complexity to incorporate temporal information limits their practical applicability. In this paper, we propose a novel approach to single image object detection, called Context Enhanced TRansformer (CETR), by incorporating temporal context into DETR using a newly designed memory module. To efficiently store temporal information, we construct a class-wise memory that collects contextual information across data. Additionally, we present a classification-based sampling technique to selectively utilize the relevant memory for the current image. In the testing, We introduce a test-time memory adaptation method that updates individual memory functions by considering the test distribution. Experiments with CityCam and ImageNet VID datasets exhibit the efficiency of the framework on various video systems. The project page and code will be made available at: https://ku-cvlab.github.io/CETR.

Generalizable Fourier Augmentation for Unsupervised Video Object Segmentation

The performance of existing unsupervised video object segmentation methods typically suffers from severe performance degradation on test videos when tested in out-of-distribution scenarios. The primary reason is that the test data in real- world may not follow the independent and identically distribution (i.i.d.) assumption, leading to domain shift. In this paper, we propose a generalizable fourier augmentation method during training to improve the generalization ability of the model. To achieve this, we perform Fast Fourier Transform (FFT) over the intermediate spatial domain features in each layer to yield corresponding frequency representations, including amplitude components (encoding scene-aware styles such as texture, color, contrast of the scene) and phase components (encoding rich semantics). We produce a variety of style features via Gaussian sampling to augment the training data, thereby improving the generalization capability of the model. To further improve the cross-domain generalization performance of the model, we design a phase feature update strategy via exponential moving average using phase features from past frames in an online update manner, which could help the model to learn cross-domain-invariant features. Extensive experiments show that our proposed method achieves the state-of-the-art performance on popular benchmarks.

Dynamic risk assessment of gas pipeline operation process by fusing visual and olfactory monitoring

With the rapid increase in urban gas consumption, the frequency of maintenance and repair of gas pipelines has escalated, leading to a rise in safety accidents during these processes. The traditional manual supervision model presents challenges such as inaccurate monitoring results, incomplete risk factor analysis, and a lack of quantitative risk assessment. This research focuses on developing a dynamic risk assessment technology for gas emergency repair operations by integrating the monitoring outcomes of artificial olfactory for gas leakage information and video object recognition for visual safety factor monitoring data. To quantitatively evaluate the risk of the operation process, a three-dimensional risk assessment model combining gas leakage with risk-correlated sensitivity was established as well as a separate three-dimensional risk assessment model integrating visual risk factors with predictable risk disposition. Furthermore, a visual risk quantification expression mode based on the risk matrix-radar map method was introduced. Additionally, a risk quantification model based on the fusion of visual and olfactory results was formulated. The verification results of simulation scenarios based on field data indicate that the visual-olfactory fusion risk assessment method can more accurately reflect the dynamic risk level of the operation process compared to simple visual safety factor monitoring. The outcomes of this research can contribute to the identification of safety status and early warning of risks related to personnel, equipment, and environmental factors in emergency repair operations. Moreover, these results can be extended to other operational scenarios, such as oil and gas production stations and long-distance pipeline operations.

Cascade transformers with dynamic attention for video question answering

Visual question answering (VQA) has become a hot study topic with challenging motivation of correctly answering the videos or images questions in recent years. However, the existing VQA model mostly aimed at answering questions about images and performed poorly in the video question answering (VideoQA) domain. VideoQA needs to simultaneously consider the correlations between video frames and the dynamic information of multiple objects in video. Therefore, we propose a novel Cascade Transformers with Dynamic Attention for Video Question Answering (CTDA-QA), which aims to simultaneously solve the above considerations. Specifically, the proposed CTDA-QA model utilizes multiple transformers structure to encode videos for reasoning complex spatial and temporal information, which is different from the previous recurrent neural network methods. Besides, in order to effectively capture the dynamic information from various scenarios in videos, a flexible attention module has been proposed to explore the essential relations between objects in a dynamic timeline. Finally, to avoid spurious answers and fully explore the cross-modal relationships, a mixed-supervised learning strategy is designed for optimizing the reasoning tasks. The experiments on several benchmark video question–answer datasets clearly verify the performance and effectiveness of CTDA-QA, which contains the results in contrast to the state-of-the-art methods. Besides, the provided ablation study and visualization results further reveal the potential of CTDA-QA.

Detection of an in-housed pig using modified YOLOv5 model

Object detection has made significant strides in recent years, but it remains a challenging task to accurately and quickly identify and detect objects. While humans can easily recognize objects in images or videos regardless of their appearance, computers face difficulties in this task. Object detection plays a crucial role in computer vision and finds applications in various domains such as healthcare, security, agriculture, home automation and more. To address the challenges of object detection, several techniques have been developed including RCNN, Faster RCNN, YOLO and Single Shot Detector (SSD). In this paper, we propose a modified YOLOv5s architecture that aims to improve detection performance. Our modified architecture incorporates the C3Ghost module along with the SPP and SPPF modules in the YOLOv5s backbone network. We also utilize the Adam and Stochastic Gradient Descent (SGD) optimizers. The paper also provides an overview of three major versions of the YOLO object detection model: YOLOv3, YOLOv4 and YOLOv5. We discussed their respective performance analyses. For our evaluation, we collected a database of pig images from the ICAR-National Research Centre on Pig farm. We assessed the performance using four metrics such as Precision (P), Recall (R), F1-score and mAP @ 0.50. The computational results demonstrate that our method YOLOv5s architecture achieves a 0.0414 higher mAP while utilizing less memory space compared to the original YOLOv5s architecture. This research contributes to the advancement of object detection techniques and showcases the potential of our modified YOLOv5s architecture for improved performance in real world applications.

A comparison of deep learning-based object detection for unmanned aerial vehicle

Unmanned aerial vehicle (UAV) plays a critical role in the field of object detection, and machine learning techniques have significantly advanced the field in recent years. This paper provides a comprehensive overview of machine learning developments for UAV object detection. The process involves multiple steps. Firstly, deep learning and Convolutional Neural Network (CNN) are widely utilized to extract precise features and train object detection models for accurate classification and efficient recognition of target objects in images and videos. Secondly, classical object detection algorithms such as Faster R-CNN, You Only Look Once (YOLO), and Single Shot MultiBox Detector (SSD), have been enhanced to improve accuracy and real-time performance. In this review, we primarily focus on comparing the principles of CNN and YOLOv5 themselves, as well as their applications in object detection and image recognition. Ultimately, it becomes apparent that CNN is better suited for processing image data, automatically extracting features, and achieving more accurate classification and detection. On the other hand, YOLOv5 directly performs detection on large images, significantly reducing computation time compared to CNN.

Modulated Memory Network for Video Object Segmentation

Existing video object segmentation (VOS) methods based on matching techniques commonly employ a reference set comprising historical segmented frames, referred to as ‘memory frames’, to facilitate the segmentation process. However, these methods suffer from the following limitations: (i) Inherent segmentation errors in memory frames can propagate and accumulate errors when utilized as templates for subsequent segmentation. (ii) The non-local matching technique employed in top-leading solutions often fails to incorporate positional information, potentially leading to incorrect matching. In this paper, we introduce the Modulated Memory Network (MMN) for VOS. Our MMN enhances matching-based VOS methods in the following ways: (i) Introducing an Importance Modulator, which adjusts memory frames using adaptive weight maps generated based on the segmentation confidence associated with each frame. (ii) Incorporating a Position Modulator that encodes spatial and temporal positional information for both memory frames and the current frame. The proposed modulator improves matching accuracy by embedding positional information. Meanwhile, the Importance Modulator mitigates error propagation and accumulation by incorporating confidence-based modulation. Through extensive experimentation, we demonstrate the effectiveness of our proposed MMN, which also achieves promising performance on VOS benchmarks.

Enhancing Embedded Object Tracking: A Hardware Acceleration Approach for Real-Time Predictability.

While Siamese object tracking has witnessed significant advancements, its hard real-time behaviour on embedded devices remains inadequately addressed. In many application cases, an embedded implementation should not only have a minimal execution latency, but this latency should ideally also have zero variance, i.e., be predictable. This study aims to address this issue by meticulously analysing real-time predictability across different components of a deep-learning-based video object tracking system. Our detailed experiments not only indicate the superiority of Field-Programmable Gate Array (FPGA) implementations in terms of hard real-time behaviour but also unveil important time predictability bottlenecks. We introduce dedicated hardware accelerators for key processes, focusing on depth-wise cross-correlation and padding operations, utilizing high-level synthesis (HLS). Implemented on a KV260 board, our enhanced tracker exhibits not only a speed up, with a factor of 6.6, in mean execution time but also significant improvements in hard real-time predictability by yielding 11 times less latency variation as compared to our baseline. A subsequent analysis of power consumption reveals our approach's contribution to enhanced power efficiency. These advancements underscore the crucial role of hardware acceleration in realizing time-predictable object tracking on embedded systems, setting new standards for future hardware-software co-design endeavours in this domain.

Fresnel incoherent compressive holography toward 3D videography via dual-channel simultaneous phase-shifting interferometry.

Fresnel incoherent correlation holography (FINCH) enables high-resolution 3D imaging of objects from several 2D holograms under incoherent light and has many attractive applications in motionless 3D fluorescence imaging. However, FINCH has difficulty implementing 3D imaging of dynamic scenes since multiple phase-shifting holograms need to be recorded for removing the bias term and twin image in the reconstructed scene, which requires the object to remain static during this progress. Here, we propose a dual-channel Fresnel noncoherent compressive holography method. First, a pair of holograms with π phase shifts obtained in a single shot are used for removing the bias term noise. Then, a physic-driven compressive sensing (CS) algorithm is used to achieve twin-image-free reconstruction. In addition, we analyze the reconstruction effect and suitability of the CS algorithm and two-step phase-shift filtering algorithm for objects with different complexities. The experimental results show that the proposed method can record hologram videos of 3D dynamic objects and scenes without sacrificing the imaging field of view or resolution. Moreover, the system refocuses images at arbitrary depth positions via computation, hence providing a new method for fast high-throughput incoherent 3D imaging.

An efficient algorithm for multi-scale maritime object detection and recognition

Due to the complexity of the maritime environment and the diversity of the volume and shape of monitored objects in the maritime, existing object detection algorithms based on Convolutional Neural Networks (CNN) are challenging to balance the requirements of high accuracy and high real-time simultaneously in the field of maritime object detection. In response to the characteristics of complex backgrounds, significant differences in object size between categories, and the characteristic of having a large number of small objects in maritime surveillance videos and images, the Maritime dataset with rich scenes and object categories was self-made, and the OS-YOLOv7 algorithm was proposed based on the YOLOv7 algorithm. Firstly, a feature enhancement module named the TC-ELAN module based on the self-attention mechanism was designed, which enables the feature map used for detection to obtain enhanced semantic information fused from multiple scale features. Secondly, in order to enhance the attention to the area of dense small objects and further improve the positioning accuracy of occluded small objects, this study redesigned the SPPCSPC structure. Then, the network structure was improved to alleviate the problem of decreased object detection accuracy caused by the loss of semantic feature information. Finally, experimental results on self-made datasets and mainstream maritime object detection datasets show that OS-YOLOv7 has a better object detection effect compared to other state-of-the-art (SOTA) object detection algorithms at the cost of reasonable inference time and parameter quantity and can achieve good object detection accuracy on mainstream datasets with high real-time performance.

Object Detection using an Advanced Deep Learning Algorithm: YOLOv4

This research proposes implementing YOLOv4, a real-time object detection system that can accurately and efficiently recognize objects in photos and videos. The goals include creating the YOLOv4 architecture, generating datasets, training the model, evaluating its performance, deploying it in real-world applications, and providing extensive documentation. Deep learning frameworks such as TensorFlow or PyTorch are used in the implementation, along with advanced techniques such as transfer learning and data augmentation. By curating annotated datasets and refining training techniques, the model hopes to attain high accuracy, precision, and recall in object detection tasks. Performance analysis will compare the model's outcomes to those of cutting-edge systems, assessing its strengths and weaknesses. The deployment phase involves integrating the model into existing systems or creating separate applications for real-world scenarios such as pedestrian and vehicle detection. Comprehensive documentation and a user guide will help developers and users make the best use of the trained model. Overall, this research intends to demonstrate YOLOv4's usefulness and feasibility in developing computer vision technology and supporting the creation of intelligent systems with real-time object identification capabilities.

Neural Radiance Fields (NeRF) for 3D Visualization Rendering Based on 2D Images

We are living in a 3D world, but oftentimes we visualize things in 2D scopes like in pictures and videos of different objects. The world of 3D visualization has been constantly evolving over the past few years, still growing with the introduction of cutting-edge computer vision and deep learning methods. Reconstructing 3D scenes and being able to interact with them in virtual environments used to be something beyond our capabilities but now with the use of Neural Radiance Fields (NeRF) and Instant Neural Graphic Primitives (InstantNGP) we are able to create a world in a virtual space, opening the gateways to applications in object detection and generation, augmented reality, medical imaging, and more. Access to these methods will allow for more immersive development by creators, hands-on educational resources, and more experimentation and research, stimulating future applications and techniques.

Suspicious Activity Tracking

Abstract: Our offering includes a deep neural network model that can identify firearms in photographs and a machine learning and computer vision pipeline that can detect abandoned luggage in order to identify potential gun-based crime and circumstances involving abandoned luggage in surveillance film. Unusual behavior the technique of identifying undesired human activity in locations and circumstances is called detection. To do this, footage is converted into frames, and the processed frames are then used to analyze the people's sports. YOLOv3 is used to find a niche, in dubious sports like lock breaking and bag snatching, among others. Our gadget has a superb processing pace in addition to appropriate accuracy of detection. It is harder for computers to detect things in videos compared to images because of issues like blurriness or things getting blocked. They propose a solution called Shot Video Object Detector, which is a faster kind of detector for videos. It works by combining information from nearby frames to make better guesses about where objects are. Unlike other methods, Shot Video Object Detector does this by figuring out how things move between frames and then using that info to combine features. It also creates new features by borrowing information directly from neighboring frames using a special structure. Automated surveillance in public areas plays a crucial role in upholding law and order and proactively identifying potential risks to the public. Not only does the procedure automatically identify and detect known crooks, but it also tracks people's and things' movements and uses machine learning algorithms to alert the authorities to any questionable activity

Occlusion-aware visual object tracking based on multi-template updating Siamese network

Visual object tracking is a crucial area of computer vision research. It aims to accurately track objects in videos with challenges such as occlusion, deformation, and lighting variations. Existing algorithms face difficulties when objects leave the camera or reappear after being occluded, and they also struggle to track objects with significant appearance changes. To address these issues, this study proposed a novel tracking algorithm. It combines tracking and detection, enabling global searching when objects are occluded or disappear and redetecting them when they reappear. A multi-template updating mechanism was used to adapt to changes in appearance. The study proposed the OTB100-AB and UAV123-AB datasets to evaluate the tracker's ability to handle target disappearance, along with the TNR metric. The proposed algorithm was evaluated on these datasets, as were OTB50 and UAV20L, which outperformed state-of-the-art algorithms and significantly improved tracking performance.

Mitigating Distractor Challenges in Video Object Segmentation through Shape and Motion Cues

The purpose of semi-supervised video object segmentation (VOS) is to predict and generate object masks in subsequent video frames after being provided with the initial frame’s object mask. Currently, mainstream methods leverage historical frame information for enhancing the network’s performance. However, this approach faces the following issues: (1) They often overlook important shape information, leading to decreased accuracy in segmenting object-edge areas. (2) They often use pixel-level motion estimation to guide the matching for addressing distractor objects. However, this brings heavy computation costs and struggle against occlusion or fast/blurry motion. For the first problem, this paper introduces an object shape extraction module that exploits both the high-level and low-level features to obtain object shape information, by which the shape information can be used to further refine the predicted masks. For the second problem, this paper introduces a novel object-level motion prediction module, in which it stores the representative motion features during the training stage, and predicts the object motion by retrieving them during the inference stage. We evaluate our method on benchmark datasets compared with recent state-of-the-art methods, and the results demonstrate the effectiveness of the proposed method.

Evolution of CNNs in human tracking applications

Tracking the movement of objects in videos or footage from CCTV systems plays an integral role in crime investigations, surveillance, security predictions, and many other domains. Historically, this task was primarily entrusted to dedicated observers or analysts who would be summoned to review pre-recorded footage post-event. With the advent of machine learning and AI, convolutional neural networks (CNNs) have paved the way for computers to augment human capabilities in analyzing streaming videos or archived recordings. Among the various tracking methodologies that machine learning offers, YOLO (You Only Look Once) and R-CNN (Region-based Convolutional Neural Networks), along with its iterations, stand out as some of the most reliable and precise. However, the scope of analysis often extends beyond these techniques. To enhance accuracy and provide an adept classification mechanism, the Deep SORT (Simple Online and Realtime Tracking) algorithm emerges as pivotal. Its synergy with human detection remains a significant area of discussion and will be deliberated upon in this study. This review aims to elucidate the intricacies of these state-of-the-art methods and their interplay in modern tracking systems.

STAVOS: A Medaka Larval Cardiac Video Segmentation Method Based on Deep Learning

Medaka (Oryzias latipes), as a crucial model organism in biomedical research, holds significant importance in fields such as cardiovascular diseases. Currently, the analysis of the medaka ventricle relies primarily on visual observation under a microscope, involving labor-intensive manual operations and visual assessments that are cumbersome and inefficient for biologists. Despite attempts by some scholars to employ machine learning methods, limited datasets and challenges posed by the blurred edges of the medaka ventricle have constrained research to relatively simple tasks such as ventricle localization and heart rate statistics, lacking precise segmentation of the medaka ventricle edges. To address these issues, we initially constructed a video object segmentation dataset comprising over 7000 microscopic images of medaka ventricles. Subsequently, we proposed a semi-supervised video object segmentation model named STAVOS, incorporating a spatial-temporal attention mechanism. Additionally, we developed an automated system capable of calculating various parameters and visualizing results for a medaka ventricle using the provided video. The experimental results demonstrate that STAVOS has successfully achieved precise segmentation of medaka ventricle contours. In comparison to the conventional U-Net model, where a mean accuracy improvement of 0.392 was achieved, our model demonstrates significant progress. Furthermore, when compared to the state-of-the-art Tackling Background Distraction (TBD) model, there is an additional enhancement of 0.038.

Words and arbitrary actions in early object categorization: weak evidence for a word advantage.

Both words and gestures have been shown to influence object categorization, often even overriding perceptual similarities to cue category membership. However, gestures are often meaningful to infants while words are arbitrarily related to an object they refer to, more similar to arbitrary actions that can be performed on objects. In this study, we examine how words and arbitrary actions shape category formation. Across three conditions (word cue, action cue, word-action cue), we presented infants (N = 90) with eight videos of single-category objects which vary in colour and other perceptual features. The objects were either accompanied by a word and/or an action that is being performed on the object. Infants in the word and action condition showed a decrease in looking over the course of the familiarization phase indicating habituation to the category, but infants in the word-action condition did not. At test, infants saw a novel object of the just-learned category and a novel object from another category side-by-side on the screen. There was some evidence for an advantage for words in shaping early object categorization, although we note that this was not robust across analyses.

Multibeam Forward-Looking Sonar Video Object Tracking Using Truncated - Sparsity and Aberrances Repression Regularization

Multibeam Forward-Looking Sonar Video Object Tracking Using Truncated - Sparsity and Aberrances Repression Regularization