Moving Object Segmentation Research Articles

Moving Object Segmentation Network for Multiview Fusion of Vehicle-Mounted LiDAR

Moving Object Segmentation Network for Multiview Fusion of Vehicle-Mounted LiDAR

Fast Motion State Estimation Based on Point Cloud by Combing Deep Learning and Spatio-Temporal Constraints

Moving objects in the environment have a higher priority and more challenges in growing domains like unmanned vehicles and intelligent robotics. Estimating the motion state of objects based on point clouds in outdoor scenarios is currently a challenging area of research. This is due to factors such as limited temporal information, large volumes of data, extended network processing times, and the ego-motion. The number of points in a point cloud frame is typically 60,000–120,000 points, but most current motion state estimation methods for point clouds only downsample to a few thousand points for fast processing. The downsampling step will lead to the loss of scene information, which means these methods are far from being used in practical applications. Thus, this paper proposes a motion state estimation method that combines spatio-temporal constraints and deep learning. It starts by estimating and compensating the ego-motion of multi-frame point cloud data and mapping multi-frame data to a unified coordinate system; then the point cloud motion segmentation model on the multi-frame point cloud is proposed for motion object segmentation. Finally, spatio-temporal constraints are utilized to correlate the moving object at different moments and estimate the motion vectors. Experiments on KITTI, nuScenes, and real captured data show that the proposed method has good results, with an average vector deviation of only 0.036 m and 0.043 m in KITTI and nuScenes under a processing time of about 80 ms. The EPE3D error under the KITTI data is only 0.076 m, which proves the effectiveness of the method.

MosViT: towards vision transformers for moving object segmentation based on Lidar point cloud

Abstract Moving object segmentation is fundamental for various downstream tasks in robotics and autonomous driving, providing crucial information for them. Effectively extracting spatial-temporal information from consecutive frames and addressing the scarcity of dataset is paramount for learning-based 3D LiDAR Moving Object Segmentation (LIDAR-MOS). In this work, we propose a novel deep neural network based on Vision Transformers (ViTs) to tackle this problem. We first validate the feasibility of Transformer networks for this task, offering an alternative to CNNs. Specifically, we utilize a dual-branch structure based on range-image data to extract spatial-temporal information from consecutive frames and fuse it using a motion-guided attention mechanism. Furthermore, we employ the ViT as the backbone, keeping its architecture unchanged from what is used for RGB images. This enables us to leverage pre-trained models from RGB images to improve results, addressing the issue of limited LIDAR point cloud data, which is cheaper compared to acquiring and annotating point cloud data. We validate the effectiveness of our approach on the LIDAR-MOS benchmark of SemanticKitti and achieve comparable results to methods that use CNNs on range image data. The source code and trained models are available at https://github.com/mafangniu/MOSViT.git.

An Intelligent Moving Object Segmentation Using Hybrid IFCM-CSS Clustering Model

In this study, a novel hybrid deep clustering approach is proposed for the effective moving object segmentation. Initially, the data is collected, and the keyframe selection is performed using the threshold-based Kennard–Stone method. Then, the preprocessing step involves noise filtering using bilateral wavelet thresholding and binary color conversion. The blob detection is performed using normalized Laplacian of Gaussian. Finally, the segmentation of moving objects is performed using a hybrid clustering approach called improved fuzzy C-mean (IFCM) clustering with chaotic salp swarm (CSS) optimization algorithm (Hybrid IFCM-CSS). The overall evaluation is done in MATLAB. The performance of the hybrid IFCM-CSS is compared to other approaches based on some measures. The proposed Hybrid IFCM-CSS achieves the highest precision of 0.971, using the SBM-RGBD dataset.

RadarMOSEVE: A Spatial-Temporal Transformer Network for Radar-Only Moving Object Segmentation and Ego-Velocity Estimation

Moving object segmentation (MOS) and Ego velocity estimation (EVE) are vital capabilities for mobile systems to achieve full autonomy. Several approaches have attempted to achieve MOSEVE using a LiDAR sensor. However, LiDAR sensors are typically expensive and susceptible to adverse weather conditions. Instead, millimeter-wave radar (MWR) has gained popularity in robotics and autonomous driving for real applications due to its cost-effectiveness and resilience to bad weather. Nonetheless, publicly available MOSEVE datasets and approaches using radar data are limited. Some existing methods adopt point convolutional networks from LiDAR-based approaches, ignoring the specific artifacts and the valuable radial velocity information of radar measurements, leading to suboptimal performance. In this paper, we propose a novel transformer network that effectively addresses the sparsity and noise issues and leverages the radial velocity measurements of radar points using our devised radar self- and cross-attention mechanisms. Based on that, our method achieves accurate EVE of the robot and performs MOS using only radar data simultaneously. To thoroughly evaluate the MOSEVE performance of our method, we annotated the radar points in the public View-of-Delft (VoD) dataset and additionally constructed a new radar dataset in various environments. The experimental results demonstrate the superiority of our approach over existing state-of-the-art methods. The code is available at https://github.com/ORCAUboat/RadarMOSEVE.

3D convolutional long short-term encoder-decoder network for moving object segmentation

Moving object segmentation (MOS) is one of the important and wellstudied computer vision tasks that is used in a variety of applications, such as video surveillance systems, human tracking, self-driving cars, and video compression. While traditional approaches to MOS rely on hand-crafted features or background modeling, deep learning methods using Convolution Neural Networks (CNNs) have been shown to be more effective in extracting features and achieving better accuracy. However, most deep learning-based methods for MOS offer scene-dependent solutions, leading to reduced performance when tested on previously unseen video content. Because spatial features are insufficient to represent the motion information, the spatial and temporal features should be used together to succeed in unseen videos. To address this issue, we propose the MOS-Net deep framework, an encoder-decoder network that combines spatial and temporal features using the flux tensor algorithm, 3D CNNs, and ConvLSTM in its different variants. MOS-Net 2.0 is an enhanced version of the base MOS-Net structure, where additional ConvLSTM modules are added to 3D CNNs for extracting long-term spatiotemporal features. In the final stage of the framework the output of the encoder-decoder network, the foreground probability map, is thresholded for producing a binary mask where moving objects are in the foreground and the rest forms the background. In addition, an ablation study has been conducted to evaluate different combinations as inputs to the proposed network, using the ChangeDetection2014 (CDnet2014) which includes challenging videos such as those with dynamic backgrounds, bad weather, and illumination changes. In most approaches, the training and test strategy are not announced, making it difficult to compare the algorithm results. In addition, the proposed method can be evaluated differently as video-optimized or video-agnostic. In video-optimized approaches, the training and test set is obtained randomly and separated from the overall dataset. The results of the proposed method are compared with competitive methods from the literature using the same evaluation strategy. It has been observed that the introduced MOS networks give highly competitive results on the CDnet2014 dataset. The source code for the simulations provided in this work is available online.

Learning Temporal Distribution and Spatial Correlation Toward Universal Moving Object Segmentation.

The goal of moving object segmentation is separating moving objects from stationary backgrounds in videos. One major challenge in this problem is how to develop a universal model for videos from various natural scenes since previous methods are often effective only in specific scenes. In this paper, we propose a method called Learning Temporal Distribution and Spatial Correlation (LTS) that has the potential to be a general solution for universal moving object segmentation. In the proposed approach, the distribution from temporal pixels is first learned by our Defect Iterative Distribution Learning (DIDL) network for a scene-independent segmentation. Notably, the DIDL network incorporates the use of an improved product distribution layer that we have newly derived. Then, the Stochastic Bayesian Refinement (SBR) Network, which learns the spatial correlation, is proposed to improve the binary mask generated by the DIDL network. Benefiting from the scene independence of the temporal distribution and the accuracy improvement resulting from the spatial correlation, the proposed approach performs well for almost all videos from diverse and complex natural scenes with fixed parameters. Comprehensive experiments on standard datasets including LASIESTA, CDNet2014, BMC, SBMI2015 and 128 real world videos demonstrate the superiority of proposed approach compared to state-of-the-art methods with or without the use of deep learning networks. To the best of our knowledge, this work has high potential to be a general solution for moving object segmentation in real world environments. The code and real-world videos can be found on GitHub https://github.com/guanfangdong/LTS-UniverisalMOS.

AWV-MOS-LIO: Adaptive Window Visibility based Moving Object Segmentation with LiDAR Inertial Odometry

AWV-MOS-LIO: Adaptive Window Visibility based Moving Object Segmentation with LiDAR Inertial Odometry

SSF-MOS: Semantic Scene Flow Assisted Moving Object Segmentation for Autonomous Vehicles

SSF-MOS: Semantic Scene Flow Assisted Moving Object Segmentation for Autonomous Vehicles

MotionBEV: Attention-Aware Online LiDAR Moving Object Segmentation With Bird's Eye View Based Appearance and Motion Features

MotionBEV: Attention-Aware Online LiDAR Moving Object Segmentation With Bird's Eye View Based Appearance and Motion Features

An efficient point cloud place recognition approach based on transformer in dynamic environment

Place recognition retrieves scenes from a prior map and identifies previously visited locations. It is essential to autonomous-driving and robotics in terms of long-term navigation and global localization. Although the semantic graph has made great progress in 3D place recognition, its construction is greatly affected by the complex dynamic environment. In response, this paper designs a scene graph transformation network to solve place recognition in a dynamic environment. To remove the interference of moving objects, we introduce a moving object segmentation (MOS) module. Besides, we design a scene graph transformer-attention module to generate a more discriminative and representative global scene graph descriptor, which significantly improves the performance of place recognition. In addition, we integrate our place recognition method for loop closure with an existing LiDAR-based odometry, boosting its localization accuracy. We evaluate our method on the KITTI and Oxford RobotCar datasets. Many experimental results show that our method can effectively accomplish place recognition, and its accuracy and robustness improve by at least 3% when compared with existing state-of-the-art methods, such as DiSCO. To illustrate the generalization capabilities of our method, we evaluate it on the KITTI-360 and NCLT datasets while using only KITTI for training. The experiments show that our scene graph descriptor can achieve accurate loop-closure and global localization in never-seen environments.

Multi-sensor fusion for robust localization with moving object segmentation in complex dynamic 3D scenes

Multi-sensor fusion for robust localization with moving object segmentation in complex dynamic 3D scenes

An efficient image-guided-based 3D point cloud moving object segmentation with transformer-attention in autonomous driving

An efficient image-guided-based 3D point cloud moving object segmentation with transformer-attention in autonomous driving

Retracted: Design and Analysis of Interactive Multimedia Online Physical Education Platform considering Moving Object Segmentation Algorithm

Retracted: Design and Analysis of Interactive Multimedia Online Physical Education Platform considering Moving Object Segmentation Algorithm

A systematic review of deep learning frameworks for moving object segmentation

A systematic review of deep learning frameworks for moving object segmentation

A Blockchain-Based Trustworthy Model Evaluation Framework for Deep Learning and Its Application in Moving Object Segmentation.

Model evaluation is critical in deep learning. However, the traditional model evaluation approach is susceptible to issues of untrustworthiness, including insecure data and model sharing, insecure model training, incorrect model evaluation, centralized model evaluation, and evaluation results that can be tampered easily. To minimize these untrustworthiness issues, this paper proposes a blockchain-based model evaluation framework. The framework consists of an access control layer, a storage layer, a model training layer, and a model evaluation layer. The access control layer facilitates secure resource sharing. To achieve fine-grained and flexible access control, an attribute-based access control model combining the idea of a role-based access control model is adopted. A smart contract is designed to manage the access control policies stored in the blockchain ledger. The storage layer ensures efficient and secure storage of resources. Resource files are stored in the IPFS, with the encrypted results of their index addresses recorded in the blockchain ledger. Another smart contract is designed to achieve decentralized and efficient management of resource records. The model training layer performs training on users' servers, and, to ensure security, the training data must have records in the blockchain. The model evaluation layer utilizes the recorded data to evaluate the recorded models. A method in the smart contract of the storage layer is designed to enable evaluation, with scores automatically uploaded as a resource attribute. The proposed framework is applied to deep learning-based motion object segmentation, demonstrating its key functionalities. Furthermore, we validated the storage strategy adopted by the framework, and the trustworthiness of the framework is also analyzed.

GAMA: Geometric analysis based motion-aware architecture for moving object segmentation

Moving object segmentation in real-world scenes is of critical significance for many computer vision applications. However, there are many challenges in moving object segmentation. It is difficult to distinguish objects with motion degeneracy. Besides, complex scenes and noisy 2D optical flows also effect the result of moving object segmentation. In this paper, to address difficulties caused by motion degeneracy, we analyze the classic motion degeneracy from a new geometric perspective. To identify objects with motion degeneracy, we propose a reprojection cost and an optical flow contrast cost which are fed into the network to enrich motion features. Furthermore, a novel geometric constraint called bidirectional motion constraint is proposed to detect moving objects with weak motion features. In order to tackle more complex scenes, we also introduce a motion-aware architecture to predict instance masks of moving objects. Extensive experiments are conducted on the KITTI dataset, the JNU-UISEE dataset and the KittiMoSeg dataset, and our proposed method achieves excellent performance.

Deep Learning-based Moving Object Segmentation: Recent Progress and Research Prospects

Moving object segmentation (MOS), aiming at segmenting moving objects from video frames, is an important and challenging task in computer vision and with various applications. With the development of deep learning (DL), MOS has also entered the era of deep models toward spatiotemporal feature learning. This paper aims to provide the latest review of recent DL-based MOS methods proposed during the past three years. Specifically, we present a more up-to-date categorization based on model characteristics, then compare and discuss each category from feature learning (FL), and model training and evaluation perspectives. For FL, the methods reviewed are divided into three types: spatial FL, temporal FL, and spatiotemporal FL, then analyzed from input and model architectures aspects, three input types, and four typical preprocessing subnetworks are summarized. In terms of training, we discuss ideas for enhancing model transferability. In terms of evaluation, based on a previous categorization of scene dependent evaluation and scene independent evaluation, and combined with whether used videos are recorded with static or moving cameras, we further provide four subdivided evaluation setups and analyze that of reviewed methods. We also show performance comparisons of some reviewed MOS methods and analyze the advantages and disadvantages of reviewed MOS methods in terms of technology. Finally, based on the above comparisons and discussions, we present research prospects and future directions.

Real-Time LiDAR Point-Cloud Moving Object Segmentation for Autonomous Driving

The key to autonomous navigation in unmanned systems is the ability to recognize static and moving objects in the environment and to support the task of predicting the future state of the environment, avoiding collisions, and planning. However, because the existing 3D LiDAR point-cloud moving object segmentation (MOS) convolutional neural network (CNN) models are very complex and have large computation burden, it is difficult to perform real-time processing on embedded platforms. In this paper, we propose a lightweight MOS network structure based on LiDAR point-cloud sequence range images with only 2.3 M parameters, which is 66% less than the state-of-the-art network. When running on RTX 3090 GPU, the processing time is 35.82 ms per frame and it achieves an intersection-over-union(IoU) score of 51.3% on the SemanticKITTI dataset. In addition, the proposed CNN successfully runs the FPGA platform using an NVDLA-like hardware architecture, and the system achieves efficient and accurate moving-object segmentation of LiDAR point clouds at a speed of 32 fps, meeting the real-time requirements of autonomous vehicles.

Moving objects segmentation using generative adversarial modeling

Moving Objects Segmentation (MOS) is a crucial step in various computer vision applications, such as visual object tracking, autonomous vehicles, human activity analysis, surveillance, and security. Existing MOS approaches suffer from performance degradation due to extreme challenging conditions in real world complex environments such as varying illumination conditions, camouflage objects, dynamic backgrounds, shadows, bad weathers and camera jitters. To address these problems we proposed a novel generative adversarial based framework for moving objects segmentation. Our framework works with one classifier discriminator, one representation learning network and one generator jointly trained to perform MOS in various challenging scenarios. During training the discriminator network acts as a decision maker between real and fake training samples using conditional least squares loss. While the representation learning network provides the difference between the deep features of real and fake training samples using content loss formulation. Another loss term we have exploited to train our generator network is the reconstruction loss that minimizes the difference between the spatial information of real and fake training samples. Moreover, we also propose a novel modified U-net architecture for our generator network showing improved performance over Vanilla U-net model. Experimental evaluations of our proposed method on four benchmark datasets in comparison with thirty-two existing methods has demonstrated the strength of our proposed model.