Moving Object Segmentation Research Articles

End-to-End Recurrent Generative Adversarial Network for Traffic and Surveillance Applications

In video frame segmentation, many existing deep networks and contemporary approaches give a remarkable performance with the assumption that the only foreground is moving, and the background is stationary. However, in the presence of infrequent motion of foreground objects, sudden illumination changes in the background, bad weather, and dynamic background, the accurate foreground object(s) segmentation is a challenging task. Generative adversarial networks (GAN) based training shows fruitful results in various fields like image-to-image style transfer, image enhancement, semantic segmentation, image super-resolution, etc. The limited results of hand-crafted approaches for moving object segmentation (MOS) and the robustness of adversarial training for a given task inspired us to propose a novel approach for moving object segmentation (MOS). In this context, an end-to-end generative adversarial network (two generators) with recurrent technique is proposed for MOS and is named as RMS-GAN. The proposed RMS-GAN is able to incorporate foreground probability knowledge with residual and weight sharing based recurrent technique for accurate segmentation. The recurrent technique helps us to exhibit the temporal behavior between successive video frames, which is more prominent for any video processing applications. Also, to enhance the spatial coherence of the obtained foreground probability map using the generator-1 network, the cascaded architecture of two generators is proposed. The effectiveness of the proposed approach is evaluated both qualitatively and quantitatively on three benchmark video datasets for MOS. Experimental result analysis shows that the proposed network outperforms the existing state-of-the-art methods on three benchmark datasets for MOS.

Moving Object Segmentation and Detection for Robust RGBD-SLAM in Dynamic Environments

Localization accuracy is a fundamental requirement for Simultaneous Localization and Mapping (SLAM) systems. Traditional visual SLAM (vSLAM) schemes are usually based upon the assumption of static environments, so they do not perform well in dynamic environments. While a number of vSLAM frameworks have been reported for dynamic environments, the localization accuracy is usually unsatisfactory. In this article, we present a novel motion detection and segmentation method using Red Green Blue-Depth (RGB-D) data to improve the localization accuracy of feature-based RGB-D SLAM in dynamic environments. To overcome the problem due to undersegmentation generated by the semantic segmentation network, a mask inpainting method is developed to ensure the completeness of object segmentation. In the meantime, an optical flow-based motion detection method is proposed to detect dynamic objects from moving cameras, allowing robust detection by removing irrelevant information. Experiments performed on the public Technical University of Munich (TUM) RGB-D data set show that the presented scheme outperforms the state-of-art RGB-D SLAM systems in terms of trajectory accuracy, improving the localization accuracy of RGB-D SLAM in dynamic environments.

Accurate moving object segmentation in unconstraint videos based on robust seed pixels selection

Due to the clutter background motion, accurate moving object segmentation in unconstrained videos remains a significant open problem, especially for the slow-moving object. This article proposes an accurate moving object segmentation method based on robust seed selection. The seed pixels of the object and background are selected robustly by using the optical flow cues. Firstly, this article detects the moving object’s rough contour according to the local difference in the weighted orientation cues of the optical flow. Then, the detected rough contour is used to guide the object and the background seed pixel selection. The object seed pixels in the previous frame are propagated to the current frame according to the optical flow to improve the robustness of the seed selection. Finally, we adopt the random walker algorithm to segment the moving object accurately according to the selected seed pixels. Experiments on publicly available data sets indicate that the proposed method shows excellent performance in segmenting moving objects accurately in unconstraint videos.

Content-aware rate control scheme for HEVC based on static and dynamic saliency detection

High efficiency video coding (HEVC) greatly outperforms previous standards H.264/AVC in terms of coding bit rate and video quality. However, it does not take into account the human visual system (HVS), that people pay more attention to specific areas and moving objects. In this paper, we present a content-aware rate control scheme for HEVC based on static and dynamic saliency detection. The proposed strategy mainly consists of three techniques, static saliency detection, dynamic saliency detection, and adaptive bit rate allocation. Firstly, we train a deep convolution network (DCN) model to extract the static saliency map by highlighting semantically salient regions. Compared to traditional texture-based or color-based region of interest (ROI) extraction techniques, our models are more in line with the HVS. Secondly, we develop a moving object segmentation technique to automatically extract the dynamic salient regions for each frame. Furthermore, according to the fusion saliency map, a coding tree unit (CTU) level bit control technique is exploited to realize flexible and adaptive bit rate allocation. As a result, the quality of salient regions is improved by allocating more bits, while allocating fewer bits to the non-salient regions. We verified the proposed method on both the JCT-VC recommended data set and eye-tracking data set. Experiment results show that the PSNR of salient regions can improve by an average of 1.85 dB without adding bit rate burden, which significantly improves the visual experience.

Integration of new moving object segmentation and classification techniques using optimal salp swarm-based feature fusion with linear multi k-SVM classifier

The feature extraction technique is applied on least enclosing rectangle (LER) of the segmented object to increase the processing speed. The main intuition of this salp swarm algorithm relays on reducing the computational load of the proposed classifier by removing the repetitive and unrelated features from the feature vector. Also, increased training samples of similarly shaped classes when applied on the classifier can generate the misclassification results. Thus, a new layered kernel-based support vector machine (k-SVM) classifier is developed by means of integrating the k-neural network classifier and layered SVM classifier. Because of the high dimensional features, a difficulty occurs in the application of a single classifier. In order to ease the computational load, this multi classifier is integrated with a shadow elimination technique to classify the object categories of intelligent transportations system such as motorcycles, bicycles, cars, and pedestrians.

Deep detector classifier (DeepDC) for moving objects segmentation and classification in video surveillance

In this study, the authors present a new approach to segment and classify moving objects in video sequences by combining an unsupervised anomaly discovery framework called DeepSphere and generative adversarial networks. The proposed deep detector classifier employs and validates DeepSphere, which aims mainly to identify the anomalous cases in the spatial and temporal context in order to perform foreground objects segmentation. For post-processing, some morphological operations are considered to better segment and extract the desired objects. Finally, they take advantage of the power of generative models, which recognise the problem of semi-supervised learning as a specific missing data imputation task in order to classify the segmented objects. They evaluate the method with multiple datasets and the results confirm the effectiveness of the proposed approach, which achieves superior performance over the state-of-the-art methods having the capabilities of segmenting and classifying moving objects from videos surveillance.

Graph Moving Object Segmentation

Moving Object Segmentation (MOS) is a fundamental task in computer vision. Due to undesirable variations in the background scene, MOS becomes very challenging for static and moving camera sequences. Several deep learning methods have been proposed for MOS with impressive performance. However, these methods show performance degradation in the presence of unseen videos; and usually, deep learning models require large amounts of data to avoid overfitting. Recently, graph learning has attracted significant attention in many computer vision applications since they provide tools to exploit the geometrical structure of data. In this work, concepts of graph signal processing are introduced for MOS. First, we propose a new algorithm that is composed of segmentation, background initialization, graph construction, unseen sampling, and a semi-supervised learning method inspired by the theory of recovery of graph signals. Second, theoretical developments are introduced, showing one bound for the sample complexity in semi-supervised learning, and two bounds for the condition number of the Sobolev norm. Our algorithm has the advantage of requiring less labeled data than deep learning methods while having competitive results on both static and moving camera videos. Our algorithm is also adapted for Video Object Segmentation (VOS) tasks and is evaluated on six publicly available datasets outperforming several state-of-the-art methods in challenging conditions.

CVABS: moving object segmentation with common vector approach for videos

Background modelling is a fundamental step for several real‐time computer vision applications that requires security systems and monitoring. An accurate background model helps to detect the activity of moving objects in the video. In this work, the authors have developed a new subspace‐based background‐modelling algorithm using the concept of common vector approach (CVA) with Gram–Schmidt orthogonalisation. Once the background model that involves the common characteristic of different views corresponding to the same scene is acquired, a smart foreground detection and background updating procedure is applied based on dynamic control parameters. A variety of experiments is conducted on different problem types related to dynamic backgrounds. Several types of metrics are utilised as objective measures and the obtained visual results are judged subjectively. It was observed that the proposed method stands successfully for all problem types reported on CDNet2014 dataset by updating the background frames with a self‐learning feedback mechanism.

Joint Stereo Video Deblurring, Scene Flow Estimation and Moving Object Segmentation.

Stereo videos for the dynamic scenes often show unpleasant blurred effects due to the camera motion and the multiple moving objects with large depth variations. Given consecutive blurred stereo video frames, we aim to recover the latent clean images, estimate the 3D scene flow and segment the multiple moving objects. These three tasks have been previously addressed separately, which fail to exploit the internal connections among these tasks and cannot achieve optimality. In this paper, we propose to jointly solve these three tasks in a unified framework by exploiting their intrinsic connections. To this end, we represent the dynamic scenes with the piece-wise planar model, which exploits the local structure of the scene and expresses various dynamic scenes. Under our model, these three tasks are naturally connected and expressed as the parameter estimation of 3D scene structure and camera motion (structure and motion for the dynamic scenes). By exploiting the blur model constraint, the moving objects and the 3D scene structure, we reach an energy minimization formulation for joint deblurring, scene flow and segmentation. We evaluate our approach extensively on both synthetic datasets and publicly available real datasets with fast-moving objects, camera motion, uncontrolled lighting conditions and shadows. Experimental results demonstrate that our method can achieve significant improvement in stereo video deblurring, scene flow estimation and moving object segmentation, over state-of-the-art methods.

Unsupervised Moving Object Segmentation from Stationary or Moving Camera based on Multi-frame Homography Constraints.

Moving object segmentation is the most fundamental task for many vision-based applications. In the past decade, it has been performed on the stationary camera, or moving camera, respectively. In this paper, we show that the moving object segmentation can be addressed in a unified framework for both type of cameras. The proposed method consists of two stages: (1) In the first stage, a novel multi-frame homography model is generated to describe the background motion. Then, the inliers and outliers of that model are classified as background trajectories and moving object trajectories by the designed cumulative acknowledgment strategy. (2) In the second stage, a super-pixel-based Markov Random Fields model is used to refine the spatial accuracy of initial segmentation and obtain final pixel level labeling, which has integrated trajectory classification information, a dynamic appearance model, and spatial temporal cues. The proposed method overcomes the limitations of existing object segmentation algorithms and resolves the difference between stationary and moving cameras. The algorithm is tested on several challenging open datasets. Experiments show that the proposed method presents significant performance improvement over state-of-the-art techniques quantitatively and qualitatively.

Active Perception for Foreground Segmentation: An RGB-D Data-Based Background Modeling Method

Foreground moving object segmentation is a fundamental problem in many computer vision applications. As a solution for foreground segmentation, background modeling has been intensively studied over past years and many effective algorithms have been developed. However, accurate foreground segmentation is still a difficult problem. Currently, most of the algorithms work solely within the color space, in which the segmentation performance is prone to be degraded by a multitude of challenges, such as illumination changes, shadows, automatic camera adjustments, and color camouflage. RGB-D cameras are active visual sensors that provide depth measurements along with color images. We present in this paper an innovative background modeling method by using both the color and depth information from an RGB-D camera. The proposed method is evaluated using a public RGB-D data set. Various experiments confirm that our method is able to achieve superior performance compared with existing well-known methods. Note to Practitioners —This paper investigates background modeling for foreground segmentation with active perception. Recent RGB-D cameras that leverage the active perception technology have advanced many computer vision algorithms. In this paper, we develop a background modeling method to achieve superior performance by using an RGB-D camera instead of a color camera. Due to the use of the active sensing technology, the proposed method is characterized by its robustness to common challenges. Our method could be used for improving existing infrastructures, such as visual surveillance systems for parking spaces. Moreover, the simple design of our method allows it to be easily deployed on various computing platforms, which facilitates many practical applications that usually require embedded computing devices. However, our method cannot run real timely at the current status. We believe that it can be further improved using parallel programming techniques to meet the real-time requirement.

Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding.

Learning to estimate 3D geometry in a single frame and optical flow from consecutive frames by watching unlabeled videos via deep convolutional network has made significant progress recently. Current state-of-the-art (SoTA) methods treat the two tasks independently. One important assumption of the existing depth estimation methods is that the scenes contain no moving object. In this paper, we propose to address the two tasks as a whole, i.e. to jointly understand per-pixel 3D geometry and motion. This eliminates the need of static scene assumption and enforces the inherent geometrical consistency during the learning process, yielding significantly improved results for both tasks. We call our method as "Every Pixel Counts++" or "EPC++". Various loss terms are formulated to jointly supervise the learning across geometrical cues and effective adaptive training strategy is proposed to achieve better performance. Comprehensive experiments were conducted on datasets with different scenes, including driving scenario (KITTI 2012 and KITTI 2015 datasets), mixed outdoor/indoor scenes (Make3D) and synthetic animation (MPI Sintel dataset). Performance on the five tasks of depth estimation, optical flow estimation, odometry, moving object segmentation and scene flow estimation shows that our approach outperforms other SoTA methods, demonstrating the effectiveness of each module of our proposed method.

A novel GPU-aware Histogram-based algorithm for supporting moving object segmentation in big-data-based IoT application scenarios

Multimedia data are a popular case of Big Data that expose the classical 3V characteristics (i.e., volume, velocity and variety). Such kind of data are likely to be processed within the core layer of Internet of Things (IoT) platforms, where a multiple, typically high, number of “things” (e.g., sensors, devices, actuators, and so forth) collaborate to massively process big data for supporting intelligent algorithms running over them. In such platforms, the computational bottleneck is very often represented by the component running the main algorithm, while communication and cooperation costs still remain relevant. Inspired by this emerging trend of big-data-based IoT applications, in this paper we focus on the specific application context represented by the problem of supporting moving object segmentation over images originated in the context of big multimedia data, and we propose an innovative background maintenance approach to this end. In particular, we provide a novel GPU-aware Histogram-based Moving Object Segmentation algorithm that adopts a pixel-oriented approach and it is based on Graphic Processing Units (GPU), called pixHMOS_gpu. pixHMOS_gpu allows us to achieve higher performance, hence making the computational gap of big-data-based IoT applications decisively smaller. Experimental results clearly confirm our arguments.

Anfis Classifier Based Moving Object Detection and Segmentation in Indoor and Outdoor Environments

Background: Moving object detection in dynamic environment video is more complex than the static environment videos. In this paper, moving objects in video sequences are detected and segmented using feature extraction based Adaptive Neuro-Fuzzy Inference System (ANFIS) classifier approach. The proposed moving object detection methodology is tested on different video sequences in both indoor and outdoor environments. Methods: This proposed methodology consists of background subtraction and classification modules. The absolute difference image is constructed in background subtraction module. The features are extracted from this difference image and these extracted features are trained and classified using ANFIS classification module. Results: The proposed moving object detection methodology is analyzed in terms of Accuracy, Recall, Average Accuracy, Precision and F-measure. The proposed moving object segmentation methodology is executed on different Central Processing Unit (CPU) processor as 1.8 GHz and 2.4 GHz for evaluating the performance during moving object segmentation. At present, some moving object detection systems used 1.8 GHz CPU processor. Recently, many systems for moving object detection are using 2.4 GHz CPU processor. Hence, CPU processors 1.8 GHz and 2.4 GHz are used in this paper for detecting the moving objects in video sequences. Table 1 shows the performance evaluation of proposed moving object detection on CPU processor 1.8 GHz (100 sequence). Table 2 shows the performance evaluation of proposed moving object detection on CPU processor 2.8 GHz (100 sequence). The average moving object detection time on CPU processor 1.8 GHz for fountain sequence is 62.5 seconds, for airport sequence is 64.7 seconds, for meeting room sequence is 71.6 seconds and for Lobby sequence is 73.5 seconds, respectively, as depicted in Table 3. The average elapsed time for moving object detection on 100 sequences is 68.07 seconds. The average moving object detection time on CPU processor 2.4 GHz for fountain sequence is 56.5 seconds, for airport sequence is 54.7 seconds, for meeting room sequence is 65.8 seconds and for Lobby sequence is 67.5 seconds, respectively, as depicted in Table 4. The average elapsed time for moving object detection on 100 sequences is 61.12 seconds. It is very clear from Table 3 and Table 4; the moving object detection time is reduced when the frequency of the CPU processor increases. Conclusion: In this paper, moving object is detected and segmented using ANFIS classifier. The proposed method initially segments the background image and then features are extracted from the threshold image. These features are trained and classified using ANFIS classification method. The proposed moving object detection method is tested on different video sequences which are obtained from different indoor and outdoor environments. The performance of the proposed moving object detection and segmentation methodology is analyzed in terms of Accuracy, Recall, Average Accuracy, Precision and F-measure.

Embedded GPU-based parallel optimization for moving objects segmentation algorithm

在光电监视系统中，广泛应用于运动目标分割的PBAS(pixel base adaptive segmenter)算法计算复杂、参数量大，难以达到实时分割的要求。针对PBAS算法是对图像中每个像素点进行独立处理，特别适合于GPU并行加速的特点，对其在嵌入式GPU平台Jetson TX2上进行了并行优化实现。在数据存储结构、共享内存使用、随机数产生机制3个方面对该算法进行了优化设计。实验结果表明，对于480×320像素分辨率的中波红外视频序列，该并行优化方法可以达到132 fps的处理速度，满足了实时处理的要求。

Foreground segmentation using convolutional neural networks for multiscale feature encoding

Several methods have been proposed to solve moving objects segmentation problem accurately in different scenes. However, many of them lack the ability of handling various difficult scenarios such as illumination changes, background or camera motion, camouflage effect, shadow etc. To address these issues, we propose two robust encoder-decoder type neural networks that generate multi-scale feature encodings in different ways and can be trained end-to-end using only a few training samples. Using the same encoder-decoder configurations, in the first model, a triplet of encoders take the inputs in three scales to embed an image in a multi-scale feature space; in the second model, a Feature Pooling Module (FPM) is plugged on top of a single input encoder to extract multi-scale features in the middle layers. Both models use a transposed convolutional network in the decoder part to learn a mapping from feature space to image space. In order to evaluate our models, we entered the Change Detection 2014 Challenge (changedetection.net) and our models, namely FgSegNet_M and FgSegNet_S, outperformed all the existing state-of-the-art methods by an average F-Measure of 0.9770 and 0.9804, respectively. We also evaluate our models on SBI2015 and UCSD Background Subtraction datasets. Our source code is made publicly available at https://github.com/lim-anggun/FgSegNet.

Real-Time Moving Object Segmentation and Classification From HEVC Compressed Surveillance Video

Moving object segmentation and classification from compressed video plays an important role in intelligent video surveillance. Compared with H.264/AVC, High Efficiency Video Coding (HEVC) introduces a host of new coding features that can be further exploited for moving object segmentation and classification. In this paper, we present a real-time approach to segment and classify moving objects using unique features directly extracted from the HEVC compressed domain for video surveillance. In the proposed method, first, motion vector (MV) interpolation for intra-coded prediction unit (PU) and MV outlier removal are employed for preprocessing. Second, blocks with nonzero MVs are clustered into the connected foreground regions using the four-connectivity component labeling algorithm. Third, object region tracking based on temporal consistency is applied to the connected foreground regions to remove the noise regions. The boundary of moving object region is further refined by the coding unit size and PU size. Finally, a person–vehicle classification model using bag of spatial–temporal HEVC syntax words is trained to classify the moving objects, either persons or vehicles. The experimental results demonstrate that the proposed method provides solid performance and can classify moving persons and vehicles accurately.

Moving objects segmentation and extraction based on motion blur features

In order to restore the spatially-varying blur images whose background is sharp, it is necessary that the moving objects are segmented and extracted from the blur images. So this paper proposes a novel segmentation and extraction method based on motion blur features. Firstly, the blur features are extracted by using contourlet transform and considered as the prior information. Then an energy function is proposed for extracting the moving object from the original images. Moreover, considering that there exists some noise in segmented images, some morphological methods are introduced to remove the noise. The experimental results demonstrate the proposed scheme achieves high-quality segmentation performance and it outperforms the existing methods when used for moving objects segmentation.

Unified Framework for Moving Object Segmentation from Stationary or Moving Camera based on Multi-frame Homography Constraints

Unified Framework for Moving Object Segmentation from Stationary or Moving Camera based on Multi-frame Homography Constraints

Moving Object Segmentation Using Various Features from Aerial Images: A Review

Moving Object Segmentation Using Various Features from Aerial Images: A Review