Background Appearance Models Research Articles

Motion and appearance based background subtraction for freely moving cameras

With increasing popularity of smart phone cameras and wearable cameras, it is imperative to develop robust vision systems in analyzing videos captured by these freely moving cameras. In this paper, we propose a novel motion and appearance based algorithm for foreground/background segmentation of these videos. Unlike existing methods, it does not require any prior information nor does it restrict camera motion or scene geometry. The proposed algorithm first estimates a dense motion field between two consecutive frames, and obtains a motion-based foreground probability estimate for each pixel by comparing the motion field with its low-rank approximation. In parallel, color features are extracted by sliding a fixed-size neighborhood window over the entire image. Using the motion-based probability estimates, highly probable foreground and background color features are identified and used to learn foreground and background appearance models. These models then generate appearance-based probability estimate for each pixel. To overcome the inaccuracies in appearance modeling and background motion approximation, we incorporate an innovative Mega-pixel denoising process that uses color segmentation to smooth out the probability estimates. Finally, the denoised probability estimates are combined with the image gradient map to produce the output foreground mask under the Graph-Cut optimization framework. To cope with non-stationary dynamic scenes, the foreground and background appearance models are continuously updated with highly probable foreground and background color features. Extensive evaluations on publicly available test sequences show that the proposed technique outperforms six state-of-the-art algorithms.

Background Appearance Modeling with Applications to Visual Object Detection in an Open‐Pit Mine

This paper addresses the problem of detecting people and vehicles on a surface mine by presenting an architecture that combines the complementary strengths of deep convolutional networks (DCN) with cluster‐based analysis. We highlight that using a DCN in a naïve black box approach results in a significantly high rate of errors due to the lack of mining‐specific training data and the unique landscape in a mine site. In this work, we propose a background model that exploits the abundance of background‐only images to discover the natural clusters in visual appearance using features extracted from the DCN. Both a simple nearest cluster‐based background model and an extended model with cosine features are investigated for their ability to identify and suppress potential false positives made by the DCN. Furthermore, localization of objects of interest is enabled through region proposals, which have been tuned to increase recall within the constraints of a computational budget. Finally, a soft fusion framework is presented to combine the estimates of both the DCN and background model to improve the accuracy of the detection. Our system is tested on over 11 km of real mine site data in both day and night conditions where we were able to detect both light and heavy vehicles along with mining personnel. We show that the introduction of our background model improves the detection performance. In particular, soft fusion of the background model and the DCN output produces a relative improvement in the F1 score of 46% and 28% compared to a baseline pretrained DCN and a DCN retrained with mining images, respectively.

Fast Appearance Modeling for Automatic Primary Video Object Segmentation.

Automatic segmentation of the primary object in a video clip is a challenging problem as there is no prior knowledge of the primary object. Most existing techniques thus adapt an iterative approach for foreground and background appearance modeling, i.e., fix the appearance model while optimizing the segmentation and fix the segmentation while optimizing the appearance model. However, these approaches may rely on good initialization and can be easily trapped in local optimal. In addition, they are usually time consuming for analyzing videos. To address these limitations, we propose a novel and efficient appearance modeling technique for automatic primary video object segmentation in the Markov random field (MRF) framework. It embeds the appearance constraint as auxiliary nodes and edges in the MRF structure, and can optimize both the segmentation and appearance model parameters simultaneously in one graph cut. The extensive experimental evaluations validate the superiority of the proposed approach over the state-of-the-art methods, in both efficiency and effectiveness.

PWP3D: Real-Time Segmentation and Tracking of 3D Objects

We formulate a probabilistic framework for simultaneous region-based 2D segmentation and 2D to 3D pose tracking, using a known 3D model. Given such a model, we aim to maximise the discrimination between statistical foreground and background appearance models, via direct optimisation of the 3D pose parameters. The foreground region is delineated by the zero-level-set of a signed distance embedding function, and we define an energy over this region and its immediate background surroundings based on pixel-wise posterior membership probabilities (as opposed to likelihoods). We derive the differentials of this energy with respect to the pose parameters of the 3D object, meaning we can conduct a search for the correct pose using standard gradient-based non-linear minimisation techniques. We propose novel enhancements at the pixel level based on temporal consistency and improved online appearance model adaptation. Furthermore, straightforward extensions of our method lead to multi-camera and multi-object tracking as part of the same framework. The parallel nature of much of the processing in our algorithm means it is amenable to GPU acceleration, and we give details of our real-time implementation, which we use to generate experimental results on both real and artificial video sequences, with a number of 3D models. These experiments demonstrate the benefit of using pixel-wise posteriors rather than likelihoods, and showcase the qualities, such as robustness to occlusions and motion blur (and also some failure modes), of our tracker.

Segmenting moving objects from a freely moving camera with an effective segmentation cue

It is challenging to segment moving objects in a video sequence viewed from a freely moving camera. Within the conditional random field framework, we propose a new segmentation cue to solve this problem. The cue is defined as a spatial–color likelihood ratio map by combining the foreground and background appearance models which are extracted from the previous segmentation result. Specifically, to make the cue suitable for the current segmentation task in the presence of complex camera motions, we introduce a camera motion compensation step in the process of evaluating the likelihood ratio map. We validate the proposed segmentation cue using several video sequences taken by hand-held cameras in outdoor urban scenes.

Background estimation under rapid gain change in thermal imagery

We consider detection of moving ground vehicles in airborne sequences recorded by a thermal sensor with automatic gain control, using an approach that integrates dense optic flow over time to maintain a model of background appearance and a foreground occlusion layer mask. However, the automatic gain control of the thermal sensor introduces rapid changes in intensity that makes this difficult. In this paper we show that an intensity-clipped affine model of sensor gain is sufficient to describe the behavior of our thermal sensor. We develop a method for gain estimation and compensation that uses sparse flow of corner features to compute the affine background scene motion that brings pairs of frames into alignment prior to estimating change in pixel brightness. Dense optic flow and background appearance modeling is then performed on these motion-compensated and brightness-compensated frames. Experimental results demonstrate that the resulting algorithm can segment ground vehicles from thermal airborne video while building a mosaic of the background layer, despite the presence of rapid gain changes.