Foreground Detection Algorithm Research Articles

A Novel Framework to Generate Synthetic Video for Foreground Detection in Highway Surveillance Scenarios

Foreground detection (FD) plays an important role in the domain of video surveillance for highway. The design of advanced FD algorithms requires large-scale and diverse video dataset. However, collecting and labeling real dataset is still time-consuming, labor-intensive, and highly subjective. To address this issue, we first use computer graphics (CG) to clone real highway scenarios (HS) and generate synthetic multi-challenge video datasets, called “Synthetic-HS (CG)”, automatically labeled with accurate pixel-level ground truth. The Synthetic-HS (CG) dataset contains eight imaging condition sequences for computer vision research. Then, we design an image translation (IT) model that translates source domain (Synthetic-HS (CG)) to target domain (real). This model uses skip connections and attention module to generate realistic synthetic images “Synthetic-HS (IT)”. We use publicly available Synthetic-HS in combination with the corresponding real video sequence to conduct experiments. The experiment results suggest that: 1) The Synthetic-HS (CG) dataset enables us to provide precise quantitative evaluation of the drawbacks of foreground detection methods 2) The realistic Synthetic-HS (IT) images can be used to promote the visual perception in highway video surveillance.

Automatic Foreground Detection at 784 FPS for Ultra-High-Speed Human–Machine Interactions

Human-machine interactive systems show increasing demand for analysing fast moving objects in high-frame-rate videos. Robust foreground detection, which is able to reduce large amount of redundant background data from high-frame-rate video, becomes the essence to achieve ultra-high-speed human-machine interactions. This paper proposes a local spatial propagation based background model generation, a local linear illumination correction based background model update, and a regional central coordinates and edge keypoints constrained foreground region reselection. The three proposals make up a robust and hardware-friendly foreground detection method. Experimental results prove that the proposed hardware-friendly algorithm achieves high accuracy and robustness on various kinds of challenging cases. Meanwhile, the hardware implementation utilizes little hardware resources and achieves realtime processing of high-frame-rate (784 frame/second) video with the delay less than 1 ms/frame in image processing core. In addition, a practical system is implemented by combing a PC, a high-speed camera and a field programmable gate array (FPGA) for realworld applications. This work will significatively promote the development and application of high-speed human machine interaction. A demo of the proposed vision system working at 784 FPS is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://wcms.waseda.jp/em/5f84f75136a6</uri> . Note to Practitioners—This paper was motivated by the problem of high-frame-rate video contains large amount of redundant background pixels which makes ultra-high-speed human-machine interactions inaccessible. Existing approaches are mainly focused on designing complex background models, but processing speed, which is the most important issue for ultra-high-speed human-machine interactions, has received relatively little attention. This paper suggests a robust and hardware-friendly foreground detection algorithm which has been implemented as a hardware system by using an FPGA, a high-frame-rate camera, and a PC. We show that the hardware implementation utilizes less hardware resources and achieves real-time processing speed of 784 FPS with the delay less than 1 ms/frame in the image processing core. This work is a pioneering attempt of ultra-high-speed foreground detection, which will significatively speed up the wide applications of ultra-high-speed human machine interactions.

Foreground Detection Based on Superpixel and Semantic Segmentation.

Foreground detection is an essential step in computer vision and video processing. Accurate foreground object extraction is crucial for subsequent high-level tasks such as target recognition and tracking. Although many foreground detection algorithms have been proposed, foreground detection in complex scenes is still a challenging problem. This paper presents a foreground detection algorithm based on superpixel and semantic segmentation. It first uses multiscale superpixel segmentation to obtain the initial foreground mask. At the same time, a semantic segmentation network is applied to separate potential foreground objects, and then use the defined rules to combine the results of superpixel and semantic segmentation to get the final foreground object. Finally, the background model is updated with the refined foreground result. Experiments on the CDNet2014 dataset demonstrate the effectiveness of the proposed algorithm, which can accurately segment foreground objects in complex scenes.

The development of ViBe foreground detection algorithm using Lévy flights random update strategy and Kinect laser imaging sensor

The development of ViBe foreground detection algorithm using Lévy flights random update strategy and Kinect laser imaging sensor

Slope Collapse Detection Based on Image Processing

In this paper, we study a task of slope collapse detection (SCD) for river embankment and formulate it as the tasks of motion detection and image recognition. Specifically, we introduce an SCD method based on motion detection and image recognition technologies to help inspector attendants detect the slope collapse. In this method, we use the foreground motion detection algorithm to identify the slope collapse of the scene of the river embankment. Since the moving targets in the foreground may not only be the slope collapse but also maybe some biology, we further use the image feature extraction and image recognition technology to recognize the foreground motion area, thus eliminating the influence of the biology on the detection results. Experimental results on the relevant scene data show that the proposed method can identify the slope collapse in real-time, and can effectively eliminate the motion interference of the biology, which has a high practical value.

Dual Camera-Based Supervised Foreground Detection for Low-End Video Surveillance Systems

Deep learning-based algorithms showed promising prospects in the computer vision domain. However, their deployment in real-time systems is challenging due to their computational complexity, high-end hardware prerequisites, and the amount of annotated data for training. This paper proposes an efficient foreground detection (EFDNet) algorithm based on deep spatial features extracted from an RGB input image using VGG-16 convolutional neural networks (CNN). The VGG-16 CNN is modified by concatenated residual (CR) blocks to learn better global contextual features and recover lost feature information due to several convolution operations. A new upsampling network is designed using bilinear interpolation sandwiched between $3\times 3$ convolutions to upsample and refine feature maps for pixel-wise prediction. This helps to propagate loss errors from the upsampling network during backpropagation. The experiments showed the effectiveness of the EFDNet in outperforming top-ranked foreground detection algorithms. EFDNet trains faster on low-end hardware and demonstrated promising results with a minimum of 50 training frames with binary ground-truth.

Underwater Pipeline Oil Spill Detection Based on Structure of Root and Branch Cells

The existing oil spill detection methods mainly rely on physical sensors or numerical models cannot locate the spill position accurately and in time. To solve this problem, combining with underwater image processing technology, an unsupervised detection algorithm for oil spill in underwater pipelines is proposed for the first time. First, the oil spill region to be detected is regarded as the moving target, and the foreground detection algorithm is applied to the processed images. Then, the HSV (Hue, Saturation, Value) color space of the image is used to screen the oil spill region meeting the threshold requirements. Next, the bitwise of foreground mask and HSV mask into cells are divided. Finally, according to the characteristics of the oil spill image, false detection is eliminated by classifying cells three times. After qualitative and quantitative analysis, it is proved that the proposed algorithm can detect oil spill region accurately.

Deep Atrous Spatial Features-Based Supervised Foreground Detection Algorithm for Industrial Surveillance Systems

Camera-based surveillance systems largely perform an intrusion detection task for sensitive areas. The task may seem trivial but is quite challenging due to environmental changes and object behaviors such as those due to night-time, sunlight, IR camera, camouflage, and static foreground objects, etc. Convolutional neural network based algorithms have shown promise in dealing with these challenges. However, they are exclusively focused on accuracy. This article proposes an efficient supervised foreground detection (SFDNet) algorithm based on atrous deep spatial features. The features are extracted using atrous convolution kernels to enlarge the field-of-view of a kernel mask, thereby encoding rich context features without increasing the number of parameters. The network further benefits from a residual dense block strategy that mixes the mid and high-level features to retain the foreground information lost in low-resolution high-level features. The extracted features are expanded using a novel pyramid upsampling network. The feature maps are upsampled using bilinear interpolation and pass through a 3x3 convolutional kernel. The expanded feature maps are concatenated with the corresponding mid and low-level feature maps from an atrous feature extractor to further refine the expanded feature maps. The SFDNet showed better performance than high-ranked foreground detection algorithms on the three standard databases. The testing demo can be found at https://drive.google.com/file/d/1z_zEj9Yp7GZeM2gSIwYKvSzQlxMAiarw/view?usp=sharing.

Foreground algorithms for detection and extraction of an object in multimedia

Background Subtraction of a foreground object in multimedia is one of the major preprocessing steps involved in many vision-based applications. The main logic for detecting moving objects from the video is difference of the current frame and a reference frame which is called “background image” and this method is known as frame differencing method. Background Subtraction is widely used for real-time motion gesture recognition to be used in gesture enabled items like vehicles or automated gadgets. It is also used in content-based video coding, traffic monitoring, object tracking, digital forensics and human-computer interaction. Now-a-days due to advent in technology it is noticed that most of the conferences, meetings and interviews are done on video calls. It’s quite obvious that a conference room like atmosphere is not always readily available at any point of time. To eradicate this issue, an efficient algorithm for foreground extraction in a multimedia on video calls is very much needed. This paper is not to just build Background Subtraction application for Mobile Platform but to optimize the existing OpenCV algorithm to work on limited resources on mobile platform without reducing the performance. In this paper, comparison of various foreground detection, extraction and feature detection algorithms are done on mobile platform using OpenCV. The set of experiments were conducted to appraise the efficiency of each algorithm over the other. The overall performances of these algorithms were compared on the basis of execution time, resolution and resources required.

The effect of downsampling–upsampling strategy on foreground detection algorithms

In video surveillance systems which incorporate stationary cameras, the first phase of movement object detection is crucial for the correct modelling of the behavior of these objects, as well as being the most complex in terms of execution time. There are many algorithms that provide a reliable and adequate segmentation mask, obtaining real-time ratios for reduced image sizes. However, due to the increased performance of camera hardware, the application of previous methods to sequences with higher resolutions (from 640 × 480 to 1920 × 1080) is not carried out in real time, compromising their use in real video surveillance systems. In this paper we propose a methodology to reduce the computational requirements of the algorithms, consisting of a reduction of the input frame and, subsequently, an interpolation of the segmentation mask of each method to recover the original frame size. In addition, the viability of this meta-model is analyzed together with the different selected algorithms, evaluating the quality of the resulting segmentation and its gain in terms of computation time.

Foreground detection by probabilistic modeling of the features discovered by stacked denoising autoencoders in noisy video sequences

A robust foreground detection system is presented, which is resilient to noise in video sequences. The proposed model divides each video frame in patches that are fed to a stacked denoising autoencoder, which is responsible for the extraction of significant features from each image patch. After that, a probabilistic model that is composed of a mixture of Gaussian distributions decides whether the given feature vector describes a patch belonging to the background or the foreground. In order to test the model robustness, several trials with noise of different types and intensities have been carried out. A comparison with other ten state of the art foreground detection algorithms has been drawn. The algorithms have been ranked according to the obtained results, and our proposal appears among the first three positions in most case and its the one that best performs on average.

Foreground Detection for Infrared Videos With Multiscale 3-D Fully Convolutional Network

Foreground detection for infrared (IR) videos is an important and fundamental problem in many applications, e.g., IR surveillance, IR object tracking, and so on. Conventional foreground detection algorithms developed for visible videos do not focus on the problems for IR videos, e.g., low contrast, coarse texture, lack of color information, and so on. Recent foreground detection methods based on deep neural network (DNN) demonstrated significant improvement, but most of them still use only spatial features, which is less obvious in IR images. In this letter, we add deeply learned multiscale temporal features to improve the performance of background subtraction for IR videos. We propose a novel multiscale 3-D fully convolutional network (MFC3-D) to establish a mapping from image sequences to pixelwise classification results and to learn deep and hierarchical multiscale spatial–temporal features of the input images sequence. The experimental results show that the MFC3-D can learn spatial–temporal features effectively and achieved state-of-the-art results on the test data set, comparing to other DNN-based methods and traditional background subtraction methods.

Video Foreground Detection Algorithm Based on Fast Principal Component Pursuit and Motion Saliency.

Aiming at the shortcoming of being unsuitable for dynamic background and high computational complexity of the existing RPCA- (robust principal component analysis-) based block-sparse moving object detection method, this paper proposes a two-stage foreground detection framework based on motion saliency for video sequence. At the first stage, the observed image sequence is regarded as the sum of a low-rank background matrix and a sparse outlier matrix, and then the decomposition is solved by the RPCA method via fast PCP (principal component pursuit). At the second stage, the sparse foreground blocks are obtained according to the spectral residuals and the spatial correlation of the foreground region. Finally, the block-sparse RPCA algorithm through fast PCP is used to estimate foreground areas dynamically and to reconstruct the foreground objects. Extensive experiments demonstrate that our method can exclude the interference of background motion and change, simultaneously improving the detection rate of small targets.

Combining background subtraction algorithms with convolutional neural network

Accurate and fast extraction of foreground objects is a key prerequisite for a wide range of computer vision applications, such as object tracking and recognition. Thus, many background subtraction (BGS) methods for foreground object detection have been proposed in recent decades. However, this is still regarded as a tough problem due to a variety of challenges, such as illumination variations, camera jitter, dynamic backgrounds, and shadows. Currently, there is no single method that can handle all the challenges in a robust way. We try to solve this problem from a perspective of combining different state-of-the-art BGS algorithms to create a more robust and more advanced foreground detection algorithm. More specifically, an encoder–decoder fully convolutional neural network architecture is adapted and trained to automatically learn how to leverage the characteristics of different algorithms to fuse the results produced by different BGS algorithms and produce a more precise result. Comprehensive experiments evaluated on the CDnet 2014 dataset demonstrate that the proposed method outperforms all the considered single BGS algorithms. We show that our solution is more efficient than other BGS combination strategies.

Comparative study of illumination-invariant foreground detection

Foreground detection plays a vital role in finding the moving objects of a scene. For the last two decades, many methods were introduced to tackle the issue of illumination variation in foreground detection. In this article, we proposed a method to segment moving objects under abrupt illumination change and analyzed the merits and demerits of the proposed method with seven other algorithms commonly used for illumination-invariant foreground detection. The proposed method calculates the entropy of the video scene to determine the level of illumination change occurred and select the update model based on the difference in entropy values. Benchmark datasets possessing different challenging illumination conditions are used to analyze the efficiency of the foreground detection algorithms. Experimental studies demonstrate the performance of the proposed algorithm with several algorithms under various illumination conditions and its low time complexity.

Sensitivity of a Video Surveillance System Based on Motion Detection

The implementation of a stand-alone system developed in JAVA language for motion detection has been discussed. The open-source OpenCV library has been adopted for video surveillance image processing thus implementing Background Subtraction algorithm also known as foreground detection algorithm. Generally the region of interest of a body or object to detect is related to a precise objects (people, cars, etc.) emphasized on a background. This technique is widely used for tracking a moving objects. In particular, the BackgroundSubtractorMOG2 algorithm of OpenCV has been applied. This algorithm is based on Gaussian distributions and offers better adaptability to different scenes due to changes in lighting and the detection of shadows as well. The implemented webcam system relies on saving frames and creating GIF and JPGs files with previously saved frames. In particular the Background Subtraction function, find Contours, has been adopted to detect the contours. The numerical quantity of these contours has been compared with the tracking points of sensitivity obtained by setting an user-modifiable slider able to save the frames as GIFs composed by different merged JPEGs. After a full design of the image processing prototype different motion test have been performed. The results showed the importance to consider few sensitivity points in order to obtain more frequent image storages also concerning minor movements. Sensitivity points can be modified through a slider function and are inversely proportional to the number of saved images. For small object in motion will be detected a low percentage of sensitivity points. Experimental results proves that the setting condition are mainly function of the typology of moving object rather than the light conditions. The proposed prototype system is suitable for video surveillance smart camera in industrial systems.

Semi-Automated Segmentation of the Tumor Vasculature in Contrast-Enhanced Ultrasound Data

The vascular architecture in tumors contains relevant information for tumor classification and evaluation of therapy responses. To develop a reliable and user-independent analysis tool, a foreground detection algorithm was combined with a maximum-intensity projection to obtain a high signal-to-noise image from contrast-enhanced B-mode data sets, enabling vessel segmentation by thresholding. Parameters describing the density of the vascular network, the number of vessels and the number of branches were extracted. The highly angiogenic A431 tumors had a relative blood volume of 49%, a mean pixel distance to the next vessel of 1.8 ± 0.3 px, 51 ± 29 individual vessels and 478 ± 184 branching points, whereas the more mature and heterogeneous vascularized human epithelial ovarian carcinoma (MLS) and A549 tumors had values of 30%, 3.7 ± 2.7 px, 65 ± 12 and 220 ± 159, and 13%, 7.4 ± 2 px, 31 ± 9 and 59 ± 40, respectively. Thus, our semi-automated analysis method enables the extraction of quantitative vascular features that may help to simplify and standardize tumor characterization.

Improving performance of coronary artery segmentation using calculated vessel location from the angiogram

Vessel segmentation in X-Ray Coronary Angiography (XCA) images, which is used for the diagnosis and treatment of anomalies in coronary vessels, has been an intriguing topic in recent years. Due to dynamic background and ambiguous brightness distribution, success rate in segmentation is low and realization has been a challenging process. This paper presents a new approach to increase the success rate of segmentation in XCA images. For this purpose, the vessel structure position is first determined with foreground detection algorithm in the XCA image sequences. Next, state of the art methods of vessel identification and image thresholding used for vessel structure segmentation are being experimented in full and foreground images. The contribution of the suggested method to the success of the segmentation is revealed with the experiments carried out. The scales of sensitivity, specificity and accuracy are given in the paper to evaluate the success of the segmentation. The results were compared with the angiography images segmented manually by a specialist physician. Without proposed detection of candidate vessel location, highest accuracy results 92.4 is achieved with Multiscale Gabor filter and PTile thresholding. Frangi Filter and Otsu thresholding algorithm reached the highest success rate of 93.81 by using the proposed method. All the experiments have shown that accuracy of vessel segmentation is increase by the proposed approach.

Sensitivity of a Video Surveillance System Based on Motion Detection

The implementation of a stand-alone system developed in JAVA language for motion detection has been discussed. The open-source OpenCV library has been adopted for video surveillance image processing thus implementing Background Subtraction algorithm also known as foreground detection algorithm. Generally the region of interest of a body or object to detect is related to a precise objects (people, cars, etc.) emphasized on a background. This technique is widely used for tracking a moving objects. In particular, the BackgroundSubtractorMOG2 algorithm of OpenCV has been applied. This algorithm is based on Gaussian distributions and offers better adaptability to different scenes due to changes in lighting and the detection of shadows as well. The implemented webcam system relies on saving frames and creating GIF and JPGs files with previously saved frames. In particular the Background Subtraction function, find Contours, has been adopted to detect the contours. The numerical quantity of these contours has been compared with the tracking points of sensitivity obtained by setting an user-modifiable slider able to save the frames as GIFs composed by different merged JPEGs. After a full design of the image processing prototype different motion test have been performed. The results showed the importance to consider few sensitivity points in order to obtain more frequent image storages also concerning minor movements. Sensitivity points can be modified through a slider function and are inversely proportional to the number of saved images. For small object in motion will be detected a low percentage of sensitivity points. Experimental results proves that the setting condition are mainly function of the typology of moving object rather than the light conditions. The proposed prototype system is suitable for video surveillance smart camera in industrial systems.

Foreground detection using texture-based codebook method for monitoring systems

Monitoring systems have been extensively applied in daily life as a result of technological advancements. Foreground detection is an important technique to refine monitoring systems. Dynamic backgrounds in the environment form a rather challenging issue in foreground detection. For example, fluttering leaves, wavelets on water, flashes on monitor screens, shadow changes are all likely to lead to erroneous analysis in foreground detection. This study proposes a foreground detection algorithm, called texture-based codebook method, which using a codebook model in combination with texture features. The codebook model used to construct a background model can effectively compress data to achieve optimal processing speeds, whereas the local binary pattern is adopted to identify texture features because of its powerful capacity for texture description and fast computation. Subsequently, texture-based connected component labeling is applied to the results from the codebook model to increase accuracy and reduce error rates. In addition, to adapt to the environment, a short-term information model has been added for better background model updates. Gradient-based time difference is also incorporated to cope with global light changes. The experiment outcome shows that the algorithm proposed is capable of real-time processing and also has good adaptability. It is able to achieve decent recognition rates when tested in different environments.