Foreground-background Segmentation Research Articles

Sea–Land Segmentation of Remote-Sensing Images with Prompt Mask-Attention

Remote-sensing technology has gradually become one of the most important ways to extract sea–land boundaries due to its large scale, high efficiency, and low cost. However, sea–land segmentation (SLS) is still a challenging problem because of data diversity and inconsistency, “different objects with the same spectrum” or “the same object with different spectra”, and noise and interference problems, etc. In this paper, a new sea–land segmentation method (PMFormer) for remote-sensing images is proposed. The contributions are mainly two points. First, based on Mask2Former architecture, we introduce the prompt mask by normalized difference water index (NDWI) of the target image and prompt encoder architecture. The prompt mask provides more reasonable constraints for attention so that the segmentation errors are alleviated in small region boundaries and small branches, which are caused by insufficiency of prior information by large data diversity or inconsistency. Second, for the large intra-class difference problem in the foreground–background segmentation in sea–land scenes, we use deep clustering to simplify the query vectors and make them more suitable for binary segmentation. Then, traditional NDWI and eight other deep-learning methods are thoroughly compared with the proposed PMFormer on three open sea–land datasets. The efficiency of the proposed method is confirmed, after the quantitative analysis, qualitative analysis, time consumption, error distribution, etc. are presented by detailed contrast experiments.

Moving object detection based on bioinspired background subtraction

Flying insects rely mainly upon visual motion to detect and track objects. There has been a lot of research on fly inspired algorithms for object detection, but few have been developed based on visual motion alone. One of the daunting difficulties is that the neural and circuit mechanisms underlying the foreground-background segmentation are still unclear. Our previous modeling study proposed that the lobula held parallel pathways with distinct directional selectivity, each of which could retinotopically discriminate figures moving in its own preferred direction based on relative motion cues. The previous model, however, did not address how the multiple parallel pathways gave the only detection output at their common downstream. Since the preferred directions of the pathways along either horizontal or vertical axis were opposite to each other, the background moving in the opposite direction to an object also activated the corresponding lobula pathway. Indiscriminate or ungated projection from all the pathways to their downstream would mix objects with the moving background, making the previous model fail with non-stationary background. Here, we extend the previous model by proposing that the background motion-dependent gating of individual lobula projections is the key to object detection. Large-field lobula plate tangential cells are hypothesized to perform the gating to realize bioinspired background subtraction. The model is shown to be capable of implementing a robust detection of moving objects in video sequences with either a moving camera that induces translational optic flow or a static camera. The model sheds light on the potential of the concise fly algorithm in real-world applications.

Leveraging foreground–background cues for semantically-driven, training-free moving object detection

Deep learning algorithms and semantic cue-based methods are increasingly used to detect moving objects. The semantic label changes in pixel location over time are used for moving object detection in semantic cue-based studies. In this approach, traditional background subtraction is employed only when there is a lack of semantic background. However, the change in semantic information in each pixel is insufficient for accurate detection of moving objects due to similar labels assignments for different objects and limited use of pixel intensity data. Moreover, challenges arise in detecting moving objects, such as a lack of diversity in labeled objects, an uneven distribution of labels or classes, and an insufficient amount of unseen datasets. In this study, we propose the Moving Semantic Object Detector (MSemOD), a train-less moving object detection method based on a traditional method enhanced with semantic information provided by pre-trained semantic segmentation models. The proposed method includes a region-based decision step, utilizing the traditional methods and semantic information as spatial–temporal cues. Threshold values are applied in the method to determine the temporal and spatial movements of each segment in the image. To improve the consistency of the semantic information, the proposed method converts the semantic segmentation output into a semantic probability map by enhancing the temporal and spatial state. The performance of the proposed method is evaluated on the LASIESTA, CDNet2014, and the BMC datasets and the results are compared with the state-of-the-art algorithms. The results show that using region-based semantic information can significantly reduce false positives and enhance foreground–background segmentation in moving object detection.

Accurate segmentation of infrared images for circuit board diagnosis using an improved Deeplabv3+ network

An effective infrared image segmentation algorithm is essential for non-contact fault diagnosis of circuit boards. However, the uneven grayscale of the infrared images, multiple target regions, and large radiation noise pose challenges to achieving accurate segmentation and efficient data extraction for the interested regions. In this paper, we propose a segmentation algorithm based on the Deeplabv3+ network, using the lightweight MobileNetV2 as a replacement for the original Xception backbone network to improve computational efficiency and reduce overfitting. We also employ a composite loss function and cosine annealing learning rate to balance foreground-background segmentation and avoid local optima. Furthermore, we integrate the Convolutional Block Attention Module (CBAM) to extract and combine important spatial and channel features, allowing the algorithm to focus on identifying elements of the circuit board instead of background pixels, thereby improving segmentation accuracy. Experimental results demonstrate that our proposed algorithm achieves state-of-the-art performance in terms of both segmentation accuracy and computational efficiency on our self-built infrared circuit board dataset, with a MIoU of 90.34%, MPA of 95.26%, and processing speed of 25.19 fps. Overall, our proposed segmentation algorithm can effectively identify the key regions of interest in infrared images of circuit boards, providing technical support for non-contact diagnosis.

ILSGAN: Independent Layer Synthesis for Unsupervised Foreground-Background Segmentation

Unsupervised foreground-background segmentation aims at extracting salient objects from cluttered backgrounds, where Generative Adversarial Network (GAN) approaches, especially layered GANs, show great promise. However, without human annotations, they are typically prone to produce foreground and background layers with non-negligible semantic and visual confusion, dubbed "information leakage", resulting in notable degeneration of the generated segmentation mask. To alleviate this issue, we propose a simple-yet-effective explicit layer independence modeling approach, termed Independent Layer Synthesis GAN (ILSGAN), pursuing independent foreground-background layer generation by encouraging their discrepancy. Specifically, it targets minimizing the mutual information between visible and invisible regions of the foreground and background to spur interlayer independence. Through in-depth theoretical and experimental analyses, we justify that explicit layer independence modeling is critical to suppressing information leakage and contributes to impressive segmentation performance gains. Also, our ILSGAN achieves strong state-of-the-art generation quality and segmentation performance on complex real-world data.

Foreground-Back ground Segmentation using K-Means Clustering Algorithm and Support Vector Machine

Foreground-background image segmentation has been an important research problem. It is one of the main tasks in the field of computer vision whose purpose is detecting variations in image sequences. It provides candidate objects for further attentional selection, e.g., in video surveillance. In this paper, we introduce an automatic and efficient Foreground-background segmentation. The proposed method starts with the detection of visually salient image regions with a saliency map that uses Fourier transform and a Gaussian filter. Then, each point in the maps classifies as salient or non-salient using a binary threshold. Next, a hole filling operator is applied for filling holes in the achieved image, and the area-opening method is used for removing small objects from the image. For better separation of the foreground and background, dilation and erosion operators are also used. Erosion and dilation operators are applied for shrinking and expanding the achieved region. Afterward, the foreground and background samples are achieved. Because the number of these data is large, K-means clustering is used as a sampling technique to restrict computational efforts in the region of interest. K cluster centers for each region are set for training of Support Vector Machine (SVM). SVM, as a powerful binary classifier, is used to segment the interest area from the background. The proposed method is applied on a benchmark dataset consisting of 1000 images and experimental results demonstrate the supremacy of the proposed method to some other foreground-background segmentation methods in terms of ER, VI, GCE, and PRI.

Background-foreground segmentation for interior sensing in automotive industry

To ensure safety in automated driving, the correct perception of the situation inside the car is as important as its environment. Thus, seat occupancy detection and classification of detected instances play an important role in interior sensing. By the knowledge of the seat occupancy status, it is possible to, e.g., automate the airbag deployment control. Furthermore, the presence of a driver, which is necessary for partially automated driving cars at the automation levels two to four can be verified. In this work, we compare different statistical methods from the field of image segmentation to approach the problem of background-foreground segmentation in camera based interior sensing. In the recent years, several methods based on different techniques have been developed and applied to images or videos from different applications. The peculiarity of the given scenarios of interior sensing is, that the foreground instances and the background both contain static as well as dynamic elements. In data considered in this work, even the camera position is not completely fixed. We review and benchmark three different methods ranging, i.e., Gaussian Mixture Models (GMM), Morphological Snakes and a deep neural network, namely a Mask R-CNN. In particular, the limitations of the classical methods, GMM and Morphological Snakes, for interior sensing are shown. Furthermore, it turns, that it is possible to overcome these limitations by deep learning, e.g. using a Mask R-CNN. Although only a small amount of ground truth data was available for training, we enabled the Mask R-CNN to produce high quality background-foreground masks via transfer learning. Moreover, we demonstrate that certain augmentation as well as pre- and post-processing methods further enhance the performance of the investigated methods.

Automated Cell Foreground-Background Segmentation with Phase-Contrast Microscopy Images: An Alternative to Machine Learning Segmentation Methods with Small-Scale Data.

Cell segmentation is a critical step for image-based experimental analysis. Existing cell segmentation methods are neither entirely automated nor perform well under basic laboratory microscopy. This study proposes an efficient and automated cell segmentation method involving morphological operations to automatically achieve cell segmentation for phase-contrast microscopes. Manual/visual counting of cell segmentation serves as the control group (156 images as ground truth) to evaluate the proposed method’s performance. The proposed technology’s adaptive performance is assessed at varying conditions, including artificial blurriness, illumination, and image size. Compared to the Trainable Weka Segmentation method, the Empirical Gradient Threshold method, and the ilastik segmentation software, the proposed method achieved better segmentation accuracy (dice coefficient: 90.07, IoU: 82.16%, and 6.51% as the average relative error on measuring cell area). The proposed method also has good reliability, even under unfavored imaging conditions at which manual labeling or human intervention is inefficient. Additionally, similar degrees of segmentation accuracy were confirmed when the ground truth data and the generated data from the proposed method were applied individually to train modified U-Net models (16848 images). These results demonstrated good accuracy and high practicality of the proposed cell segmentation method with phase-contrast microscopy image data.

Visual perception and local features for foreground‐background segmentation

The traditional foreground-background segmentation models mainly depend on the low-level features of the image, while ignoring the visual effect. Combining visual perception and local features, a top-down segmentation model is proposed. This model regards foreground-background segmentation as a reasoning problem based on visual perception, and calculates the association between two-pixel blocks through Kullback–Leibler divergence, which solves the ill-posed problem of traditional single-pixel recognition. Meanwhile, local features are used to optimize the overall segmentation results in detail and improve the segmentation accuracy. The experimental results on the CMU-Cornell iCoseg database and the BSDS500 database show that visual perception and local features can help improve the segmentation performance to a certain extent.

A weakly supervised framework for real-world point cloud classification

Real-world point cloud objects pose great challenges in point cloud classification as objects acquired by scanning devices from real-world scans are often cluttered with background, and are partial due to occlusions as well as reconstruction errors. In the literature, few works tackle the problem of real-world point cloud classification while existing methods require fully point-level annotated training samples. However, large-scale dense point-level foreground–background labeling for real-world point clouds is a labor-intensive and time-consuming job. Leveraging two auxiliary modules, called semi-supervised point-level pseudo labels generation and noise-robust multi-task loss, the framework can integrate well with existing supervised point cloud classification network. A relational graph convolutional network on the local and non-local graph (PointRGCN) is first proposed to predict point-level foreground–background pseudo labels for each object with sparse ground-truth point-level foreground–background labels in training datasets. Then, a weakly supervised classification network, which combines with an auxiliary foreground–background segmentation branch, is employed to classify real-world point clouds. To cope with noise-containing point-level foreground–background labels generated above, a noise-robust multi-task loss is proposed to train the network accurately. Experimental results show that the performance of the proposed framework which is trained with only 1% point-level labels is comparable with many popular or state-of-the-art fully supervised methods. The source code will be available at http://zhiyongsu.github.io.

Multi‐frame based adversarial learning approach for video surveillance

Foreground-background segmentation (FBS) is one of the prime tasks for automated video-based applications like traffic analysis and surveillance. The different practical scenarios like weather degraded videos, irregular moving objects, dynamic background, etc., make FBS a challenging task. The existing FBS algorithms mainly depend on one of the three different factors, namely (1) complicated training process, (2) additionally trained modules for other applications, or (3) neglect the inter-frame spatio-temporal structural dependencies. In this paper, a novel multi-frame-based adversarial learning network is proposed with multi-scale inception and residual module for FBS. As, FBS is a temporal enlightenment-based problem, a temporal encoding mechanism with decreasing variable intervals is proposed for the input frame selection. The proposed network comprises multi-scale inception and residual connection-based dense modules to learn prominent features of the foreground object(s). Also, feedback of the estimated foreground map of previous frame is utilized to exhibit more temporal consistency. Learning of the network is concentrated in different ways like cross-data, disjoint, and global training-testing for FBS. The qualitative and quantitative experimental analysis of the proposed approach is done on three benchmark datasets for FBS. Experimental analysis on three benchmark datasets proves the significance of the proposed approach as compared to state-of-the-art FBS approaches.

Research on fingerprint image recognition based on convolution neural network

In order to overcome the problem of poor image matching performance of the image recognition method, a method of fingerprint image recognition based on convolution neural network is proposed. In this method, the defaced fingerprint image is pre-processed by smoothing, convergence, equalisation, background foreground segmentation and distortion correction, and the feature points of the defaced fingerprint image are extracted by combining the neighbourhood judgment method, and the information pseudo feature points are removed by fusing the feature points, the centre points are extracted from the feature points of the defaced fingerprint image, and the centre block image is identified by convolution neural network, so as to realise the defaced fingerprint image distinguish. The experimental results show that the performance of restoration and reconstruction is improved. The rejection rate (FRR) is 3.75%, the false recognition rate (FAR) is 1.25%, and the correct recognition rate (CR) is 99.25%.

PIXEL-BASED FOREGROUND DETECTION IN REPETITIVE TIME-SERIES REGION

Currently, many state-of-the-art background subtraction techniques cannot deal properly with the area of periodic changing background, while some continue classifying them as foreground at intervals, others simply mask that area as a non-region of interest. To cope with this issue, a novel method of detecting repetitive temporal patterns based on the image sequences was proposed in this paper. The main emphasis of the proposed approach is on classifying those pixels as a background and identifying foreground objects in their relevant areas. As for the foreground detection, a model of time series pattern found in each pixel is individually built first; and then, any changes beyond the allowance of model periodicity are then determined as foreground objects. The proposed method could be used and run in parallel with any state-of-the-art background subtraction technique, allowing more accurate foreground-background segmentation. Experimental results showed that using Y channel, the proposed method of detecting time-series background area could achieve 92.9% of recall rate with less than 1% false positives. The recall of foreground detection in an area of repetitive time-series pattern was about 87%; while F-measure was about 0.73 on average. The false positives of foreground detection were also less than 1%. Accordingly, the proposed time-delay detection technique could significantly help to suppress the foreground error on time series background area, especially during the change from one sub-pattern to another which causes a camera sensor to capture both sub-pattern values in one frame. Performance comparison with state-of-the-art methods showed that our proposed method was able to reduce 80% of the average false alarm and improve F-measure to 28% while the computational efficiency was reduced by only 1%.

Target-Aware Correlation Filter Tracking in RGBD Videos

Recently, the surge of depth sensors makes RGBD data available and facilitates the development of RGBD tracking. Inspired by the popularity of discriminative correlation filter (DCF)-based trackers in RGB tracking, we propose a target-aware framework for RGBD tracking based on the existing DCF-based trackers in this paper. The framework comprehensively utilizes the prior, depth, and color information to compute a fine foreground-background segmentation map. This map is masked on the extracted features for adaptive reweighting and adjusts the DCF-based trackers to focus on the area with higher target possibility. Meanwhile, we propose an occlusion detection and handling mechanism based on this segmentation map, which helps to detect an occlusion early and avoids contaminations of the target model. Our proposed framework overcomes the limitations of the cosine window generally adopted in the existing DCF-based trackers and could be incorporated into these trackers. Experiments on the princeton tracking benchmark (PTB) demonstrate that DCF-based trackers outperform their baselines after integrated with our framework. Among these enhanced trackers, our proposed tracker ECO_TA (target-aware) based on efficient convolution operators (ECOs) for tracking achieves the best performance on the PTB and another RGBD benchmarks.

Image Segmentation using a Fuzzy Roughness Measure

Measures of uncertainty are highly useful for determining the information content of a system. In this paper, new measures of information on fuzzy approximation spaces are introduced based on divergence measures of fuzzy sets. The proposed fuzzy rough uncertainty measure is used to develop an algorithm for histogram based foreground background segmentation of a grey level image and it is experimented with twelve standard test images. It is observed that the overlapping of the foreground background pixels in the images segmented using the proposed method is lesser than those produced by OTSU and FCM methods. The segmented images are compared using their root mean square error values.

Deep Learning Approaches for Whiteboard Image Quality Enhancement

Different whiteboard image degradations highly reduce the legibility of pen-stroke content as well as the overall quality of the images. Consequently, different researchers addressed the problem through different image enhancement techniques. Most of the state-of-the-art approaches applied common image processing techniques such as background foreground segmentation, text extraction, contrast and color enhancements and white balancing. However, such types of conventional enhancement methods are incapable of recovering severely degraded pen-stroke contents and produce artifacts in the presence of complex pen-stroke illustrations. In order to surmount such problems, the authors have proposed a deep learning based solution. They have contributed a new whiteboard image data set and adopted two deep convolutional neural network architectures for whiteboard image quality enhancement applications. Their different evaluations of the trained models demonstrated their superior performances over the conventional methods.

Robust Pan/Tilt Compensation for Foreground-Background Segmentation.

In this paper, we describe a robust method for compensating the panning and tilting motion of a camera, applied to foreground–background segmentation. First, the necessary internal camera parameters are determined through feature-point extraction and tracking. From these parameters, two motion models for points in the image plane are established. The first model assumes a fixed tilt angle, whereas the second model allows simultaneous pan and tilt. At runtime, these models are used to compensate for the motion of the camera in the background model. We will show that these methods provide a robust compensation mechanism and improve the foreground masks of an otherwise state-of-the-art unsupervised foreground–background segmentation method. The resulting algorithm is always able to obtain scores above on every daytime video in our test set when a minimal number of only eight feature matches are used to determine the background compensation, whereas the standard approaches need significantly more feature matches to produce similar results.

Illumination-Aware Multi-Task GANs for Foreground Segmentation

Foreground-background segmentation has been an active research area over the years. However, conventional models fail to produce accurate results when challenged with the videos of challenging illumination conditions. In this paper, we present a robust model that allows accurately extracting the foreground even in exceptionally dark or bright scenes and in continuously varying illumination in a video sequence. This is accomplished by a triple multi-task generative adversarial network (TMT-GAN) that effectively models the semantic relationship between the dark and bright images and performs binary segmentation end-to-end. Our contribution is twofold: first, we show that by jointly optimizing the GAN loss and the segmentation loss, our network simultaneously learns both tasks that mutually benefit each other. Second, fusing features of images with varying illumination into the segmentation branch vastly improve the performance of the network. Comparative evaluations on highly challenging real and synthetic benchmark datasets (ESI and SABS) demonstrate the robustness of TMT-GAN and its superiority over state-of-the-art approaches.

Extraction of residential building instances in suburban areas from mobile LiDAR data

In the recent years, mobile LiDAR data has become an important data source for building mapping. However, it is challenging to extract building instances in residential areas where buildings of different structures are closely distributed and surrounded by cluttered objects such as vegetations. In this paper, we present a new “localization then segmentation” framework to tackle these problems. First, a hypothesis and selection method is proposed to localize buildings. Rectangle proposals which indicate building locations are generated using projections of vertical walls obtained by region growing. The selection of rectangles is formulated as a constrained maximization problem, which is solved by linear programming. Then, point clouds are divided into groups, each of which contains one building instance. A foreground-background segmentation method is then proposed to extract buildings from complex surroundings in each group. Based on the graph of points, an objective function which integrates local geometric features and shape priors is minimized by the graph cuts. The experiments are conducted in two large and complex scenes, Calgary and Kentucky residential areas. The completeness and correctness of building localization in the former dataset are 87.2% and 91.34%, respectively. In the latter dataset, the completeness and correctness of building localization are 100% and 96.3%, respectively. Based on the tests, our binary segmentation method outperforms existing methods regarding the F1 measure. These results demonstrate the feasibility and effectiveness of our framework in extracting instance-level residential buildings from mobile LiDAR point clouds in suburban areas.

Foreground-Background Segmentation Revealed during Natural Image Viewing.

One of the major challenges in visual neuroscience is represented by foreground-background segmentation. Data from nonhuman primates show that segmentation leads to two distinct, but associated processes: the enhancement of neural activity during figure processing (i.e., foreground enhancement) and the suppression of background-related activity (i.e., background suppression). To study foreground-background segmentation in ecological conditions, we introduce a novel method based on parametric modulation of low-level image properties followed by application of simple computational image-processing models. By correlating the outcome of this procedure with human fMRI activity, measured during passive viewing of 334 natural images, we produced easily interpretable “correlation images” from visual populations. Results show evidence of foreground enhancement in all tested regions, from V1 to lateral occipital complex (LOC), while background suppression occurs in V4 and LOC only. Correlation images derived from V4 and LOC revealed a preserved spatial resolution of foreground textures, indicating a richer representation of the salient part of natural images, rather than a simplistic model of object shape. Our results indicate that scene segmentation occurs during natural viewing, even when individuals are not required to perform any particular task.