Foreground-background Segmentation Research Articles

DL-CSPF: deep-learning-based cell segmentation with a physical framework for digital holographic microscopy

Digital holographic microscopy (DHM) offers label-free, full-field imaging of live-cell samples by capturing optical path differences to produce quantitative phase images. Accurate cell segmentation from phase images is crucial for long-term quantitative analysis. However, complicated cellular states (e.g., cell adhesion, proliferation, and apoptosis) and imaging conditions (e.g., noise and magnification) pose significant challenge to the accuracy of cell segmentation. Here, we introduce DL-CSPF, a deep-learning-based cell segmentation method with a physical framework designed for high-precision live-cell analysis. DL-CSPF utilizes two neural networks for foreground-background segmentation and cell detection, generating foreground edges and “seed points.” These features serve as input for a marker-controlled watershed algorithm to segment cells. By focusing on foreground edges and “seed points”, which have lower information entropy than complete cell contours, DL-CSPF achieves accurate segmentation with a reduced dataset and without manual parameter tuning. We validated the feasibility and generalization of DL-CSPF using various open-source and DHM-collected datasets, including HeLa, pollen, and COS-7 cells. Long-term live-cell imaging results further demonstrate that DL-CSPF reliably characterized and quantitatively analyzed the morphological metrics across the cellular lifecycle, rendering it a promising tool for biomedical research.

Unsupervised co-generation of foreground–background segmentation from Text-to-Image synthesis

Text-to-Image (T2I) synthesis is a challenging task requiring modelling both textual and image domains and their relationship. The substantial improvement in image quality achieved by recent works has paved the way for numerous applications such as language-aided image editing, computer-aided design, text-based image retrieval, and training data augmentation. In this work, we ask a simple question: Along with realistic images, can we obtain any useful by-product (e.g. foreground/background or multi-class segmentation masks, detection labels) in an unsupervised way that will also benefit other computer vision tasks and applications?. In an attempt to answer this question, we explore generating realistic images and their corresponding foreground/background segmentation masks from the given text. To achieve this, we experiment the concept of co-segmentation along with GAN. Specifically, a novel GAN architecture called Co-Segmentation Inspired GAN (COS-GAN) is proposed that generates two or more images simultaneously from different noise vectors and utilises a spatial co-attention mechanism between the image features to produce realistic segmentation masks for each of the generated images. The advantages of such an architecture are two-fold: (1) The generated segmentation masks can be used to focus on foreground and background exclusively to improve the quality of generated images, and (2) the segmentation masks can be used as a training target for other tasks, such as object localisation and segmentation. Extensive experiments conducted on CUB, Oxford-102, and COCO datasets show that COS-GAN is able to improve visual quality and generate reliable foreground/background masks for the generated images.

Sea–Land Segmentation of Remote-Sensing Images with Prompt Mask-Attention

Remote-sensing technology has gradually become one of the most important ways to extract sea–land boundaries due to its large scale, high efficiency, and low cost. However, sea–land segmentation (SLS) is still a challenging problem because of data diversity and inconsistency, “different objects with the same spectrum” or “the same object with different spectra”, and noise and interference problems, etc. In this paper, a new sea–land segmentation method (PMFormer) for remote-sensing images is proposed. The contributions are mainly two points. First, based on Mask2Former architecture, we introduce the prompt mask by normalized difference water index (NDWI) of the target image and prompt encoder architecture. The prompt mask provides more reasonable constraints for attention so that the segmentation errors are alleviated in small region boundaries and small branches, which are caused by insufficiency of prior information by large data diversity or inconsistency. Second, for the large intra-class difference problem in the foreground–background segmentation in sea–land scenes, we use deep clustering to simplify the query vectors and make them more suitable for binary segmentation. Then, traditional NDWI and eight other deep-learning methods are thoroughly compared with the proposed PMFormer on three open sea–land datasets. The efficiency of the proposed method is confirmed, after the quantitative analysis, qualitative analysis, time consumption, error distribution, etc. are presented by detailed contrast experiments.

Moving object detection based on bioinspired background subtraction

Flying insects rely mainly upon visual motion to detect and track objects. There has been a lot of research on fly inspired algorithms for object detection, but few have been developed based on visual motion alone. One of the daunting difficulties is that the neural and circuit mechanisms underlying the foreground-background segmentation are still unclear. Our previous modeling study proposed that the lobula held parallel pathways with distinct directional selectivity, each of which could retinotopically discriminate figures moving in its own preferred direction based on relative motion cues. The previous model, however, did not address how the multiple parallel pathways gave the only detection output at their common downstream. Since the preferred directions of the pathways along either horizontal or vertical axis were opposite to each other, the background moving in the opposite direction to an object also activated the corresponding lobula pathway. Indiscriminate or ungated projection from all the pathways to their downstream would mix objects with the moving background, making the previous model fail with non-stationary background. Here, we extend the previous model by proposing that the background motion-dependent gating of individual lobula projections is the key to object detection. Large-field lobula plate tangential cells are hypothesized to perform the gating to realize bioinspired background subtraction. The model is shown to be capable of implementing a robust detection of moving objects in video sequences with either a moving camera that induces translational optic flow or a static camera. The model sheds light on the potential of the concise fly algorithm in real-world applications.

Leveraging foreground–background cues for semantically-driven, training-free moving object detection

Deep learning algorithms and semantic cue-based methods are increasingly used to detect moving objects. The semantic label changes in pixel location over time are used for moving object detection in semantic cue-based studies. In this approach, traditional background subtraction is employed only when there is a lack of semantic background. However, the change in semantic information in each pixel is insufficient for accurate detection of moving objects due to similar labels assignments for different objects and limited use of pixel intensity data. Moreover, challenges arise in detecting moving objects, such as a lack of diversity in labeled objects, an uneven distribution of labels or classes, and an insufficient amount of unseen datasets. In this study, we propose the Moving Semantic Object Detector (MSemOD), a train-less moving object detection method based on a traditional method enhanced with semantic information provided by pre-trained semantic segmentation models. The proposed method includes a region-based decision step, utilizing the traditional methods and semantic information as spatial–temporal cues. Threshold values are applied in the method to determine the temporal and spatial movements of each segment in the image. To improve the consistency of the semantic information, the proposed method converts the semantic segmentation output into a semantic probability map by enhancing the temporal and spatial state. The performance of the proposed method is evaluated on the LASIESTA, CDNet2014, and the BMC datasets and the results are compared with the state-of-the-art algorithms. The results show that using region-based semantic information can significantly reduce false positives and enhance foreground–background segmentation in moving object detection.

Accurate segmentation of infrared images for circuit board diagnosis using an improved Deeplabv3+ network

An effective infrared image segmentation algorithm is essential for non-contact fault diagnosis of circuit boards. However, the uneven grayscale of the infrared images, multiple target regions, and large radiation noise pose challenges to achieving accurate segmentation and efficient data extraction for the interested regions. In this paper, we propose a segmentation algorithm based on the Deeplabv3+ network, using the lightweight MobileNetV2 as a replacement for the original Xception backbone network to improve computational efficiency and reduce overfitting. We also employ a composite loss function and cosine annealing learning rate to balance foreground-background segmentation and avoid local optima. Furthermore, we integrate the Convolutional Block Attention Module (CBAM) to extract and combine important spatial and channel features, allowing the algorithm to focus on identifying elements of the circuit board instead of background pixels, thereby improving segmentation accuracy. Experimental results demonstrate that our proposed algorithm achieves state-of-the-art performance in terms of both segmentation accuracy and computational efficiency on our self-built infrared circuit board dataset, with a MIoU of 90.34%, MPA of 95.26%, and processing speed of 25.19 fps. Overall, our proposed segmentation algorithm can effectively identify the key regions of interest in infrared images of circuit boards, providing technical support for non-contact diagnosis.

ILSGAN: Independent Layer Synthesis for Unsupervised Foreground-Background Segmentation

Unsupervised foreground-background segmentation aims at extracting salient objects from cluttered backgrounds, where Generative Adversarial Network (GAN) approaches, especially layered GANs, show great promise. However, without human annotations, they are typically prone to produce foreground and background layers with non-negligible semantic and visual confusion, dubbed "information leakage", resulting in notable degeneration of the generated segmentation mask. To alleviate this issue, we propose a simple-yet-effective explicit layer independence modeling approach, termed Independent Layer Synthesis GAN (ILSGAN), pursuing independent foreground-background layer generation by encouraging their discrepancy. Specifically, it targets minimizing the mutual information between visible and invisible regions of the foreground and background to spur interlayer independence. Through in-depth theoretical and experimental analyses, we justify that explicit layer independence modeling is critical to suppressing information leakage and contributes to impressive segmentation performance gains. Also, our ILSGAN achieves strong state-of-the-art generation quality and segmentation performance on complex real-world data.

Foreground-Back ground Segmentation using K-Means Clustering Algorithm and Support Vector Machine

Foreground-background image segmentation has been an important research problem. It is one of the main tasks in the field of computer vision whose purpose is detecting variations in image sequences. It provides candidate objects for further attentional selection, e.g., in video surveillance. In this paper, we introduce an automatic and efficient Foreground-background segmentation. The proposed method starts with the detection of visually salient image regions with a saliency map that uses Fourier transform and a Gaussian filter. Then, each point in the maps classifies as salient or non-salient using a binary threshold. Next, a hole filling operator is applied for filling holes in the achieved image, and the area-opening method is used for removing small objects from the image. For better separation of the foreground and background, dilation and erosion operators are also used. Erosion and dilation operators are applied for shrinking and expanding the achieved region. Afterward, the foreground and background samples are achieved. Because the number of these data is large, K-means clustering is used as a sampling technique to restrict computational efforts in the region of interest. K cluster centers for each region are set for training of Support Vector Machine (SVM). SVM, as a powerful binary classifier, is used to segment the interest area from the background. The proposed method is applied on a benchmark dataset consisting of 1000 images and experimental results demonstrate the supremacy of the proposed method to some other foreground-background segmentation methods in terms of ER, VI, GCE, and PRI.

Background-foreground segmentation for interior sensing in automotive industry

To ensure safety in automated driving, the correct perception of the situation inside the car is as important as its environment. Thus, seat occupancy detection and classification of detected instances play an important role in interior sensing. By the knowledge of the seat occupancy status, it is possible to, e.g., automate the airbag deployment control. Furthermore, the presence of a driver, which is necessary for partially automated driving cars at the automation levels two to four can be verified. In this work, we compare different statistical methods from the field of image segmentation to approach the problem of background-foreground segmentation in camera based interior sensing. In the recent years, several methods based on different techniques have been developed and applied to images or videos from different applications. The peculiarity of the given scenarios of interior sensing is, that the foreground instances and the background both contain static as well as dynamic elements. In data considered in this work, even the camera position is not completely fixed. We review and benchmark three different methods ranging, i.e., Gaussian Mixture Models (GMM), Morphological Snakes and a deep neural network, namely a Mask R-CNN. In particular, the limitations of the classical methods, GMM and Morphological Snakes, for interior sensing are shown. Furthermore, it turns, that it is possible to overcome these limitations by deep learning, e.g. using a Mask R-CNN. Although only a small amount of ground truth data was available for training, we enabled the Mask R-CNN to produce high quality background-foreground masks via transfer learning. Moreover, we demonstrate that certain augmentation as well as pre- and post-processing methods further enhance the performance of the investigated methods.

Automated Cell Foreground-Background Segmentation with Phase-Contrast Microscopy Images: An Alternative to Machine Learning Segmentation Methods with Small-Scale Data.

Cell segmentation is a critical step for image-based experimental analysis. Existing cell segmentation methods are neither entirely automated nor perform well under basic laboratory microscopy. This study proposes an efficient and automated cell segmentation method involving morphological operations to automatically achieve cell segmentation for phase-contrast microscopes. Manual/visual counting of cell segmentation serves as the control group (156 images as ground truth) to evaluate the proposed method’s performance. The proposed technology’s adaptive performance is assessed at varying conditions, including artificial blurriness, illumination, and image size. Compared to the Trainable Weka Segmentation method, the Empirical Gradient Threshold method, and the ilastik segmentation software, the proposed method achieved better segmentation accuracy (dice coefficient: 90.07, IoU: 82.16%, and 6.51% as the average relative error on measuring cell area). The proposed method also has good reliability, even under unfavored imaging conditions at which manual labeling or human intervention is inefficient. Additionally, similar degrees of segmentation accuracy were confirmed when the ground truth data and the generated data from the proposed method were applied individually to train modified U-Net models (16848 images). These results demonstrated good accuracy and high practicality of the proposed cell segmentation method with phase-contrast microscopy image data.

Visual perception and local features for foreground‐background segmentation

The traditional foreground-background segmentation models mainly depend on the low-level features of the image, while ignoring the visual effect. Combining visual perception and local features, a top-down segmentation model is proposed. This model regards foreground-background segmentation as a reasoning problem based on visual perception, and calculates the association between two-pixel blocks through Kullback–Leibler divergence, which solves the ill-posed problem of traditional single-pixel recognition. Meanwhile, local features are used to optimize the overall segmentation results in detail and improve the segmentation accuracy. The experimental results on the CMU-Cornell iCoseg database and the BSDS500 database show that visual perception and local features can help improve the segmentation performance to a certain extent.

A weakly supervised framework for real-world point cloud classification

Real-world point cloud objects pose great challenges in point cloud classification as objects acquired by scanning devices from real-world scans are often cluttered with background, and are partial due to occlusions as well as reconstruction errors. In the literature, few works tackle the problem of real-world point cloud classification while existing methods require fully point-level annotated training samples. However, large-scale dense point-level foreground–background labeling for real-world point clouds is a labor-intensive and time-consuming job. Leveraging two auxiliary modules, called semi-supervised point-level pseudo labels generation and noise-robust multi-task loss, the framework can integrate well with existing supervised point cloud classification network. A relational graph convolutional network on the local and non-local graph (PointRGCN) is first proposed to predict point-level foreground–background pseudo labels for each object with sparse ground-truth point-level foreground–background labels in training datasets. Then, a weakly supervised classification network, which combines with an auxiliary foreground–background segmentation branch, is employed to classify real-world point clouds. To cope with noise-containing point-level foreground–background labels generated above, a noise-robust multi-task loss is proposed to train the network accurately. Experimental results show that the performance of the proposed framework which is trained with only 1% point-level labels is comparable with many popular or state-of-the-art fully supervised methods. The source code will be available at http://zhiyongsu.github.io.

Multi‐frame based adversarial learning approach for video surveillance

Foreground-background segmentation (FBS) is one of the prime tasks for automated video-based applications like traffic analysis and surveillance. The different practical scenarios like weather degraded videos, irregular moving objects, dynamic background, etc., make FBS a challenging task. The existing FBS algorithms mainly depend on one of the three different factors, namely (1) complicated training process, (2) additionally trained modules for other applications, or (3) neglect the inter-frame spatio-temporal structural dependencies. In this paper, a novel multi-frame-based adversarial learning network is proposed with multi-scale inception and residual module for FBS. As, FBS is a temporal enlightenment-based problem, a temporal encoding mechanism with decreasing variable intervals is proposed for the input frame selection. The proposed network comprises multi-scale inception and residual connection-based dense modules to learn prominent features of the foreground object(s). Also, feedback of the estimated foreground map of previous frame is utilized to exhibit more temporal consistency. Learning of the network is concentrated in different ways like cross-data, disjoint, and global training-testing for FBS. The qualitative and quantitative experimental analysis of the proposed approach is done on three benchmark datasets for FBS. Experimental analysis on three benchmark datasets proves the significance of the proposed approach as compared to state-of-the-art FBS approaches.

Research on fingerprint image recognition based on convolution neural network

In order to overcome the problem of poor image matching performance of the image recognition method, a method of fingerprint image recognition based on convolution neural network is proposed. In this method, the defaced fingerprint image is pre-processed by smoothing, convergence, equalisation, background foreground segmentation and distortion correction, and the feature points of the defaced fingerprint image are extracted by combining the neighbourhood judgment method, and the information pseudo feature points are removed by fusing the feature points, the centre points are extracted from the feature points of the defaced fingerprint image, and the centre block image is identified by convolution neural network, so as to realise the defaced fingerprint image distinguish. The experimental results show that the performance of restoration and reconstruction is improved. The rejection rate (FRR) is 3.75%, the false recognition rate (FAR) is 1.25%, and the correct recognition rate (CR) is 99.25%.

PIXEL-BASED FOREGROUND DETECTION IN REPETITIVE TIME-SERIES REGION

Currently, many state-of-the-art background subtraction techniques cannot deal properly with the area of periodic changing background, while some continue classifying them as foreground at intervals, others simply mask that area as a non-region of interest. To cope with this issue, a novel method of detecting repetitive temporal patterns based on the image sequences was proposed in this paper. The main emphasis of the proposed approach is on classifying those pixels as a background and identifying foreground objects in their relevant areas. As for the foreground detection, a model of time series pattern found in each pixel is individually built first; and then, any changes beyond the allowance of model periodicity are then determined as foreground objects. The proposed method could be used and run in parallel with any state-of-the-art background subtraction technique, allowing more accurate foreground-background segmentation. Experimental results showed that using Y channel, the proposed method of detecting time-series background area could achieve 92.9% of recall rate with less than 1% false positives. The recall of foreground detection in an area of repetitive time-series pattern was about 87%; while F-measure was about 0.73 on average. The false positives of foreground detection were also less than 1%. Accordingly, the proposed time-delay detection technique could significantly help to suppress the foreground error on time series background area, especially during the change from one sub-pattern to another which causes a camera sensor to capture both sub-pattern values in one frame. Performance comparison with state-of-the-art methods showed that our proposed method was able to reduce 80% of the average false alarm and improve F-measure to 28% while the computational efficiency was reduced by only 1%.

Target-Aware Correlation Filter Tracking in RGBD Videos

Recently, the surge of depth sensors makes RGBD data available and facilitates the development of RGBD tracking. Inspired by the popularity of discriminative correlation filter (DCF)-based trackers in RGB tracking, we propose a target-aware framework for RGBD tracking based on the existing DCF-based trackers in this paper. The framework comprehensively utilizes the prior, depth, and color information to compute a fine foreground-background segmentation map. This map is masked on the extracted features for adaptive reweighting and adjusts the DCF-based trackers to focus on the area with higher target possibility. Meanwhile, we propose an occlusion detection and handling mechanism based on this segmentation map, which helps to detect an occlusion early and avoids contaminations of the target model. Our proposed framework overcomes the limitations of the cosine window generally adopted in the existing DCF-based trackers and could be incorporated into these trackers. Experiments on the princeton tracking benchmark (PTB) demonstrate that DCF-based trackers outperform their baselines after integrated with our framework. Among these enhanced trackers, our proposed tracker ECO_TA (target-aware) based on efficient convolution operators (ECOs) for tracking achieves the best performance on the PTB and another RGBD benchmarks.

Image Segmentation using a Fuzzy Roughness Measure

Measures of uncertainty are highly useful for determining the information content of a system. In this paper, new measures of information on fuzzy approximation spaces are introduced based on divergence measures of fuzzy sets. The proposed fuzzy rough uncertainty measure is used to develop an algorithm for histogram based foreground background segmentation of a grey level image and it is experimented with twelve standard test images. It is observed that the overlapping of the foreground background pixels in the images segmented using the proposed method is lesser than those produced by OTSU and FCM methods. The segmented images are compared using their root mean square error values.

Deep Learning Approaches for Whiteboard Image Quality Enhancement

Different whiteboard image degradations highly reduce the legibility of pen-stroke content as well as the overall quality of the images. Consequently, different researchers addressed the problem through different image enhancement techniques. Most of the state-of-the-art approaches applied common image processing techniques such as background foreground segmentation, text extraction, contrast and color enhancements and white balancing. However, such types of conventional enhancement methods are incapable of recovering severely degraded pen-stroke contents and produce artifacts in the presence of complex pen-stroke illustrations. In order to surmount such problems, the authors have proposed a deep learning based solution. They have contributed a new whiteboard image data set and adopted two deep convolutional neural network architectures for whiteboard image quality enhancement applications. Their different evaluations of the trained models demonstrated their superior performances over the conventional methods.

Robust Pan/Tilt Compensation for Foreground-Background Segmentation.

In this paper, we describe a robust method for compensating the panning and tilting motion of a camera, applied to foreground–background segmentation. First, the necessary internal camera parameters are determined through feature-point extraction and tracking. From these parameters, two motion models for points in the image plane are established. The first model assumes a fixed tilt angle, whereas the second model allows simultaneous pan and tilt. At runtime, these models are used to compensate for the motion of the camera in the background model. We will show that these methods provide a robust compensation mechanism and improve the foreground masks of an otherwise state-of-the-art unsupervised foreground–background segmentation method. The resulting algorithm is always able to obtain scores above on every daytime video in our test set when a minimal number of only eight feature matches are used to determine the background compensation, whereas the standard approaches need significantly more feature matches to produce similar results.

Illumination-Aware Multi-Task GANs for Foreground Segmentation

Foreground-background segmentation has been an active research area over the years. However, conventional models fail to produce accurate results when challenged with the videos of challenging illumination conditions. In this paper, we present a robust model that allows accurately extracting the foreground even in exceptionally dark or bright scenes and in continuously varying illumination in a video sequence. This is accomplished by a triple multi-task generative adversarial network (TMT-GAN) that effectively models the semantic relationship between the dark and bright images and performs binary segmentation end-to-end. Our contribution is twofold: first, we show that by jointly optimizing the GAN loss and the segmentation loss, our network simultaneously learns both tasks that mutually benefit each other. Second, fusing features of images with varying illumination into the segmentation branch vastly improve the performance of the network. Comparative evaluations on highly challenging real and synthetic benchmark datasets (ESI and SABS) demonstrate the robustness of TMT-GAN and its superiority over state-of-the-art approaches.