Foreground-background Separation Research Articles

Learning Robust and Sparse Principal Components with the α-Divergence.

In this paper, novel robust principal component analysis (RPCA) methods are proposed to exploit the local structure of datasets. The proposed methods are derived by minimizing the α-divergence between the sample distribution and the Gaussian density model. The α-divergence is used in different frameworks to represent variants of RPCA approaches including orthogonal, non-orthogonal, and sparse methods. We show that the classical PCA is a special case of our proposed methods where the α-divergence is reduced to the Kullback-Leibler (KL) divergence. It is shown in simulations that the proposed approaches recover the underlying principal components (PCs) by down-weighting the importance of structured and unstructured outliers. Furthermore, using simulated data, it is shown that the proposed methods can be applied to fMRI signal recovery and Foreground-Background (FB) separation in video analysis. Results on real world problems of FB separation as well as image reconstruction are also provided.

End-To-End Compression for Surveillance Video With Unsupervised Foreground-Background Separation

With the exponential growth of surveillance video, efficient video coding method is in great demand. The learning-based methods emerge which either directly use a general video compression framework, or separate the foreground and background and then compress them in two stages. However, they do not take into account the relatively static background fact of surveillance video, or simply separate foreground and background in offline mode which reduces the separation performance because the temporal domain correlation is not considered very well. In this paper, we propose an end-to-end Unsupervised foreground-background separation based Video Compression neural Networks, dubbed as UVCNet. Our method mainly consists of three parts. First, the Mask Net unsupervisely separates foreground and background online which sufficiently uses the temporal correlation prior. Then, a traditional motion estimation-based residual coding module is applied to foreground compression. Simultaneously, a background compression module is applied to compress background residual and update the background by sufficiently using the relatively static property. Compared with previous approaches, our method does not separate foreground and background in advance but in an end-to-end manner. So we can not only use the relatively static background property to save bit rate, but also achieve end-to-end online video compression. Experimental results demonstrate that the proposed UVCNet achieves superior performance compared with the state-of-the-art methods. Specifically, UVCNet can achieve 2.11 dB average improvement on Peak Signal-to-Noise Ratio (PSNR) compared with H.265 on surveillance datasets.

TVRPCA＋: Low-rank and sparse decomposition based on spectral norm and structural sparsity-inducing norm

Traditional low-rank sparse decomposition algorithms have trouble obtaining a clear and complete foreground representation in foreground–background separation due to the complex video environment and the noise. For this issue, We propose a more robust and higher-performance low-rank and sparse decomposition algorithm named TVRPCA＋ based on spectral norm, structured sparse norm and total variation (TV) regularization. The structured sparse norm and TV regularization are exploited to suppress noise and obtain much cleaner foregrounds. Spectral norm is used in our algorithm for the low-rank component to address the issue of over-punishment and restore more foreground information. Moreover, an efficient algorithm based on the inexact augmented Lagrange multiplier method is designed to solve the proposed optimization problem. Experimental results show that TVRPCA＋ obtained five top F-measures and three of the second-highest F-measures in eight noise-free test video sequences with complex backgrounds, while the highest average F-measure was also achieved in all ten experimental groups with noise.

Collaborative Foreground, Background, and Action Modeling Network for Weakly Supervised Temporal Action Localization

In this paper, we explore the problem of Weakly-supervised Temporal Action Localization (W-TAL), where the task is to localize the temporal boundaries of all action instances in an untrimmed video with only video-level supervision. The existing W-TAL methods achieve a good action localization performance by separating the discriminative action and background frames. However, there is still a large performance gap between the weakly and fully supervised methods. The main reason comes from that there are plenty of ambiguous action and background frames in addition to the discriminative action and background frames. Due to the lack of temporal annotations in W-TAL, the ambiguous background frames may be localized as foreground and the ambiguous action frames may be suppressed as background, which result in false positives and false negatives, respectively. In this paper, we introduce a novel collaborative Foreground, Background, and Action Modeling Network (FBA-Net) to suppress the background (i.e., both the discriminative and ambiguous background) frames, and localize the actual-action-related (i.e., both the discriminative and ambiguous action) frames as foreground, for the precise temporal action localization. We design our FBA-Net with three branches: the foreground modeling (FM) branch, the background modeling (BM) branch, and the class-specific action and background modeling (CM) branch. The CM branch learns to highlight the video frames related to <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</i> action classes, and separate the action-related frames of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</i> action classes from the ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</i> + 1)th background class. The collaboration between FM and CM regularizes the consistency between the FM and the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</i> action classes of CM, which reduces the false negative rate by localizing different actual-action-related (i.e., both the discriminative and ambiguous action) frames in a video as foreground. On the other hand, the collaboration between BM and CM regularizes the consistency between the BM and the ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</i> + 1)th background class of CM, which reduces the false positive rate by suppressing both the discriminative and ambiguous background frames. Furthermore, the collaboration between FM and BM enforces more effective foreground-background separation. To evaluate the effectiveness of our FBA-Net, we perform extensive experiments on two challenging datasets, THUMOS14 and ActivityNet1.3. The experiments show that our FBA-Net attains superior results.

Aesthetic-aware image retargeting based on foreground–background separation and PSO optimization

Aesthetic-aware image retargeting based on foreground–background separation and PSO optimization

Looking at Boundary: Siamese Densely Cooperative Fusion for Salient Object Detection

Though deep learning-based saliency detection methods have achieved gratifying performance recently, the predicted saliency maps still suffer from the boundary challenge. From the perspective of foreground-background separation, this article attempts to extract the edge information of objects by exploiting the difference between different color channels in the RGB color space and establishes a novel multicolor contrast extraction (MCE) mechanism to improve the learning ability of exquisite boundary information of the network. To make full use of the MCE outputs and RGB colors, and well depict and capture the complementary information between them, we devise a novel Siamese densely cooperative fusion (DCF) network (SDFNet) for saliency detection, which consists of two effective components: boundary-directed feature learning (BDFL) and DCF. The BDFL provides joint learning for both MCE and RGB modalities through a Siamese network, while the DCF module is devised for complementary feature discovery, in order to effectively combine the features learned from two modalities. Experiments on five well-known benchmark datasets demonstrate that the proposed method outperforms the state-of-the-art approaches in terms of different evaluation metrics. We provide a detailed analysis of these results and indicate that our joint modeling of MCE and RGB colors helps to better capture the object details, especially in the object boundaries.

Transformer-Based Approach Via Contrastive Learning for Zero-Shot Detection.

Zero-shot detection (ZSD) aims to locate and classify unseen objects in pictures or videos by semantic auxiliary information without additional training examples. Most of the existing ZSD methods are based on two-stage models, which achieve the detection of unseen classes by aligning object region proposals with semantic embeddings. However, these methods have several limitations, including poor region proposals for unseen classes, lack of consideration of semantic representations of unseen classes or their inter-class correlations, and domain bias towards seen classes, which can degrade overall performance. To address these issues, the Trans-ZSD framework is proposed, which is a transformer-based multi-scale contextual detection framework that explicitly exploits inter-class correlations between seen and unseen classes and optimizes feature distribution to learn discriminative features. Trans-ZSD is a single-stage approach that skips proposal generation and performs detection directly, allowing the encoding of long-term dependencies at multiple scales to learn contextual features while requiring fewer inductive biases. Trans-ZSD also introduces a foreground-background separation branch to alleviate the confusion of unseen classes and backgrounds, contrastive learning to learn inter-class uniqueness and reduce misclassification between similar classes, and explicit inter-class commonality learning to facilitate generalization between related classes. Trans-ZSD addresses the domain bias problem in end-to-end generalized zero-shot detection (GZSD) models by using balance loss to maximize response consistency between seen and unseen predictions, ensuring that the model does not bias towards seen classes. The Trans-ZSD framework is evaluated on the PASCAL VOC and MS COCO datasets, demonstrating significant improvements over existing ZSD models.

Multi-scale foreground-background separation for light field depth estimation with deep convolutional networks

There are three viewpoints to divide the depth from light field (DFLF) methods: light field (LF) representation, signal processing, and learning viewpoints. In the signal processing viewpoint in details, the DFLF methods can be divided into three groups: cost-based, foreground-background separation (FBS)-based, and depth model-based methods. Although the deep learning techniques are widely implemented in the cost-based and depth model-based methods for the DFLF, they are not implemented in the FBS-based method, yet. In this paper, we propose a DFLF method based on the FBS with the deep learning techniques. Based on an observation that the angular aliasing artifacts are reduced in the resulting disparity maps obtained from spatially low-scale LF images, multi-scale (MS) FBS score maps are used in the proposed methods. To combine the MS FBS score maps into a single FBS score maps, four directional LF gradients, features from the 4-D LF, and features from the center view images are utilized. Based on the sensitivity and specificity loss, the gradient matching loss between the predicted and ground truth disparity maps is used to improve the performance around the occlusion boundary in the resulting disparity maps. The experimental results show that the proposed method performs reasonably in both the synthetic and real LF datasets. Especially, in the texture-less region in the real dataset, the resulting disparity maps obtained by the proposed method show that the artifacts are reduced compared to those by the existing DFLF methods.

Weighted Schatten p-norm and Laplacian scale mixture-based low-rank and sparse decomposition for foreground–background separation

Low-rank and sparse decomposition (LRSD) plays a vital role in foreground–background separation. The existing LRSD methods have the drawback: imprecise surrogate functions of rank and sparsity. We propose the weighted Schatten p-norm (WSN) and Laplacian scale mixture (LSM) method based on LRSD for foreground–background separation, which introduces the WSN and LSM to improve this drawback. To demonstrate the performance of the proposed method, it is applied to foreground–background separation and gets the highest average F-measure score.

Fruit Classification using Colorized Depth Images

Fruit classification is a computer vision task that aims to classify fruit classes correctly, given an image. Nearly all fruit classification studies have used RGB color images as inputs, a few have used costly hyperspectral images, and a few classical ML-based have used colorized depth images. Depth images have apparent benefits such as invariance to lighting, less storage requirement, better foreground-background separation, and more pronounced curvature details and object edge discontinuities. However, the use of depth images in CNN-based fruit classification remains unexplored. The purpose of this study is to investigate the use of colorized depth images in fruit classification with four CNN models, namely, AlexNet, GoogleNet, ResNet101, and VGG16, and compare their performance and computational efficiency, as well as the impact of transfer learning. Depth images of apple, orange, mango, banana and rambutan (Nephelium Lappaceum) were manually collected using a depth sensor with sub-millimeter accuracy and subjected to jet, uniform, and inverse colorization to produce three sets of dataset. Results show that depth images can be used to train CNN models for fruit classification with ResNet101 achieving the best accuracy of 96%on the inverse dataset. It achieved 100% accuracy after transfer learning. GoogleNet showed the most significant improvement after transfer learning on the uniform dataset, at 12.27%. It also exhibited the lowest training and inference times. The results show the potential use of depth images for fruit classification and similar computer vision tasks.

Iteratively Reweighted Minimax-Concave Penalty Minimization for Accurate Low-rank Plus Sparse Matrix Decomposition.

Low-rank plus sparse matrix decomposition (LSD) is an important problem in computer vision and machine learning. It has been solved using convex relaxations of the matrix rank and l0-pseudo-norm, which are the nuclear norm and l1-norm, respectively. Convex approximations are known to result in biased estimates, to overcome which, nonconvex regularizers such as weighted nuclear-norm minimization and weighted Schatten p-norm minimization have been proposed. However, works employing these regularizers have used heuristic weight-selection strategies. We propose weighted minimax-concave penalty (WMCP) as the nonconvex regularizer and show that it admits an equivalent representation that enables weight adaptation. Similarly, an equivalent representation to the weighted matrix gamma norm (WMGN) enables weight adaptation for the low-rank part. The optimization algorithms are based on the alternating direction method of multipliers technique. We show that the optimization frameworks relying on the two penalties, WMCP and WMGN, coupled with a novel iterative weight update strategy, result in accurate low-rank plus sparse matrix decomposition. The algorithms are also shown to satisfy descent properties and convergence guarantees. On the applications front, we consider the problem of foreground-background separation in video sequences. Simulation experiments and validations on standard datasets, namely, I2R, CDnet 2012, and BMC 2012 show that the proposed techniques outperform the benchmark techniques.

REAL-TIME FOREGROUND SEGMENTATION FOR SURVEILLANCE APPLICATIONS IN NRCS LIDAR SEQUENCES

Abstract. In this paper, we propose a point-level foreground-background separation technique for the segmentation of measurement sequences of a Non-repetitive Circular Scanning (NRCS) Lidar sensor, which is used as a 3D surveillance camera mounted in a fixed position. We show that by applying the NRCS Lidar technology, we can overcome various limitations of rotating multi-beam Lidar sensors, such as low vertical measurement resolution, which is disadvantageous in surveillance applications. As the main challenge, we need to efficiently balance between the spatial and the temporal resolution of the recorded range data. For this reason, we automatically generate and maintain a very high-resolution background model of the sensor’s Field of View, while for enabling real-time analysis of dynamic objects we use low integration time to extract the consecutive time frames. As a result, the laser reflections from foreground objects reflect sparse, but geometrically accurate samples of the silhouettes providing valuable input for higher-level shape description or event analysis steps. We demonstrate the efficiency of the new approach in different realistic NRCS Lidar measurements sequences, obtaining a 0.76 overall F1-score on the measured dataset.

Home Environment Augmented Reality System Based on 3D Reconstruction of a Single Furniture Picture.

With the popularization of the concept of “metaverse”, Augmented Reality (AR) technology is slowly being applied to people’s daily life as its underlying technology support. In recent years, rapid 3D reconstruction of interior furniture to meet AR shopping needs has become a new method. In this paper, a virtual home environment system is designed and the related core technologies in the system are studied. Background removal and instance segmentation are performed for furniture images containing complex backgrounds, and a Bayesian Classifier and GrabCut (BCGC) algorithm is proposed to improve on the traditional foreground background separation technique. The reconstruction part takes the classical occupancy network reconstruction algorithm as the network basis and proposes a precise occupancy network (PONet) algorithm, which can reconstruct the structural details of furniture images, and the model accuracy is improved. Because the traditional 3D registration model is prone to the problems of model position shift and inaccurate matching with the scene, the AKAZE-based tracking registration algorithm is improved, and a Multiple Filtering-AKAZE (MF-AKAZE) based on AKAZE is proposed to remove the matching points. The matching accuracy is increased by improving the RANSAC filtering mis-matching algorithm based on further screening of the matching results. Finally, the system is verified to realize the function of the AR visualization furniture model, which can better complete the reconstruction as well as registration effect.

Detecting abnormality with separated foreground and background: Mutual Generative Adversarial Networks for video abnormal event detection

As one of the most important tasks in intelligent video analysis, video abnormal event detection has been extensively studied. Prior arts have made a great process in designing frameworks to capture spatio-temporal features of video frames. However, video frames usually consist of various objects. It is challenging to grasp the nuances of anomalies against noisy backgrounds. To tackle the bottleneck, we propose a novel Foreground–Background Separation Mutual Generative Adversarial Network (FSM-GAN) framework. The FSM-GAN permits the separation of video frames into the foreground and background. The separated foreground and background are utilized as the input of mutual generative adversarial networks, which transform raw-pixel images in optical-flow representations and vice versa. In the networks, the background is regarded as known conditions and the model focuses on learning the high-level spatio-temporal foreground features to represent the event with the given conditions during the mutual adversarial training. In the test stage, these high-level features instead of low-level visual primitives are utilized to measure the abnormality in the semantic level. Compared with state-of-the-art methods and other abnormal event detection approaches, the proposed framework demonstrates its effectiveness and reliability across various scenes and events.

Occlusion Handling by Successively Excluding Foregrounds for Light Field Depth Estimation Based on Foreground-Background Separation

This paper proposes a depth from light field (DFLF) method specifically to deal with occlusion based on the foreground-background separation (FBS). The FBS-based methods infer the disparity maps by accumulating the binary maps which divide whether each pixel is the foreground or background. Although there have been widely studied to handle the occlusion problem with the cost-based method, there are not enough researches to handle the occlusion problem with the FBS-based methods yet. We found that errors around the occlusion boundary in the resulting disparity maps of the FBS-based methods arise from the fattened foreground by the light field reprameterization. To avoid fattened foregrounds, the inferred foreground maps in the front region with respect to the disparity axis could be utilized in the back region in the three-dimensional volume construction, which corresponds to the cost volume construction in the cost-based methods. With the front-to-back scanning manner of the FBS-based method, by successively excluding inferred foreground maps, errors around occlusion boundary could be effectively reduced in the resulting disparity maps. With synthetic and real LF images, the proposed method shows reasonable performance compared to the existing methods and better performance than existing FBS-based methods.

Robust CUR Decomposition: Theory and Imaging Applications

This paper considers the use of Robust PCA in a CUR decomposition framework and applications thereof. Our main algorithms produce a robust version of column-row factorizations of matrices $\mathbf{D}=\mathbf{L}+\mathbf{S}$ where $\mathbf{L}$ is low-rank and $\mathbf{S}$ contains sparse outliers. These methods yield interpretable factorizations at low computational cost, and provide new CUR decompositions that are robust to sparse outliers, in contrast to previous methods. We consider two key imaging applications of Robust PCA: video foreground-background separation and face modeling. This paper examines the qualitative behavior of our Robust CUR decompositions on the benchmark videos and face datasets, and find that our method works as well as standard Robust PCA while being significantly faster. Additionally, we consider hybrid randomized and deterministic sampling methods which produce a compact CUR decomposition of a given matrix, and apply this to video sequences to produce canonical frames thereof.

Generalized nuclear norm and Laplacian scale mixture based low-rank and sparse decomposition for video foreground-background separation

Low-rank and sparse decomposition (LRSD) poses a big challenge problem in video foreground-background separation and many other fields due to its difficulties in approximations of low-rank and sparse parts. The rank and sparsity may not be well approximated in practice since these conventional approaches suffer from the suboptimal issues in many cases. In this paper, we adopt the generalized nuclear norm (GNN) and the Laplacian scale mixture (LSM) modeling to approximate the low-rank and sparse matrices, respectively, and propose a generalized formulation which called GNNLSM for nonconvex low-rank and sparse decomposition based on the GNN and the LSM. And then, we adopt the alternating direction method of multipliers (ADMM) to solve our proposed problem. Simulation results and discussions on video foreground-background separation are given to validate the superiority and the effectiveness of our proposed method.

Foreground-Background Separation via Generalized Nuclear Norm and Structured Sparse Norm Based Low-Rank and Sparse Decomposition

Low-rank and sparse decomposition (LRSD) has attracted wide attention in video foreground-background separation and many other fields. However, the traditional LRSD methods have many tough problems, such as the problems of the low accuracy of the surrogate functions of rank and sparsity, ignoring the spatial information of the videos and sensitivity to noise, etc. To deal with these problems, this paper proposes the generalized nuclear norm and structured sparse norm (GNNSSN) method based LRSD for video foreground-background separation, which introduces the generalized nuclear norm (GNN) and the structured sparse norm (SSN) to approximate the rank function and the l 0 -norm of the LRSD method. In addition, we extend our proposed model to a robust model against noise for practical applications, and we called the extended method as the robust generalized nuclear norm and structured sparse norm (RGNNSSN) method. At last, we use the alternating direction method of multipliers (ADMM) to solve our proposed two methods. Experimental results and discussions on video foreground-background separation demonstrate that our proposed two methods have better performances than other LRSD based foreground-background separation methods.

A Tensor Foreground-Background Separation Algorithm Based on Dynamic Dictionary Update and Active Contour Detection

Foreground-background separation of surveillance video, that models static background and extracts moving foreground simultaneously, attracts increasing attentions in building a smart city. Conventional techniques towards this always consider the background as primary target and tend to adopt low-rank constraint as its estimator, which provides finite (equal to the value of rank) alternatives when constructing the background. However, in practical missions, although general sketch of background is stable, some details change constantly. Aimed at this, we propose to represent the general background by a linear combination of some atoms and record the detailed background by spatiotemporal clustered patches. Then, the moving foreground is considered as a mixture of active contours and continuous contents. Eventually, joint optimization is conducted under a unified framework, i.e., alternating direction multipliers method (ADMM), and produces our tensor model for hierarchical background and hierarchical foreground separation (THHS). The employed tensor space, which agrees with the instinct structure of video data, benefits all the spatiotemporal designs in both background modular and foreground part. Experimental results show that THHS is more adaptive to the dynamic background and produces more accurate foreground when compared against current state-of-the-art techniques.

Low Rank and Sparse Decomposition of Ultrasound Color Flow Images for Suppressing Clutter in Real-Time.

In this work, a novel technique for real-time clutter rejection in ultrasound Color Flow Imaging (CFI) is proposed. Suppressing undesired clutter signal is important because clutter prohibits an unambiguous view of the vascular network. Although conventional eigen-based filters are potentially efficient in suppressing clutter signal, their performance is highly dependent on proper selection of a clutter to blood boundary which is done manually. Herein, we resolve this limitation by formulating the clutter suppression problem as a foreground-background separation problem to extract the moving blood component. To that end, we adapt the fast Robust Matrix Completion (fRMC) algorithm, and utilize the in-face extended Frank-Wolfe method to minimize the rank of the matrix of ultrasound frames. Our method automates the clutter suppression process, which is critical for clinical use. We name the method RAPID (Robust mAtrix decomPosition for suppressIng clutter in ultrasounD) since the automation step can substantially streamline clutter suppression. The technique is validated with simulation, flow phantom and two sets of in-vivo data. RAPID code as well as most of the data in this paper can be downloaded from RAPID.sonography.ai.