Foreground Object Segmentation Research Articles

Tomato pose estimation using the association of tomato body and sepal

In facility horticulture smart farms, harvesting robotic systems to automate harvesting tasks are challenging due to the complex environment, irregular growth and fruits pose. To harvest a fruits, the vision information required by a harvesting robot is accurate position and pose. Especially, fruit pose information is essential for planning paths to avoid damaging stems, leaves, branches, and obstacles, preventing damage to the fruit and harvesting robots, and planning efficient harvest sequences. This paper presents a method that uses information from tomato sepals to estimate the orientation of tomatoes, which are a commonly grown crop in horticulture. First, we train a YOLOv8 model to detect and segment bodies and sepals of tomatoes. For robust training of the model, the training data is constructed using effective data augmentation, which synthesizes segmented foreground objects and inserting them into the background. Then, for finding association between the body and sepal of a tomato, we apply IoU score based matching and the Hungarian algorithm. Consequently, we obtain point clouds for both parts using RGB-D data. Finally, we compute the center point of each object by using spherical fitting and the statistics of the point cloud, respectively. Then, we estimate the pose of the tomato as a vector between two center points of the body and sepal. To accurately and practically evaluate the proposed method, we generated ground truth data using calibration patterns in tomato greenhouse. As the experimental results show, The segmentation results show that AP50,sepal is 94.7, AP50,tomato is 96.3, and mAP is 61.5. The result of the pose estimation for the pose validation dataset is a mean error angle of 6.79 ± 3.18 and angle errors of less than 10 degrees account for 87.2%. The total algorithm proposed in this paper requires about 0.038s for each tomato greenhouse image. As a result, our proposed pose estimation can be practically utilized in robot systems for tomato harvesting.

CGMGM: A Cross-Gaussian Mixture Generative Model for Few-Shot Semantic Segmentation

Few-shot semantic segmentation (FSS) aims to segment unseen objects in a query image using a few pixel-wise annotated support images, thus expanding the capabilities of semantic segmentation. The main challenge lies in extracting sufficient information from the limited support images to guide the segmentation process. Conventional methods typically address this problem by generating single or multiple prototypes from the support images and calculating their cosine similarity to the query image. However, these methods often fail to capture meaningful information for modeling the de facto joint distribution of pixel and category. Consequently, they result in incomplete segmentation of foreground objects and mis-segmentation of the complex background. To overcome this issue, we propose the Cross Gaussian Mixture Generative Model (CGMGM), a novel Gaussian Mixture Models~(GMMs)-based FSS method, which establishes the joint distribution of pixel and category in both the support and query images. Specifically, our method initially matches the feature representations of the query image with those of the support images to generate and refine an initial segmentation mask. It then employs GMMs to accurately model the joint distribution of foreground and background using the support masks and the initial segmentation mask. Subsequently, a parametric decoder utilizes the posterior probability of pixels in the query image, by applying the Bayesian theorem, to the joint distribution, to generate the final segmentation mask. Experimental results on PASCAL-5i and COCO-20i datasets demonstrate our CGMGM's effectiveness and superior performance compared to the state-of-the-art methods.

SS-CPGAN: Self-Supervised Cut-and-Pasting Generative Adversarial Network for Object Segmentation.

This paper proposes a novel self-supervised based Cut-and-Paste GAN to perform foreground object segmentation and generate realistic composite images without manual annotations. We accomplish this goal by a simple yet effective self-supervised approach coupled with the U-Net discriminator. The proposed method extends the ability of the standard discriminators to learn not only the global data representations via classification (real/fake) but also learn semantic and structural information through pseudo labels created using the self-supervised task. The proposed method empowers the generator to create meaningful masks by forcing it to learn informative per-pixel and global image feedback from the discriminator. Our experiments demonstrate that our proposed method significantly outperforms the state-of-the-art methods on the standard benchmark datasets.

Hierarchical Co-attention Propagation Network for Zero-Shot Video Object Segmentation.

Zero-shot video object segmentation (ZS-VOS) aims to segment foreground objects in a video sequence without prior knowledge of these objects. However, existing ZS-VOS methods often struggle to distinguish between foreground and background or to keep track of the foreground in complex scenarios. The common practice of introducing motion information, such as optical flow, can lead to overreliance on optical flow estimation. To address these challenges, we propose an encoder-decoder-based hierarchical co-attention propagation network (HCPN) capable of tracking and segmenting objects. Specifically, our model is built upon multiple collaborative evolutions of the parallel co-attention module (PCM) and the cross co-attention module (CCM). PCM captures common foreground regions among adjacent appearance and motion features, while CCM further exploits and fuses cross-modal motion features returned by PCM. Our method is progressively trained to achieve hierarchical spatio-temporal feature propagation across the entire video. Experimental results demonstrate that our HCPN outperforms all previous methods on public benchmarks, showcasing its effectiveness for ZS-VOS. Code and pre-trained model can be found at https://github.com/NUST-Machine-Intelligence-Laboratory/HCPN.

基于机器视觉的羊只目标提取算法设计与实现

In order to improve the quality of sheep foreground object segmentation, images are segmented using the watershed algorithm in combination with a growing region algorithm, and the pixel-by-pixel comparison of segmentation is optimized to reduce the processing time. Compared with other algorithms, the optimized watershed algorithm can achieve more complete target extraction, and its processing time is improved by over 50% compared with the other six algorithms. Moreover, the optimized watershed algorithm has the optimal overall image quality indicators. This algorithm can provide a reference for the real-time extraction of the activity state of sheep.

Reviewing ViBe, a Popular Background Subtraction Algorithm for Real-Time Applications

Background subtraction or foreground segmentation is commonly the first step in video processing pipelines, where foreground objects and moving entities are detected for further analysis. Background subtraction is still an open problem in computer vision due to the wide range of possible scenarios and challenging cases, such as weather conditions, illumination variation, dynamic backgrounds or low frame-rates. In this article, we review a well-known general algorithm for background subtraction named visual background extractor or ViBe. More than 10 years after its publication, ViBe is still widely used due to its simplicity and low computational load, as its authors claim it can process up to 200 fps. We analyze the algorithm's mechanism to segment foreground objects, comment on its strengths and drawbacks, and describe an effective variant of the method to quickly dissipate ghost effects using 3-frame difference. Furthermore, we provide an easy-to-use demo that allows to quickly test the performance of ViBe on custom data, modify its parameters and download the results. **This is an MLBriefs article, the source code has not been reviewed!**<br> The ViBe algorithm is covered by several patents and its code is not provided with this publication. <a href="https://orbi.uliege.be/handle/2268/145853">Its code is only shared for research purposes.</a><br>

Foreground Detection Based on Superpixel and Semantic Segmentation.

Foreground detection is an essential step in computer vision and video processing. Accurate foreground object extraction is crucial for subsequent high-level tasks such as target recognition and tracking. Although many foreground detection algorithms have been proposed, foreground detection in complex scenes is still a challenging problem. This paper presents a foreground detection algorithm based on superpixel and semantic segmentation. It first uses multiscale superpixel segmentation to obtain the initial foreground mask. At the same time, a semantic segmentation network is applied to separate potential foreground objects, and then use the defined rules to combine the results of superpixel and semantic segmentation to get the final foreground object. Finally, the background model is updated with the refined foreground result. Experiments on the CDNet2014 dataset demonstrate the effectiveness of the proposed algorithm, which can accurately segment foreground objects in complex scenes.

Context-Unsupervised Adversarial Network for Video Sensors.

Foreground object segmentation is a crucial first step for surveillance systems based on networks of video sensors. This problem in the context of dynamic scenes has been widely explored in the last two decades, but it still has open research questions due to challenges such as strong shadows, background clutter and illumination changes. After years of solid work based on statistical background pixel modeling, most current proposals use convolutional neural networks (CNNs) either to model the background or to make the foreground/background decision. Although these new techniques achieve outstanding results, they usually require specific training for each scene, which is unfeasible if we aim at designing software for embedded video systems and smart cameras. Our approach to the problem does not require specific context or scene training, and thus no manual labeling. We propose a network for a refinement step on top of conventional state-of-the-art background subtraction systems. By using a statistical technique to produce a rough mask, we do not need to train the network for each scene. The proposed method can take advantage of the specificity of the classic techniques, while obtaining the highly accurate segmentation that a deep learning system provides. We also show the advantage of using an adversarial network to improve the generalization ability of the network and produce more consistent results than an equivalent non-adversarial network. The results provided were obtained by training the network on a common database, without fine-tuning for specific scenes. Experiments on the unseen part of the CDNet database provided 0.82 a F-score, and 0.87 was achieved for LASIESTA databases, which is a database unrelated to the training one. On this last database, the results outperformed by 8.75% those available in the official table. The results achieved for CDNet are well above those of the methods not based on CNNs, and according to the literature, among the best for the context-unsupervised CNNs systems.

Real-time foreground object segmentation networks using long and short skip connections

Foreground object segmentation is an important task with various applications in outdoor surveillance and navigation. Most existing methods focus on accuracy and therefore, are computationally expensive and low in speed, making them difficult to use in actual applications. In this study, we aim to address the issue of accuracy and efficiency trade-off. In particular, in contrast with existing methods that use fine-tuning routine on heavyweight pretrained models and/or optimization techniques to enhance results, we propose a lightweight end-to-end network that can be trained from scratch effectively and efficiently. First, long and short skip connections are used among convolutional blocks and within the bottleneck block. By doing so, information flow within the networks is enhanced during the training stage, and thus, the use rate of parameters in the model is increased, allowing a more compact and efficient network design. Second, we use feature fusions based on element-wise summing before each up-sampling layer to reduce the size of the decoder, accelerate the up-sampling process, and stabilize training convergence. Our proposed method is tested rigorously. In particular, we achieved 1000 times higher speed compared with state-of-the-art methods on CD2014 and SBI2015 datasets with comparable accuracy.

Efficient foreground object segmentation from video by Probability Weighted Moments

Segmentation of foreground objects is an important issue in computer vision. Since, there exist no predefined classes for unsupervised learning, which makes it computationally expensive. Also, the contrast of foreground to background needs to be tackled effectively. To overcome the aforementioned problems, in this paper an efficient approach named Fused Probability weighted moments and Principal component analysis Video Object Segmentation (FPPVOS), is proposed by the fusion of Probability Weighted Moments (PWM) with Principal Component Analysis (PCA). FPPVOS significantly reduces the computational cost by taking into account the intra-class variability due to its less sensitivity towards the outliers and low sample variability. Theoretically, this fusion ensures the efficiency of the unsupervised learning where classification is assured, even though if no grouping or labels of the objects are available. The results are evaluated on the standard YouTube-Objects dataset. We achieved the preeminent performance by using FPPVOS while solving the problems of object separation and segmentation from its background in a reasonable time interval comparative to the state-of-the-art methods.

End-to-End Recurrent Generative Adversarial Network for Traffic and Surveillance Applications

In video frame segmentation, many existing deep networks and contemporary approaches give a remarkable performance with the assumption that the only foreground is moving, and the background is stationary. However, in the presence of infrequent motion of foreground objects, sudden illumination changes in the background, bad weather, and dynamic background, the accurate foreground object(s) segmentation is a challenging task. Generative adversarial networks (GAN) based training shows fruitful results in various fields like image-to-image style transfer, image enhancement, semantic segmentation, image super-resolution, etc. The limited results of hand-crafted approaches for moving object segmentation (MOS) and the robustness of adversarial training for a given task inspired us to propose a novel approach for moving object segmentation (MOS). In this context, an end-to-end generative adversarial network (two generators) with recurrent technique is proposed for MOS and is named as RMS-GAN. The proposed RMS-GAN is able to incorporate foreground probability knowledge with residual and weight sharing based recurrent technique for accurate segmentation. The recurrent technique helps us to exhibit the temporal behavior between successive video frames, which is more prominent for any video processing applications. Also, to enhance the spatial coherence of the obtained foreground probability map using the generator-1 network, the cascaded architecture of two generators is proposed. The effectiveness of the proposed approach is evaluated both qualitatively and quantitatively on three benchmark video datasets for MOS. Experimental result analysis shows that the proposed network outperforms the existing state-of-the-art methods on three benchmark datasets for MOS.

Retrieving point cloud models of target objects in a scene from photographed images

Image-based reconstruction is devoted to recovering the 3D point cloud models of target objects from scene images photographed at different viewpoints, and the existing methods often produce a large number of redundant background points, which causes inconvenience to 3D modeling or other related applications. To solve this issue, this work proposes an improved framework that combines image segmentation in the point cloud retrieving procedure, so as it only reconstructs the objects of interest in a scene. This framework provides two options for foreground object segmentation, and users can determine the appropriate method to obtain accurate segmentation for different scenes. Then, the feature matches are extracted from the segmented images, and the point cloud model is recovered via two phases of dense diffusion, feature diffusion and patch diffusion. In the diffusion stage, we introduce a new normalized metric that deals with both the illumination change and low texture case to enhance the robustness of the reconstruction. The experimental results show that proposed framework can effectively avoid reconstructing the irrelevant background data while outputting more even and detailed point cloud models.

Semantic Image Segmentation with Deep Convolutional Neural Networks and Quick Shift

Semantic image segmentation, as one of the most popular tasks in computer vision, has been widely used in autonomous driving, robotics and other fields. Currently, deep convolutional neural networks (DCNNs) are driving major advances in semantic segmentation due to their powerful feature representation. However, DCNNs extract high-level feature representations by strided convolution, which makes it impossible to segment foreground objects precisely, especially when locating object boundaries. This paper presents a novel semantic segmentation algorithm with DeepLab v3+ and super-pixel segmentation algorithm-quick shift. DeepLab v3+ is employed to generate a class-indexed score map for the input image. Quick shift is applied to segment the input image into superpixels. Outputs of them are then fed into a class voting module to refine the semantic segmentation results. Extensive experiments on proposed semantic image segmentation are performed over PASCAL VOC 2012 dataset, and results that the proposed method can provide a more efficient solution.

Text non-text classification based on area occupancy of equidistant pixels

Text detection and localization from text embedded natural images is still considered as a challenging problem in complex imagery environment. Foreground object segmentation followed by classification is a popular approach for this task. Component level object classification in clutter environment is therefore an important sub-problem. Appropriate extraction of foreground objects leads to effective classification that may certainly increase the performance of text detection. In this paper, a novel feature vector is developed based on area occupancy profile of equidistant pixels is reported for text/non-text classification. The generated feature descriptors are script invariant and much effective in practical scenario. This proposed feature set is evaluated on our dataset using five different pattern classifiers and experimental result shows that the said feature set yields more than 86% classification accuracy irrespective of scripts.

Optimisation of linear dependence energy for object co‐segmentation in a set of images with heterogeneous contents

This work proposes a framework for simultaneously segmenting foreground objects in a collection of images having heterogeneous contents. Rather than resorting to image co-segmentation to segment similar objects in multiple images, which requires the use of categorised images, the authors’ idea disseminates segmentation information within images. In this way, it becomes easier to detect foreground objects in all of them simultaneously, mainly under the hypothesis of using similar or different images. General information is aggregated, on foregrounds as well as on backgrounds, from a set of images for joint segmentation of category-independent objects. The key idea is to estimate the linear dependence of the foreground histograms of the input images to optimise a Markov random field-based energy function. Iterative optimisation of each image permits after that the enhancement of the final segmentation results. Extensive experiments demonstrate that the proposed method (PM) enables full-object segmentation of foreground objects within a collection of images composed of different classes. Indeed, the validation of the accuracy on five challenging datasets (iCoseg, Oxford Flowers, MicroSoft Research Cambridge (MSRC), Caltech101 and Berkeley) shows that the PM achieves satisfactory results as compared with state-of-the-art methods. Besides, it has the challenging ability to efficiently deal with uncategorised objects.

Genetic programming based feature construction methods for foreground object segmentation

Foreground object segmentation is a crucial preprocessing step for many high-level computer vision tasks, e.g. object recognition. It is still challenging to achieve accurate segmentation, especially for complex images (e.g. with high variations). Feature construction can help to improve the segmentation performance by extracting more distinctive features for foreground/background regions from the original features. However, commonly-used feature construction methods (e.g. principle component analysis) often involve certain assumptions/constraints, and the constructed features cannot be interpreted. To address these problems, genetic programming (GP) is employed in this paper, which is a well-suited feature construction technique.The aim of this work is to design new feature construction methods using GP, and analyse/compare popular GP-based feature construction methods for foreground object segmentation, especially on complex image datasets with high variations. Specifically, one new feature construction method that incorporates the subtree technique in GP is designed, which can construct multiple features simultaneously (called SubtMFC, Subtree Multiple Feature Construction). Moreover, a parsimony pressure technique is introduced to improve SubtMFC for bloat control (a common issue for GP-based methods), which forms the method, PSubtMFC (Parsimony SubtMFC). In addition, comparison of popular GP-based feature construction methods for foreground object segmentation is conducted for the first time. Results show that SubtMFC achieves better or similar performance compared with three reference methods. In addition, compared with SubtMFC that does not control bloat, PSubtMFC can significantly reduce the solution size while maintain similar performance in the segmentation accuracy. The GP-based feature construction framework is further extended for feature representation based knowledge transfer, which can handle the problem of the scare labelled training data. Moreover, after GP is thoroughly investigated on benchmark datasets with one type of foreground objects (i.e. the Weizmann horse dataset and Pascal aeroplane dataset), it is considered whether the GP methods can perform well on datasets containing multiple types of foreground objects. Compared with three other well-performing GP-based feature construction methods, the proposed method achieves better or comparable results for the given segmentation tasks. In addition, this paper thoroughly compares/analyses popular GP-based feature construction methods for complex figure-ground segmentation for the first time. Moreover, further analyses on the input features frequently used by the GP-evolved feature construction functions reflect the effectiveness of the extracted high-level features.

PerSeg : segmenting salient objects from bag of single image perturbations

Salient object segmentation is an important computer vision problem having applications in numerous areas such as video surveillance, scene parsing, autonomous navigation etc. For images, this task is quite challenging due to clutter/texture present in the background, low resolution and/or low contrast of the object(s) of interest etc. In case of videos, additional issues such as object deformation, camera motion and presence of multiple moving objects make the foreground object segmentation a significantly difficult and open problem. However, motion pattern can also act as an important cue to identify the foreground objects against the background. This is exploited by the recent approaches via aggregation of temporally perturbed information from a series of consecutive frames. Unfortunately for images, this additional cue is not available. In this paper, we propose to emulate the effect of such perturbations by constructing a bag of multiple augmentations applied on a single input image. Saliency features are estimated independently from each perturbed image in this bag, which are further combined using a novel aggregation strategy based on a convolutional gated recurrent encoder-decoder unit. Through extensive experiments on the benchmark datasets, we show better or very competitive performance when compared with the state-of-the-art methods. We further observe that even with a bag constructed using simple affine transformations, we achieve impressive performances, proving the robustness of the proposed framework.

Coarse-to-Fine Semantic Segmentation From Image-Level Labels.

Deep neural network-based semantic segmentation generally requires large-scale cost extensive annotations for training to obtain better performance. To avoid pixel-wise segmentation annotations that are needed for most methods, recently some researchers attempted to use object-level labels (e.g., bounding boxes) or image-level labels (e.g., image categories). In this paper, we propose a novel recursive coarse-to-fine semantic segmentation framework based on only image-level category labels. For each image, an initial coarse mask is first generated by a convolutional neural network-based unsupervised foreground segmentation model and then is enhanced by a graph model. The enhanced coarse mask is fed to a fully convolutional neural network to be recursively refined. Unlike the existing image-level label-based semantic segmentation methods, which require labeling of all categories for images that contain multiple types of objects, our framework only needs one label for each image and can handle images that contain multi-category objects. Only trained on ImageNet, our framework achieves comparable performance on the PASCAL VOC dataset with other image-level label-based state-of-the-art methods of semantic segmentation. Furthermore, our framework can be easily extended to foreground object segmentation task and achieves comparable performance with the state-of-the-art supervised methods on the Internet object dataset.

Unsupervised Learning of Foreground Object Segmentation

Unsupervised learning represents one of the most interesting challenges in computer vision today. The task has an immense practical value with many applications in artificial intelligence and emerging technologies, as large quantities of unlabeled images and videos can be collected at low cost. In this paper, we address the unsupervised learning problem in the context of segmenting the main foreground objects in single images. We propose an unsupervised learning system, which has two pathways, the teacher and the student, respectively. The system is designed to learn over several generations of teachers and students. At every generation the teacher performs unsupervised object discovery in videos or collections of images and an automatic selection module picks up good frame segmentations and passes them to the student pathway for training. At every generation multiple students are trained, with different deep network architectures to ensure a better diversity. The students at one iteration help in training a better selection module, forming together a more powerful teacher pathway at the next iteration. In experiments, we show that the improvement in the selection power, the training of multiple students and the increase in unlabeled data significantly improve segmentation accuracy from one generation to the next. Our method achieves top results on three current datasets for object discovery in video, unsupervised image segmentation and saliency detection. At test time, the proposed system is fast, being one to two orders of magnitude faster than published unsupervised methods. We also test the strength of our unsupervised features within a well known transfer learning setup and achieve competitive performance, proving that our unsupervised approach can be reliably used in a variety of computer vision tasks.

Predicting How to Distribute Work Between Algorithms and Humans to Segment an Image Batch

Foreground object segmentation is a critical step for many image analysis tasks. While automated methods can produce high-quality results, their failures disappoint users in need of practical solutions. We propose a resource allocation framework for predicting how best to allocate a fixed budget of human annotation effort in order to collect higher quality segmentations for a given batch of images and automated methods. The framework is based on a prediction module that estimates the quality of given algorithm-drawn segmentations. We demonstrate the value of the framework for two novel tasks related to predicting how to distribute annotation efforts between algorithms and humans. Specifically, we develop two systems that automatically decide, for a batch of images, when to recruit humans versus computers to create (1) coarse segmentations required to initialize segmentation tools and (2) final, fine-grained segmentations. Experiments demonstrate the advantage of relying on a mix of human and computer efforts over relying on either resource alone for segmenting objects in images coming from three diverse modalities (visible, phase contrast microscopy, and fluorescence microscopy).