Object Segmentation Approach Research Articles

Regularized Loss With Hyperparameter Estimation for Weakly Supervised Single Class Segmentation.

We propose a new image level weakly supervised segmentation approach for datasets with a single object class of interest. Our approach is based on a regularized loss function inspired by the classical Conditional Random Field (CRF) modeling. Our loss models properties of generic objects, and we use it to guide CNN towards segments that are more likely to correspond to the object, thus avoiding the need for pixel precise annotations. Training CNN with regularized loss is a difficult task for gradient descent. We develop an annealing algorithm which is crucial for a successful training. Furthermore, we develop an approach for hyperparameter setting for the most important components of our regularized loss. This is far from trivial, since there is no pixel precise ground truth for guidance. The advantage of our method is that we use a standard CNN architecture and an easy to interpret loss function, derived from classical CRF models. Furthermore, we apply the same loss function for any task/dataset. We first evaluate our approach for salient object segmentation and co-segmentation. These tasks naturally involve one object class of interest. Then we adapt our approach to image level weakly supervised multi-class semantic segmentation. We obtain state-of-the-art results.

Region Aware Video Object Segmentation With Deep Motion Modeling.

Current semi-supervised video object segmentation (VOS) methods often employ the entire features of one frame to predict object masks and update memory. This introduces significant redundant computations. To reduce redundancy, we introduce a Region Aware Video Object Segmentation (RAVOS) approach, which predicts regions of interest (ROIs) for efficient object segmentation and memory storage. RAVOS includes a fast object motion tracker to predict object ROIs in the next frame. For efficient segmentation, object features are extracted based on the ROIs, and an object decoder is designed for object-level segmentation. For efficient memory storage, we propose motion path memory to filter out redundant context by memorizing the features within the motion path of objects. In addition to RAVOS, we also propose a large-scale occluded VOS dataset, dubbed OVOS, to benchmark the performance of VOS models under occlusions. Evaluation on DAVIS and YouTube-VOS benchmarks and our new OVOS dataset show that our method achieves state-of-the-art performance with significantly faster inference time, e.g., 86.1 J & F at 42 FPS on DAVIS and 84.4 J & F at 23 FPS on YouTube-VOS. Project page: ravos.netlify.app.

Beyond Appearance: Multi-Frame Spatio-Temporal Context Memory Networks for Efficient and Robust Video Object Segmentation.

Current video object segmentation approaches primarily rely on frame-wise appearance information to perform matching. Despite significant progress, reliable matching becomes challenging due to rapid changes of the object's appearance over time. Moreover, previous matching mechanisms suffer from redundant computation and noise interference as the number of accumulated frames increases. In this paper, we introduce a multi-frame spatio-temporal context memory (STCM) network to exploit discriminative spatio-temporal cues in multiple adjacent frames by utilizing a multi-frame context interaction module (MCI) for memory construction. Based on the proposed MCI module, a sparse group memory reader is developed to enable efficient sparse matching during memory reading. Our proposed method is generic and achieves state-of-the-art performance with real-time speed on benchmark datasets such as DAVIS and YouTube-VOS. In addition, our model exhibits robustness to sparse videos with low frame rates.

CT-VOS: Cutout prediction and tagging for self-supervised video object segmentation

We propose a novel self-supervised Video Object Segmentation (VOS) approach that strives to achieve better visual correspondence across frames and object-background discriminability for accurate object segmentation. Distinct from previous self-supervised VOS methods, our approach is based on a learning loss formulation that takes into account both object and background information to ensure object-background discriminability, rather than using only object appearance. The objective function comprises cutout-based reconstruction (cutout region represents part of a frame, whose pixels are replaced with some constant values) and tag prediction loss terms. The cutout-based reconstruction term utilizes a simple cutout scheme to learn the pixel-wise correspondence between the current and previous frames in order to reconstruct the original current frame with added cutout region in it. The introduced cutout patch guides the model to focus on the reappearance of scene parts , thereby implicitly equipping the model to address occlusion-based scenarios Next, the tag prediction term encourages object-background separability by grouping tags of all pixels in the cutout region that are similar, while separating them from the tags of the rest of the reconstructed pixels. Additionally, we introduce a zoom-in scheme that addresses the problem of small object segmentation by capturing fine structural information at multiple scales. Our proposed approach, termed CT-VOS, achieves state-of-the-art results on two challenging benchmarks: DAVIS-2017 and Youtube-VOS. A detailed ablation showcases the importance of the proposed loss formulation to effectively establish correspondences, object-background discriminability, and the impact of our zoom-in scheme to accurately segment small-sized objects.

A weak-labelling and deep learning approach for in-focus object segmentation in 3D widefield microscopy

Three-dimensional information is crucial to our understanding of biological phenomena. The vast majority of biological microscopy specimens are inherently three-dimensional. However, conventional light microscopy is largely geared towards 2D images, while 3D microscopy and image reconstruction remain feasible only with specialised equipment and techniques. Inspired by the working principles of one such technique—confocal microscopy, we propose a novel approach to 3D widefield microscopy reconstruction through semantic segmentation of in-focus and out-of-focus pixels. For this, we explore a number of rule-based algorithms commonly used for software-based autofocusing and apply them to a dataset of widefield focal stacks. We propose a computation scheme allowing the calculation of lateral focus score maps of the slices of each stack using these algorithms. Furthermore, we identify algorithms preferable for obtaining such maps. Finally, to ensure the practicality of our approach, we propose a surrogate model based on a deep neural network, capable of segmenting in-focus pixels from the out-of-focus background in a fast and reliable fashion. The deep-neural-network-based approach allows a major speedup for data processing making it usable for online data processing.

The fast image segmentation algorithms using dynamic programming for modals of image histograms

A machine learning method is now considered capable of accurately segmenting images. However, one significant disadvantage of this strategy is that it requires a lengthy training phase and an extensive training dataset. This article uses an image segmentation by histogram thresholding approach that does not require training to overcome this difficulty. This article proposes straightforward and time-optimal algorithms, which are guaranteed by mathematical proofs. Furthermore, we experiment with the proposed algorithms using 100 images from a standard database. The results show that, while their performances are not significantly different, the two proposed methods are roughly 10 and 20 times faster than the most simple and optimal method, Brute Force. They also show that the proposed algorithms can deal with bimodal images and images with various shapes of the image histogram. Because our proposed algorithms are the most efficient and effective. As a result, they can be used for real-time segmentations and as a pre-processing approach for multiple object segmentation.

Bin Picking for Ship-Building Logistics Using Perception and Grasping Systems

Bin picking is a challenging task involving many research domains within the perception and grasping fields, for which there are no perfect and reliable solutions available that are applicable to a wide range of unstructured and cluttered environments present in industrial factories and logistics centers. This paper contributes with research on the topic of object segmentation in cluttered scenarios, independent of previous object shape knowledge, for textured and textureless objects. In addition, it addresses the demand for extended datasets in deep learning tasks with realistic data. We propose a solution using a Mask R-CNN for 2D object segmentation, trained with real data acquired from a RGB-D sensor and synthetic data generated in Blender, combined with 3D point-cloud segmentation to extract a segmented point cloud belonging to a single object from the bin. Next, it is employed a re-configurable pipeline for 6-DoF object pose estimation, followed by a grasp planner to select a feasible grasp pose. The experimental results show that the object segmentation approach is efficient and accurate in cluttered scenarios with several occlusions. The neural network model was trained with both real and simulated data, enhancing the success rate from the previous classical segmentation, displaying an overall grasping success rate of 87.5%.

A Morphological Post-Processing Approach for Overlapped Segmentation of Bacterial Cell Images

Scanning electron microscopy (SEM) techniques have been extensively performed to image and study bacterial cells with high-resolution images. Bacterial image segmentation in SEM images is an essential task to distinguish an object of interest and its specific region. These segmentation results can then be used to retrieve quantitative measures (e.g., cell length, area, cell density) for the accurate decision-making process of obtaining cellular objects. However, the complexity of the bacterial segmentation task is a barrier, as the intensity and texture of foreground and background are similar, and also, most clustered bacterial cells in images are partially overlapping with each other. The traditional approaches for identifying cell regions in microscopy images are labor intensive and heavily dependent on the professional knowledge of researchers. To mitigate the aforementioned challenges, in this study, we tested a U-Net-based semantic segmentation architecture followed by a post-processing step of morphological over-segmentation resolution to achieve accurate cell segmentation of SEM-acquired images of bacterial cells grown in a rotary culture system. The approach showed an 89.52% Dice similarity score on bacterial cell segmentation with lower segmentation error rates, validated over several cell overlapping object segmentation approaches with significant performance improvement.

Full-duplex strategy for video object segmentation

Previous video object segmentation approaches mainly focus on simplex solutions linking appearance and motion, limiting effective feature collaboration between these two cues. In this work, we study a novel and efficient full-duplex strategy network (FSNet) to address this issue, by considering a better mutual restraint scheme linking motion and appearance allowing exploitation of cross-modal features from the fusion and decoding stage. Specifically, we introduce a relational cross-attention module (RCAM) to achieve bidirectional message propagation across embedding sub-spaces. To improve the model’s robustness and update inconsistent features from the spatiotemporal embeddings, we adopt a bidirectional purification module after the RCAM. Extensive experiments on five popular benchmarks show that our FSNet is robust to various challenging scenarios (e.g., motion blur and occlusion), and compares well to leading methods both for video object segmentation and video salient object detection. The project is publicly available at https://github.com/GewelsJI/FSNet.

FASTDLO: Fast Deformable Linear Objects Instance Segmentation

In this paper, an approach for fast and accurate segmentation of Deformable Linear Objects (DLOs) named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FASTDLO</i> is presented. A deep convolutional neural network is employed for background segmentation, generating a binary mask that isolates DLOs in the image. Thereafter, the obtained mask is processed with a skeletonization algorithm and the intersections between different DLOs are solved with a similarity-based network. Apart from the usual pixel-wise color-mapped image, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FASTDLO</i> also describes each DLO instance with a sequence of 2D coordinates, enabling the possibility of modeling the DLO instances with splines curves, for example. Synthetically generated data are exploited for the training of the data-driven methods, avoiding expensive collection and annotations of real data. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FASTDLO</i> is experimentally compared against both a DLO-specific approach and general-purpose deep learning instance segmentation models, achieving better overall performances and a processing rate higher than 20 FPS.

RoiSeg: An Effective Moving Object Segmentation Approach Based on Region-of-Interest with Unsupervised Learning

Traditional video object segmentation often has low detection speed and inaccurate results due to the jitter caused by the pan-and-tilt or hand-held devices. Deep neural network (DNN) has been widely adopted to address these problems; however, it relies on a large number of annotated data and high-performance computing units. Therefore, DNN is not suitable for some special scenarios (e.g., no prior knowledge or powerful computing ability). In this paper, we propose RoiSeg, an effective moving object segmentation approach based on Region-of-Interest (ROI), which utilizes unsupervised learning method to achieve automatic segmentation of moving objects. Specifically, we first hypothesize that the central n × n pixels of images act as the ROI to represent the features of the segmented moving object. Second, we pool the ROI to a central point of the foreground to simplify the segmentation problem into a classification problem based on ROI. Third but not the least, we implement a trajectory-based classifier and an online updating mechanism to address the classification problem and the compensation of class imbalance, respectively. We conduct extensive experiments to evaluate the performance of RoiSeg and the experimental results demonstrate that RoiSeg is more accurate and faster compared with other segmentation algorithms. Moreover, RoiSeg not only effectively handles ambient lighting changes, fog, salt and pepper noise, but also has a good ability to deal with camera jitter and windy scenes.

Learning disentangled representation for self-supervised video object segmentation

This paper proposes a novel self-supervised method for one-shot video object segmentation where the object annotations are only provided in the first frame. The current self-supervised video object segmentation approaches are implemented by modeling the pairwise correspondence between the target and reference frames. The pairwise correspondence only maintains spatio-temporal consistency. However, the VOS tasks not only require a spatio-temporal relationship between the two frames but also require the salient object information for each frame. In order to achieve this goal, we propose a disentangled representation strategy to disentangle the temporal correspondence into the pairwise term and unary term. The pairwise and unary terms capture inter-frame spatio-temporal and intra-frame salient object information, respectively. To demonstrate the importance of the disentangled representation, we apply the proposed approach to DAVIS-2017 and YouTube-VOS datasets. Experimental results confirm the effectiveness of the proposed solution.

Separable Structure Modeling for Semi-Supervised Video Object Segmentation

In this paper, we propose a separable structure modeling approach for semi-supervised video object segmentation. Unlike most existing methods which preclude the semantically structural information of target objects, our method not only captures pixel-level similarity relationships between the reference and target frames but also reveals the separable structure of the specified objects in target frames. Specifically, we first compute a pixel-wise similarity matrix by using representations of reference and target pixels and then select top rank reference pixels for target pixel classification. According to the prior knowledge from these top-rank reference pixels, we further appoint the representative target pixels for object structure modeling. Particularly, in the structure modeling branch, we extract the shared and individual features that can well represent the whole object and its components, respectively. Moreover, the proposed method is a fast algorithm without online fine-tuning and any post-processing. We conduct extensive experiments and ablation studies on the DAVIS-16, DAVIS-17, and YouTube-VOS datasets, and experimental results on three widely-used datasets demonstrate that our method achieves a superior performance, compared with state-of-the-art semi-supervised video object segmentation approaches in terms of speed and accuracy.

Automated Identification of Skull Fractures With Deep Learning: A Comparison Between Object Detection and Segmentation Approach.

Objective: Skull fractures caused by head trauma can lead to life-threatening complications. Hence, timely and accurate identification of fractures is of great importance. Therefore, this study aims to develop a deep learning system for automated identification of skull fractures from cranial computed tomography (CT) scans.Method: This study retrospectively analyzed CT scans of 4,782 patients (median age, 54 years; 2,583 males, 2,199 females; development set: n = 4,168, test set: n = 614) diagnosed with skull fractures between September 2016 and September 2020. Additional data of 7,856 healthy people were included in the analysis to reduce the probability of false detection. Skull fractures in all the scans were manually labeled by seven experienced neurologists. Two deep learning approaches were developed and tested for the identification of skull fractures. In the first approach, the fracture identification task was treated as an object detected problem, and a YOLOv3 network was trained to identify all the instances of skull fracture. In the second approach, the task was treated as a segmentation problem and a modified attention U-net was trained to segment all the voxels representing skull fracture. The developed models were tested using an external test set of 235 patients (93 with, and 142 without skull fracture).Results: On the test set, the YOLOv3 achieved average fracture detection sensitivity and specificity of 80.64, and 85.92%, respectively. On the same dataset, the modified attention U-Net achieved a fracture detection sensitivity and specificity of 82.80, and 88.73%, respectively.Conclusion: Deep learning methods can identify skull fractures with good sensitivity. The segmentation approach to fracture identification may achieve better results.

Unsupervised video object segmentation with distractor-aware online adaptation

Unsupervised video object segmentation is a crucial application in video analysis when there is no prior information about the objects. It becomes tremendously challenging when multiple objects occur and interact in a video clip. In this paper, a novel unsupervised video object segmentation approach via distractor-aware online adaptation (DOA) is proposed. DOA models spatiotemporal consistency in video sequences by capturing background dependencies from adjacent frames. Instance proposals are generated by the instance segmentation network for each frame and they are grouped by motion information as positives or hard negatives. To adopt high-quality hard negatives, the block matching algorithm is then applied to preceding frames to track the associated hard negatives. General negatives are also introduced when there are no hard negatives in the sequence. The experimental results demonstrate these two kinds of negatives are complementary. Finally, we conduct DOA using positive, negative, and hard negative masks to update the foreground and background segmentation. The proposed approach achieves state-of-the-art results on two benchmark datasets, the DAVIS 2016 and the Freiburg-Berkeley motion segmentation (FBMS)-59.

One Shot Segmentation: Unifying Rigid Detection and Non-Rigid Segmentation Using Elastic Regularization.

This paper proposes a novel approach for the non-rigid segmentation of deformable objects in image sequences, which is based on one-shot segmentation that unifies rigid detection and non-rigid segmentation using elastic regularization. The domain of application is the segmentation of a visual object that temporally undergoes a rigid transformation (e.g., affine transformation) and a non-rigid transformation (i.e., contour deformation). The majority of segmentation approaches to solve this problem are generally based on two steps that run in sequence: a rigid detection, followed by a non-rigid segmentation. In this paper, we propose a new approach, where both the rigid and non-rigid segmentation are performed in a single shot using a sparse low-dimensional manifold that represents the visual object deformations. Given the multi-modality of these deformations, the manifold partitions the training data into several patches, where each patch provides a segmentation proposal during the inference process. These multiple segmentation proposals are merged using the classification results produced by deep belief networks (DBN) that compute the confidence on each segmentation proposal. Thus, an ensemble of DBN classifiers is used for estimating the final segmentation. Compared to current methods proposed in the field, our proposed approach is advantageous in four aspects: (i) it is a unified framework to produce rigid and non-rigid segmentations; (ii) it uses an ensemble classification process, which can help the segmentation robustness; (iii) it provides a significant reduction in terms of the number of dimensions of the rigid and non-rigid segmentations search spaces, compared to current approaches that divide these two problems; and (iv) this lower dimensionality of the search space can also reduce the need for large annotated training sets to be used for estimating the DBN models. Experiments on the problem of left ventricle endocardial segmentation from ultrasound images, and lip segmentation from frontal facial images using the extended Cohn-Kanade (CK+) database, demonstrate the potential of the methodology through qualitative and quantitative evaluations, and the ability to reduce the search and training complexities without a significant impact on the segmentation accuracy.

Depth-aware salient object segmentation

Depth-aware salient object segmentation

Anisotropic gradient-based filtering for object segmentation in medical images

ABSTRACT A four-step approach for image filtering and object segmentation is explored. The key steps are i) a low-pass digital differentiator is used to compute the image gradient (vector) field; ii) the regularised anisotropic diffusion method is used to filter this vector field; iii) the modified image is reconstructed from the filtered gradient field as a least-squares best fit; iv) object segmentation is performed on the reconstructed image. The advantages of this approach is the easier identification of noise in the gradient field, and consequently the image filtering can be more effective. The least-squares fit allows for non-local adjustment to the image to improve overall smoothness while enhancing object contrast. Importantly one can also ensure that relevant feature boundary locations are preserved while appearing smoother due to the filtering. The proposed filtering methodology is compared to image filtering applied directly to the image intensity. A set of challenging medical images of mammography exams and confocal microscopy experiments are used as numerical tests.

Unsupervised Video Object Segmentation Based on Mixture Models and Saliency Detection

In this paper, we propose an unsupervised video object segmentation approach which is mainly based on a saliency detection method and the Gaussian mixture model with Markov random field. In our approach, the saliency detection method is developed as a preprocessing technique to calculate the probability of each pixel as the target object. In contrast to traditional saliency detection methods which are normally difficult to obtain the object’s precise boundary and are therefore hard to segment consistent objects, the developed saliency detection method can calculate the saliency of each frame in the video sequence and extract the position and region of the target object with more accurate object boundary. The refined extracted object region is then taken as the prior information and incorporated into the Gaussian mixture model with Markov random field to obtain the precise pixel-wise segmentation result of each frame. The effectiveness of the proposed unsupervised video object segmentation approach is validated through experimental results using both the SegTrack and the SegTrack v2 data sets.

Sparse Multi-Bending Snakes.

Active contour models are one of the most emblematic algorithms of computer vision. Their strong theoretical foundations and high user interoperability turned them into a reference approach for object segmentation and tracking tasks. A high number of modifications have already been proposed in order to overcome the known problems of traditional snakes, such as initialization dependence and poor convergence to concavities. In this paper, we address the scenario where the user wants to segment an object that has multiple dynamic regions but some of them do not correspond to the true object boundary. We propose a novel parametric active contour model, the Sparse Multi-Bending snake, which is capable of dividing the contour into a set of contiguous regions with different bending properties. We derive a new energy function that induces such behavior and presents a group optimization strategy that can be used to find the optimal bending resistance parameter for each point of the contour. We show the flexibility of our model in a set of synthetic images. In addition, we consider two real applications, lung segmentation in Computerized Tomography data and hand segmentation in depth images. We show how the proposed method is able to improve the segmentations obtained in both applications, when compared with other active contour models.