Object Segmentation Framework Research Articles

Learning spatiotemporal relationships with a unified framework for video object segmentation

Learning spatiotemporal relationships with a unified framework for video object segmentation

Implicit Sparse Shape Representation: A Unified Framework for Object Segmentation and Recognition

Given a classified probabilistic shape dictionary, and an image with a shape similar to some of the elements in the dictionary, this letter introduces a sparse representation based framework with a twofold goal. First, to select a sparse shape combination from the dictionary that best represents the shape, and second, to accurately segment the image taking into account both the sparse shape combination and the image information. A new energy function that combines the region-based segmentation with sparse representation is introduced to accomplish both goals simultaneously. The experimental results show the superior segmentation and recognition capabilities of the proposed model.

Edge-assisted Object Segmentation Using Multimodal Feature Aggregation and Learning

Object segmentation aims to perfectly identify objects embedded in the surrounding environment and has a wide range of applications. Most previous methods of object segmentation only use RGB images and ignore geometric information from disparity images. Making full use of heterogeneous data from different devices has proved to be a very effective strategy for improving segmentation performance. The key challenge of the multimodal fusion-based object segmentation task lies in the learning, transformation, and fusion of multimodal information. In this article, we focus on the transformation of disparity images and the fusion of multimodal features. We develop a multimodal fusion object segmentation framework, termed the Hybrid Fusion Segmentation Network (HFSNet). Specifically, HFSNet contains three key components, i.e., disparity convolutional sparse coding (DCSC), asymmetric dense projection feature aggregation (ADPFA), and multimodal feature fusion (MFF). The DCSC is designed based on convolutional sparse coding. It not only has better interpretability but also preserves the key geometric information of the object. ADPFA is designed to enhance texture and geometric information to fully exploit nonadjacent features. MFF is used to perform multimodal feature fusion. Extensive experiments show that our HFSNet outperforms existing state-of-the-art models on two challenging datasets.

Generalized framework for image and video object segmentation using affinity learning and message passing GNNS

Despite significant amount of work reported in the computer vision literature, segmenting images or videos based on multiple cues such as objectness, texture and motion, is still a challenge. This is particularly true when the number of objects to be segmented is not known or there are objects that are not classified in the training data (unknown objects). A possible remedy to this problem is to utilize graph-based clustering techniques such as Correlation Clustering. It is known that using long range affinities (Lifted multicut), makes correlation clustering more accurate than using only adjacent affinities (Multicut). However, the former is computationally expensive and hard to use. In this paper, we introduce a new framework to perform image/motion segmentation using an affinity learning module and a Message Passing Graph Neural Network (MPGNN). The affinity learning module uses a permutation invariant affinity representation to overcome the multi-object problem. The paper shows, both theoretically and empirically, that the proposed MPGNN aggregates higher order information and thereby converts the Lifted Multicut Problem (LMP) to a Multicut Problem (MP), which is easier and faster to solve. Importantly, the proposed method can be generalized to deal with different clustering problems with the same MPGNN architecture. For instance, our method produces competitive results for single image segmentation (on BSDS dataset) as well as unsupervised video object segmentation (on DAVIS17 dataset), by only changing the feature extraction part. In addition, using an ablation study on the proposed MPGNN architecture, we show that the way we update the parameterized affinities directly contributes to the accuracy of the results.

Video Object Segmentation Using Multi-Scale Attention-Based Siamese Network

Video target segmentation is a fundamental problem in computer vision that aims to segment targets from a background by learning their appearance information and movement information. In this study, a video target segmentation network based on the Siamese structure was proposed. This network has two inputs: the current video frame, used as the main input, and the adjacent frame, used as the auxiliary input. The processing modules for the inputs use the same structure, optimization strategy, and encoder weights. The input is encoded to obtain features with different resolutions, from which good target appearance features can be obtained. After processing using the encoding layer, the motion features of the target are learned using a multi-scale feature fusion decoder based on an attention mechanism. The final predicted segmentation results were calculated from a layer of decoded features. The video object segmentation framework proposed in this study achieved optimal results on CDNet2014 and FBMS-3D, with scores of 78.36 and 86.71, respectively. It outperformed the second-ranked method by 4.3 on the CDNet2014 dataset and by 0.77 on the FBMS-3D dataset. Suboptimal results were achieved on the video primary target segmentation datasets SegTrackV2 and DAVIS2016, with scores of 60.57 and 81.08, respectively.

SiamMask: A Framework for Fast Online Object Tracking and Segmentation.

In this article, we introduce SiamMask, a framework to perform both visual object tracking and video object segmentation, in real-time, with the same simple method. We improve the offline training procedure of popular fully-convolutional Siamese approaches by augmenting their losses with a binary segmentation task. Once the offline training is completed, SiamMask only requires a single bounding box for initialization and can simultaneously carry out visual object tracking and segmentation at high frame-rates. Moreover, we show that it is possible to extend the framework to handle multiple object tracking and segmentation by simply re-using the multi-task model in a cascaded fashion. Experimental results show that our approach has high processing efficiency, at around 55 frames per second. It yields real-time state-of-the art results on visual-object tracking benchmarks, while at the same time demonstrating competitive performance at a high speed for video object segmentation benchmarks.

Fruit maturity grading framework for small dataset using single image multi-object sampling and Mask R-CNN

Fruit maturity grading is the key factor for fruit export where maturity consistence and standard are required. This paper proposes a non-destructive approach for pineapple maturity grading and pineapple localization based on object segmentation framework which has enhanced for training a robust model with small dataset. We introduced a multi-object sampling technique in augmentation process to generate images from a small dataset taken under controlled conditions to generalize the model for a more practical dataset. We identify the robustness of object segmentation models, Mask R-CNN over other models, e.g., Faster R-CNN, RetinaNet and CenterMask through mean average precision (mAP), detection ratio to explore the false positives detected, precision-recall curve and computational time. The optimal threshold selection which is crucial especially for sensitive small dataset to achieve high detection performance is proposed in this work using mAP and detection ratio. Our proposed framework enhances the model generalization and achieves mAP of 86.7%, AP50 and AP75 at 97.98% and APunripe, APpartially_ripe and APfully_ripe of 99.20%, 96.58% and 98.63%, respectively. Additional insights of the models developed with our small dataset are explained along with the experimental results which are suggested for future.

RVMOS: Range-View Moving Object Segmentation Leveraged by Semantic and Motion Features

Detecting traffic participants is an essential and age-old problem in autonomous driving. Recently, the recognition of moving objects has emerged as a major issue in this field for safe driving. In this paper, we present RVMOS, a LiDAR Range-View-based Moving Object Segmentation framework that segments moving objects given a sequence of range-view images. In contrast to the conventional method, our network incorporates both motion and semantic features, each of which encodes the motion of objects and the surrounding circumstance of the objects. In addition, we design a new feature extraction module suitably designed for range-view images. Lastly, we introduce simple yet effective data augmentation methods: time interval modulation and zero residual image synthesis. With these contributions, we achieve a 19% higher performance (mIoU) with 10% faster computational time (34 FPS on RTX 3090) than the state-of-the-art method with the SemanticKitti benchmark. Extensive experiments demonstrate the effectiveness of our network design and data augmentation scheme.

Generalized Framework for Image and Video Object Segmentation Using Affinity Learning and Message Passing Gnns

Generalized Framework for Image and Video Object Segmentation Using Affinity Learning and Message Passing Gnns

Superpixel-based object boundary gimmicking using optimized conditional random fields with random associations

Superpixel-based clustering approach gains attention as an efficient pre-processing step for image segmentation during the last decade. In general, color similarity and position of the pixels are used as similarity metrics for segmentation. In this proposed work, a two-level object segmentation framework is proposed, where the mid-level cues are meritoriously utilized for efficient object segmentation. In the first level, a superpixel-based object boundary gimmicking algorithm with improved distance measure is used to gimmick the cluster boundaries. To make the picture visualization as human vision friendly, the regular clusters are decomposed literally only at the required places to adhere with object boundaries. Spurious Clusters are identified and merged to remove the noisy superpixels. As a post-processing step, superpixel-based optimized conditional random field (CRF) with random association (SOCRA) algorithm is proposed for efficient segmentation. Here homogeneity of the superpixel is imposed by optimized CRF to augment the object clusters with random associations. CRFs are employed for its excellent capability to characterize the relationship among the random variables. Extensive performance evaluation shows the proposed boundary gimmicking approach significantly competes with the state-of-the-art methods. Results obtained with Berkeley segmentation dataset (BSDS300) are compared with state-of-the-art methods in terms of under segmentation error, boundary recall, and achievable segmentation accuracy.

Meta-VOS: Learning to Adapt Online Target-Specific Segmentation.

The task of video object segmentation is a fundamental but challenging problem in the field of computer vision. To deal with large variations in target objects and background clutter, we propose an online adaptive video object segmentation (VOS) framework, named Meta-VOS, that learns to adapt the target-specific segmentation. Meta-VOS builds an online adaptive learning process by exploiting cumulative expertise after searching for confidence patterns across different videos/frames, and then dynamically improves the model learning from two aspects: Meta-seg learner (i.e., module updating) and Meta-seg criterion (i.e., rule of expertise). As our goal is to rapidly determine which patterns best represent the essential characteristics of specific targets in a video, Meta-seg learner is introduced to adaptively learn to update the parameters and hyperparameters of segmentation network in very few gradient descent steps. Furthermore, a Meta-seg criterion of learned expertise, which is constructed to evaluate the Meta-seg learner for the online adaptation of the segmentation network, can confidently online update positive/negative patterns under the guidance of motion cues, object appearances and learned knowledge. Comprehensive evaluations on several benchmark datasets demonstrate the superiority of our proposed Meta-VOS when compared with other state-of-the-art methods applied to the VOS problem.

A salient object segmentation framework using diffusion-based affinity learning

In this paper, a salient object segmentation framework by using diffusion-based affinity learning and based on absorbing Markov chain (AMC) is proposed. Traditional approaches for structural modeling of images via local information and pairwise similarity graph by using, e.g. Gaussian heat kernel function, are insufficient for capturing the faithful relationships among the regions. According to the AMC principles, the more strong relationships result in lowering the time that a transient node becomes an absorbed one and consequently increases the transition probability between those nodes. To this end, a dense transition probability matrix is constructed based on an affinity matrix which learned using a diffusion process. Computing tensor product of the initial similarity graph with itself provides credible information about inter-relationships of nodes. Since conducting similarity propagation over such a tensor product graph imposes high computational costs, an iterative diffusion process is leveraged that introduces the same complexity as applying traditional diffusion processes on the original graph. As a fundamental benefit, such a process will enhance the accuracy and preciseness of saliency detection. Finally, as a complementary step, the saliency map will be refined by revisiting the saliency value of every single pixel. The experimental results on three major benchmark datasets demonstrate the efficiency of the proposed framework. More specifically, as expected, taking advantage of full learned affinity matrix can significantly improve the precision of the process.

Automated Segmentation and Morphological Analyses of Stockpile Aggregate Images using Deep Convolutional Neural Networks

Particle size and morphological/shape properties ensure the reliable and sustainable use of all aggregate skeleton materials placed as constructed layers in transportation applications. The composition and packing of these aggregate assemblies rely heavily on particle size and morphological properties, which affect layer strength, modulus, and deformation response under vehicular loading and therefore facilitate the quality assurance/quality control (QA/QC) process. Aggregate imaging systems developed to date for size and shape characterization, however, have primarily focused on measurement of separated or slightly contacting aggregate particles. Development of efficient computer vision algorithms is urgently needed for image-based evaluations of densely stacked (or stockpile) aggregates, which requires image segmentation of a stockpile for the size and morphological properties of individual particles. This paper presents an innovative approach for automated segmentation and morphological analyses of stockpile aggregate images based on deep learning techniques. A task-specific stockpile aggregate image dataset is established from images collected from various quarries in Illinois. Individual particles from the stockpile images are manually labeled on each image associated with particle locations and regions. A state-of-the-art object detection and segmentation framework called Mask R-CNN is then used to train the image segmentation kernel, which enables user-independent segmentation of stockpile aggregate images. The segmentation results show good agreement with ground-truth labeling and improve the efficiency of size and morphological analyses conducted on densely stacked and overlapping particle images. Based on the presented approach, stockpile aggregate image analysis promises to become an efficient and innovative application for field-scale and in-place evaluations of aggregate materials.

MASK-RL: Multiagent Video Object Segmentation Framework Through Reinforcement Learning.

Integrating human-provided location priors into video object segmentation has been shown to be an effective strategy to enhance performance, but their application at large scale is unfeasible. Gamification can help reduce the annotation burden, but it still requires user involvement. We propose a video object segmentation framework that leverages the combined advantages of user feedback for segmentation and gamification strategy by simulating multiple game players through a reinforcement learning (RL) model that reproduces human ability to pinpoint moving objects and using the simulated feedback to drive the decisions of a fully convolutional deep segmentation network. Experimental results on the DAVIS-17 benchmark show that: 1) including user-provided prior, even if not precise, yields high performance; 2) our RL agent replicates satisfactorily the same variability of humans in identifying spatiotemporal salient objects; and 3) employing artificially generated priors in an unsupervised video object segmentation model reaches state-of-the-art performance.

Reliable and Dynamic Appearance Modeling and Label Consistency Enforcing for Fast and Coherent Video Object Segmentation With the Bilateral Grid

We propose a novel optimization framework for video object segmentation, given the initial annotations of objects in the keyframes of an input video sequence. In this work, video data is represented by a Markov Random Field model, and segmentation is achieved by finding the minimum graph cut label assignment. More specifically, we first create a bilateral representation of the input video sequence which reduces the size of the graph that the min-cut must operate on. We then introduce dynamic appearance models to learn the segmentation likelihoods, and the reliability of likelihoods is measured to identify false likelihoods that may cause segmentation errors. Thus, the model accurately describes changes in the object’s appearance that have evolved over time. Furthermore, we augment spatial and temporal connections using a soft higher-order potential, ensuring long-range label consistency in the segmentation. We provide extensive analysis and evaluation with respect to the influence of each component of the framework through the ablation study. Experiments on three benchmark datasets (DAVIS 2016, YouTube-Objects and SegTrack v2) show that our method achieves competitive performance compared to state-of-the-art while having the order of magnitude faster runtime.

Unsupervised Online Video Object Segmentation With Motion Property Understanding.

Unsupervised video object segmentation aims to automatically segment moving objects over an unconstrained video without any user annotation. So far, only few unsupervised online methods have been reported in the literature, and their performance is still far from satisfactory because the complementary information from future frames cannot be processed under online setting. To solve this challenging problem, in this paper, we propose a novel unsupervised online video object segmentation (UOVOS) framework by construing the motion property to mean moving in concurrence with a generic object for segmented regions. By incorporating the salient motion detection and the object proposal, a pixel-wise fusion strategy is developed to effectively remove detection noises, such as dynamic background and stationary objects. Furthermore, by leveraging the obtained segmentation from immediately preceding frames, a forward propagation algorithm is employed to deal with unreliable motion detection and object proposals. Experimental results on several benchmark datasets demonstrate the efficacy of the proposed method. Compared to state-of-the-art unsupervised online segmentation algorithms, the proposed method achieves an absolute gain of 6.2%. Moreover, our method achieves better performance than the best unsupervised offline algorithm on the DAVIS-2016 benchmark dataset. Our code is available on the project website: https://www.github.com/visiontao/uovos.

ObjectFusion: An object detection and segmentation framework with RGB-D SLAM and convolutional neural networks

Given the driving advances on CNNs (Convolutional Neural Networks) [1], deep neural networks being deployed for accurate detection and semantic reconstruction in SLAM (Simultaneous Localization and Mapping) has become a trend. However, as far as we know, almost all existing methods focus on design a specific CNN architecture for single task. In this paper, we propose a novel framework which employs a general object detection CNN to fuse with a SLAM system towards obtaining better performances on both detection and semantic segmentation in 3D space. Our approach first use CNN-based detection network to obtain the 2D object proposals which can be used to establish the local target map. We then use the results estimated from SLAM to update the dynamic global target map based on the local target map obtained by CNNs. Finally, we are able to obtain the detection result for the current frame by projecting the global target map into 2D space. On the other hand, we send the estimation results back to SLAM and update the semantic surfel model in SLAM system. Therefore, we can acquire the segmentation result by projecting the updated 3D surfel model into 2D. Our fusion scheme privileges in object detection and segmentation by integrating with SLAM system to preserve the spatial continuity and temporal consistency. Evaluation performances on four datasets demonstrate the effectiveness and robustness of our method.

Segmentation of moving objects in image sequence based on perceptual similarity of local texture and photometric features

The segmentation of moving objects in image sequence can be formulated as a background subtraction problem—the separation of objects from the background in each image frame. The background scene is learned and modeled. A pixelwise process is employed to classify each pixel as an object or background based on its similarity with the background model. The segmentation is challenging due to the occurrence of dynamic elements such as illumination change and background motions. We propose a framework for object segmentation with a novel feature for background representation and new mechanisms for updating the background model. A ternary pattern is employed to characterize the local texture. The pattern and photometric features are used for background modeling. The classification of pixel is performed based on the perceptual similarity between the current pixel and the background model. The segmented object is refined by taking into account the spatial consistency of the image feature. For the background model update, we propose two mechanisms that are able to adapt to abrupt background change and also merge new background elements into the model. We compare our framework with various background subtraction algorithms on video datasets.

A Framework for Acoustic Segmentation Using Order Statistic-Constant False Alarm Rate in Two Dimensions From Sidescan Sonar Data

This paper describes a framework for object segmentation from sidescan sonar acoustic data. The current techniques consume a great deal of computational resources to accurately carry out object segmentation. They also involve the tuning of many parameters to obtain good quality images. This is due to the handling of the large data volume generated by these devices and environmental fluctuations such as salinity, density, temperature, and others variations. The framework proposed uses a migration and adaptation of a technique widely used in radar technology for detecting moving objects. This radar technique is known as order statistic-constant false alarm rate (OS-CFAR) applied in 2-D. OS-CFAR 2-D rank orders the samples obtained from a sliding window to make a segmentation of the image. This segmentation is done into several types of regions: acoustic highlight, shadow, and different seafloor reverberation areas. OS-CFAR 2-D is less sensitive than other methods to the presence of the speckle noise due to the use of order statistics. This proposal was contrasted experimentally on real images. Likewise, an experimental comparison with the results obtained with the undecimated discrete wavelet transform, active contours, Markov random field, and accumulated cell averaging CFAR applied in two dimensions technique is also presented.

SSG: superpixel segmentation and GrabCut-based salient object segmentation

Saliency detection is a popular topic for image processing recently. In this paper, we propose a simple, robust and fast salient object segmentation framework. Firstly, we develop a novel saliency map segmentation strategy, named SSG which consists of superpixel region growing, superpixel Density-Based Spatial Clustering of Applications with Noise (DBSCAN) clustering and iterated graph cuts (GrabCut), where DBSCAN makes similar background regions cluster as a whole, region growing groups similar regions together as much as possible, GrabCut segments salient objects accurately. Then, the proposed SSG is combined with saliency detection to abstract salient objects. Experimental results on three benchmark datasets demonstrate that the proposed method achieves the favorable performance than many recent state-of-the-art methods in terms of precision, recall, F-measure and execution time.