Object Segmentation Research Articles

Unsupervised video object segmentation with mask transformer: boosting accuracy and efficiency through feature fusion

Unsupervised video object segmentation with mask transformer: boosting accuracy and efficiency through feature fusion

MVSRF: Point cloud semantic segmentation and optimization method for granular construction objects

MVSRF: Point cloud semantic segmentation and optimization method for granular construction objects

Using deep learning model integration to build a smart railway traffic safety monitoring system

According to the importance of railway safety, it is crucial to build a smart railway traffic safety system in Taiwan, especially there are often to see related accidents. Therefore, this study aimed to build a smart railway traffic safety system using the integration of object detection, segmentation, machine learning, and notification system. First, the Mask R-CNN model was applied to automatically build the digital boundaries of railway, which achieved an average Interest of Union (IOU) of over 0.9. Then, the YOLO v3 model was applied to detect intrusions of railway, especially humans’ intrusion. The above object detection model achieved an Overall accuracy (OA) of over 90% for different classes, and an OA of 95.68% for human detection. The YOLO v3 model was also able to detect intrusion within different scenarios, such as nighttime, rainy daytime, and rainy nighttime. Moreover, the XGBoost model was applied to predict the sizes of intruding objects, which has a low MAE of 0.54 cm and an R2 score of 0.997. Finally, the LINE bot was applied to notify the related operators, including the above information, such as time of intrusion, locations, classes of intruding objects, sizes and the image of intrusion. The above implementation can be helpful for railway traffic safety monitoring, which may help related accidents prevention.

Continuous-time Object Segmentation using High Temporal Resolution Event Camera.

Event cameras are novel bio-inspired sensors, where individual pixels operate independently and asynchronously, generating intensity changes as events. Leveraging the microsecond resolution (no motion blur) and high dynamic range (compatible with extreme light conditions) of events, there is considerable promise in directly segmenting objects from sparse and asynchronous event streams in various applications. However, different from the rich cues in video object segmentation, it is challenging to segment complete objects from the sparse event stream. In this paper, we present the first framework for continuous-time object segmentation from event stream. Given the object mask at the initial time, our task aims to segment the complete object at any subsequent time in event streams. Specifically, our framework consists of a Recurrent Temporal Embedding Extraction (RTEE) module based on a novel ResLSTM, a Cross-time Spatiotemporal Feature Modeling (CSFM) module which is a transformer architecture with long-term and short-term matching modules, and a segmentation head. The historical events and masks (reference sets) are recurrently fed into our framework along with current-time events. The temporal embedding is updated as new events are input, enabling our framework to continuously process the event stream. To train and test our model, we construct both real-world and simulated event-based object segmentation datasets, each comprising event streams, APS images, and object annotations. Extensive experiments on our datasets demonstrate the effectiveness of the proposed recurrent architecture. Our code and dataset are available at https://sites.google.com/view/ecos-net/.

Promoting camouflaged object detection through novel edge-target interaction and frequency-spatial fusion

Promoting camouflaged object detection through novel edge-target interaction and frequency-spatial fusion

LBSNet: Lightweight Joint Boundary Detection and Semantic Segmentation for Transparent and Reflective Objects

LBSNet: Lightweight Joint Boundary Detection and Semantic Segmentation for Transparent and Reflective Objects

Object segmentation of near surface magnetic field data based on deep convolutional neural networks

Object segmentation of near surface magnetic field data based on deep convolutional neural networks

GobletNet: Wavelet-Based High-Frequency Fusion Network for Semantic Segmentation of Electron Microscopy Images.

Semantic segmentation of electron microscopy (EM) images is crucial for nanoscale analysis. With the development of deep neural networks (DNNs), semantic segmentation of EM images has achieved remarkable success. However, current EM image segmentation models are usually extensions or adaptations of natural or biomedical models. They lack the full exploration and utilization of the intrinsic characteristics of EM images. Furthermore, they are often designed only for several specific segmentation objects and lack versatility. In this study, we quantitatively analyze the characteristics of EM images compared with those of natural and other biomedical images via the wavelet transform. To better utilize these characteristics, we design a high-frequency (HF) fusion network, GobletNet, which outperforms state-of-the-art models by a large margin in the semantic segmentation of EM images. We use the wavelet transform to generate HF images as extra inputs and use an extra encoding branch to extract HF information. Furthermore, we introduce a fusion-attention module (FAM) into GobletNet to facilitate better absorption and fusion of information from raw images and HF images. Extensive benchmarking on seven public EM datasets (EPFL, CREMI, SNEMI3D, UroCell, MitoEM, Nanowire and BetaSeg) demonstrates the effectiveness of our model. The code is available at https://github.com/Yanfeng-Zhou/GobletNet.

ProtoSAM-3D: Interactive semantic segmentation in volumetric medical imaging via a Segment Anything Model and mask-level prototypes.

Semantic segmentation of volumetric medical images is essential for accurate delineation of anatomic structures and pathology, enabling quantitative analysis in precision medicine applications. While volumetric segmentation has been extensively studied, most existing methods require full supervision and struggle to generalize to new classes at inference time, particularly for irregular, ill-defined targets such as tumors, where fine-grained, high-salience segmentation is required. Consequently, conventional semantic segmentation methods cannot easily offer zero/few-shot generalization to segment objects of interest beyond their closed training set. Foundation models, such as the Segment Anything Model (SAM), have demonstrated promising zero-shot generalization for interactive instance segmentation based on user prompts. However, these models sacrifice semantic knowledge for generalization capabilities that largely rely on collaborative user prompting to inject semantics. For volumetric medical image analysis, a unified approach that combines the semantic understanding of conventional segmentation methods with the flexible, prompt-driven capabilities of SAM is essential for comprehensive anatomical delineation On the one hand, it is natural to exploit anatomic knowledge to enable semantic segmentation without any user interaction. On the other hand, SAM-like approaches to segment unknown classes via prompting provide the needed flexibility to segment structures beyond the closed training set, enabling quantitative analysis. To address these needs in a unified framework, we introduce ProtoSAM-3D, which extends SAMs to semantic segmentation of volumetric data via a novel mask-level prototype prediction approach while retaining the flexibility of SAM Our model utilizes an innovative spatially-aware Transformer to fuse instance-specific intermediate representations from the SAM encoder and decoder, obtaining a comprehensive feature embedding for each mask. These embeddings are then classified by computing similarity with learned prototypes. By predicting prototypes instead of classes directly, ProtoSAM-3D gains the flexibility to rapidly adapt to new classes with minimal retraining. Furthermore, we introduce an auto-prompting method to enable semantic segmentation of known classes without user interaction. We demonstrate state-of-the-art zero/few-shot performance on multi-organ segmentation in CT and MRI. Experimentally, ProtoSAM-3D achieves competitive performance compared to fully supervised methods. Our work represents a step towards interactive semantic segmentation models with SAM for volumetric medical image processing.

Using the Mask R-CNN model for segmentation of real estate objects in aerial photographs

Using the Mask R-CNN model for segmentation of real estate objects in aerial photographs

Modeling Objects with Artificial Intelligence Based Image Processing Techniques: Object Detection with Mask R-CNN

Object detection and classification on digital images have great importance in the digitalizing world. After deep learning methods started being implemented for object detection, classification and segmentation a rapid development has been monitored in the field. One of the most successful methods in the field is Mask R-CNN. It can be used in order to detect and segment purposes for many different objects. This study contains the use of Mask R-CNN for weapon detection, specifically handguns. Nowadays, there are many cameras in public areas and these cameras can detect weapons before a forensic incident. Our model achieved a mean average precision (mAP) of 0.78 in the detection of handguns on test data. Our findings verify the potential of deep learning in security by detecting threats in images and live videos.

GCNet: A Deep Learning Framework for Enhanced Grape Cluster Segmentation and Yield Estimation Incorporating Occluded Grape Detection with a Correction Factor for Indoor Experimentation

Object segmentation algorithms have heavily relied on deep learning techniques to estimate the count of grapes which is a strong indicator for the yield success of grapes. The issue with using object segmentation algorithms for grape analytics is that they are limited to counting only the visible grapes, thus omitting hidden grapes, which affect the true estimate of grape yield. Many grapes are occluded because of either the compactness of the grape bunch cluster or due to canopy interference. This introduces the need for models to be able to estimate the unseen berries to give a more accurate estimate of the grape yield by improving grape cluster segmentation. We propose the Grape Counting Network (GCNet), a novel framework for grape cluster segmentation, integrating deep learning techniques with correction factors to address challenges in indoor yield estimation. GCNet incorporates occlusion adjustments, enhancing segmentation accuracy even under conditions of foliage and cluster compactness, and setting new standards in agricultural indoor imaging analysis. This approach improves yield estimation accuracy, achieving a R² of 0.96 and reducing mean absolute error (MAE) by 10% compared to previous methods. We also propose a new dataset called GrapeSet which contains visible imagery of grape clusters imaged indoors, along with their ground truth mask, total grape count, and weight in grams. The proposed framework aims to encourage future research in determining which features of grapes can be leveraged to estimate the correct grape yield count, equip grape harvesters with the knowledge of early yield estimation, and produce accurate results in object segmentation algorithms for grape analytics.

Semantic Object Segmentation using Modified HRNet Deep Learning Model

This paper presents a modified High-Resolution Network (HRNet) architecture designed for enhanced semantic object segmentation, addressing the prevalent challenges of accuracy and computational efficiency in complex image analyses. Through strategic modifications to the conventional HRNet, including optimized feature blocks and advanced feature extraction techniques, our model demonstrates significant improvements in segmentation performance. Utilizing the Cityscapes dataset, renowned for its comprehensive urban scene representation, our enhanced HRNet model achieved a notable increase in validation accuracy to 85.8% and mean Intersection over Union (mIoU) to 63.43%, surpassing the original HRNet benchmarks by 3.39% and 3.43%, respectively. These results highlight not only the precision of our model in delineating intricate urban landscapes but also its robustness in handling diverse object scales and complexities. The qualitative analysis, underscored by comparison images, reveals our model's ability to produce more defined segmentation contours and accurate object identifications, setting a new standard for HRNet-based semantic segmentation. This work not only advances the field of high-resolution image segmentation but also offers a foundational model for future research aimed at solving increasingly complex image processing challenges

Comparing Above and Underwater Visibility of Fluorescent Quinine-Based Liquid and Solid Tracers to Estimate Suspended Sediment Concentrations Under Low Luminosity Conditions

While exploring the use and visibility of fluorescent tracers for estimating flow velocities under varying water turbidity, this study introduces a technique for estimating suspended sediment concentration (SSC) in water. Laboratory and field experiments were conducted using a dual above/underwater optical camera monitoring system to measure and compare the visibility (brightness intensity) of fluorescent tracers applied into water under varying SSC loads. Based on the fluorescent properties of quinine under low luminosity conditions, a quinine solution, in both liquid and solid (ice cube) states, was applied to the water surface. Its visibility was recorded using the monitoring system, and brightness intensity was quantified using image processing techniques such as grayscale conversion, object segmentation, and pixel intensity extraction. Results indicate that the tracer brightness intensity decreases as SSC loads increase, with the underwater camera recording lower visibility than the above-water camera, particularly at higher SSC loads. Empirical regression equations were developed to describe the relationship between the tracer brightness intensity recorded by the underwater camera and the SSC loads. These findings suggest the ability of the presented monitoring system, using quinine solution in both liquid and solid states, along with an underwater camera, to estimate a relevant range of SSC loads under low luminosity conditions.

A semi-supervised fracture-attention model for segmenting tubular objects with improved topological connectivity.

Ensuring connectivity and preventing fractures in tubular object segmentation are critical for downstream analyses. Despite advancements in deep neural networks that have significantly improved tubular object segmentation, existing methods still face limitations. They often rely heavily on precise annotations, hindering their scalability to large-scale unlabeled image datasets. Additionally, current evaluation metrics are insufficient for effectively capturing segmentation fractures. To address these challenges, we propose a semi-supervised fracture-attention model (SSFA) for tubular object segmentation. SSFA enhances connectivity, reduces fractures, and maintains volumetric accuracy. It outperforms state-of-the-art models in topological performance. Extensive experiments on four public datasets validate the effectiveness of SSFA. Furthermore, we introduce a novel evaluation metric, the fracture rate, which provides an intuitive and quantitative assessment of segmentation fractures. Our source code is available at http://github.com/Yanfeng-Zhou/SSFA.

Gradient Enhancement Techniques and Motion Consistency Constraints for Moving Object Segmentation in 3D LiDAR Point Clouds

The ability to segment moving objects from three-dimensional (3D) LiDAR scans is critical to advancing autonomous driving technology, facilitating core tasks like localization, collision avoidance, and path planning. In this paper, we introduce a novel deep neural network designed to enhance the performance of 3D LiDAR point cloud moving object segmentation (MOS) through the integration of image gradient information and the principle of motion consistency. Our method processes sequential range images, employing depth pixel difference convolution (DPDC) to improve the efficacy of dilated convolutions, thus boosting spatial information extraction from range images. Additionally, we incorporate Bayesian filtering to impose posterior constraints on predictions, enhancing the accuracy of motion segmentation. To handle the issue of uneven object scales in range images, we develop a novel edge-aware loss function and use a progressive training strategy to further boost performance. Our method is validated on the SemanticKITTI-based LiDAR MOS benchmark, where it significantly outperforms current state-of-the-art (SOTA) methods, all while working directly on two-dimensional (2D) range images without requiring mapping.

Unraveling Spatial Heterogeneity in Mass Spectrometry Imaging Data with GraphMSI.

Mass spectrometry imaging (MSI) provides valuable insights into metabolic heterogeneity by capturing in situ molecular profiles within organisms. One challenge of MSI heterogeneity analysis is performing an objective segmentation to differentiate the biological tissue into distinct regions with unique characteristics. However, current methods struggle due to the insufficient incorporation of biological context and high computational demand. To address these challenges, a novel deep learning-based approach is proposed, GraphMSI, which integrates metabolic profiles with spatial information to enhance MSI data analysis. Our comparative results demonstrate GraphMSI outperforms commonly used segmentation methods in both visual inspection and quantitative evaluation. Moreover, GraphMSI can incorporate partial or coarse biological contexts to improve segmentation results and enable more effective three-dimensional MSI segmentation with reduced computational requirements. These are facilitated by two optional enhanced modes: scribble-interactive and knowledge-transfer. Numerous results demonstrate the robustness of these two modes, ensuring that GraphMSI consistently retains its capability to identify biologically relevant sub-regions in complex practical applications. It is anticipated that GraphMSI will become a powerful tool for spatial heterogeneity analysis in MSI data.

Semi-supervised one-shot learning for video object segmentation in dynamic environments

Semi-supervised one-shot learning for video object segmentation in dynamic environments

Image segmentation with traveling waves in an exactly solvable recurrent neural network

We study image segmentation using spatiotemporal dynamics in a recurrent neural network where the state of each unit is given by a complex number. We show that this network generates sophisticated spatiotemporal dynamics that can effectively divide an image into groups according to a scene’s structural characteristics. We then demonstrate a simple algorithm for object segmentation that generalizes across inputs ranging from simple geometric objects in grayscale images to natural images. Using an exact solution of the recurrent network’s dynamics, we present a precise description of the mechanism underlying object segmentation in the network dynamics, providing a clear mathematical interpretation of how the algorithm performs this task. Object segmentation across all images is accomplished with one recurrent neural network that has a single, fixed set of weights. This demonstrates the expressive potential of recurrent neural networks when constructed using a mathematical approach that brings together their structure, dynamics, and computation.

A novel W13 deep CNN structure for improved semantic segmentation of multiple objects in remote sensing imagery

AbstractThis paper proposes a novel convolutional neural network (CNN) architecture designed for semantic segmentation in remote sensing images. The proposed W13 Net model addresses the inherent challenges of segmentation tasks through a carefully crafted architecture, combining the strengths of multistage encoding–decoding, skip connections, combined weighted output, and concatenation techniques. Compared with different segmentation models, the suggested model performs better. A comprehensive analysis of different segmentation models has been carried out, resulting in an extensive comparison between the proposed W13 Net and five existing state-of-the-art segmentation architectures. Utilizing two standardized datasets, the Dense Labeling Remote Sensing Dataset Termed (DLRSD), and the Mohammad Bin Rashid Space Center (MBRSC) Dubai Aerial Imagery Dataset, the evaluation entails training, testing, and validation across different classes. The W13 Net demonstrates adaptability, generalization capabilities, and superior results in key metrics, all while displaying robustness across a variety of datasets. A number of metrics, including accuracy, precision, recall, F1 score, and IOU, were used to evaluate the system’s performance. According to the experimental results, the W13 Net model obtained an accuracy of 87.8%, precision of 0.88, recall of 0.88, F1 score of 0.88, and IOU of 0.74. The suggested model showed a significant improvement in segmentation IOU, with an increase of up to 18%, when compared to other with the recent segmentation models taking into consideration the model’s comparatively low number of parameter (2.2 million) in comparison with the recent models.