Instance Segmentation Research Articles

Utilizing an Instance Segmentation Network Capable of Balancing Precision and Speed to Achieve Fine-Grained Vibration Displacement Measurement of Rotating Bodies

Utilizing an Instance Segmentation Network Capable of Balancing Precision and Speed to Achieve Fine-Grained Vibration Displacement Measurement of Rotating Bodies

Enhancing sustainable Chinese cabbage production: a comparative analysis of multispectral image instance segmentation techniques

Accurate instance segmentation of individual crops is crucial for field management and crop monitoring in smart agriculture. To address the limitations of traditional remote sensing methods in individual crop analysis, this study proposes a novel instance segmentation approach combining UAVs with the YOLOv8-Seg model. The YOLOv8-Seg model supports independent segmentation masks and detection at different scales, utilizing Path Aggregation Feature Pyramid Networks (PAFPN) for multi-scale feature integration and optimizing sample matching through the Task-Aligned Assigner. We collected multispectral data of Chinese cabbage using UAVs and constructed a high-quality dataset via semi-automatic annotation with the Segment Anything Model (SAM). Using mAP as the evaluation metric, we compared YOLO series algorithms with other mainstream instance segmentation methods and analyzed model performance under different spectral band combinations and spatial resolutions. The results show that YOLOv8-Seg achieved 86.3% mAP under the RGB band and maintained high segmentation accuracy at lower spatial resolutions (1.33 ~ 1.14 cm/pixel), successfully extracting key metrics such as cabbage count and average leaf area. These findings highlight the potential of integrating UAV technology with advanced segmentation models for individual crop monitoring, supporting precision agriculture applications.

High-Quality Damaged Building Instance Segmentation Based on Improved Mask Transfiner Using Post-Earthquake UAS Imagery: A Case Study of the Luding Ms 6.8 Earthquake in China

Unmanned aerial systems (UASs) are increasingly playing a crucial role in earthquake emergency response and disaster assessment due to their ease of operation, mobility, and low cost. However, post-earthquake scenes are complex, with many forms of damaged buildings. UAS imagery has a high spatial resolution, but the resolution is inconsistent between different flight missions. These factors make it challenging for existing methods to accurately identify individual damaged buildings in UAS images from different scenes, resulting in coarse segmentation masks that are insufficient for practical application needs. To address these issues, this paper proposed DB-Transfiner, a building damage instance segmentation method for post-earthquake UAS imagery based on the Mask Transfiner network. This method primarily employed deformable convolution in the backbone network to enhance adaptability to collapsed buildings of arbitrary shapes. Additionally, it used an enhanced bidirectional feature pyramid network (BiFPN) to integrate multi-scale features, improving the representation of targets of various sizes. Furthermore, a lightweight Transformer encoder has been used to process edge pixels, enhancing the efficiency of global feature extraction and the refinement of target edges. We conducted experiments on post-disaster UAS images collected from the 2022 Luding earthquake with a surface wave magnitude (Ms) of 6.8 in the Sichuan Province of China. The results demonstrated that the average precisions (AP) of DB-Transfiner, APbox and APseg, are 56.42% and 54.85%, respectively, outperforming all other comparative methods. Our model improved the original model by 5.00% and 4.07% in APbox and APseg, respectively. Importantly, the APseg of our model was significantly higher than the state-of-the-art instance segmentation model Mask R-CNN, with an increase of 9.07%. In addition, we conducted applicability testing, and the model achieved an average correctness rate of 84.28% for identifying images from different scenes of the same earthquake. We also applied the model to the Yangbi earthquake scene and found that the model maintained good performance, demonstrating a certain level of generalization capability. This method has high accuracy in identifying and assessing damaged buildings after earthquakes and can provide critical data support for disaster loss assessment.

Cucumber Leaf Segmentation Based on Bilayer Convolutional Network

When monitoring crop growth using top-down images of the plant canopies, leaves in agricultural fields appear very dense and significantly overlap each other. Moreover, the image can be affected by external conditions such as background environment and light intensity, impacting the effectiveness of image segmentation. To address the challenge of segmenting dense and overlapping plant leaves under natural lighting conditions, this study employed a Bilayer Convolutional Network (BCNet) method for accurate leaf segmentation across various lighting environments. The major contributions of this study are as follows: (1) Utilized Fully Convolutional Object Detection (FCOS) for plant leaf detection, incorporating ResNet-50 with the Convolutional Block Attention Module (CBAM) and Feature Pyramid Network (FPN) to enhance Region of Interest (RoI) feature extraction from canopy top-view images. (2) Extracted the sub-region of the RoI based on the position of the detection box, using this region as input for the BCNet, ensuring precise segmentation. (3) Utilized instance segmentation of canopy top-view images using BCNet, improving segmentation accuracy. (4) Applied the Varifocal Loss Function to improve the classification loss function in FCOS, leading to better performance metrics. The experimental results on cucumber canopy top-view images captured in glass greenhouse and plastic greenhouse environments show that our method is highly effective. For cucumber leaves at different growth stages and under various lighting conditions, the Precision, Recall and Average Precision (AP) metrics for object recognition are 97%, 94% and 96.57%, respectively. For instance segmentation, the Precision, Recall and Average Precision (AP) metrics are 87%, 83% and 84.71%, respectively. Our algorithm outperforms commonly used deep learning algorithms such as Faster R-CNN, Mask R-CNN, YOLOv4 and PANet, showcasing its superior capability in complex agricultural settings. The results of this study demonstrate the potential of our method for accurate recognition and segmentation of highly overlapping leaves in diverse agricultural environments, significantly contributing to the application of deep learning algorithms in smart agriculture.

Three-Dimensional Instance Segmentation Using the Generalized Hough Transform and the Adaptive n-Shifted Shuffle Attention

The progress of 3D instance segmentation techniques has made it essential for several applications, such as augmented reality, autonomous driving, and robotics. Traditional methods usually have challenges with complex indoor scenes made of multiple objects with different occlusions and orientations. In this work, the authors present an innovative model that integrates a new adaptive n-shifted shuffle (ANSS) attention mechanism with the Generalized Hough Transform (GHT) for robust 3D instance segmentation of indoor scenes. The proposed technique leverages the n-shifted sigmoid activation function, which improves the adaptive shuffle attention mechanism, permitting the network to dynamically focus on relevant features across various regions. A learnable shuffling pattern is produced through the proposed ANSS attention mechanism to spatially rearrange the relevant features, thus augmenting the model’s ability to capture the object boundaries and their fine-grained details. The integration of GHT furnishes a vigorous framework to localize and detect objects in the 3D space, even when heavy noise and partial occlusions are present. The authors evaluate the proposed method on the challenging Stanford 3D Indoor Spaces Dataset (S3DIS), where it establishes its superiority over existing methods. The proposed approach achieves state-of-the-art performance in both mean Intersection over Union (IoU) and overall accuracy, showcasing its potential for practical deployment in real-world scenarios. These results illustrate that the integration of the ANSS and the GHT yields a robust solution for 3D instance segmentation tasks.

YOLOv8-seg-CP: a lightweight instance segmentation algorithm for chip pad based on improved YOLOv8-seg model.

Real-time detection and accurate segmentation of chip pads are important tasks to ensure chip alignment and position correction. To address the challenges of small target chip pad detection, segmentation accuracy and model lightweight, this paper proposes a lightweight chip pad instance segmentation algorithm based on an improved YOLOv8-seg, named YOLOv8-seg-CP (chip pad). Firstly, we integrate the next-generation lightweight StarNet into the original backbone network to enhance fine feature capture capabilities while reducing the number of parameters. Then, we construct the C2f-Star module in the neck network, which enhances the feature extraction performance for small chip pad targets. This maintains accuracy, reduces computational load, and improves detection and segmentation speed. On this basis, we introduce a lightweight shared convolution segmentation head (LSCSH), significantly reducing both parameter count and computational load while enhancing segmentation performance. Additionally, we propose a CGCAFusion convolutional attention fusion module. This module uses a content-guided convolutional attention fusion mechanism to dynamically adjust attention weights based on the content of input features, capturing both global and local feature information and enhancing multimodal feature fusion. Experiments on the chip pad dataset demonstrate that our algorithm achieves a detection and segmentation accuracy of 89.8%. The model size, parameters, and FLOPs are 3.7M, 1.7M, and 8.2G respectively, representing reductions of 45.6%, 50%, and 31.7% compared to the baseline model. The FPS is 1399.3, an improvement of 25.8% over the baseline model. The inference time is 0.72ms, which is 0.18ms less than the baseline model. Extensive experimental results on COCO, carparts-seg, and crack-seg datasets further show that the improved YOLOv8n-seg model outperforms many existing advanced methods in terms of performance. This approach holds significant industrial application value for fully automated chip testing and sorting integrated production.

I-DINO: High-Quality Object Detection for Indoor Scenes

Object Detection in Complex Indoor Scenes is designed to identify and categorize objects in indoor settings, with applications in areas such as smart homes, security surveillance, and home service robots. It forms the basis for advanced visual tasks including visual question answering, video description generation, and instance segmentation. Nonetheless, the task faces substantial hurdles due to background clutter, overlapping objects, and significant size differences. To tackle these challenges, this study introduces an indoor object detection approach utilizing an enhanced DINO framework. To cater to the needs of indoor object detection, an Indoor-COCO dataset was developed from the COCO object detection dataset. The model incorporates an advanced Res2net as the backbone feature extraction network, complemented by a deformable attention mechanism to better capture detailed object features. An upgraded Bi-FPN module is employed to replace the conventional feature fusion module, and SIoU loss is utilized to expedite convergence. The experimental outcomes indicate that the refined model attains an mAP of 62.3%, marking a 5.2% improvement over the baseline model. These findings illustrate that the DINO-based indoor object detection model exhibits robust generalization abilities and practical utility for multi-scale object detection in complex environments.

AER-Net: Attention-Enhanced Residual Refinement Network for Nuclei Segmentation and Classification in Histology Images

The acurate segmentation and classification of nuclei in histological images are crucial for the diagnosis and treatment of colorectal cancer. However, the aggregation of nuclei and intra-class variability in histology images present significant challenges for nuclei segmentation and classification. In addition, the imbalance of various nuclei classes exacerbates the difficulty of nuclei classification and segmentation using deep learning models. To address these challenges, we present a novel attention-enhanced residual refinement network (AER-Net), which consists of one encoder and three decoder branches that have same network structure. In addition to the nuclei instance segmentation branch and nuclei classification branch, one branch is used to predict the vertical and horizontal distance from each pixel to its nuclear center, which is combined with output by the segmentation branch to improve the final segmentation results. The AER-Net utilizes an attention-enhanced encoder module to focus on more valuable features. To further refine predictions and achieve more accurate results, an attention-enhancing residual refinement module is employed at the end of each encoder branch. Moreover, the coarse predictions and refined predictions are combined by using a loss function that employs cross-entropy loss and generalized dice loss to efficiently tackle the challenge of class imbalance among nuclei in histology images. Compared with other state-of-the-art methods on two colorectal cancer datasets and a pan-cancer dataset, AER-Net demonstrates outstanding performance, validating its effectiveness in nuclear segmentation and classification.

Spontaneous breaking of symmetry in overlapping cell instance segmentation using diffusion models

Abstract Instance segmentation is the task of assigning unique identifiers to individual objects in images. Solving this task requires breaking the inherent symmetry that semantically similar objects must result in distinct outputs. Deep learning algorithms bypass this break-of-symmetry by training specialized predictors or by utilizing intermediate label representations. However, many of these approaches break down when faced with overlapping labels that are ubiquitous in biomedical imaging, for instance for segmenting cell layers. Here, we discuss the reason for this failure and offer a novel approach for instance segmentation based on diffusion models that breaks this symmetry spontaneously. Our method outputs pixel-level instance segmentations matching the performance of models such as cellpose on the cellpose fluorescent cell dataset while also permitting overlapping labels.

Instance Segmentation XXL-CT Challenge of a Historic Airplane

Instance segmentation of compound objects in XXL-CT imagery poses a unique challenge in non-destructive testing. This complexity arises from the lack of known reference segmentation labels, limited applicable segmentation tools, as well as partially degraded image quality. To asses recent advancements in the field of machine learning-based image segmentation, the ‘Instance Segmentation XXL-CT Challenge of a Historic Airplane’ was conducted. The challenge aimed to explore automatic or interactive instance segmentation methods for an efficient delineation of the different aircraft components, such as screws, rivets, metal sheets or pressure tubes. We report the organization and outcome of this challenge and describe the capabilities and limitations of the submitted segmentation methods.

Unsupervised cervical cell instance segmentation method integrating cellular characteristics.

Cell instance segmentation is a key technology for cervical cancer auxiliary diagnosis systems. However, pixel-level annotation is time-consuming and labor-intensive, making it difficult to obtain a large amount of annotated data. This results in the model not being fully trained. In response to these problems, this paper proposes an unsupervised cervical cell instance segmentation method that integrates cell characteristics. Cervical cells have a clear corresponding structure between the nucleus and cytoplasm. This method fully takes this feature into account by building a dual-flow framework to locate the nucleus and cytoplasm and generate high-quality pseudo-labels. In the nucleus segmentation stage, the position and range of the nucleus are determined using the standard cell-restricted nucleus segmentation method. In the cytoplasm segmentation stage, a multi-angle collaborative segmentation method is used to achieve the positioning of the cytoplasm. First, taking advantage of the self-similarity characteristics of pixel blocks in cells, a cytoplasmic segmentation method based on self-similarity map iteration is proposed. The pixel blocks are mapped from the perspective of local details, and the iterative segmentation is repeated. Secondly, using low-level features such as cell color and shape, a self-supervised heatmap-aware cytoplasm segmentation method is proposed to obtain the activation map of the cytoplasm from the perspective of global attention. The two methods are fused to determine cytoplasmic regions, and combined with nuclear locations, high-quality pseudo-labels are generated. These pseudo-labels are used to train the model cyclically, and the loss strategy is used to encourage the model to discover new object masks, thereby obtaining a segmentation model with better performance. Experimental results show that this method achieves good results in cytoplasm segmentation. On the three datasets of ISBI, MS_CellSeg, and Cx22, 54.32%, 44.64%, and 66.52% AJI were obtained, respectively, which is better than other typical unsupervised methods selected in this article.

TopoPIS: Topology-constrained pipe instance segmentation via adaptive curvature convolution

Precise and fast pipe instance segmentation is a critical component in industrial automatic assembly, facilitating accurate object detection and pose estimation, optimizing and supervising the assembly process. However, this problem is challenging due to topological errors on fine-scale structures caused by the pipes being complex and slender. To address these challenges, we propose a topology-constrained pipe instance segmentation network (TopoPIS) for complex stacking scene to achieve accurate segmentation with topological correctness. To better extract the features of complex and variable morphological pipes, adaptive curvature convolution is introduced to dynamically adapt to the slender pipe structure and capture critical features. To handle topological errors like broken connections, we propose a novel topological constraint loss function based on persistent homology, which greatly improves the topological continuity of the segmentation. Experimental results on real-world and unseen datasets demonstrate that our TopoPIS outperforms other methods regrading segmentation accuracy and topological continuity.

LACOSTE: Exploiting stereo and temporal contexts for surgical instrument segmentation

Surgical instrument segmentation is instrumental to minimally invasive surgeries and related applications. Most previous methods formulate this task as single-frame-based instance segmentation while ignoring the natural temporal and stereo attributes of a surgical video. As a result, these methods are less robust against the appearance variation through temporal motion and view change. In this work, we propose a novel LACOSTE model that exploits Location-Agnostic COntexts in Stereo and TEmporal images for improved surgical instrument segmentation. Leveraging a query-based segmentation model as core, we design three performance-enhancing modules. Firstly, we design a disparity-guided feature propagation module to enhance depth-aware features explicitly. To generalize well for even only a monocular video, we apply a pseudo stereo scheme to generate complementary right images. Secondly, we propose a stereo-temporal set classifier, which aggregates stereo-temporal contexts in a universal way for making a consolidated prediction and mitigates transient failures. Finally, we propose a location-agnostic classifier to decouple the location bias from mask prediction and enhance the feature semantics. We extensively validate our approach on three public surgical video datasets, including two benchmarks from EndoVis Challenges and one real radical prostatectomy surgery dataset GraSP. Experimental results demonstrate the promising performances of our method, which consistently achieves comparable or favorable results with previous state-of-the-art approaches.

Fast Building Instance Proxy Reconstruction for Large Urban Scenes.

Digitalization of large-scale urban scenes (in particular buildings) has been a long-standing open problem, which attributes to the challenges in data acquisition, such as incomplete scene coverage, lack of semantics, low efficiency, and low reliability in path planning. In this paper, we address these challenges in urban building reconstruction from aerial images, and we propose an effective workflow and a few novel algorithms for efficient 3D building instance proxy reconstruction for large urban scenes. Specifically, we propose a novel learning-based approach to instance segmentation of urban buildings from aerial images followed by a voting-based algorithm to fuse the multi-view instance information to a sparse point cloud (reconstructed using a standard Structure from Motion pipeline). Our method enables effective instance segmentation of the building instances from the point cloud. We also introduce a layer-based surface reconstruction method dedicated to the 3D reconstruction of building proxies from extremely sparse point clouds. Extensive experiments on both synthetic and real-world aerial images of large urban scenes have demonstrated the effectiveness of our approach. The generated scene proxy models can already provide a promising 3D surface representation of the buildings in large urban scenes, and when applied to aerial path planning, the instance-enhanced building proxy models can significantly improve data completeness and accuracy, yielding highly detailed 3D building models.

Semantics and instance interactive learning for labeling and segmentation of vertebrae in CT images

Automatically labeling and segmenting vertebrae in 3D CT images compose a complex multi-task problem. Current methods progressively conduct vertebra labeling and semantic segmentation, which typically include two separate models and may ignore feature interaction among different tasks. Although instance segmentation approaches with multi-channel prediction have been proposed to alleviate such issues, their utilization of semantic information remains insufficient. Additionally, another challenge for an accurate model is how to effectively distinguish similar adjacent vertebrae and model their sequential attribute. In this paper, we propose a Semantics and Instance Interactive Learning (SIIL) paradigm for synchronous labeling and segmentation of vertebrae in CT images. SIIL models semantic feature learning and instance feature learning, in which the former extracts spinal semantics and the latter distinguishes vertebral instances. Interactive learning involves semantic features to improve the separability of vertebral instances and instance features to help learn position and contour information, during which a Morphological Instance Localization Learning (MILL) module is introduced to align semantic and instance features and facilitate their interaction. Furthermore, an Ordinal Contrastive Prototype Learning (OCPL) module is devised to differentiate adjacent vertebrae with high similarity (via cross-image contrastive learning), and simultaneously model their sequential attribute (via a temporal unit). Extensive experiments on several datasets demonstrate that our method significantly outperforms other approaches in labeling and segmenting vertebrae. Our code is available at https://github.com/YuZhang-SMU/Vertebrae-Labeling-Segmentation

An iterative up-sampling convolutional neural network for glass curtain crack detection using unmanned aerial vehicles

It is inefficient, subjective, and potentially life-threatening for both inspectors and pedestrians to manually inspect glass curtain cracks of high-rise buildings. Due to the development of image instance segmentation techniques for deep learning and the operation of compact unmanned aerial vehicles (UAVs), automatic, accurate, and safe glass curtain crack inspection can be achieved. A novel approach proposes to segment glass curtain cracks efficiently and accurately, combining a convolutional neural network (CNN), the region of interest alignment (RoIAlign), and an iterative up-sampling method. The approach proposed achieves excellent performance with an accuracy=99.89%, precision=91.1%, recall=72.1%, F1-score=79.7%, and mIoU=78.7%. Comparative studies have been conducted to examine the performance of the proposed approach relative to Solo, Swin, Mask RCNN, and ConvNext V2. The results show that the proposed network outperforms the networks used for comparison and that it can more accurately detect and segment a variety of crack types in complex reflective environments of glass curtains.

Vision-Based Detection of Unsafe Worker Guardrail Climbing Based on Posture and Instance Segmentation Data Fusion

Vision-Based Detection of Unsafe Worker Guardrail Climbing Based on Posture and Instance Segmentation Data Fusion

Spatio-temporal Analysis of Agroforestry Systems in Hotan Using Multi-source Remote Sensing and Deep Learning

In the Hotan region of Xinjiang, where arable land is scarce, an agroforestry system integrating walnut trees with crops has been implemented to maximize land-use efficiency. While this dense planting enhances land use, it also limits the availability of light to the understory crops, potentially impacting their yield and quality. To address this issue and enhance the system's sustainability and productivity, precise delineation of the planting structure is critical. This study proposes a novel framework that leverages multi-source remote sensing data combined with advanced deep learning techniques to analyze the agroforestry planting structure. The methodological approach consists of three key phases. First, an instance segmentation model was employed to extract farmland parcels from high-resolution imagery, providing a basis for vegetation classification. Next, a time series model using irregular satellite image time series (irSITS) tracked the growth dynamics of the vegetation. Finally, the spatial planting structure of the walnut trees was quantified using the D-LinkNet model, integrated with a template filling algorithm.The results demonstrated a classification accuracy of 97.85% in extracting parcel-level planting structures, identifying 42,955 farmland parcels, including 21,153 intercropped parcels. The temporal and spatial characteristics of the agroforestry system were then analyzed, leading to a grading of canopy cover and walnut tree density within the intercropped areas. This comprehensive spatiotemporal planting structure offers a valuable foundation for informed local agricultural policy adjustments. In conclusion, this approach advances the understanding of complex agroforestry systems and provides a robust scientific basis for optimizing intercropping practices, contributing to sustainable agricultural development in arid regions.

On-the-go table grape ripeness estimation via proximal snapshot hyperspectral imaging

The monitoring of grapes for ripeness estimation is a practice that enables fruit harvesting at the optimal time. Hyperspectral Imaging (HSI) represents a non-destructive and high-throughput alternative to traditional laboratory analyses. Current literature approaches perform hyperspectral measurements using line scan sensors or low-resolution static snapshot cameras, which hinder a fast per-bunch ripeness characterization. We propose a framework for on-the-go collection and processing of proximal snapshot hyperspectral images to estimate single bunch ripeness parameters. Focusing on table grapes (Vitis vinifera L. cv. Red Globe), we collected images under natural illumination with a hyperspectral camera (500–900 nm) mounted on a moving vehicle in an experimental block sited in Piacenza, Italy. We investigated images collected in August and September 2021 representing two ripening stages. The composition of the imaged grape bunches was determined through laboratory chemical analyses to predict Total Soluble Solids (TSS) and anthocyanin concentration. The images were pre-processed via multimodal image registration to correct the unalignment of bands due to the vehicle motion, and the single bunches were automatically identified on false RGB images through a Mask Region-Convolutional Neural Network (Mask R-CNN) instance segmentation network. The mean spectra of the bunches were used as input features of a Partial Least Squares Regression (PLSR) model to predict the chemical parameters at single bunch and whole vine scales. The regression model of TSS had an R2 (10-fold nested cross-validation) of 0.75 and 0.85 on a per-bunch and per-vine basis, respectively. The regression model of anthocyanin had an R2 of 0.68 and 0.49 on a per-bunch and per-vine basis, respectively. The results suggest the potential of using snapshot hyperspectral images for high-throughput analysis of a per-bunch grape ripeness estimation. The method described in this study could give valuable information to improve grape ripening monitoring and management of harvest operations and even allow for precise and automated robotic harvesting.

Concealed Hazardous Object Instance Segmentation for Terahertz Security Inspection Images With Channel-Spatialized Transformer

Concealed Hazardous Object Instance Segmentation for Terahertz Security Inspection Images With Channel-Spatialized Transformer