Instance Segmentation Algorithm Research Articles

An Instance Segmentation Method for Insulator Defects Based on an Attention Mechanism and Feature Fusion Network

Among the existing insulator defect detection methods, the automatic detection of inspection robots based on the instance segmentation algorithm is relatively more efficient, but the problem of the limited accuracy of the segmentation algorithm is still a bottleneck for increasing inspection efficiency. Therefore, we propose a single-stage insulator instance defect segmentation method based on both an attention mechanism and improved feature fusion network. YOLACT is selected as the basic instance segmentation model. Firstly, to improve the segmentation speed, MobileNetV2 embedded with an scSE attention mechanism is introduced as the backbone network. Secondly, a new feature map that combines semantic and positional information is obtained by improving the FPN module and fusing the feature maps of each layer, during which, an attention mechanism is introduced to further improve the quality of the feature map. Thirdly, in view of the problems that affect the insulator segmentation, a Restrained-IoU (RIoU) bounding box loss function which covers the area deviation, center deviation, and shape deviation is designed for object detection. Finally, for the validity evaluation of the proposed method, experiments are performed on the insulator defect data set. It is shown in the results that the improved algorithm achieves a mask accuracy improvement of 5.82% and a detection speed of 37.4 FPS, which better complete the instance segmentation of insulator defect images.

Instance segmentation by blend U‐Net and VOLO network

AbstractInstance segmentation is still challengeable to correctly distinguish different instances on overlapping, dense and large number of target objects. To address this, the authors simplify the instance segmentation problem to an instance classification problem and propose a novel end‐to‐end trained instance segmentation algorithm CotuNet. Firstly, the algorithm combines convolutional neural networks (CNN), Outlooker and Transformer to design a new hybrid Encoder (COT) to further feature extraction. It consists of extracting low‐level features of the image using CNN, which is passed through the Outlooker to extract more refined local data representations. Then global contextual information is generated by aggregating the data representations in local space using Transformer. Finally, the combination of cascaded upsampling and skip connection modules is used as Decoders (C‐UP) to enable the blend of multiple different scales of high‐resolution information to generate accurate masks. By validating on the CVPPP 2017 dataset and comparing with previous state‐of‐the‐art methods, CotuNet shows superior competitiveness and segmentation performance.

Design of Logistics Sorting Algorithm Based on Deep Learning and Sampling Evaluation

The advancement of automation and Internet of Things technology has bolstered the automation process in the logistics sector. To address the challenge of localizing and generating grasping positions for intelligent robots in logistics sorting, this study developed an algorithm for item localization. The algorithm relies on enhanced YOLOv3 target detection and instance segmentation technologies to design a position generation algorithm for the robotic arm, which was further refined using sampling evaluation. The experimental results showed that the research-improved target detection model performed better on different datasets in terms of F1 value, accuracy and Area under the Curve (AUC) metrics, with the highest values of 95.77%, 94.05%, and 91.30%, respectively, which was effective in localizing document-like parcels. Meanwhile, the instance segmentation algorithm with fused features took significantly lower values than other target detection algorithms in terms of average absolute value error and root mean square error. The accuracy rate and all-class average precision value were higher than other target detection models, and the fluctuation of the value taken was smaller, which was suitable for logistics parcel localization. The position generation model, based on a sampling evaluation, yielded significantly different values compared to other algorithms. The relative position error and absolute trajectory error indexes were all below 0.4. The combined indexes of grasping accuracy and error indicate the superior performance of the research-designed algorithms. They can effectively enhance the sorting effects of real logistics scenarios. This research contributes to the improvement of the automated sorting system through the use of visual robotic arm technology. Additionally, it encourages the development of logistics automation and the establishment of intelligent logistics factories.

Instance segmentation algorithm for sorting dismantling components of end-of-life vehicles

In the recycling and dismantling procedures of End-of-Life Vehicles, the diverse thicknesses of disassembled parts necessitate specialized recycling processes, underscoring the imperative for efficient classification and recycling methods. Presently, the classification of disassembled components from scrapped vehicles relies heavily on manual visual inspection and caliper measurements, resulting in inefficiency and low accuracy. To overcome the challenges associated with thickness classification for disassembled components of End-of-Life Vehicles, propose a paradigm shift by introducing an intelligent grading strategy. This approach aims to significantly improve efficiency and accuracy in the classification process, offering a more effective solution for the recycling of dismantled parts. In response to the inefficiencies, low accuracy, and safety concerns associated with manual sorting, propose a deep learning model, incorporating an attention mechanism, for accomplishing instance segmentation tasks of End-of-Life Vehicles components. The model introduces a global attention mechanism into the backbone network for feature extraction, which is further integrated into the instance segmentation branch, resulting in a significant enhancement of instance segmentation performance. The model is trained and optimized using a dataset collected from an End-of-Life Vehicles dismantling facility in China. Experimental results show that when the intersection over union is 0.5, the mean Average Precision of the model in target detection and instance segmentation reaches 93.2% and 92.1% respectively., and 77.2% and 72.4% in the intersection over union is 0.5–0.95. In comparison to the unmodified baseline model, the model exhibits improvements of 1.7%, 2%, 5.8%, and 3.1% across evaluation metrics. When contrasted with traditional manual sorting methods, the proposed model exhibits pronounced advantages in terms of accuracy and fairness. In conclusion, the model effectively addresses the challenge of intelligent sorting of dismantled components from End-of-Life Vehicles during the dismantling process.

A Novel Boundary-Guided Global Feature Fusion Module for Instance Segmentation

The task of instance segmentation is widely acknowledged as being one of the most formidable challenges in the field of computer vision. Current methods have low utilization of boundary information, especially in dense scenes with occlusion and complex shapes of object instances, the boundary information may become ineffective. This results in coarse object boundary masks that fail to cover the entire object. To address this challenge, we are introducing a novel method called boundary-guided global feature fusion (BGF) which is based on the Mask R-CNN network. We designed a boundary branch that includes a Boundary Feature Extractor (BFE) module to extract object boundary features at different stages. Additionally, we constructed a binary image dataset containing instance boundaries for training the boundary branch. We also trained the boundary branch separately using a dedicated dataset before training the entire network. We then input the Mask R-CNN features and boundary features into a feature fusion module where the boundary features provide shape information needed for detection and segmentation. Finally, we use a global attention module (GAM) to further fuse features. Through extensive experiments, we demonstrate that our approach outperforms state-of-the-art instance segmentation algorithms, producing finer and more complete instance masks while also improving model capability.

Improved senescent cell segmentation on bright‐field microscopy images exploiting representation level contrastive learning

AbstractMesenchymal stem cells (MSCs) are stromal cells which have multi‐lineage differentiation and self‐renewal potentials. Accurate estimation of total number of senescent cells in MSCs is crucial for clinical applications. Traditional manual cell counting using an optical bright‐field microscope is time‐consuming and needs an expert operator. In this study, the senescence cells were segmented and counted automatically by deep learning algorithms. However, well‐performing deep learning algorithms require large numbers of labeled datasets. The manual labeling is time consuming and needs an expert. This makes deep learning‐based automated counting process impractically expensive. To address this challenge, self‐supervised learning based approach was implemented. The approach incorporates representation level contrastive learning component into the instance segmentation algorithm for efficient senescent cell segmentation with limited labeled data. Test results showed that the proposed model improves mean average precision and mean average recall of downstream segmentation task by 8.3% and 3.4% compared to original segmentation model.

INCAD: 2D VECTOR DRAWINGS CREATION USING INSTANCE SEGMENTATION

Abstract. Orthoimages are a common product used as a base in CAD software for vectorization purposes. In fact, vectorization of orthoimages constitutes a tedious and labour-intensive process which should be supervised by experts e.g. architects, chemical engineers etc. On the one hand, deep learning algorithms are used extensively nowadays achieving high quality results. On the other hand, deep learning algorithms require a huge amount of manually annotated data to be trained on, which is a very difficult process especially at pixel level applications like semantic segmentation and instance segmentation. However, the transformation of 2D CAD drawings into a suitable deep learning dataset (CAD2DLD) is underexplored ignoring a large source of data, created by experts. In this effort, the InCAD algorithm is proposed, which aims to automatically create 2D vector drawings using the YOLOv8 instance segmentation algorithm which was trained on CAD2DLD data. Additionally, a methodology for transforming 2D CAD drawings into a suitable deep learning dataset for instance segmentation, is presented. Finally, the proposed workflow is evaluated on the creation of 2D vector drawings of stones of a fortification wall achieving promising results (78.34 mIoU).

Intrarow Uncut Weed Detection Using You-Only-Look-Once Instance Segmentation for Orchard Plantations.

Mechanical weed management is a drudging task that requires manpower and has risks when conducted within rows of orchards. However, intrarow weeding must still be conducted by manual labor due to the restricted movements of riding mowers within the rows of orchards due to their confined structures with nets and poles. However, autonomous robotic weeders still face challenges identifying uncut weeds due to the obstruction of Global Navigation Satellite System (GNSS) signals caused by poles and tree canopies. A properly designed intelligent vision system would have the potential to achieve the desired outcome by utilizing an autonomous weeder to perform operations in uncut sections. Therefore, the objective of this study is to develop a vision module using a custom-trained dataset on YOLO instance segmentation algorithms to support autonomous robotic weeders in recognizing uncut weeds and obstacles (i.e., fruit tree trunks, fixed poles) within rows. The training dataset was acquired from a pear orchard located at the Tsukuba Plant Innovation Research Center (T-PIRC) at the University of Tsukuba, Japan. In total, 5000 images were preprocessed and labeled for training and testing using YOLO models. Four versions of edge-device-dedicated YOLO instance segmentation were utilized in this research-YOLOv5n-seg, YOLOv5s-seg, YOLOv8n-seg, and YOLOv8s-seg-for real-time application with an autonomous weeder. A comparison study was conducted to evaluate all YOLO models in terms of detection accuracy, model complexity, and inference speed. The smaller YOLOv5-based and YOLOv8-based models were found to be more efficient than the larger models, and YOLOv8n-seg was selected as the vision module for the autonomous weeder. In the evaluation process, YOLOv8n-seg had better segmentation accuracy than YOLOv5n-seg, while the latter had the fastest inference time. The performance of YOLOv8n-seg was also acceptable when it was deployed on a resource-constrained device that is appropriate for robotic weeders. The results indicated that the proposed deep learning-based detection accuracy and inference speed can be used for object recognition via edge devices for robotic operation during intrarow weeding operations in orchards.

Instance segmentation algorithm of electronic components based on improved YOLOv5

Instance segmentation algorithm of electronic components based on improved YOLOv5

Semi-supervised instance segmentation algorithm based on transfer learning

ABSTRACT Semi-supervised instance segmentation algorithms are mainly divided into algorithms based on pseudo-label generation and algorithms based on transfer learning. The algorithms based on pseudo-label generation need to design a specific pseudo-label generation process, but the process is not scalable for different types of source tasks. The algorithms based on transfer learning that started late have relatively high scalability, but the algorithm research ideas are relatively simple. To expand the research on semi-supervised instance segmentation based on transfer learning, this paper proposes a feature transfer-based semi-supervised instance segmentation algorithm Feature Transfer Mask R-CNN (FT-Mask). The FT-Mask algorithm is more scalable than algorithms based on pseudo-label generation and can be used to transfer knowledge from different types of source tasks. Compared with other semi-supervised instance segmentation algorithms based on transfer learning, FT-Mask uses the feature transfer method to achieve semi-supervised instance segmentation for the first time. The experimental results show that the FT-Mask model improves the semi-supervised instance segmentation accuracy of the Mask R-CNN benchmark model through the semi-supervised learning process, and can achieve effective transfer learning.

Fruit tree canopy segmentation from UAV orthophoto maps based on a lightweight improved U-Net

Segmenting fruit tree canopies from drone remote sensing images is a prerequisite for achieving accurate agricultural monitoring and precision aerial spraying at the individual tree (instance) level. However, current instance segmentation algorithms for canopy segmentation require repeated sampling of digital orthophoto maps (DOMs), resulting in high computational complexity and poor performance in densely planted orchards. Therefore, we propose a canopy labeling method suitable for U-Net and a lightweight segmentation network, where a lightweight backbone network, an attention mechanism, and a focal loss function are introduced to improve the U-Net decoder, greatly reducing the computational complexity required for large-scale canopy segmentation. The feasibility and effectiveness of the proposed method were validated using datasets from two seasons of two different lychee orchards. The average recognition rate of the improved U-Net was 90.98 %, 28.59 % higher than the basic model U-Net, and the floating-point operations (FLOPs) amounted to 50.86 GFLOPS, 27.67 % lower than the basic model U-Net. Under the same experimental conditions, the proposed model outperformed mainstream semantic segmentation models such as Deeplabv3 + and ResNet50-U-Net, and it was more efficient than previous instance segmentation methods based on YOLACT, as it does not require repeated sampling of the same region. For the same area, the number of sampled small images decreased from 194 to 78, resulting in a 148 % overall efficiency improvement while achieving better segmentation results. Thus, the proposed model can be used to extract and locate the crown of a lychee orchard, aiding in accurate management of a lychee orchard and in a differentiated agricultural analysis and decision-making for individual lychee trees.

C-LFNet: Central-Local Feature 3D point cloud instance segmentation Network for robot bin-picking

We propose a novel instance segmentation algorithm called Central-Local Feature Network (C-LFNet) to address the low performance of point cloud segmentation in bin-picking. First, to address the issue of batch processing data, we propose a downsampling algorithm that can sample the number of target points while preserving the density distribution of points in space as much as possible both before and after sampling. Second, we design a dynamic graph convolution backbone to better extract the local geometric invariant features, referred to as Central-Local Feature Module (C-LFM). Finally, the results of each point estimate are clustered in order to enable instance segmentation of the point cloud. The proposed network is evaluated using the Fraunhofer IPA Bin-Picking dataset and compared to existing methods. Results demonstrate that the proposed network is effective and robust in bin-picking scenes to achieve the same category of point cloud instance segmentation, resulting in improved performance.

Improvement of Road Instance Segmentation Algorithm Based on the Modified Mask R-CNN

Although the Mask region-based convolutional neural network (R-CNN) model possessed a dominant position for complex and variable road scene segmentation, some problems still existed, including insufficient feature expressive ability and low segmentation accuracy. To address these problems, a novel road scene segmentation algorithm based on the modified Mask R-CNN was proposed. The multi-scale backbone network, Res2Net, was utilized to replace the ResNet network, and aimed to improve the feature extraction capability. The soft non-maximum suppression algorithm with attenuation function (soft-NMS) was adopted to improve detection efficiency in the case of a higher overlap rate. The comparison analyses of partition accuracy for various models were performed on the adopted Cityscapes dataset. The results demonstrated that the modified Mask R-CNN effectively increased the segmentation accuracy, especially for small and highly overlapping objects. The adopted Res2Net and soft-NMS can effectively enhance the feature extraction and improve segmentation performance. The average accuracy of the modified Mask R-CNN model reached up to 0.321, and was 0.054 higher than Mask R-CNN. This work provides important guidance to design a more efficient road scene instance segmentation algorithm for further promoting the actual application in automatic driving systems.

Metal Surface Defect Detection Method Based on Improved Cascade R-CNN

Abstract In contemporary industrial systems, ensuring the quality of object surfaces has become an essential and inescapable aspect of factory inspections. Cascade Regional Convolutional Neural Network (Cascade R-CNN), an object detection and instance segmentation algorithm based on deep learning, has been widely applied in numerous industrial applications. Nonetheless, there is still space for improving the detection of defects on metal surfaces. This paper proposes an enhanced metal defect detection method based on Cascade R-CNN. Specifically, the improved backbone network is employed to acquire the features of images, which enables more precise localization. Additionally, up and down sampling is combined to extract multi-scale defect feature maps, and contrast histogram equalization enhancement is utilized to tackle the issue of unclear contrast in the data. Experimental results demonstrate that the proposed approach achieves a mean Average Precision (mAP) of 0.754 on the NEU-DET dataset, and outperforms the Cascade R-CNN model by 9.2%.

Synchronous mixed reality (SMR): A personalized virtual‐real fusion framework with high immersion and effective interaction

AbstractVirtual‐real fusion (VRF) technology plays a crucial role in the meta‐universe by bridging the gap between virtual environments (VEs) and physical environments (PEs). However, the current VRF system has the problem of single function and fixed integration, limiting the popularization and application of VRF technology. Therefore, this paper proposes a framework named synchronous mixed reality (SMR) to personalize the VRF system and maintain the balance between interaction efficiency and environmental immersion. We combine an instance segmentation algorithm with an interaction prediction algorithm to achieve this balance. To evaluate the effectiveness of the SMR framework, we design three scenarios based on the interaction properties of physical objects and measure environmental immersion, systematic interaction efficiency, and user experience. Our results demonstrate that the SMR framework meets diverse needs in various PEs while balancing immersion and interaction efficiency. Additionally, our framework significantly reduces collisions and negative emotions experienced by users in VEs. We anticipate that this framework will serve as a guide for constructing virtual‐real fusion systems in the future.

A novel approach for pavement distress detection and quantification using RGB-D camera and deep learning algorithm

This study presents an innovative and cost-effective approach for pavement distress detection and quantification using an RGB-D camera in conjunction with an instance segmentation algorithm. The proposed method employs the instance segmentation algorithm to acquire 2D coordinate information pertaining to the affected pixels and integrates the internal reference matrix and depth data from the depth camera to transform the 2D images of the distressed areas into 3D point cloud data, consequently enabling distress quantification. To assess the dependability of the proposed approach, high-density polystyrene foam was used to simulate potholes and cracks of the pavement. Data collection was conducted under varying experimental conditions to identify the optimal data collection scheme. Using the approach, potholes and cracks of varying severity were collected from both asphalt and cement concrete pavements. The outcomes of the study demonstrate that the proposed methodology is capable of accurately detecting and quantifying potholes on the pavement. Furthermore, the errors associated with both the calculated area and volume exhibit a gradual decrease with an increase in the severity of pavement distress. However, for cracks, the method yields a larger error in results, primarily due to the instance segmentation algorithm's imprecise segmentation of crack edge pixels, despite superior restoration of the planar contour of cracks.

VISN: virus instance segmentation network for TEM images using deep attention transformer.

The identification of viruses from negative staining transmission electron microscopy (TEM) images has mainly depended on experienced experts. Recent advances in artificial intelligence have enabled virus recognition using deep learning techniques. However, most of the existing methods only perform virus classification or semantic segmentation, and few studies have addressed the challenge of virus instance segmentation in TEM images. In this paper, we focus on the instance segmentation of severe acute respiratory syndrome coronavirus type 2 (SARS-CoV-2) and other respiratory viruses and provide experts with more effective information about viruses. We propose an effective virus instance segmentation network based on the You Only Look At CoefficienTs backbone, which integrates the Swin Transformer, dense connections and the coordinate-spatial attention mechanism, to identify SARS-CoV-2, H1N1 influenza virus, respiratory syncytial virus, Herpes simplex virus-1, Human adenovirus type 5 and Vaccinia virus. We also provide a public TEM virus dataset and conduct extensive comparative experiments. Our method achieves a mean average precision score of 83.8 and F1 score of 0.920, outperforming other state-of-the-art instance segmentation algorithms. The proposed automated method provides virologists with an effective approach for recognizing and identifying SARS-CoV-2 and assisting in the diagnosis of viruses. Our dataset and code are accessible at https://github.com/xiaochiHNU/Virus-Instance-Segmentation-Transformer-Network.

A lightweight network based on dual-stream feature fusion and dual-domain attention for white blood cells segmentation.

Accurate white blood cells segmentation from cytopathological images is crucial for evaluating leukemia. However, segmentation is difficult in clinical practice. Given the very large numbers of cytopathological images to be processed, diagnosis becomes cumbersome and time consuming, and diagnostic accuracy is also closely related to experts' experience, fatigue and mood and so on. Besides, fully automatic white blood cells segmentation is challenging for several reasons. There exists cell deformation, blurred cell boundaries, and cell color differences, cells overlapping or adhesion. The proposed method improves the feature representation capability of the network while reducing parameters and computational redundancy by utilizing the feature reuse of Ghost module to reconstruct a lightweight backbone network. Additionally, a dual-stream feature fusion network (DFFN) based on the feature pyramid network is designed to enhance detailed information acquisition. Furthermore, a dual-domain attention module (DDAM) is developed to extract global features from both frequency and spatial domains simultaneously, resulting in better cell segmentation performance. Experimental results on ALL-IDB and BCCD datasets demonstrate that our method outperforms existing instance segmentation networks such as Mask R-CNN, PointRend, MS R-CNN, SOLOv2, and YOLACT with an average precision (AP) of 87.41%, while significantly reducing parameters and computational cost. Our method is significantly better than the current state-of-the-art single-stage methods in terms of both the number of parameters and FLOPs, and our method has the best performance among all compared methods. However, the performance of our method is still lower than the two-stage instance segmentation algorithms. in future work, how to design a more lightweight network model while ensuring a good accuracy will become an important problem.

Three-dimensional quantification of apple phenotypic traits based on deep learning instance segmentation

Estimating three-dimensional (3D) fruit-level phenotypic traits of apple trees can potentially improve apple orchard breeders' management strategy. However, the phenotypic traits of apple fruits (including the quantity, 3D distribution, and volume of apples) constitute important parameters that influence yield but are difficult to quantify manually. Therefore, it is necessary to effectively and efficiently quantify apple phenotyping to monitor apple yield and support a better management system. This study developed a novel method for extracting individual 3D apple traits and 3D mapping for three apple training systems. The 3D point cloud of apples was reconstructed from multi-view images collected via a multi-camera system-based unmanned-aerial vehicle. Individual apples in the 3D point cloud were extracted via a 3D instance segmentation algorithm that included generalized sparse convolutional neural networks, a weighted discriminative loss function, and a varying density-based 3D clustering method. The developed apple trait extraction algorithm can help compute the position and volume of an individual apple. The R2 values with the average weighted-mean-absolute-percentage error (WMAPE) of apple counting and apple volume estimation were 0.84–0.99 (VMAPE = 3.41–12.75%) and 0.84–0.90 (WMAPE = 4.74–7.03%), respectively. The 3D spatial and volumetric distribution of apples were obtained and analyzed. This study developed an effective method combining 3D photography and 3D instance segmentation that can accurately estimate individual apple phenotypic traits from different types of apple training systems in orchards and can also be utilized for the analysis of other fruit traits.

A boundary optimization model of instance segmentation combined with wavelet transform on Buildings

Data driven deep learning methods have become the mainstream method of building extraction from remote sensing images. In this paper, deep learning algorithm is used to classify and extract buildings from remote sensing images of rural areas around the Great Wall in the suburbs of Beijing captured by unmanned aerial vehicles. Aiming at the shortcomings of the current mainstream instance segmentation algorithm Mask R-CNN in feature fusion and poor prediction of instance mask boundaries, this paper proposes a boundary optimization algorithm for building instance segmentation based on discrete wavelet transform. Firstly, the discrete wavelet transform is introduced into the segmentation task branch of Mask R-CNN algorithm to extract the low-frequency and high-frequency information of the real mask, in which the high-frequency information includes the boundary information. Secondly, the pixel by pixel prediction of the mask turns into the learning of the low-frequency and high-frequency information of the real mask. The learning of the high-frequency information helps the segmentation network to learn the boundary features better. Finally, using the reversibility of discrete wavelet transform, the low-frequency and high-frequency information of the learned mask is inversely transformed to reconstruct the final mask. The improved algorithm is evaluated on the dataset COCO, and applied to the automatic extraction of buildings. The DWT Mask R-CNN algorithm model achieved 70.2% segmentation accuracy and 71.4% detection accuracy, which were improved by 1% and 0.7% respectively compared with the Mask R-CNN and Cascade Mask R-CNN models. The experimental results show that the instance segmentation edge optimization algorithm combined with wavelet transform has achieved better results on the segmentation boundary, improved the poor effect of mask edge detection and achieved higher detection accuracy, and can accurately extract village buildings.