Instance Segmentation Method Research Articles

Vision-based robotic system for on-site construction and demolition waste sorting and recycling

Construction and demolition waste (CDW) causes severe pollution on a global scale, as we can see from western countries' as well as Chinese statistics. CDW accounts over 25% of Hong Kong’s solid waste and about 40% for Australia and EU. The classification and recycling of CDW on construction sites make a vital contribution to the successful recycling of CDW. Currently, recycling of onsite construction waste is very time consuming and labor-intensive, which is probably one of the most critical reasons for inefficient CDW management. Previous experiences found that computer vision performed well in CDW recycling, but in special cases, the accuracy of the patrol and the picking method may cause failure. To reduce the occurrence of failure, this research adopts the Simultaneous Localization and Mapping (SLAM) technology and the instance segmentation method to enable the robot to cope with complex site situations. A database of CDW was developed and used to train a computer vision model for recognizing residual pipes and cables. The feasibility of developed methods and algorithms was evaluated through both lab and site experiments. The system can be improved and applied to other objects, which is useful for researchers and practitioners engaged in similar tasks.

Single-stage Instance Segmentation

Albeit the highest accuracy of object detection is generally acquired by multi-stage detectors, like R-CNN and its extension approaches, the single-stage object detectors also achieve remarkable performance with faster execution and higher scalability. Inspired by this, we propose a single-stage framework to tackle the instance segmentation task. Building on a single-stage object detection network in hand, our model outputs the detected bounding box of each instance, the semantic segmentation result, and the pixel affinity simultaneously. After that, we generate the final instance masks via a fast post-processing method with the help of the three outputs above. As far as we know, it is the first attempt to segment instances in a single-stage pipeline on challenging datasets. Extensive experiments demonstrate the efficiency of our post-processing method, and the proposed framework obtains competitive results as a single-stage instance segmentation method. We achieve 32.5 box AP and 26.0 mask AP on the COCO validation set with 500 pixels input scale and 22.9 mask AP on the Cityscapes test set.

Improved deep learning framework for fish segmentation in underwater videos

Deep learning networks have become increasingly popular in recent years due to promising breakthroughs achieved in several areas. The importance of deep learning lies in the localisation and classification of an object based on frames. This study focuses on fish recognition methods in underwater videos and addresses the underlying challenges of these methods. It is important to develop effective methods to recognise fish and their movements using underwater videos. From a practical and scientific perspective, this is extremely useful to automatically recognise fish through their movement and to monitor and collect biomass in marine bodies. More importantly, it allows researchers to collect and analyse information related to the health and well-being of the Marine ecosystem. As most of the current methods work on static images, the issue arises when these methods are applied to images from videos. The existing multiple fish detection methods for underwater videos have a low detection rate due to the inherent underwater conditions such as the presence of coral reefs and other challenges which include the different sizes, shapes, colour, and speed of fish as well as marine behaviours such as the overlapping of fish. Therefore, the use of improved methods based on the latest deep learning algorithms has been proposed for multiple fish detection. This paper provides a novel framework for fish instance segmentation in underwater videos. The proposed model for improved recognition methods is composed of four main stages: 1) pre-processing method to reduce external factors in the videos for better detection and recognition of fish in underwater videos, 2) use of deep learning approach for enhanced detection of fish using RESENT, 3) enhanced detection of multiple fish based on the Region Proposal Network (RPN) architecture, and 4) use of a dynamic instance segmentation method. The results of this study indicate that the proposed framework has a better performance capability than other state-of-the-art models for multi-fish instance segmentation.

Surgical tool segmentation and localization using spatio-temporal deep network.

Laparoscopic cholecystectomy surgery is a minimally invasive surgery to remove the gallbladder, where surgical instruments are inserted through small incisions in the abdomen with the help of a laparoscope. Identification of tool presence and precise segmentation of tools from the video is very important in understanding the quality of the surgery and training budding surgeons. Precise segmentation of tools is required to track the tools during real-time surgeries. In this paper, a new pixel-wise instance segmentation algorithm is proposed, which segments and localizes the surgical tool using spatio-temporal deep network. The performance of the proposed has been compared with the state-of-the-art image-based instance segmentation method using the Cholec80 dataset. It is also compared with methods in the literature using frame-level presence detection and spatial detection with good results.

Deep learning-based mobile augmented reality for task assistance using 3D spatial mapping and snapshot-based RGB-D data

This paper proposes a new deep learning-based mobile AR for intelligent task assistance by conducting 3D spatial mapping without pre-registration using AR markers, which can match virtual AR objects to their corresponding physical objects automatically and accurately using single snapshot-based RGB-D data. Firstly, the proposed approach applies a deep learning-based instance segmentation method to the snapshot-based RGB-D data to detect real object instances and to segment their surrounding regions in 3D point cloud data. Then, an iterative closest point (ICP) algorithm is used to perform a 3D spatial mapping between the segmented point cloud of the real object and its corresponding virtual model. Therefore, the virtual information can be seamlessly and automatically synchronized with its corresponding real object. To prove the effectiveness of the proposed method, we performed comparative experiments quantitatively and qualitatively, which evaluated the accuracy, basic task performance, and usability. Experimental results verify that the proposed deep learning-based 3D spatial mapping approach is more accurate and more suitable for mobile AR-based visualization and interaction than previous studies. We have also implemented several applications in actual working situations, which verifies the applicability and extensibility of the proposed approach.

Automated Detection and Segmentation of Early Gastric Cancer from Endoscopic Images Using Mask R-CNN

Gastrointestinal endoscopy is widely conducted for the early detection of gastric cancer. However, it is often difficult to detect early gastric cancer lesions and accurately evaluate the invasive regions. Our study aimed to develop a detection and segmentation method for early gastric cancer regions from gastrointestinal endoscopic images. In this method, we first collected 1208 healthy and 533 cancer images. The gastric cancer region was detected and segmented from endoscopic images using Mask R-CNN, an instance segmentation method. An endoscopic image was provided to the Mask R-CNN, and a bounding box and a label image of the gastric cancer region were obtained. As a performance evaluation via five-fold cross-validation, sensitivity and false positives (FPs) per image were 96.0% and 0.10 FP/image, respectively. In the evaluation of segmentation of the gastric cancer region, the average Dice index was 71%. These results indicate that our proposed scheme may be useful for the detection of gastric cancer and evaluation of the invasive region in gastrointestinal endoscopy.

Instance Segmentation Method Based on Improved Mask R-CNN for the Stacked Electronic Components

Object-detection methods based on deep learning play an important role in achieving machine automation. In order to achieve fast and accurate autonomous detection of stacked electronic components, an instance segmentation method based on an improved Mask R-CNN algorithm was proposed. By optimizing the feature extraction network, the performance of Mask R-CNN was improved. A dataset of electronic components containing 1200 images (992 × 744 pixels) was developed, and four types of components were included. Experiments on the dataset showed the model was superior in speed while being more lightweight and more accurate. The speed of our model showed promising results, with twice that of Mask R-CNN. In addition, our model was 0.35 times the size of Mask R-CNN, and the average precision (AP) of our model was improved by about two points compared to Mask R-CNN.

Research on Urban Building Extraction Method Based on Deep Learning Convolutional Neural Network

Buildings’ location information is an important basic data for the construction of refined smart city. Using remote sensing images to obtain building information is both high-quality and efficiency. However, urban village building is a very special category in remote sensing image of urban area. The distribution patterns of buildings inside the urban village is quite unique, such as high building density, narrow streets and lanes, concentrated illegal buildings, which brings various high-risk security risks. Therefore, how to use remote sensing technique to detect the structural characteristics and explore the spatial distribution patterns of urban village buildings through massive urban image data is of great significance to both planning and governing urban villages. Traditional remote sensing image building extraction methods have the problems of large workload, low efficiency and slow update, and the buildings in the urban village has complex building types and serious interference of deposits. To tackle the above issues, this paper adopts the deep learning convolutional neural network algorithm which is the advanced method in the field of computer vision, and proposes a building extraction method based on high-resolution remote sensing images. In this paper, Mask R-CNN, an instance segmentation method, is used to extract urban village buildings. After trained on 678 samples, this method reached 66% mAP on test dataset. In addition, spatial distribution patterns of the research area are analysed based on the detection result. The spatial analyzation results show that buildings in the research area are with small average building area and large building density and do not compliance with regulations of People’s Republic of China on the Planning and Design Standards for Urban Residential Areas.

SSAP: Single-Shot Instance Segmentation With Affinity Pyramid

Proposal-free instance segmentation methods mainly generate instance-agnostic semantic segmentation labels and instance-aware features to group pixels into different object instances. However, previous methods mostly employ separate modules for these two sub-tasks and require multiple passes for inference. In addition to the lack of efficiency, previous methods also failed to perform as well as proposal-based approaches. To this end, this work proposes a single-shot proposal-free instance segmentation method that requires only one single pass for prediction. Our method is based on learning an affinity pyramid, which computes the probability that two pixels belong to the same instance in a hierarchical manner. Moreover, incorporating with the learned affinity pyramid, a novel cascaded graph partition (CGP) module is presented to fuse the two predictions and segment instances efficiently. As an additional contribution, we conduct an experiment to demonstrate the benefits of proposal-free methods in capturing detailed structures from finely annotated training examples. Our approach is evaluated on the Cityscapes and COCO datasets and achieves state-of-the-art performance.

Object-Guided Instance Segmentation for Biological Images

Instance segmentation of biological images is essential for studying object behaviors and properties. The challenges, such as clustering, occlusion, and adhesion problems of the objects, make instance segmentation a non-trivial task. Current box-free instance segmentation methods typically rely on local pixel-level information. Due to a lack of global object view, these methods are prone to over- or under-segmentation. On the contrary, the box-based instance segmentation methods incorporate object detection into the segmentation, performing better in identifying the individual instances. In this paper, we propose a new box-based instance segmentation method. Mainly, we locate the object bounding boxes from their center points. The object features are subsequently reused in the segmentation branch as a guide to separate the clustered instances within an RoI patch. Along with the instance normalization, the model is able to recover the target object distribution and suppress the distribution of neighboring attached objects. Consequently, the proposed model performs excellently in segmenting the clustered objects while retaining the target object details. The proposed method achieves state-of-the-art performances on three biological datasets: cell nuclei, plant phenotyping dataset, and neural cells.

A Virtual-Real Interaction Approach to Object Instance Segmentation in Traffic Scenes

Object instance segmentation in traffic scenes is an important research topic. For training instance segmentation models, synthetic data can potentially complement real data, alleviating manual effort on annotating real images. However, the data distribution discrepancy between synthetic data and real data hampers the wide applications of synthetic data. In light of that, we propose a virtual-real interaction method for object instance segmentation. This method works over synthetic images with accurate annotations and real images without any labels. The virtual-real interaction guides the model to learn useful information from synthetic data while keeping consistent with real data. We first analyze the data distribution discrepancy from a probabilistic perspective, and divide it into image-level and instance-level discrepancies. Then, we design two components to align these discrepancies, i.e., global-level alignment and local-level alignment. Furthermore, a consistency alignment component is proposed to encourage the consistency between the global-level and the local-level alignment components. We evaluate the proposed approach on the real Cityscapes dataset by adapting from virtual SYNTHIA, Virtual KITTI, and VIPER datasets. The experimental results demonstrate that it achieves significantly better performance than state-of-the-art methods.

Instance Segmentation and Classification Method for Plant Leaf Images Based on ISC-MRCNN and APS-DCCNN

To solve the complex background problems (e.g. Noise interference, object overlap, and different illumination) that affect the classification performance on plant leaf images, this paper proposes an instance segmentation and classification method for plant leaf images based on IFPN SNMS CFFI-Mask R-CNN (ISC-MRCNN) and ACPSOSVM-Dual Channels Convolutional Neural Network (APS-DCCNN). To obtain the foreground of plant leaf images, the lateral connection structure of the feature map pyramid in ISC-MRCNN fuses the feature maps of different depths, so that the network learns more detailed features. Then, the Soft Non-Maximum Suppression Algorithm is employed to improve the detection performance of overlapping objects. Next, the pooling method of integrating the continuous function can reduce the precision loss during the alignment of the mapping between the feature map and the original image. Finally, by constructing a mask filter layer, complex backgrounds are masked. To distinguish the similarity between plant leaf images, APS-DCCNN is used to classify the foreground images. In this process, the Support Vector Machine is used to replace softmax and then an Adaptive Chaotic Particle Swarm Algorithm is employed to optimize it. The experimental results show that compared with Mask R-CNN, the average precision of ISC-MRCNN has increased by 1.89% under different thresholds. The proposed method is suitable for the object detection and instance segmentation problems with complex background. Besides, compared with traditional CNN, the average precision of the classification results obtained by APS-DCCNN has improved by 1.59%. This has shown that the proposed method is suitable for the classification of plant leaves.

A Fast and Accurate Algorithm for Nuclei Instance Segmentation in Microscopy Images

Nuclei instance segmentation within microscopy images is a fundamental task in the pathology work-flow, based on that the meaningful nuclear features can be extracted and multiple biological related analysis can be performed. However, this task is still challenging because of the large variability among different types of nuclei. Although deep learning(DL) based methods have achieved state-of-the-art results in nuclei instance segmentation tasks, these methods are usually focus on improving the accuracy and require support of powerful computing resources. In this paper, we joint the detection and segmentation simultaneously, and propose a fast and accurate box-based nuclei instance segmentation method. Mainly, we employ a fusion module based on the feature pyramid network(FPN) to combine the complementary information of the shallow layers with deep layers for detection the nuclear location by bounding boxes. Subsequently, we crop the feature maps according to the bounding boxes and feed the cropped patches into an U-net architecture as a guide to separate clustered nuclei. The experiments show that the proposed approach outperforms prior state-of-the-art methods, not only on accuracy but also on speed. The source code will be released at: https://github.com/QUAPNH/Nucleiseg.

MitoEM Dataset: Large-scale 3D Mitochondria Instance Segmentation from EM Images.

Electron microscopy (EM) allows the identification of intracellular organelles such as mitochondria, providing insights for clinical and scientific studies. However, public mitochondria segmentation datasets only contain hundreds of instances with simple shapes. It is unclear if existing methods achieving human-level accuracy on these small datasets are robust in practice. To this end, we introduce the MitoEM dataset, a 3D mitochondria instance segmentation dataset with two (30μm)3 volumes from human and rat cortices respectively, 3, 600× larger than previous benchmarks. With around 40K instances, we find a great diversity of mitochondria in terms of shape and density. For evaluation, we tailor the implementation of the average precision (AP) metric for 3D data with a 45× speedup. On MitoEM, we find existing instance segmentation methods often fail to correctly segment mitochondria with complex shapes or close contacts with other instances. Thus, our MitoEM dataset poses new challenges to the field. We release our code and data: https://donglaiw.github.io/page/mitoEM/index.html.

PoSeg: Pose-Aware Refinement Network for Human Instance Segmentation

Human instance segmentation is a core problem for human-centric scene understanding and segmenting human instances poses a unique challenge to vision systems due to large intra-class variations in both appearance and shape, and complicated occlusion patterns. In this paper, we propose a new pose-aware human instance segmentation method. Compared to the previous pose-aware methods which first predict bottom-up poses and then estimate instance segmentation on top of predicted poses, our method integrates both top-down and bottom-up cues for an instance: it adopts detection results as human proposals and jointly estimates human pose and instance segmentation for each proposal. We develop a modular recurrent deep network that utilizes pose estimation to refine instance segmentation in an iterative manner. Our refinement modules exploit pose cues in two levels: as a coarse shape prior and local part attention. We evaluate our approach on two public multi-person benchmarks: OCHuman dataset and COCOPersons dataset. The proposed method surpasses the state-of-the-art methods on OCHuman dataset by 3.0 mAP and on COCOPersons by 6.4 mAP, demonstrating the effectiveness of our approach.

AS-PANet: a chromosome instance segmentation method based on improved path aggregation network architecture

目的染色体是遗传信息的重要载体，健康的人体细胞中包含46条染色体，包括22对常染色体和1对性染色体。染色体核型化分析是产前诊断和遗产疾病诊断的重要且常用方法。染色体核型化分析是指从分裂中期的细胞显微镜图像中，分割出染色体并根据染色体的条带进行分组排列的过程。染色体核型化分析通常由细胞学家手工完成，但是这个过程非常费时、繁琐且容易出错。由于染色体的非刚性特质，多条染色体之间存在重叠及交叉现象，致使染色体实例分割非常困难。染色体分割是染色体核型化分析过程中最重要且最困难的一步，因此本文旨在解决重叠、交叉染色体实例分割问题。 方法本文基于路径增强网络（PANet）模型，提出AS-PANet（amount segmentation PANet）模型用于解决重叠染色体实例分割问题。在路径增强网络的基础上引入染色体计数领域知识预测作为模型的一个预测分支，并改进了路径增强网络的模型结构和损失函数，使图像分类、目标检测、实例分割和染色体计数4个子任务共享卷积特征，进行联合训练。在临床染色体图像数据上进行标注并构建训练集和测试集，同时提出有效的数据增广方法用以扩充染色体标注训练数据集，提升模型的训练效果。 结果在临床染色体数据集中开展实证研究实验。实验结果表明，本文方法在临床染色体数据集中，平均分割精度mAP（mean average precision）为90.63%。该结果比PANet提升了1.18%，比基线模型Mask R-CNN提升了2.85%。分割准确率为85%，相比PANet提升了2%，相比Mask R-CNN（region with convolutional neural network）提升3.75%。 结论本文染色体实例分割方法能够更有效地解决临床染色体分割问题，相比现有的方法，分割效果更好。

Deep learning–based image instance segmentation for moisture marks of shield tunnel lining

This paper presents a method for the image instance segmentation of the moisture marks of shield tunnel lining using a mask-region-based convolutional neural network (Mask R-CNN) algorithm. The authors’ previously proposed fully convolutional network (FCN) framework and the moisture-mark detection framework have been combined into a unified Mask R-CNN framework. A total of 5031 images covering five scales were collected and annotated to train this deep-learning (DL)-based algorithm to identify the moisture marks in images. Three steps are detailed for instance segmentation: feature extraction, the generation of region proposals, and moisture-mark identification. A high-quality segmentation mask for the moisture marks is generated, and the moisture-mark area is obtained by counting the pixels with a value of 1 in the polygon generated for moisture marks during the test process of the trained Mask R-CNN model. The proposed method is validated by an experimental study, and the results are compared with those obtained by the authors’ previous FCN method and two conventional methods—the region growing algorithm (RGA) and Otsu algorithm (OA). The accuracy, F1 score, and intersection over union (IoU) for the proposed method are superior than those for the FCN, RGA, and OA with respect to 503 test images. The inference time for the proposed method is considerably shorter than that for the FCN and RGA and slightly longer than that for the OA.

Determination of the Optimal State of Dough Fermentation in Bread Production by Using Optical Sensors and Deep Learning

Dough fermentation plays an essential role in the bread production process, and its success is critical to producing high-quality products. In Germany, the number of stores per bakery chain has increased within the last years as well as the trend to finish the bakery products local at the stores. There is an unsatisfied demand for skilled workers, which leads to an increasing number of untrained and inexperienced employees at the stores. This paper proposes a method for the automatic monitoring of the fermentation process based on optical techniques. By using a combination of machine learning and superellipsoid model fitting, we have developed an instance segmentation and parameter estimation method for dough objects that are positioned inside a fermentation chamber. In our method we measure the given topography at discrete points in time using a movable laser sensor system that is located at the back of the fermentation chamber. By applying the superellipsoid model fitting method, we estimated the volume of each object and achieved results with a deviation of approximately 10% on average. Thereby, the volume gradient is monitored continuously and represents the progress of the fermentation state. Exploratory tests show the reliability and the potential of our method, which is particularly suitable for local stores but also for high volume production in bakery plants.

Integrating deep convolutional neural networks with marker-controlled watershed for overlapping nuclei segmentation in histopathology images

Nuclei segmentation in histopathology images plays a crucial role in the morphological quantitative analysis of tissue structure and has become a hot research topic. Though numerous efforts have been tried in this research area, the overlapping and touching nuclei segmentation remains a challenging problem. In this paper, we present a novel and effective instance segmentation method for tackling this challenge by integrating Deep Convolutional Neural Networks with Marker-controlled Watershed. Firstly, we design a novel network architecture with multiple segmentation tasks, called Deep Interval-Marker-Aware Network, for learning the foreground, marker, and interval of nuclei, simultaneously. Then the learned interval between overlapping nuclei is used to refine the foreground result of nuclei by using the logical operators. Finally, the learned marker result and the nuclei segmentation result refined by interval are transmitted into the Marker-controlled Watershed for splitting the touching nuclei. The experiments on the standard public datasets demonstrate that our method achieves a substantial improvement compared with state-of-the-art methods. Source codes are available at https://github.com/appiek/Nuclei_Segmentation_Experiments_Demo.

MU R-CNN: A Two-Dimensional Code Instance Segmentation Network Based on Deep Learning

In the context of Industry 4.0, the most popular way to identify and track objects is to add tags, and currently most companies still use cheap quick response (QR) tags, which can be positioned by computer vision (CV) technology. In CV, instance segmentation (IS) can detect the position of tags while also segmenting each instance. Currently, the mask region-based convolutional neural network (Mask R-CNN) method is used to realize IS, but the completeness of the instance mask cannot be guaranteed. Furthermore, due to the rich texture of QR tags, low-quality images can lower intersection-over-union (IoU) significantly, disabling it from accurately measuring the completeness of the instance mask. In order to optimize the IoU of the instance mask, a QR tag IS method named the mask UNet region-based convolutional neural network (MU R-CNN) is proposed. We utilize the UNet branch to reduce the impact of low image quality on IoU through texture segmentation. The UNet branch does not depend on the features of the Mask R-CNN branch so its training process can be carried out independently. The pre-trained optimal UNet model can ensure that the loss of MU R-CNN is accurate from the beginning of the end-to-end training. Experimental results show that the proposed MU R-CNN is applicable to both high- and low-quality images, and thus more suitable for Industry 4.0.