Instance Segmentation Network Research Articles

SDSCNet: an instance segmentation network for efficient monitoring of goose breeding conditions

SDSCNet: an instance segmentation network for efficient monitoring of goose breeding conditions

TRF-Net: a transformer-based RGB-D fusion network for desktop object instance segmentation

TRF-Net: a transformer-based RGB-D fusion network for desktop object instance segmentation

A deep learning-based method for detecting granular fertilizer deposition distribution patterns in centrifugal variable-rate spreader fertilization

Beginning with the problems of high cost, low efficiency and high working intensity of traditional granular fertilizer deposition distribution detection methods, this study proposes an automatic detection system of granular fertilizer deposition distribution pattern based on deep learning method. The system analyzes the distribution pattern of granular fertilizer deposition on the ground and allows for parameter calibration, performance evaluation and structural optimization of fertilizer spreading devices. In this study, hardware and software parts of the proposed system were designed. The hardware part primarily includes a customized fertilizer calibration mat, an image acquisition device, a system control terminal, and a processing core to realize image acquisition, storage, data processing, and system control functions. For the software part, a granular fertilizer identification algorithm was designed; in addition, the granular fertilizer instance segmentation network model F-TPP (Fertilizer-Swin Transformer PANet partition) was proposed that integrated the Swin Transformer network and path aggregation network (PANet) to address the problems associated with extremely small and relatively dense targets in the detection of granular fertilizers. Moreover, the F-TPP network model optimized the Anchor generation scale for appropriate segmentation detection of granular fertilizers. The experimental results show that the F-TPP network model outperformed comparison conventional network models, such as Mask R-CNN, SOLO, and YOLOv5. Indeed, the detection efficiency of the proposed model was higher than that of conventional models for small and dense granular fertilizers. The proposed model yielded a detection precision, recall rate, and F1 score of 93.8%, 96.3%, and 0.95, respectively. The F-TPP network model improves the segmentation precision of granular fertilizer; thus, it improves the accuracy of automatically-deposited fertilizers. According to the experimental results, the proposed detection method yielded maximum absolute and relative errors of 0.59 g and 5.25%, respectively.

PerSnake: a real-time pedestrian instance segmentation network using contour representation

PerSnake: a real-time pedestrian instance segmentation network using contour representation

Road Traffic Sign Detection Method Based on RTS R-CNN Instance Segmentation Network.

With the rapid development of the autonomous driving industry, there is increasing research on related perception tasks. However, research on road surface traffic sign detection tasks is still limited. There are two main challenges to this task. First, when the target object's pixel ratio is small, the detection accuracy often decreases. Second, the existing publicly available road surface traffic sign datasets have limited image data. To address these issues, this paper proposes a new instance segmentation network, RTS R-CNN, for road surface traffic sign detection tasks based on Mask R-CNN. The network can accurately perceive road surface traffic signs and provide important information for the autonomous driving decision-making system. Specifically, CSPDarkNet53_ECA is proposed in the feature extraction stage to enhance the performance of deep convolutional networks by increasing inter-channel interactions. Second, to improve the network's detection accuracy for small target objects, GR-PAFPN is proposed in the feature fusion part, which uses a residual feature enhancement module (RFA) and atrous spatial pyramid pooling (ASPP) to optimize PAFPN and introduces a balanced feature pyramid module (BFP) to handle the imbalanced feature information at different resolutions. Finally, data augmentation is used to generate more data and prevent overfitting in specific scenarios. The proposed method has been tested on the open-source dataset Ceymo, achieving a Macro F1-score of 87.56%, which is 2.3% higher than the baseline method, while the inference speed reaches 23.5 FPS.

An Autonomous Surgical Instrument Tracking Framework With a Binocular Camera for a Robotic Flexible Laparoscope

In minimally invasive surgery (MIS), the field of view (FOV) plays a vital role. To enhance the stability of FOV and lighten the burden on surgeons, robot-assisted laparoscope systems have been developed and introduced into surgery. However, most of the existing automatic surgical tool tracking schemes with robotic laparoscopes either do not consider tracking of the depth direction or the remote center of motion (RCM) constraints. To tackle these problems, this paper presents an autonomous three-dimensional surgical instrument tracking framework for a robotic laparoscope based on a binocular camera. A 3-degree-of-freedom visual tracking scheme is derived from the kinematics of the binocular robotic laparoscope. A programmable RCM constraint is introduced due to the MIS requirement. To precisely locate the surgical instruments in FOV, a surgical instrument tip segmentation dataset with 5471 images is built and a modified real-time instance segmentation network is trained based on the dataset. Then, a semi-global matching (SGM) algorithm is utilized with the binocular camera to estimate the depth in FOV. To meet the joint limit constraint and maintain the dexterity of the robotic system, a task-priority-based control method is adopted. The simulation and experiment results demonstrate the feasibility of the proposed framework for instrument tracking with the robotic laparoscope.

Improvement of rotated object detection and instance segmentation in warship satellite remote sensing images based on convolutional neural network

ABSTRACT Object detection and instance segmentation networks are improved to realise the accurate detection and instance segmentation of rotated warship objects in satellite remote sensing images. An adaptive threshold generation scheme and segmentation annotation information are applied used to improve a rotated label generation method to obtain high-precision rotated object labels. The original RPN is combined with the bbox head with improved output dimensions to obtain a rotated RPN to generate rotated region proposals. Rotated RoIAlign is used to solve the problem of mismatch between rotated region proposals and dimensions of subsequent feature maps. A rotated detection frame is used to correct the output of the network, which alleviates false detection and omission. In addition, this removes the pixels outside the rotated detection frame that are incorrectly classified as objects. The improved networks can achieve high-precision detection and instance segmentation of rotated warship objects, and the methods used in this study have good generalisability.

NISNet3D: three-dimensional nuclear synthesis and instance segmentation for fluorescence microscopy images

The primary step in tissue cytometry is the automated distinction of individual cells (segmentation). Since cell borders are seldom labeled, cells are generally segmented by their nuclei. While tools have been developed for segmenting nuclei in two dimensions, segmentation of nuclei in three-dimensional volumes remains a challenging task. The lack of effective methods for three-dimensional segmentation represents a bottleneck in the realization of the potential of tissue cytometry, particularly as methods of tissue clearing present the opportunity to characterize entire organs. Methods based on deep learning have shown enormous promise, but their implementation is hampered by the need for large amounts of manually annotated training data. In this paper, we describe 3D Nuclei Instance Segmentation Network (NISNet3D) that directly segments 3D volumes through the use of a modified 3D U-Net, 3D marker-controlled watershed transform, and a nuclei instance segmentation system for separating touching nuclei. NISNet3D is unique in that it provides accurate segmentation of even challenging image volumes using a network trained on large amounts of synthetic nuclei derived from relatively few annotated volumes, or on synthetic data obtained without annotated volumes. We present a quantitative comparison of results obtained from NISNet3D with results obtained from a variety of existing nuclei segmentation techniques. We also examine the performance of the methods when no ground truth is available and only synthetic volumes were used for training.

FADM-SLAM: a fast and accurate dynamic intelligent motion SLAM for autonomous robot exploration involving movable objects

PurposeMany popular simultaneous localization and mapping (SLAM) techniques have low accuracy, especially when localizing environments containing dynamically moving objects since their presence can potentially cause inaccurate data associations. To address this issue, the proposed FADM-SLAM system aims to improve the accuracy of SLAM techniques in environments containing dynamically moving objects. It uses a pipeline of feature-based approaches accompanied by sparse optical flow and multi-view geometry as constraints to achieve this goal.Design/methodology/approachFADM-SLAM, which works with monocular, stereo and RGB-D sensors, combines an instance segmentation network incorporating an intelligent motion detection strategy (iM) with an optical flow technique to improve location accuracy. The proposed AS-SLAM system comprises four principal modules, which are the optical flow mask and iM, the ego motion estimation, dynamic point detection and the feature-based extraction framework.FindingsExperiment results using the publicly available RGBD-Bonn data set indicate that FADM-SLAM outperforms established visual SLAM systems in highly dynamic conditions.Originality/valueIn summary, the first module generates the indication of dynamic objects by using the optical flow and iM with geometric-wise segmentation, which is then used by the second module to compute the starting point of a posture. The third module, meanwhile, first searches for the dynamic feature points in the environment, and second, eliminates them from further processing. An algorithm based on epipolar constraints is implemented to do this. In this way, only the static feature points are retained, which are then fed to the fourth module for extracting important features.

Improving RGB-D SLAM accuracy in dynamic environments based on semantic and geometric constraints

Most current visual SLAM systems rely on static environment assumptions. However, in dynamic environments, the presence of dynamic objects can severely impair the performance of visual SLAM systems. To address this issue, we propose an algorithm that combines semantic segmentation and geometric constraints to eliminate the impact of dynamic objects on visual SLAM trajectory accuracy in dynamic indoor environments. First, a missed detection compensation algorithm based on the constant velocity model and region growing algorithm is proposed, which improves the accuracy of semantic or instance segmentation network and realizes accurate segmentation of dynamic objects at the semantic level in the new thread. Second, for the dynamic objects that cannot be recognized by the semantic segmentation network, a geometric constraint model based on the position correlation between feature points is presented to further eliminate the dynamic feature points that do not meet the motion consistency. Thereafter, the proposed method is integrated into the front-end of the ORB-SLAM2 system and evaluated using the RGB-D dataset. Experimental results demonstrate that the proposed method can improve the absolute trajectory accuracy of the ORB-SLAM2 system by an average of 96.90% in high dynamic scenes.

A Refined Apple Binocular Positioning Method with Segmentation-Based Deep Learning for Robotic Picking

An apple-picking robot is now the most widely accepted method in the substitution of low-efficiency and high-cost labor-intensive apple harvesting. Although most current research on apple-picking robots works well in the laboratory, most of them are unworkable in an orchard environment due to unsatisfied apple positioning performance. In general, an accurate, fast, and widely used apple positioning method for an apple-picking robot remains lacking. Some positioning methods with detection-based deep learning reached an acceptable performance in some orchards. However, apples occluded by apples, leaves, and branches are ignored in these methods with detection-based deep learning. Therefore, an apple binocular positioning method based on a Mask Region Convolutional Neural Network (Mask R-CNN, an instance segmentation network) was developed to achieve better apple positioning. A binocular camera (Bumblebee XB3) was adapted to capture binocular images of apples. After that, a Mask R-CNN was applied to implement instance segmentation of apple binocular images. Then, template matching with a parallel polar line constraint was applied for the stereo matching of apples. Finally, four feature point pairs of apples from binocular images were selected to calculate disparity and depth. The trained Mask R-CNN reached a detection and segmentation intersection over union (IoU) of 80.11% and 84.39%, respectively. The coefficient of variation (CoV) and positioning accuracy (PA) of binocular positioning were 5.28 mm and 99.49%, respectively. The research developed a new method to fulfill binocular positioning with a segmentation-based neural network.

DWPIS: Dynamic-Weight Parallel Instance and Skeleton Network for Railway Centerline Detection

The primary premise of autonomous railway inspection using unmanned aerial vehicles is achieving autonomous flight along the railway. In our previous work, fitted centerline-based unmanned aerial vehicle (UAV) navigation is proven to be an effective method to guide UAV autonomous flying. However, the empirical parameters utilized in the fitting procedure lacked a theoretical basis and the fitted curves were also not coherent nor smooth. To address these problems, this paper proposes a skeleton detection method, called the dynamic-weight parallel instance and skeleton network, to directly extract the centerlines that can be viewed as skeletons. This multi-task branch network for skeleton detection and instance segmentation can be trained end to end. Our method reformulates a fused loss function with dynamic weights to control the dominant branch. During training, the sum of the weights always remains constant and the branch with a higher weight changes from instance to skeleton gradually. Experiments show that our model yields 93.98% mean average precision (mAP) for instance segmentation, a 51.9% F-measure score (F-score) for skeleton detection, and 60.32% weighted mean metrics for the entire network based on our own railway skeleton and instance dataset which comprises 3235 labeled overhead-view images taken in various environments. Our method can achieve more accurate railway skeletons and is useful to guide the autonomous flight of a UAV in railway inspection.

Knowledge Transfer Based Spatial Embedding Network for Plant Leaf Instance Segmentation

This paper proposes a method to segment plant leaves using knowledge distillation. Unlike the existing knowledge distillation method aimed at lightening the model, we use knowledge distillation to achieve good performance even with a small amount of dataset. Plants have many leaves, and each leaf is very small. Therefore, the leaf instance segmentation is performed based on spatial embedding. The teacher network is trained with a large dataset and then distills its segmentation knowledge into the student network. Two types of knowledge are distilled from the teacher network: attention distillation and region affinity distillation. The results of the experiment demonstrate that better instance segmentation can be achieved when knowledge distillation is used.

Meta-Knowledge Guided Weakly Supervised Instance Segmentation for Optical and SAR Image Interpretation

The interpretation of optical and synthetic aperture radar (SAR) images in remote sensing is general for many tasks, such as environmental monitoring, marine management, and resource planning. Instance segmentation of optical and SAR images, which can simultaneously provide instance-level localization and pixel-level classification of objects of interest, is a crucial and challenging task in image interpretation. Considering that most current methods for instance segmentation of optical and SAR images rely on expensive pixel-level annotation, we develop a weakly supervised instance segmentation (WSIS) method to balance the visual processing requirements with the annotation cost. First, we decompose the prior knowledge of the mask-aware task in WSIS into three meta-knowledge components: fundamental knowledge, apparent knowledge, and detailed knowledge inspired by human visual perception habits of “whole to part” and “coarse to detailed.” Then, a meta-knowledge-guided weakly supervised instance segmentation network (MGWI-Net) is proposed. In this network, the weakly supervised mask (WSM) head can instantiate both fundamental knowledge and apparent knowledge to perform mask awareness without any annotations at the pixel level. The network also includes a mask information awareness assist (MIAA) head, which can implicitly guide the network to learn detailed information about edges through the boundary-sensitive feature of the fully connected conditional random field (CRF), facilitating the instantiation of detailed knowledge. The experimental results show that the MGWI-Net can efficiently generate instance masks for optical and SAR images and achieve the approximate instance segmentation results of the fully supervised method with about one-eighth of the annotation production time. The model parameters and processing speed of our network are also competitive. This study can provide inexpensive and convenient technical support for applying and promoting instance segmentation methods for optical and SAR images.

AMNet: a new RGB-D instance segmentation network based on attention and multi-modality

AMNet: a new RGB-D instance segmentation network based on attention and multi-modality

A Multiscale Instance Segmentation Method Based on Cleaning Rubber Ball Images.

The identification of wear rubber balls in the rubber ball cleaning system in heat exchange equipment directly affects the descaling efficiency. For the problem that the rubber ball image contains impurities and bubbles and the segmentation is low in real time, a multi-scale feature fusion real-time instance segmentation model based on the attention mechanism is proposed for the object segmentation of the rubber ball images. First, we introduce the Pyramid Vision Transformer instead of the convolution module in the backbone network and use the spatial-reduction attention layer of the transformer to improve the feature extraction ability across scales and spatial reduction to reduce computational cost; Second, we improve the feature fusion module to fuse image features across scales, combined with an attention mechanism to enhance the output feature representation; Third, the prediction head separates the mask branches separately. Combined with dynamic convolution, it improves the accuracy of the mask coefficients and increases the number of upsampling layers. It also connects the penultimate layer with the second layer feature map to achieve detection of smaller images with larger feature maps to improve the accuracy. Through the validation of the produced rubber ball dataset, the Dice score, Jaccard coefficient, and mAP of the actual segmented region of this network with the rubber ball dataset are improved by 4.5%, 4.7%, and 7.73%, respectively, and our model achieves 33.6 fps segmentation speed and 79.3% segmentation accuracy. Meanwhile, the average precision of Box and Mask can also meet the requirements under different IOU thresholds. We compared the DeepMask, Mask R-CNN, BlendMask, SOLOv1 and SOLOv2 instance segmentation networks with this model in terms of training accuracy and segmentation speed and obtained good results. The proposed modules can work together to better handle object details and achieve better segmentation performance.

An Instance Segmentation Model Based on Deep Learning for Intelligent Diagnosis of Uterine Myomas in MRI.

Uterine myomas affect 70% of women of reproductive age, potentially impacting their fertility and health. Manual film reading is commonly used to identify uterine myomas, but it is time-consuming, laborious, and subjective. Clinical treatment requires the consideration of the positional relationship among the uterine wall, uterine cavity, and uterine myomas. However, due to their complex and variable shapes, the low contrast of adjacent tissues or organs, and indistinguishable edges, accurately identifying them in MRI is difficult. Our work addresses these challenges by proposing an instance segmentation network capable of automatically outputting the location, category, and masks of each organ and lesion. Specifically, we designed a new backbone that facilitates learning the shape features of object diversity, and filters out background noise interference. We optimized the anchor box generation strategy to provide better priors in order to enhance the process of bounding box prediction and regression. An adaptive iterative subdivision strategy ensures that the mask boundary details of objects are more realistic and accurate. We conducted extensive experiments to validate our network, which achieved better average precision (AP) results than those of state-of-the-art instance segmentation models. Compared to the baseline network, our model improved AP on the uterine wall, uterine cavity, and myomas by 8.8%, 8.4%, and 3.2%, respectively. Our work is the first to realize multiclass instance segmentation in uterine MRI, providing a convenient and objective reference for the clinical development of appropriate surgical plans, and has significant value in improving diagnostic efficiency and realizing the automatic auxiliary diagnosis of uterine myomas.

A semi-supervised generative adversarial network for amodal instance segmentation of piglets in farrowing pens

Occlusions, such as farrowing pens in piggeries, hinder computer vision applications for automated animal monitoring. Amodal instance segmentation (AIS), aiming to predict a complete mask of an occluded target, is a promising solution. However, AIS usually requires amodal datasets, which are challenging to create and limit the application of AIS. To solve this problem, we proposed a novel semi-supervised generative adversarial network (GAN) for AIS, denoted “the AISGAN”. Our AISGAN only requires a regular modal dataset and generate amodal samples by random occlusions, making the AIS method more applicable. A corresponding segmentation loss was added to overcome mode collapse of GAN. The results showed that the AISGAN achieved a mean Intersection of Union (mIoU) of 0.823 and outperformed the mIoUs of Mask RCNN, Raw, and Convex Hull (0.801, 0.780, and 0.778, respectively). As a semi-supervised method, the mIoU of our AISGAN was further enhanced (by 0.6%) when we fine-tuned it with unlabeled new data, showing its extensibility to new unseen scenarios. The visualization demonstrates that the AISGAN can produce realistic masks of piglets, including details of their noses and legs, even under heavily occluded conditions. With the AISGAN, we achieved an occlusion-resistant spatial distribution analysis of the piglets in farrowing pens. Thus, the AISGAN is a promising tool to manage occlusion problems for automated animal monitoring in complex housing environments.

Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation

The detection and sizing of fruits with computer vision methods is of interest because it provides relevant information to improve the management of orchard farming. However, the presence of partially occluded fruits limits the performance of existing methods, making reliable fruit sizing a challenging task. While previous fruit segmentation works limit segmentation to the visible region of fruits (known as modal segmentation), in this work we propose an amodal segmentation algorithm to predict the complete shape, which includes its visible and occluded regions. To do so, an end-to-end convolutional neural network (CNN) for simultaneous modal and amodal instance segmentation was implemented. The predicted amodal masks were used to estimate the fruit diameters in pixels. Modal masks were used to identify the visible region and measure the distance between the apples and the camera using the depth image. Finally, the fruit diameters in millimetres (mm) were computed by applying the pinhole camera model. The method was developed with a Fuji apple dataset consisting of 3925 RGB-D images acquired at different growth stages with a total of 15,335 annotated apples, and was subsequently tested in a case study to measure the diameter of Elstar apples at different growth stages. Fruit detection results showed an F1-score of 0.86 and the fruit diameter results reported a mean absolute error (MAE) of 4.5 mm and R2 = 0.80 irrespective of fruit visibility. Besides the diameter estimation, modal and amodal masks were used to automatically determine the percentage of visibility of measured apples. This feature was used as a confidence value, improving the diameter estimation to MAE = 2.93 mm and R2 = 0.91 when limiting the size estimation to fruits detected with a visibility higher than 60%. The main advantages of the present methodology are its robustness for measuring partially occluded fruits and the capability to determine the visibility percentage. The main limitation is that depth images were generated by means of photogrammetry methods, which limits the efficiency of data acquisition. To overcome this limitation, future works should consider the use of commercial RGB-D sensors. The code and the dataset used to evaluate the method have been made publicly available at https://github.com/GRAP-UdL-AT/Amodal_Fruit_Sizing.

SIAN: STYLE-GUIDED INSTANCE-ADAPTIVE NORMALIZATION FOR MULTI-ORGAN HISTOPATHOLOGY IMAGE SYNTHESIS.

Existing deep neural networks for histopathology image synthesis cannot generate image styles that align with different organs, and cannot produce accurate boundaries of clustered nuclei. To address these issues, we propose a style-guided instance-adaptive normalization (SIAN) approach to synthesize realistic color distributions and textures for histopathology images from different organs. SIAN contains four phases, semantization, stylization, instantiation, and modulation. The first two phases synthesize image semantics and styles by using semantic maps and learned image style vectors. The instantiation module integrates geometrical and topological information and generates accurate nuclei boundaries. We validate the proposed approach on a multiple-organ dataset, Extensive experimental results demonstrate that the proposed method generates more realistic histopathology images than four state-of-the-art approaches for five organs. By incorporating synthetic images from the proposed approach to model training, an instance segmentation network can achieve state-of-the-art performance.