Nuclear instance segmentation and tracking for preimplantation mouse embryos

Simple SummaryBiodiversity monitoring is one of the primary means of ecological research. With the development of convolutional neural networks (CNNs) in the field of instance segmentation, CNNs are also used for species recognition. Almost all species recognition models apply pixel-based instance segmentation to recognize animal species. However, pixel-based instance segmentation models require a large number of annotations and labels, which makes them time-consuming and unsuitable for small datasets. Therefore, in this paper, we propose a contour-based wild animal instance segmentation model that can reach a balance between accuracy and real-time performance.Camera traps are widely used in wildlife research, conservation, and management, and abundant images are acquired every day. Efficient real-time instance segmentation networks can help ecologists label and study wild animals. However, existing deep convolutional neural networks require a large number of annotations and labels, which makes them unsuitable for small datasets. In this paper, we propose a two-stage method for the instance segmentation of wildlife, including object detection and contour approximation. In the object detection stage, we use FSOD (few-shot object detection) to recognize animal species and detect the initial bounding boxes of animals. In the case of a small wildlife dataset, this method may improve the generalization ability of the wild animal species recognition and even identify new species that only have a small number of training samples. In the second stage, deep snake is used as the contour approximation model for the instance segmentation of wild mammals. The initial bounding boxes generated in the first stage are input to deep snake to approximate the contours of the animal bodies. The model fuses the advantages of detecting new species and real-time instance segmentation. The experimental results show that the proposed method is more suitable for wild animal instance segmentation, in comparison with pixel-wise segmentation methods. In particular, the proposed method shows a better performance when facing challenging images.

Object segmentation and classification using the deep convolutional neural network (DCNN) has been widely researched in recent years. On the one hand, DCNN requires large data training sets and precise labeling, which bring about great difficulties in practical application. On the other hand, it consumes a large amount of computing resources, so it is difficult to apply it to low-cost terminal equipment. This paper proposes a method of crop organ segmentation and disease recognition that is based on weakly supervised DCNN and lightweight model. While considering the actual situation in the greenhouse, we adopt a two-step strategy to reduce the interference of complex background. Firstly, we use generic instance segmentation architecture—Mask R-CNN to realize the instance segmentation of tomato organs based on weakly supervised learning, and then the disease recognition of tomato leaves is realized by depth separable multi-scale convolution. Instance segmentation algorithms usually require accurate pixel-level supervised labels, which are difficult to collect, so we propose a weakly supervised instance segmentation assignment to solve this problem. The lightweight model uses multi-scale convolution to expand the network width, which makes the extracted features richer, and depth separable convolution is adopted to reduce model parameters. The experimental results showed that our method reached higher recognition accuracy when compared with other methods, at the same time occupied less memory space, which can realize the real-time recognition of tomato diseases on low-performance terminals, and can be applied to the recognition of crop diseases in other similar application scenarios.

Instance segmentation is an important method for high-resolution remote sensing images (HRRSIs) analysis. Traditional instance segmentation algorithms are not suitable to analyze complex HRRSIs that exhibit: 1) various shapes and sizes of targets; 2) a large number of small targets; and 3) data with long tail distribution. Here we introduce DB-BlendMask, an efficient and accurate instance segmentation method that can accommodate complex HRRSIs. It is composed of size balance coefficient (SBC), class balance module (CBM), and decomposed attention blender module (DA-Blender module). SBC consists of a fair weight allocation strategy for positive samples in object detection. CBM combines classification obtained in object detection stage to guide the semantic feature extraction. Complementary to a traditional convolutional neural network (CNN) architecture, DA-Blender module has the ability to considerably compress space complexity of attention and merge attention with semantic feature to generate the instance mask. We compare the performance of DB-BlendMask with a benchmark Mask R-CNN on two typical datasets, iSAID, and ISPRS Postdam. We obtain an average detection precision of 39.2% on iSAID and 63.6% on ISPRS Postdam, which corresponds to an improvement of 2.5% and 2.7%, respectively, compared to the benchmark in a real-time scenario.

Highlights Provides a low-cost, fast, accurate, and non-destructive method for segmenting stems and leaves of oilseed rape at the seedling stage. Different varieties and leaf shapes of oilseed rape can all be accurately and efficiently segmented at organ level. The results of accurate organ instance segmentation are used for further organ morphological structure analysis and organ-level phenotype extraction. The calculated phenotypic data is highly correlated with the true phenotypic data. Abstract. Organ-level plant point cloud instance segmentation is crucial for three-dimensional (3D) plant structure investigations and plant phenotypic formations. Due to the complexity of 3D plant structures, accurate plant organ segmentation methods remain a bottleneck in current development of plant organ phenotypes. In this work, we present a fast, accurate, low-cost, and non-destructive method of stem and leaf instance segmentation at the organ level for oilseed rape point cloud. Firstly, we use a 3D scanner to obtain the point cloud of oilseed rape plants. Then, principal component analysis (PCA) is used to perform surface feature extraction on the point cloud after preprocessing. Subsequently, the stems and leaves in different features are segmented by region growing. The irrational segmentation results are finally optimized. Segmentation results of stems and leaves of oilseed rape of different sizes and leaf shapes are evaluated using manually segmented oilseed rape point clouds as a benchmark. The precision, recall, F-score, and average accuracy of the proposed stem and leave segmentation method are 0.983, 0.897, 0.937, and 0.883, respectively. The experimental results suggest that the proposed method can accurately segment an oilseed rape 3D point cloud into instances corresponding to the organs of the plant. Additionally, the separated leaf and stem organs of oilseed rape can be further used for plant structure studies and organ phenotype extraction. The organ phenotypic parameters of oilseed rape are calculated based on the results of stem and leaf instances segmentation. The leaf area, stem length and stem angle of oilseed rape calculated from the point cloud are also compared with the corresponding manual measurements, which are highly correlated with coefficients of determination R2 (0.82-0.97). In conclusion, the proposed segmentation method can be applied as a fundamental segmentation step to extract organ phenotype data from oilseed rape plants. Namely, our research is valuable for precision breeding and basic plant research. Keywords: 3D point cloud, Oilseed rape plant, Phenotypic traits, Stem-leaf segmentation.

Separating and labeling each nuclear instance (instance-aware segmentation) is the key challenge in nuclear image segmentation. Deep Convolutional Neural Networks have been demonstrated to solve nuclear image segmentation tasks across different imaging modalities, but a systematic comparison on complex immunofluorescence images has not been performed. Deep learning based segmentation requires annotated datasets for training, but annotated fluorescence nuclear image datasets are rare and of limited size and complexity. In this work, we evaluate and compare the segmentation effectiveness of multiple deep learning architectures (U-Net, U-Net ResNet, Cellpose, Mask R-CNN, KG instance segmentation) and two conventional algorithms (Iterative h-min based watershed, Attributed relational graphs) on complex fluorescence nuclear images of various types. We propose and evaluate a novel strategy to create artificial images to extend the training set. Results show that instance-aware segmentation architectures and Cellpose outperform the U-Net architectures and conventional methods on complex images in terms of F1 scores, while the U-Net architectures achieve overall higher mean Dice scores. Training with additional artificially generated images improves recall and F1 scores for complex images, thereby leading to top F1 scores for three out of five sample preparation types. Mask R-CNN trained on artificial images achieves the overall highest F1 score on complex images of similar conditions to the training set images while Cellpose achieves the overall highest F1 score on complex images of new imaging conditions. We provide quantitative results demonstrating that images annotated by under-graduates are sufficient for training instance-aware segmentation architectures to efficiently segment complex fluorescence nuclear images.

When monitoring crop growth using top-down images of the plant canopies, leaves in agricultural fields appear very dense and significantly overlap each other. Moreover, the image can be affected by external conditions such as background environment and light intensity, impacting the effectiveness of image segmentation. To address the challenge of segmenting dense and overlapping plant leaves under natural lighting conditions, this study employed a Bilayer Convolutional Network (BCNet) method for accurate leaf segmentation across various lighting environments. The major contributions of this study are as follows: (1) Utilized Fully Convolutional Object Detection (FCOS) for plant leaf detection, incorporating ResNet-50 with the Convolutional Block Attention Module (CBAM) and Feature Pyramid Network (FPN) to enhance Region of Interest (RoI) feature extraction from canopy top-view images. (2) Extracted the sub-region of the RoI based on the position of the detection box, using this region as input for the BCNet, ensuring precise segmentation. (3) Utilized instance segmentation of canopy top-view images using BCNet, improving segmentation accuracy. (4) Applied the Varifocal Loss Function to improve the classification loss function in FCOS, leading to better performance metrics. The experimental results on cucumber canopy top-view images captured in glass greenhouse and plastic greenhouse environments show that our method is highly effective. For cucumber leaves at different growth stages and under various lighting conditions, the Precision, Recall and Average Precision (AP) metrics for object recognition are 97%, 94% and 96.57%, respectively. For instance segmentation, the Precision, Recall and Average Precision (AP) metrics are 87%, 83% and 84.71%, respectively. Our algorithm outperforms commonly used deep learning algorithms such as Faster R-CNN, Mask R-CNN, YOLOv4 and PANet, showcasing its superior capability in complex agricultural settings. The results of this study demonstrate the potential of our method for accurate recognition and segmentation of highly overlapping leaves in diverse agricultural environments, significantly contributing to the application of deep learning algorithms in smart agriculture.

Convolutional Neural Network (CNN) based scene text detection methods mostly employ the semantic segmentation (text/non-text classification) task to localize the regions of texts. However, they cannot distinguish different text-lines like instance segmentation. In this paper, we propose a novel framework based on Fully Convolutional Networks (FCN) and Recurrent Neural Network (RNN) to achieve both scene text detection and instance segmentation. The FCN is used to classify text and non-text regions, and the RNN utilizes the features extracted by FCN to simultaneously detect and segment one text instance at each time step. Meanwhile, it also extracts bounding boxes by a much simpler way than the non-maximum suppression (NMS) method. The proposed method achieves competitive results on two public benchmarks including ICDAR 2015 Incidental Scene Text Dataset and ICDAR 2013 Focused Scene Text Dataset. Moreover, the benefits of adding regression task in the RNN module are manifested.

In commercial forestry and large-scale plant propagation, the utilization of artificial intelligence techniques for automated somatic embryo analysis has emerged as a highly valuable tool. Notably, image segmentation plays a key role in the automated assessment of mature somatic embryos. However, to date, the application of Convolutional Neural Networks (CNNs) for segmentation of mature somatic embryos remains unexplored. In this study, we present a novel application of CNNs for delineating mature somatic conifer embryos from background and residual proliferating embryogenic tissue and differentiating various morphological regions within the embryos. A semantic segmentation CNN was trained to assign pixels to cotyledon, hypocotyl, and background regions, while an instance segmentation network was trained to detect individual cotyledons for automated counting. The main dataset comprised 275 high-resolution microscopic images of mature Pinus radiata somatic embryos, with 42 images reserved for testing and validation sets. The evaluation of different segmentation methods revealed that semantic segmentation achieved the highest performance averaged across classes, achieving F1 scores of 0.929 and 0.932, with IoU scores of 0.867 and 0.872 for the cotyledon and hypocotyl regions respectively. The instance segmentation approach demonstrated proficiency in accurate detection and counting of the number of cotyledons, as indicated by a mean squared error (MSE) of 0.79 and mean absolute error (MAE) of 0.60. The findings highlight the efficacy of neural network-based methods in accurately segmenting somatic embryos and delineating individual morphological parts, providing additional information compared to previous segmentation techniques. This opens avenues for further analysis, including quantification of morphological characteristics in each region, enabling the identification of features of desirable embryos in large-scale production systems. These advancements contribute to the improvement of automated somatic embryogenesis systems, facilitating efficient and reliable plant propagation for commercial forestry applications.

In this paper, an efficient instance segmentation scheme based on deep convolutional neural networks is proposed to deal with unconstrained psoriasis images for computer-aided diagnosis. To achieve instance segmentation, the You Only Look At CoefficienTs (YOLACT) network composed of backbone, feature pyramid network (FPN), Protonet, and prediction head is used to deal with psoriasis images. The backbone network is used to extract feature maps from an image, and FPN is designed to generate multiscale feature maps for effectively classifying and localizing objects with multiple sizes. The prediction head is used to predict the classification information, bounding box information, and mask coefficients of objects. Some prototypes generated by Protonet are combined with mask coefficients to estimate the pixel-level shapes for objects. To achieve instance segmentation for unconstrained psoriasis images, YOLACT++ with a pretrained model is retrained via transfer learning. To evaluate the performance of the proposed scheme, unconstrained psoriasis images with different severity levels are collected for testing. As for subjective testing, the psoriasis regions and normal skin areas can be located and classified well. The four performance indices of the proposed scheme were higher than 93% after cross validation. About object localization, the Mean Average Precision (mAP) rates of the proposed scheme were at least 85.9% after cross validation. As for efficiency, the frames per second (FPS) rate of the proposed scheme reached up to 15. In addition, the F1_score and the execution speed of the proposed scheme were higher than those of the Mask Region-Based Convolutional Neural Networks (R-CNN)-based method. These results show that the proposed scheme based on YOLACT++ can not only detect psoriasis regions but also distinguish psoriasis pixels from background and normal skin pixels well. Furthermore, the proposed instance segmentation scheme outperforms the Mask R-CNN-based method for unconstrained psoriasis images.

A fast and accurate deep learning method for strawberry instance segmentation

Instance Segmentation and Localization of Strawberries in Farm Conditions for Automatic Fruit Harvesting

Instance segmentation is a significant way for remote sensing image (RSI) interpretation. The large number, sharp variation of sizes, and complex background of objects raise higher demands for instance segmentation models. The synergistic usage of global and local features has drawn great attention due to its superior performance but has not been fully explored in mainstream instance segmentation methods. In this work, a global-local feature refusion network (GLFRNet) with two fusion procedures is proposed to fully utilize coarse-grained and fine-grained features for RSI instance segmentation. In this model, the backbone integrates both convolutional neural network (CNN)-based and VMamba-based branches to extract local and global features, respectively. Three novel models are proposed to leverage the features adaptively, i.e., the cross-dim feature fusion (CDFF) module, the semantic complementary feature fusion (SCFF) module, and the guided feature refusion module (GFRM). The CDFF module is designed to aggregate features flexibly by fusing features from two backbones with different attention modules in the first fusion procedure. The GFRM and SCFF module are proposed in the refusion procedure to generate accurate segmentation results. Inspired by agent attention, the GFRM dynamically assembles detailed features for mask generation by refusing local and global features with the guidance of fusion results from CDFF. The SCFF module complements the significant features by enhancing and integrating global, local, and detailed features, and finally generates masks of instances. Extensive experiments demonstrate that GLFRNet outperforms the second-best model by 1.9, 1.3, and 0.3 in mask average precisions (APs) on NWPU VHR-10, WHU Building, and iSAID datasets.

The fish instance segmentation task plays an important role in fish image analysis. Traditional fish analysis methods (e.g., segmenting the fish curve by hand to obtain the size of fish) cost a mass of manual labor and thus are not efficient. Convolutional Neural Networks (CNNs) become an effective scheme to replace manual labor to decrease costs and improve efficiency. The Hybrid Task Cascade (HTC) is a novel CNN model which applies cascade architecture to achieve boosted performance in the instance segmentation task. However, instance segmentation models cannot handle fish images very well due to small image size and low image quality. Furthermore, HTC still suffers from the incomplete confidence score only consisting of the classification information without the mask information, so the instance segmentation performance would be degraded. To this end, we propose an Enhanced Hybrid Task Cascade (EHTC) model to overcome these limitations. (1) The EHTC conducts data pre-processing before the instance segmentation network through an image super-resolution technology to resize images and optimize features that can be more easily understood by the later instance segmentation network. (2) Our EHTC addresses the incomplete confidence score problem in the HTC by adding one mask scoring block, named MaskIoU, to generate mask confidence scores providing the mask that improves the instance segmentation accuracy. Finally, the experimental results show that our EHTC achieves better performance than the state-of-the-art models on the Fish4knowledge dataset.

With the development of satellite technology, up to date imaging mode of synthetic aperture radar (SAR) satellite can provide higher resolution SAR imageries, which benefits ship detection and instance segmentation. Meanwhile, object detectors based on convolutional neural network (CNN) show high performance on SAR ship detection even without land-ocean segmentation; but with respective shortcomings, such as the relatively small size of SAR images for ship detection, limited SAR training samples, and inappropriate annotations, in existing SAR ship datasets, related research is hampered. To promote the development of CNN based ship detection and instance segmentation, we have constructed a High-Resolution SAR Images Dataset (HRSID). In addition to object detection, instance segmentation can also be implemented on HRSID. As for dataset construction, under the overlapped ratio of 25%, 136 panoramic SAR imageries with ranging resolution from 1m to 5m are cropped to $800 \times 800$ pixels SAR images. To reduce wrong annotation and missing annotation, optical remote sensing imageries are applied to reduce the interferes from harbor constructions. There are 5604 cropped SAR images and 16951 ships in HRSID, and we have divided HRSID into a training set (65% SAR images) and test set (35% SAR images) with the format of Microsoft Common Objects in Context (MS COCO). 8 state-of-the-art detectors are experimented on HRSID to build the baseline; MS COCO evaluation metrics are applicated for comprehensive evaluation. Experimental results reveal that ship detection and instance segmentation can be well implemented on HRSID.

We present a simple post-processing method for object instance segmentation. Instances segment images into parts with rich semantics while less texture consistency. Superpixels segment images into parts with great texture consistency while less semantics. We design a method, joining super-pixel to the instance segmentation workflow, in order to enhance the instance segmentation results. The workflow is, firstly calling a certain instance segmentation method (for example, Mask Region Convolutional Neural Network (R-CNN), Mask R-CNN) on the image to get prediction masks preliminary; then utilizing super-pixels as the assistant information to modify the prediction masks; and finally obtaining the better segmentation results. Our method is train-free, while it can refine the instance segmentation masks. Our experiments performed on multiple neural networks and the Microsoft Common Objects in Contexts (MS-COCO) dataset demonstrate the effectiveness of our method.