Mask R-CNN Research Articles

Prostate Segmentation in MRI Images using Transfer Learning based Mask RCNN.

The second highest cause of death among males is Prostate Cancer (PCa) in America. Over the globe, it's the usual case in men, and the annual PCa ratio is very surprising. Identical to other prognosis and diagnostic medical systems, deep learning-based automated recognition and detection systems (i.e., Computer Aided Detection (CAD) systems) have gained enormous attention in PCA. These paradigms have attained promising results with a high segmentation, detection, and classification accuracy ratio. Numerous researchers claimed efficient results from deep learning-based approaches compared to other ordinary systems that utilized pathological samples. This research is intended to perform prostate segmentation using transfer learning-based Mask R-CNN, which is consequently helpful in prostate cancer detection. Lastly, limitations in current work, research findings, and prospects have been discussed.

Comparing Mask R-CNN backbone architectures for human detection using thermal imaging

We introduce a method for detecting humans in thermal imaging using an end-to-end deep learning model. Our objective is to optimize the human detection process in thermal imaging by investigating the mask region-based convolutional neural network (Mask R-CNN). The model, an advancement of the faster region-based convolutional neural network (Faster R-CNN), not only captures bounding boxes encompassing human subjects but also delineates segmentation masks around them. Our investigation extends to the evaluation and comparison of various convolutional neural networks for feature learning, like residual network (ResNet) and Inception ResNet, all integrated into the Mask R-CNN framework. Furthermore, the experimental results show that our proposed technique achieves high accuracy. Specifically, the Mask R-CNN model using ResNet50-V1 achieved the best results, with an F-value of 87.85%, a recall of 79.33%, and a precision of 98.41%.

Fusion PCAM R-CNN of Automatic Segmentation for Magnetic Flux Leakage Defects.

Magnetic leakage detection technology plays an important role in the long-oil pipeline. Automatic segmentation of defecting images is crucial for the detection of magnetic flux leakage (MFL) works. At present, accurate segmentation for small defects has always been a difficult problem. In contrast to the state-of-the-art MFL detection methodologies based on convolution neural network (CNN), an optimization method is devised in our study by integrating mask region-based CNN (Mask R-CNN) and information entropy constraint (IEC). To be precise, the principal component analysis (PCA) is utilized to improve the feature learning and network segmentation ability of the convolution kernel. The similarity constraint rule of information entropy is proposed to be inserted into the convolution layer in the Mask R-CNN network. The Mask R-CNN optimizes the convolutional kernel with similar weights or higher similarity, meanwhile, the PCA network reduces the dimension of the feature image to reconstruct the original feature vector. As such, the feature extraction of MFL defects is optimized in the convolution check. The research results can be applied in the field of MFL detection.

Polygonal Approximation Learning for Convex Object Segmentation in Biomedical Images With Bounding Box Supervision.

As a common and critical medical image analysis task, deep learning based biomedical image segmentation is hindered by the dependence on costly fine-grained annotations. To alleviate this data dependence, in this article, a novel approach, called Polygonal Approximation Learning (PAL), is proposed for convex object instance segmentation with only bounding-box supervision. The key idea behind PAL is that the detection model for convex objects already contains the necessary information for segmenting them since their convex hulls, which can be generated approximately by the intersection of bounding boxes, are equivalent to the masks representing the objects. To extract the essential information from the detection model, a repeated detection approach is employed on biomedical images where various rotation angles are applied and a dice loss with the projection of the rotated detection results is utilized as a supervised signal in training our segmentation model. In biomedical imaging tasks involving convex objects, such as nuclei instance segmentation, PAL outperforms the known models (e.g., BoxInst) that rely solely on box supervision. Furthermore, PAL achieves comparable performance with mask-supervised models including Mask R-CNN and Cascade Mask R-CNN. Interestingly, PAL also demonstrates remarkable performance on non-convex object instance segmentation tasks, for example, surgical instrument and organ instance segmentation.

Identify and segment microalgae in complex backgrounds with improved YOLO

Microalgae are widely distributed in the ocean, and some species are prone to causing harmful algal blooms that threaten the marine ecological environment. At present, microscopy is the most common method for microalgae analysis, and the combination of computer vision and microscopy is the mainstream trend in algae morphology classification. However, most methods only focus on the species and quantities of algae, without obtaining contour information that can further analyze their survival status and biomass. Therefore, this article proposes a convolutional neural network (AlgaeSeg-YOLO), which can recognize the species, quantities, and contours of algae. Firstly, a dataset of microalgae microscopic images in microfluidic chips was constructed, which includes a total of 2799 annotated images of 6 harmful microalgae, including 3916 instances. Secondly, the AlgaeSeg-YOLO was constructed with stronger feature fusion and pixel-level spatial information modeling capabilities based on YOLOv8n-seg. Finally, compared to other common methods, the experimental results show that the mAP (mean Average Precision) of AlgaeSeg-YOLO reaches 95.61 %, which is 1.64 %, 1.76 %, 5.28 %, 3.34 %, and 5.39 % higher than YOLOv8n-seg, YOLOv5n-seg, Mask R-CNN, Cascade Mask R-CNN, and SOLOv2, respectively, achieving real-time and accurate segmentation of microalgae in complex backgrounds. Meanwhile, the parameters and computation remain relatively low. This work helps to achieve fully automated analysis of microalgae, reduce labor costs, and provide technical support for long-term monitoring of the ecological environment and further research.

Promote computer vision applications in pig farming scenarios: high-quality dataset, fundamental models, and comparable performance1

Computer vision is widely recognized as an influential technology in the field of precision management of animals. Emerging studies have demonstrated the potential to improve pig health and welfare through animal surveillance systems and computer vision (CV) algorithms. However, the lack of benchmark datasets and robust fundamental algorithms restrict CV applications for the commercial use. This study aims to bridge the gap between technology development and commercial applications in pig farming scenarios by introducing a general-purpose dataset (PigLife), comparing benchmark performances of foundational CV algorithms and model development workflows. The PigLife dataset contains video clips and images (38 short video clips, 2K image frames, 22K pig instances) across most pig production phases in a typical commercial pig farm: Breeding and Gestation, Farrow to Wean, Weaning & Nursery, and Growth to Finish. Three detection algorithms (Faster R-CNN, RetinaNet, TridentNet) and three segmentation algorithms (Mask R-CNN, MViTv2, Point-Rend) were trained on the PigLife dataset from scratch. Fine-tuning of pre-trained models (YOLO8-m, Faster-RCNN-r50) and no-training from zero-shot models (CLIP-SAM, Grouddino-HQSAM) were also evaluated to suggest faster CV development workflows for commercial applications in pig farming. This study emphasizes the necessity of a benchmark dataset for evaluating the robustness of algorithms and identifying the remaining difficulties and challenges across various algorithms. Furthermore, developing CV models from pre-trained algorithms or zero-shot models showed better performance and a faster process, which could reduce barriers when developing high-performance CV products in pig production industry.

HiSEG: Human assisted instance segmentation

Instance segmentation is a form of image detection which has a range of applications, such as object refinement, medical image analysis, and image/video editing, all of which demand a high degree of accuracy. However, this precision is often beyond the reach of what even state-of-the-art, fully automated instance segmentation algorithms can deliver. The performance gap becomes particularly prohibitive for small and complex objects. Practitioners typically resort to fully manual annotation, which can be a laborious process. In order to overcome this problem, we propose a novel approach to enable more precise predictions and generate higher-quality segmentation masks for high-curvature, complex and small-scale objects. Our human-assisted segmentation method, HiSEG, augments the existing Strong Mask R-CNN network to incorporate human-specified partial boundaries. We also present a dataset of hand-drawn partial object boundaries, which we refer to as “human attention maps”. In addition, the Partial Sketch Object Boundaries (PSOB) dataset contains hand-drawn partial object boundaries which represent curvatures of an object’s ground truth mask with several pixels. Through extensive evaluation using the PSOB dataset, we show that HiSEG outperforms state-of-the art methods such as Mask R-CNN, Strong Mask R-CNN, Mask2Former, and Segment Anything, achieving respective increases of +42.0, +34.9, +29.9, and +13.4 points in APMask metrics for these four models. We hope that our novel approach will set a baseline for future human-aided deep learning models by combining fully automated and interactive instance segmentation architectures.

Semantic segmentation and thermal imaging for forest fires detection and monitoring by drones

Forest ecosystems play a crucial role in providing a wide range of ecological, social, and economic benefits. However, the increasing frequency and severity of forest fires pose a significant threat to the sustainability of forests and their functions, highlighting the need for early detection and swift action to mitigate damage. The combination of drones and artificial intelligence, particularly deep learning, proves to be a cost-effective solution for accurately and efficiently detecting forest fires in real-time. Deep learning-based image segmentation models can not only be employed for forest fire detection but also play a vital role in damage assessment and support reforestation efforts. Furthermore, the integration of thermal cameras on drones can significantly enhance the sensitivity in forest fire detection. This study undertakes an in-depth analysis of recent advancements in deep learning-based semantic segmentation, with a particular focus on model’s mask region convolutional neural network (Mask R-CNN) and you only look once (YOLO) v5, v7, and v8 variants. Emphasis is placed on their suitability for forest fire monitoring using drones equipped with RGB and/or thermal cameras. The conducted experiments have yielded encouraging outcomes across various metrics, underscoring its significance as an invaluable asset for both fire detection and continuous monitoring endeavors.

A Temperature Monitoring Method for Sensor Arrays Based on Temperature Mapping and Improved Mask R-CNN

A Temperature Monitoring Method for Sensor Arrays Based on Temperature Mapping and Improved Mask R-CNN

Two deep learning methods in comparison to characterize droplet sizes in emulsification flow processes

AbstractFor reliable supervision in multiphase processes, the droplet size represents a critical quality attribute and needs to be monitored. A promising approach is the use of smart image flow sensors since optical measurement is the most commonly used technique for droplet size distribution determination. For this, two different AI-based object detection methods, Mask RCNN and YOLOv4, are compared regarding their accuracy and their applicability to an emulsification flow process. Iterative optimization steps, including data diversification and adaption of training parameters, enable the models to achieve robust detection performance across varying image qualities and compositions. YOLOv4 shows better detection performances and more accurate results which leads to a wider application window than Mask RCNN in determining droplet sizes in emulsification processes. The final droplet detection model YOLOv4 with Hough Circle (HC) for feature extraction determines reliable droplet sizes across diverse datasets of liquid-liquid flow systems (disperse phase content 1–15 vol.-%, droplet size range 5–150 μm). Evaluating the adjustment of Confidence Scores (CS) ensures statistical representation of even smaller droplets. The droplet detection performance of the final YOLOv4 model is compared with a manual image processing method to validate the model in general as well as its accuracy and reliability. Since YOLOv4 in combination with Hough Circle (HC) shows an accurate and robust detection and size determination, it is applicable for online monitoring and characterization of various liquid-liquid flow processes. Graphical abstract

GRA-Net: Group response attention for deep learning

Activation function, one of the most critical components in deep learning, enables the artificial neural networks to learn complex patterns through nonlinearity. Currently, element-wise activation functions such as ReLU are widely utilized because of their simplicity and efficiency. However, these methods do not consider the interactions among neurons, which may produce redundant features and affect performance. Inspired by neuroscience, we propose Group Response Attention (GRA). It establishes connections among neurons, which means that the state of each member is determined by the entire group. Accordingly, GRA makes the features more diverse and reduces redundancy. In addition, we propose a pure attention network without any conventional activation function, termed as GRA-Net. Experiments conducted on typical computer vision tasks show that GRA-Net achieves state-of-the-art results. For the ImageNet-1K classification task, GRA-Net-T achieves 82.9% Top-1 accuracy. With comparable parameters and FLOPs, its performance improves by 1.6% and 0.8% over Swin-T and ConvNeXt-T. On the COCO detection task, as the backbone of Mask R-CNN, GRA-Net-T outperforms Swin-T and ConvNeXt-T by 2.1% and 2.0% on box AP. Meanwhile, the mIoU of GRA-Net-T on the ADE20K semantic segmentation task is higher than Swin-T and ConvNeXt-T by 1.9% and 1.0%.

Leaf only SAM: A segment anything pipeline for zero-shot automated leaf segmentation

Segment Anything Model (SAM) is a new “foundation model” that can be used as a zero-shot object segmentation method with the use of either guide prompts such as bounding boxes, polygons, or points. Alternatively, additional post processing steps can be used to identify objects of interest after segmenting everything in an image. Here a method is presented using segment anything together with a series of post processing steps to segment potato leaves, called Leaf Only SAM. The advantage of this proposed method is that it does not require any training data to produce its results so has many applications across the field of plant phenotyping where there is limited high quality annotated data available. The performance of Leaf Only SAM is compared to a Mask R-CNN model which has been fine-tuned on a small novel potato leaf dataset. On the evaluation dataset, Leaf Only SAM finds an average recall of 73.1 and an average precision of 73.9, compared to recall of 87.6 and precision of 84.4 for Mask R-CNN. Leaf Only SAM does not perform better than the fine-tuned Mask R-CNN model on the potato leaf dataset, but the SAM based model does not require any extra training or annotation. This shows there is potential to use SAM as a zero-shot classifier with the addition of post processing steps.

Target reconstruction and process parameter decision-making for bolt intelligent assembly based on robot and multi-camera

Bolt assembly is widely used in modern manufacturing. And with the development of industrial automation, there is a growing demand for automatic assembly. To form a fully automatic and highly intelligent system for bolt assembly, a novel strategy based robot and multi-camera is proposed. In this strategy, a high assurance components reconstruction scheme is designed, wherein the Mask R-CNN is used for the identification and segmentation of the workpiece, then the shape, size and pose of the workpiece are determined by the local point cloud data and its envelope using basic geometries. The results show that the accuracy of workpiece recognition exceeds 96 %, the maximum positioning accuracy of the hole is 0.334 mm based on multi-camera system. Based on the historical fastening process data, an improved artificial jellyfish search optimizer (AJSO) BP neural network is proposed to automatically determine the optimal values for the bolt fastening speed and torque. The BP neural network optimized by AJSO reduces the maximum prediction error of process parameters from 0.117 mm/s to 0.051 mm/s for speed and from 0.052 N·m to 0.027 N·m for torque. Finally, the feasibility of the proposed strategy is demonstrated by an experimental case, which used a dual robot unit for the bearing seat assembly.

A Method for Extracting Joints on Mountain Tunnel Faces Based on Mask R-CNN Image Segmentation Algorithm

The accurate distribution of joints on the tunnel face is crucial for assessing the stability and safety of surrounding rock during tunnel construction. This paper introduces the Mask R-CNN image segmentation algorithm, a state-of-the-art deep learning model, to achieve efficient and accurate identification and extraction of joints on tunnel face images. First, digital images of tunnel faces were captured and stitched, resulting in 286 complete images suitable for analysis. Then, the joints on the tunnel face were extracted using traditional image processing algorithms, the commonly used U-net image segmentation model, and the Mask R-CNN image segmentation model introduced in this paper to address the lack of recognition accuracy. Finally, the extraction results obtained by the three methods were compared. The comparison results show that the joint extraction method based on the Mask R-CNN image segmentation deep learning model introduced in this paper achieved the best joint extraction effect with a Dice similarity coefficient of 87.48%, outperforming traditional methods and the U-net model, which scored 60.59% and 75.36%, respectively, realizing accurate and efficient acquisition of tunnel face rock joints. These findings suggest that the Mask R-CNN model can be effectively implemented in real-time monitoring systems for tunnel construction projects.

An exploratory framework to identify dust on photovoltaic panels in offshore floating solar power stations

Offshore floating solar power stations represent a new frontier in energy development. These stations maximize solar energy use, reduce land consumption and promote algae growth. However, moist marine air makes airborne particles like sand adhere more easily to PV panels, reducing photoelectric conversion efficiency and causing overheating and potential damage. Additionally, drones and fixed cameras face challenges due to complex maritime factors and complicating image analysis. To address these issues, this paper proposes an exploratory framework for identifying dust regions on photovoltaic panels specifically for offshore floating solar power stations. The framework aims to address how to accurately and reliably identify dust accumulation on PV panels, even in images with complex background interference. The framework is designed to be compatible with existing image recognition models and incorporates specialized enhancement modules to further improve recognition accuracy and efficiency. In this study, we utilized a dust feature enhancement module based on the HLS color space, combined with Mask R-CNN, to demonstrate the framework's feasibility and effectiveness. Experimental results show that, compared to a single Mask R-CNN, the framework significantly reduces misidentification and omission of dust areas, boosting the confidence level and effectively meeting the operational needs of offshore floating solar power stations.

Pig Weight Estimation Method Based on a Framework Combining Mask R-CNN and Ensemble Regression Model.

Using computer vision technology to estimate pig live weight is an important method to realize pig welfare. But there are two key issues that affect pigs' weight estimation: one is the uneven illumination, which leads to unclear contour extraction of pigs, and the other is the bending of the pig body, which leads to incorrect pig body information. For the first one, Mask R-CNN was used to extract the contour of the pig, and the obtained mask image was converted into a binary image from which we were able to obtain a more accurate contour image. For the second one, the body length, hip width and the distance from the camera to the pig back were corrected by XGBoost and actual measured information. Then we analyzed the rationality of the extracted features. Three feature combination strategies were used to predict pig weight. In total, 1505 back images of 39 pigs obtained using Azure kinect DK were used in the numerical experiments. The highest prediction accuracy is XGBoost, with an MAE of 0.389, RMSE of 0.576, MAPE of 0.318% and R2 of 0.995. We also recommend using the Mask R-CNN + RFR method because it has fairly high precision in each strategy. The experimental results show that our proposed method has excellent performance in live weight estimation of pigs.

Identification and Localization of Wind Turbine Blade Faults Using Deep Learning

This study addresses the challenges inherent in the maintenance and inspection of wind turbines through the application of deep learning methodologies for fault detection on Wind Turbine Blades (WTBs). Specifically, this research focuses on defect detection on the blades of small-scale WTBs due to the unavailability of commercial wind turbines. This research compared popular object localization architectures, YOLO and Mask R-CNN, to identify the most effective model to detect common WTB defects, including cracks, holes, and erosion. YOLOv9 C emerged as the most effective model, with the highest scores of mAP50 and mAP50-95 of 0.849 and 0.539, respectively. Modifications to Mask R-CNN, specifically integrating a ResNet18-FPN network, reduced computational complexity by 32 layers and achieved a mAP50 of 0.8415. The findings highlight the potential of deep learning and computer vision in improving WTB fault analysis and inspection.

Comparing YOLOv8 and Mask R-CNN for instance segmentation in complex orchard environments

Instance segmentation, an important image processing operation for automation in agriculture, is used to precisely delineate individual objects of interest within images, which provides foundational information for various automated or robotic tasks such as selective harvesting and precision pruning. This study compares the one-stage YOLOv8 and the two-stage Mask R-CNN machine learning models for instance segmentation under varying orchard conditions across two datasets. Dataset 1, collected in dormant season, includes images of dormant apple trees, which were used to train multi-object segmentation models delineating tree branches and trunks. Dataset 2, collected in the early growing season, includes images of apple tree canopies with green foliage and immature (green) apples (also called fruitlet), which were used to train single-object segmentation models delineating only immature green apples. The results showed that YOLOv8 performed better than Mask R-CNN, achieving good precision and near-perfect recall across both datasets at a confidence threshold of 0.5. Specifically, for Dataset 1, YOLOv8 achieved a precision of 0.90 and a recall of 0.95 for all classes. In comparison, Mask R-CNN demonstrated a precision of 0.81 and a recall of 0.81 for the same dataset. With Dataset 2, YOLOv8 achieved a precision of 0.93 and a recall of 0.97. Mask R-CNN, in this single-class scenario, achieved a precision of 0.85 and a recall of 0.88. Additionally, the inference times for YOLOv8 were 10.9 ms for multi-class segmentation (Dataset 1) and 7.8 ms for single-class segmentation (Dataset 2), compared to 15.6 ms and 12.8 ms achieved by Mask R-CNN's, respectively. These findings show YOLOv8's superior accuracy and efficiency in machine learning applications compared to two-stage models, specifically Mask-R-CNN, which suggests its suitability in developing smart and automated orchard operations, particularly when real-time applications are necessary in such cases as robotic harvesting and robotic immature green fruit thinning.

Machine Vision Analysis of Ujumqin Sheep's Walking Posture and Body Size.

The ability to recognize the body sizes of sheep is significantly influenced by posture, especially without artificial fixation, leading to more noticeable changes. This study presents a recognition model using the Mask R-CNN convolutional neural network to identify the sides and backs of sheep. The proposed approach includes an algorithm for extracting key frames through mask calculation and specific algorithms for head-down, head-up, and jumping postures of Ujumqin sheep. The study reported an accuracy of 94.70% in posture classification. We measured the body size parameters of Ujumqin sheep of different sexes and in different walking states, including observations of head-down and head-up. The errors for the head-down position of rams, in terms of body slanting length, withers height, hip height, and chest depth, were recorded as 0.08 ± 0.06, 0.09 ± 0.07, 0.07 ± 0.05, and 0.12 ± 0.09, respectively. For rams in the head-up position, the corresponding errors were 0.06 ± 0.05, 0.06 ± 0.05, 0.07 ± 0.05, and 0.13 ± 0.07, respectively. The errors for the head-down position of ewes, in terms of body slanting length, withers height, hip height, and chest depth, were recorded as 0.06 ± 0.05, 0.09 ± 0.08, 0.07 ± 0.06, and 0.13 ± 0.10, respectively. For ewes in the head-up position, the corresponding errors were 0.06 ± 0.05, 0.08 ± 0.06, 0.06 ± 0.04, and 0.16 ± 0.12, respectively. The study observed that sheep walking through a passage exhibited a more curved knee posture compared to normal measurements, often with a lowered head. This research presents a cost-effective data collection scheme for studying multiple postures in animal husbandry.

Deep learning application of vertebral compression fracture detection using mask R-CNN

Vertebral compression fractures (VCFs) of the thoracolumbar spine are commonly caused by osteoporosis or result from traumatic events. Early diagnosis of vertebral compression fractures can prevent further damage to patients. When assessing these fractures, plain radiographs are used as the primary diagnostic modality. In this study, we developed a deep learning based fracture detection model that could be used as a tool for primary care in the orthopedic department. We constructed a VCF dataset using 487 lateral radiographs, which included 598 fractures in the L1-T11 vertebra. For detecting VCFs, Mask R-CNN model was trained and optimized, and was compared to three other popular models on instance segmentation, Cascade Mask R-CNN, YOLOACT, and YOLOv5. With Mask R-CNN we achieved highest mean average precision score of 0.58, and were able to locate each fracture pixel-wise. In addition, the model showed high overall sensitivity, specificity, and accuracy, indicating that it detected fractures accurately and without misdiagnosis. Our model can be a potential tool for detecting VCFs from a simple radiograph and assisting doctors in making appropriate decisions in initial diagnosis.