Mask R-CNN Research Articles

Semantic segmentation and thermal imaging for forest fires detection and monitoring by drones

Forest ecosystems play a crucial role in providing a wide range of ecological, social, and economic benefits. However, the increasing frequency and severity of forest fires pose a significant threat to the sustainability of forests and their functions, highlighting the need for early detection and swift action to mitigate damage. The combination of drones and artificial intelligence, particularly deep learning, proves to be a cost-effective solution for accurately and efficiently detecting forest fires in real-time. Deep learning-based image segmentation models can not only be employed for forest fire detection but also play a vital role in damage assessment and support reforestation efforts. Furthermore, the integration of thermal cameras on drones can significantly enhance the sensitivity in forest fire detection. This study undertakes an in-depth analysis of recent advancements in deep learning-based semantic segmentation, with a particular focus on model’s mask region convolutional neural network (Mask R-CNN) and you only look once (YOLO) v5, v7, and v8 variants. Emphasis is placed on their suitability for forest fire monitoring using drones equipped with RGB and/or thermal cameras. The conducted experiments have yielded encouraging outcomes across various metrics, underscoring its significance as an invaluable asset for both fire detection and continuous monitoring endeavors.

A Temperature Monitoring Method for Sensor Arrays Based on Temperature Mapping and Improved Mask R-CNN

A Temperature Monitoring Method for Sensor Arrays Based on Temperature Mapping and Improved Mask R-CNN

Two deep learning methods in comparison to characterize droplet sizes in emulsification flow processes

AbstractFor reliable supervision in multiphase processes, the droplet size represents a critical quality attribute and needs to be monitored. A promising approach is the use of smart image flow sensors since optical measurement is the most commonly used technique for droplet size distribution determination. For this, two different AI-based object detection methods, Mask RCNN and YOLOv4, are compared regarding their accuracy and their applicability to an emulsification flow process. Iterative optimization steps, including data diversification and adaption of training parameters, enable the models to achieve robust detection performance across varying image qualities and compositions. YOLOv4 shows better detection performances and more accurate results which leads to a wider application window than Mask RCNN in determining droplet sizes in emulsification processes. The final droplet detection model YOLOv4 with Hough Circle (HC) for feature extraction determines reliable droplet sizes across diverse datasets of liquid-liquid flow systems (disperse phase content 1–15 vol.-%, droplet size range 5–150 μm). Evaluating the adjustment of Confidence Scores (CS) ensures statistical representation of even smaller droplets. The droplet detection performance of the final YOLOv4 model is compared with a manual image processing method to validate the model in general as well as its accuracy and reliability. Since YOLOv4 in combination with Hough Circle (HC) shows an accurate and robust detection and size determination, it is applicable for online monitoring and characterization of various liquid-liquid flow processes. Graphical abstract

GRA-Net: Group response attention for deep learning

Activation function, one of the most critical components in deep learning, enables the artificial neural networks to learn complex patterns through nonlinearity. Currently, element-wise activation functions such as ReLU are widely utilized because of their simplicity and efficiency. However, these methods do not consider the interactions among neurons, which may produce redundant features and affect performance. Inspired by neuroscience, we propose Group Response Attention (GRA). It establishes connections among neurons, which means that the state of each member is determined by the entire group. Accordingly, GRA makes the features more diverse and reduces redundancy. In addition, we propose a pure attention network without any conventional activation function, termed as GRA-Net. Experiments conducted on typical computer vision tasks show that GRA-Net achieves state-of-the-art results. For the ImageNet-1K classification task, GRA-Net-T achieves 82.9% Top-1 accuracy. With comparable parameters and FLOPs, its performance improves by 1.6% and 0.8% over Swin-T and ConvNeXt-T. On the COCO detection task, as the backbone of Mask R-CNN, GRA-Net-T outperforms Swin-T and ConvNeXt-T by 2.1% and 2.0% on box AP. Meanwhile, the mIoU of GRA-Net-T on the ADE20K semantic segmentation task is higher than Swin-T and ConvNeXt-T by 1.9% and 1.0%.

Leaf only SAM: A segment anything pipeline for zero-shot automated leaf segmentation

Segment Anything Model (SAM) is a new “foundation model” that can be used as a zero-shot object segmentation method with the use of either guide prompts such as bounding boxes, polygons, or points. Alternatively, additional post processing steps can be used to identify objects of interest after segmenting everything in an image. Here a method is presented using segment anything together with a series of post processing steps to segment potato leaves, called Leaf Only SAM. The advantage of this proposed method is that it does not require any training data to produce its results so has many applications across the field of plant phenotyping where there is limited high quality annotated data available. The performance of Leaf Only SAM is compared to a Mask R-CNN model which has been fine-tuned on a small novel potato leaf dataset. On the evaluation dataset, Leaf Only SAM finds an average recall of 73.1 and an average precision of 73.9, compared to recall of 87.6 and precision of 84.4 for Mask R-CNN. Leaf Only SAM does not perform better than the fine-tuned Mask R-CNN model on the potato leaf dataset, but the SAM based model does not require any extra training or annotation. This shows there is potential to use SAM as a zero-shot classifier with the addition of post processing steps.

Target reconstruction and process parameter decision-making for bolt intelligent assembly based on robot and multi-camera

Bolt assembly is widely used in modern manufacturing. And with the development of industrial automation, there is a growing demand for automatic assembly. To form a fully automatic and highly intelligent system for bolt assembly, a novel strategy based robot and multi-camera is proposed. In this strategy, a high assurance components reconstruction scheme is designed, wherein the Mask R-CNN is used for the identification and segmentation of the workpiece, then the shape, size and pose of the workpiece are determined by the local point cloud data and its envelope using basic geometries. The results show that the accuracy of workpiece recognition exceeds 96 %, the maximum positioning accuracy of the hole is 0.334 mm based on multi-camera system. Based on the historical fastening process data, an improved artificial jellyfish search optimizer (AJSO) BP neural network is proposed to automatically determine the optimal values for the bolt fastening speed and torque. The BP neural network optimized by AJSO reduces the maximum prediction error of process parameters from 0.117 mm/s to 0.051 mm/s for speed and from 0.052 N·m to 0.027 N·m for torque. Finally, the feasibility of the proposed strategy is demonstrated by an experimental case, which used a dual robot unit for the bearing seat assembly.

A Method for Extracting Joints on Mountain Tunnel Faces Based on Mask R-CNN Image Segmentation Algorithm

The accurate distribution of joints on the tunnel face is crucial for assessing the stability and safety of surrounding rock during tunnel construction. This paper introduces the Mask R-CNN image segmentation algorithm, a state-of-the-art deep learning model, to achieve efficient and accurate identification and extraction of joints on tunnel face images. First, digital images of tunnel faces were captured and stitched, resulting in 286 complete images suitable for analysis. Then, the joints on the tunnel face were extracted using traditional image processing algorithms, the commonly used U-net image segmentation model, and the Mask R-CNN image segmentation model introduced in this paper to address the lack of recognition accuracy. Finally, the extraction results obtained by the three methods were compared. The comparison results show that the joint extraction method based on the Mask R-CNN image segmentation deep learning model introduced in this paper achieved the best joint extraction effect with a Dice similarity coefficient of 87.48%, outperforming traditional methods and the U-net model, which scored 60.59% and 75.36%, respectively, realizing accurate and efficient acquisition of tunnel face rock joints. These findings suggest that the Mask R-CNN model can be effectively implemented in real-time monitoring systems for tunnel construction projects.

An exploratory framework to identify dust on photovoltaic panels in offshore floating solar power stations

Offshore floating solar power stations represent a new frontier in energy development. These stations maximize solar energy use, reduce land consumption and promote algae growth. However, moist marine air makes airborne particles like sand adhere more easily to PV panels, reducing photoelectric conversion efficiency and causing overheating and potential damage. Additionally, drones and fixed cameras face challenges due to complex maritime factors and complicating image analysis. To address these issues, this paper proposes an exploratory framework for identifying dust regions on photovoltaic panels specifically for offshore floating solar power stations. The framework aims to address how to accurately and reliably identify dust accumulation on PV panels, even in images with complex background interference. The framework is designed to be compatible with existing image recognition models and incorporates specialized enhancement modules to further improve recognition accuracy and efficiency. In this study, we utilized a dust feature enhancement module based on the HLS color space, combined with Mask R-CNN, to demonstrate the framework's feasibility and effectiveness. Experimental results show that, compared to a single Mask R-CNN, the framework significantly reduces misidentification and omission of dust areas, boosting the confidence level and effectively meeting the operational needs of offshore floating solar power stations.

Pig Weight Estimation Method Based on a Framework Combining Mask R-CNN and Ensemble Regression Model.

Using computer vision technology to estimate pig live weight is an important method to realize pig welfare. But there are two key issues that affect pigs' weight estimation: one is the uneven illumination, which leads to unclear contour extraction of pigs, and the other is the bending of the pig body, which leads to incorrect pig body information. For the first one, Mask R-CNN was used to extract the contour of the pig, and the obtained mask image was converted into a binary image from which we were able to obtain a more accurate contour image. For the second one, the body length, hip width and the distance from the camera to the pig back were corrected by XGBoost and actual measured information. Then we analyzed the rationality of the extracted features. Three feature combination strategies were used to predict pig weight. In total, 1505 back images of 39 pigs obtained using Azure kinect DK were used in the numerical experiments. The highest prediction accuracy is XGBoost, with an MAE of 0.389, RMSE of 0.576, MAPE of 0.318% and R2 of 0.995. We also recommend using the Mask R-CNN + RFR method because it has fairly high precision in each strategy. The experimental results show that our proposed method has excellent performance in live weight estimation of pigs.

Identification and Localization of Wind Turbine Blade Faults Using Deep Learning

This study addresses the challenges inherent in the maintenance and inspection of wind turbines through the application of deep learning methodologies for fault detection on Wind Turbine Blades (WTBs). Specifically, this research focuses on defect detection on the blades of small-scale WTBs due to the unavailability of commercial wind turbines. This research compared popular object localization architectures, YOLO and Mask R-CNN, to identify the most effective model to detect common WTB defects, including cracks, holes, and erosion. YOLOv9 C emerged as the most effective model, with the highest scores of mAP50 and mAP50-95 of 0.849 and 0.539, respectively. Modifications to Mask R-CNN, specifically integrating a ResNet18-FPN network, reduced computational complexity by 32 layers and achieved a mAP50 of 0.8415. The findings highlight the potential of deep learning and computer vision in improving WTB fault analysis and inspection.

Comparing YOLOv8 and Mask R-CNN for instance segmentation in complex orchard environments

Instance segmentation, an important image processing operation for automation in agriculture, is used to precisely delineate individual objects of interest within images, which provides foundational information for various automated or robotic tasks such as selective harvesting and precision pruning. This study compares the one-stage YOLOv8 and the two-stage Mask R-CNN machine learning models for instance segmentation under varying orchard conditions across two datasets. Dataset 1, collected in dormant season, includes images of dormant apple trees, which were used to train multi-object segmentation models delineating tree branches and trunks. Dataset 2, collected in the early growing season, includes images of apple tree canopies with green foliage and immature (green) apples (also called fruitlet), which were used to train single-object segmentation models delineating only immature green apples. The results showed that YOLOv8 performed better than Mask R-CNN, achieving good precision and near-perfect recall across both datasets at a confidence threshold of 0.5. Specifically, for Dataset 1, YOLOv8 achieved a precision of 0.90 and a recall of 0.95 for all classes. In comparison, Mask R-CNN demonstrated a precision of 0.81 and a recall of 0.81 for the same dataset. With Dataset 2, YOLOv8 achieved a precision of 0.93 and a recall of 0.97. Mask R-CNN, in this single-class scenario, achieved a precision of 0.85 and a recall of 0.88. Additionally, the inference times for YOLOv8 were 10.9 ms for multi-class segmentation (Dataset 1) and 7.8 ms for single-class segmentation (Dataset 2), compared to 15.6 ms and 12.8 ms achieved by Mask R-CNN's, respectively. These findings show YOLOv8's superior accuracy and efficiency in machine learning applications compared to two-stage models, specifically Mask-R-CNN, which suggests its suitability in developing smart and automated orchard operations, particularly when real-time applications are necessary in such cases as robotic harvesting and robotic immature green fruit thinning.

Machine Vision Analysis of Ujumqin Sheep's Walking Posture and Body Size.

The ability to recognize the body sizes of sheep is significantly influenced by posture, especially without artificial fixation, leading to more noticeable changes. This study presents a recognition model using the Mask R-CNN convolutional neural network to identify the sides and backs of sheep. The proposed approach includes an algorithm for extracting key frames through mask calculation and specific algorithms for head-down, head-up, and jumping postures of Ujumqin sheep. The study reported an accuracy of 94.70% in posture classification. We measured the body size parameters of Ujumqin sheep of different sexes and in different walking states, including observations of head-down and head-up. The errors for the head-down position of rams, in terms of body slanting length, withers height, hip height, and chest depth, were recorded as 0.08 ± 0.06, 0.09 ± 0.07, 0.07 ± 0.05, and 0.12 ± 0.09, respectively. For rams in the head-up position, the corresponding errors were 0.06 ± 0.05, 0.06 ± 0.05, 0.07 ± 0.05, and 0.13 ± 0.07, respectively. The errors for the head-down position of ewes, in terms of body slanting length, withers height, hip height, and chest depth, were recorded as 0.06 ± 0.05, 0.09 ± 0.08, 0.07 ± 0.06, and 0.13 ± 0.10, respectively. For ewes in the head-up position, the corresponding errors were 0.06 ± 0.05, 0.08 ± 0.06, 0.06 ± 0.04, and 0.16 ± 0.12, respectively. The study observed that sheep walking through a passage exhibited a more curved knee posture compared to normal measurements, often with a lowered head. This research presents a cost-effective data collection scheme for studying multiple postures in animal husbandry.

Deep learning application of vertebral compression fracture detection using mask R-CNN

Vertebral compression fractures (VCFs) of the thoracolumbar spine are commonly caused by osteoporosis or result from traumatic events. Early diagnosis of vertebral compression fractures can prevent further damage to patients. When assessing these fractures, plain radiographs are used as the primary diagnostic modality. In this study, we developed a deep learning based fracture detection model that could be used as a tool for primary care in the orthopedic department. We constructed a VCF dataset using 487 lateral radiographs, which included 598 fractures in the L1-T11 vertebra. For detecting VCFs, Mask R-CNN model was trained and optimized, and was compared to three other popular models on instance segmentation, Cascade Mask R-CNN, YOLOACT, and YOLOv5. With Mask R-CNN we achieved highest mean average precision score of 0.58, and were able to locate each fracture pixel-wise. In addition, the model showed high overall sensitivity, specificity, and accuracy, indicating that it detected fractures accurately and without misdiagnosis. Our model can be a potential tool for detecting VCFs from a simple radiograph and assisting doctors in making appropriate decisions in initial diagnosis.

Multimodal rapid identification of growth stages and discrimination of growth status for Morchella

We introduce a multimodal rapid identification and growth status discrimination method for morchella. Based on the unique biological characteristics and growth environmental requirements of morchella, the efficient and accurate identification of key growth stages of morchella is achieved through the integration of multimodal information acquisition technology. During the rapid identification process of the growth stage of Morchella, the Multi Stage Vision Enhanced Position Encoding Vision Transformer (MS-EP ViT) model is adopted. By introducing multi-stage input embedding, enhanced position encoding, and optimized Transformer Encoder layers, the performance of the model in identifying different growth stages of Morchella mushrooms is significantly improved. In the multimodal Morchella growth state discrimination method, text and image modalities are integrated, a Non downsampled Contourlet Transform Mask Region based Convolutional Neural Network (NSCT Mask R-CNN) model is designed, and a multimodal feature extraction strategy combining Non downsampled Contourlet Transform (NSCT) features with environmental features is explored. This strategy effectively achieves the goals of object detection and instance segmentation, and thus we have accurately evaluated the growth status of Morchella in the later stages of mulberry, young mushroom, and mature. The experimental results show that both models have achieved significant improvements in recognition accuracy and stability, and the rationality of hyperparameter settings has been verified through convergence and parameter sensitivity experiments. Overall, we provide a more accurate and efficient identification method for monitoring the growth of Morchella, which helps to better understand the growth of Morchella and provides scientific basis for optimizing its growth environment.

TTIS-YOLO: a traffic target instance segmentation paradigm for complex road scenarios

Abstract The instance segmentation of traffic targets in complex road scenes is one of the most challenging tasks in autonomous driving. Unlike the bounding box localization for object detection and the category perception mask for semantic segmentation, instance segmentation requires accurate identification of each object under each category and more precise segmentation and positioning of these target objects. Although instance segmentation has apparent advantages, methods, for instance segmentation in complex road scenes, still need to be discovered. In this paper, we proposed an efficient instance segmentation method TTIS-YOLO(Traffic Target Instance Segmentation - YOLO) based on YOLOV5-7.0 for traffic object segmentation of complex road scenes. Our main work is as follows: to propose a MECSP(Multiscale Efficient Cross Stage Partial Network) module, which has fewer parameters, better cross-layer information exchange, and feature representation capabilities. Propose an Efficient Bidirectional Cross Scale Connection optimization method that enables the network to perform more detailed and efficient feature fusion without losing original information, refining the mask flow. WIoU Loss is used as the loss function of positioning and segmentation, and the positioning performance of the model is effectively improved through the strategy of dynamically allocating gradient gains. Experiments have shown that our proposed TTIS-YOLO outperforms baseline models and other mainstream instances segmentation algorithms such as Mask RCNN, YOLACT, SOLO, and SOLOV2 with the highest segmentation accuracy and fastest inference speed. Our proposed TTIS-YOLO-S achieves the best balance between segmentation accuracy and inference speed. Compared to the baseline model, the AP50 and recall values on the Cityscapes validation set increased by 1.7% and 0.9%, respectively, with a parameter reduction of 20.6%, inference speed of 78.1fps on GeForce RTX 3090Ti. Meanwhile, TTIS-YOLO-L achieved the highest segmentation accuracy, with an AP50 value of 27%, and the model parameter quantity decreased by 35.4% compared to the baseline model.

Manipulation of a Complex Object Using Dual-Arm Robot with Mask R-CNN and Grasping Strategy

Hot forging is one of the common manufacturing processes for producing brass workpieces. However forging produces flash which is a thin metal part around the desired part formed with an excessive material. Using robots with vision system to manipulate this workpiece has encountered several challenging issues, e.g. the uncertain shape of flash, color, reflection of brass surface, different lighting condition, and the uncertainty surrounding the position and orientation of the workpiece. In this research, Mask region-based convolutional neural network together with image processing is used to resolve these issues. The depth camera can provide images for visual detection. Machine learning Mask region-based convolutional neural network model was trained with color images and the position of the object is determined by the depth image. A dual arm 7 degree of freedom collaborative robot with proposed grasping strategy is used to grasp the workpiece that can be in inappropriate position and pose. Eventually, experiments were conducted to assess the visual detection process and the grasp planning of the robot.

Detecting Internal Defects in FRP-Reinforced Concrete Structures through the Integration of Infrared Thermography and Deep Learning.

This study represents a significant advancement in structural health monitoring by integrating infrared thermography (IRT) with cutting-edge deep learning techniques, specifically through the use of the Mask R-CNN neural network. This approach targets the precise detection and segmentation of hidden defects within the interfacial layers of Fiber-Reinforced Polymer (FRP)-reinforced concrete structures. Employing a dual RGB and thermal camera setup, we captured and meticulously aligned image data, which were then annotated for semantic segmentation to train the deep learning model. The fusion of the RGB and thermal imaging significantly enhanced the model's capabilities, achieving an average accuracy of 96.28% across a 5-fold cross-validation. The model demonstrated robust performance, consistently identifying true negatives with an average specificity of 96.78% and maintaining high precision at 96.42% in accurately delineating damaged areas. It also showed a high recall rate of 96.91%, effectively recognizing almost all actual cases of damage, which is crucial for the maintenance of structural integrity. The balanced precision and recall culminated in an average F1-score of 96.78%, highlighting the model's effectiveness in comprehensive damage assessment. Overall, this synergistic approach of combining IRT and deep learning provides a powerful tool for the automated inspection and preservation of critical infrastructure components.

Research on Segmentation Method of Maize Seedling Plant Instances Based on UAV Multispectral Remote Sensing Images.

The accurate instance segmentation of individual crop plants is crucial for achieving a high-throughput phenotypic analysis of seedlings and smart field management in agriculture. Current crop monitoring techniques employing remote sensing predominantly focus on population analysis, thereby lacking precise estimations for individual plants. This study concentrates on maize, a critical staple crop, and leverages multispectral remote sensing data sourced from unmanned aerial vehicles (UAVs). A large-scale SAM image segmentation model is employed to efficiently annotate maize plant instances, thereby constructing a dataset for maize seedling instance segmentation. The study evaluates the experimental accuracy of six instance segmentation algorithms: Mask R-CNN, Cascade Mask R-CNN, PointRend, YOLOv5, Mask Scoring R-CNN, and YOLOv8, employing various combinations of multispectral bands for a comparative analysis. The experimental findings indicate that the YOLOv8 model exhibits exceptional segmentation accuracy, notably in the NRG band, with bbox_mAP50 and segm_mAP50 accuracies reaching 95.2% and 94%, respectively, surpassing other models. Furthermore, YOLOv8 demonstrates robust performance in generalization experiments, indicating its adaptability across diverse environments and conditions. Additionally, this study simulates and analyzes the impact of different resolutions on the model's segmentation accuracy. The findings reveal that the YOLOv8 model sustains high segmentation accuracy even at reduced resolutions (1.333 cm/px), meeting the phenotypic analysis and field management criteria.

Enhancements in image quality and block detection performance for Reinforced Soil-Retaining Walls under various illuminance conditions

To ensure continuous monitoring of reinforced soil-retaining walls (RSWs) even under low-illuminance conditions, such as during the night, it is imperative to evaluate the performance of deep learning-based detection. In this study, we constructed a laboratory RSW model and generated a dataset with varying illuminance levels to assess the impact of image enhancement and block detection. Various image enhancement methods were applied to improve image quality and evaluate their effect on deep learning. RGB optimization (RO) was proposed to optimize RGB intensity and compared with gamma correction, histogram equalization, and low-light image enhancement with illumination map estimation. RO demonstrated outstanding image enhancement performance, as evidenced by lightness order error, peak signal-to-noise ratio, and structural similarity index measure, ensuring high image quality. The trained RO model using Mask R-CNN exhibited excellent accuracy, recall, and F1 score, delivering remarkable detection performance under low illuminance conditions, resulting in a 7.44 % improvement in the F1 score. Image enhancement techniques that maintain similarity, such as lightness order error and structural similarity, across varying illuminance conditions contribute to enhancing the block detection performance of Mask R-CNNs.

ALdamage-seg: A Lightweight Model for Instance Segmentation of Aluminum Profiles

Aluminum profiles are widely used in various manufacturing sectors due to their flexibility and chemical properties. However, these profiles are susceptible to defects during manufacturing and transportation. Detecting these defects is crucial, but existing object detection models like Mask R-CNN and YOLOv8-seg are not optimized for this task. These models are large and computationally intensive, making them unsuitable for edge devices used in industrial inspections. To address this issue, this study proposes a novel lightweight instance segmentation model called AL-damage-seg, inspired by the YOLOv8n-seg architecture. This model utilizes MobileNetV3 as the backbone. In YOLOv8n-seg, the role of C2f is to enhance the nonlinear representation of the model to capture complex image features more efficiently. We upgraded and improved it to form multilayer feature extraction module (MFEM) and integrates a large separable kernel attention (LSKA) mechanism in the C2f module, resulting in C2f_LSKA, to further optimize the performance of the model. Additionally, depth-wise separable convolutions are employed in the feature fusion process. The ALdamage-seg’s weight on the Alibaba Tian-chi aluminum profile dataset constitutes 43.9% of that of YOLOv8n-seg, with its GFLOPs reduced to 53% relative to YOLOv8-seg, all the while achieving an average precision (mAP) of 99% relative to YOLOv8-seg. With its compact size and lower computational requirements, this model is well-suited for deployment on edge devices with limited processing capabilities.