Pixel-level Labels Research Articles

An One-step Triple Enhanced weakly supervised semantic segmentation using image-level labels.

Weakly supervised semantic segmentation, based on image-level labels, abandons the pixel-level labels relied upon by traditional semantic segmentation algorithms. It only utilizes images as supervision information, thereby reducing the time cost and human resources required for marking pixel data. The prevailing approach in weakly supervised segmentation involves two-step method, introducing an additional network and numerous parameters, thereby complicating the model structure. Furthermore, image-level labels typically furnishes only category information for the entire image, lacking specific location details and accurate target boundaries during model training. We propose an innovative One-Step Triple Enhanced weakly supervised semantic segmentation network(OSTE). OSTE streamlines the model structure, which can accomplish both pseudo-labels generation and semantic segmentation tasks in just one step. Furthermore, we augment the weakly supervised semantic segmentation network in three key aspects based on the class activation map construction method, thereby enhancing segmentation accuracy: Firstly, by integrating local information from the activation map with the image, we can enhance the network's localization and expansion capabilities to obtain more accurate and rich location information. Then, we refine the seed areas of the class activation map by exploiting the correlation between multi-level feature. Finally, we incorporate conditional random field theory to generate pseudo-labels with higher confidence and richer boundary information. In comparison to the prevailing two-step weakly supervised semantic segmentation schemes, the segmentation network proposed in this paper achieves a more competitive mean Intersection over Union (mIoU) score of 58.47% on Pascal VOC. Additionally, it enhances the mIoU score by at least 5.03% when compared to existing end-to-end schemes.

Segmentation-assisted classification model with convolutional neural network for weld defect detection

Detecting weld defects in battery trays is crucial for the safety of new energy vehicles. Existing methods for weld surface defect detection, relying on traditional computer vision algorithms and convolutional neural networks with substantial image-level labeled data, face challenges in accurately identifying small defects, especially with limited samples. To address these issues, we developed an innovative Segmentation-Assisted Classification with Convolutional Neural Networks (SACNN) model. SACNN integrates a common feature extraction subnet, a segmentation subnet enhanced by a multi-scale feature fusion module, and a classification subnet specifically designed for precise defect detection. A joint loss function co-trains the segmentation and classification subnets using both image-level and pixel-level labels, enhancing the model’s ability to accurately detect small defect regions. Our model demonstrates notable improvement, achieving accuracy gains ranging from 2% to 18% compared to existing state-of-the-art methods, with an overall accuracy of 94.09% on an industrial dataset of battery tray welds. To further evaluate the generalization capability of our model, we evaluated it on the publicly available Magnetic Tile dataset, achieving state-of-the-art results in this challenging context. Additionally, we conducted comprehensive ablation studies to validate the contribution of each component in our approach and utilized visualization techniques to enhance the interpretability of our model. These advancements represent a significant contribution to the state of the art in aluminum alloy weld defect detection.

WSSS-CRAM: precise segmentation of histopathological images via class region activation mapping.

Fast, accurate, and automatic analysis of histopathological images using digital image processing and deep learning technology is a necessary task. Conventional histopathological image analysis algorithms require the manual design of features, while deep learning methods can achieve fast prediction and accurate analysis, but rely on the drive of a large amount of labeled data. In this work, we introduce WSSS-CRAM a weakly-supervised semantic segmentation that can obtain detailed pixel-level labels from image-level annotated data. Specifically, we use a discriminative activation strategy to generate category-specific image activation maps via class labels. The category-specific activation maps are then post-processed using conditional random fields to obtain reliable regions that are directly used as ground-truth labels for the segmentation branch. Critically, the two steps of the pseudo-label acquisition and training segmentation model are integrated into an end-to-end model for joint training in this method. Through quantitative evaluation and visualization results, we demonstrate that the framework can predict pixel-level labels from image-level labels, and also perform well when testing images without image-level annotations. Future, we consider extending the algorithm to different pathological datasets and types of tissue images to validate its generalization capability.

Decoupling foreground and background with Siamese ViT networks for weakly-supervised semantic segmentation

Due to the coarse granularity of information extraction in image-level annotation-based weakly supervised semantic segmentation algorithms, there exists a significant gap between the generated pseudo-labels and the real pixel-level labels. In this paper, we propose the DeFB-SV framework, which consists of a dual-branch Siamese network structure. This framework separates the foreground and background of images by generating unified resolution and mixed resolution class activation maps, which are then fused to obtain pseudo-labels. The mixed-resolution class activation maps are produced by a new mixed-resolution patch partition method, where we introduce a semantically heuristic patch scorer to divide the image into patches of different sizes based on semantics. Additionally, a novel multi-confidence region division mechanism is proposed to enable the adaptive extraction of the effective parts of pseudo-labels, further enhancing the accuracy of weakly supervised semantic segmentation algorithms. The proposed semantic segmentation framework, DeFB-SV, is evaluated on the PASCAL VOC 2012 and MS COCO 2014 datasets, demonstrating comparable segmentation performance with state-of-the-art methods.

AeroClick: An advanced single-click interactive framework for aeroengine defect segmentation

In aeroengine defect inspection, traditional algorithms are often encumbered by overreliance on pixel-level labels and an inability to adapt to variations in domain-specific data, resulting in a lack of efficiency and generalizability. To address these challenges, this paper proposes an innovative solution using a single-click interactive framework that combines user-friendly interaction and computational efficiency. First, user clicks are incorporated into an additional channel coupled with the original images. Second, superpixel-guided interactive signals and corresponding Gaussian heat maps are utilized to embed user clicks into the training and prediction processes, tolerating certain click errors without additional corrections. Third, the tailored backbone handles defect variations and captures intricate details. Furthermore, the Bayesian-optimization-based refinement process further enhances accuracy and generalizability. The experimental results from four industrial scenarios demonstrate that our method not only excels in aeroengine defect segmentation but also has the potential to be effectively applied as a semi-automatic annotation tool in other industrial inspection fields.

Multi-scale feature correspondence and pseudo label retraining strategy for weakly supervised semantic segmentation

Recently, the performance of semantic segmentation using weakly supervised learning has significantly improved. Weakly supervised semantic segmentation (WSSS) that uses only image-level labels has received widespread attention, it employs Class Activation Maps (CAM) to generate pseudo labels. Compared to traditional use of pixel-level labels, this technique greatly reduces annotation costs by utilizing simpler and more readily available image-level annotations. Besides, due to the local perceptual ability of Convolutional Neural Networks (CNN), the generated CAM cannot activate the entire object area. Researchers have found that this CNN limitation can be compensated for by using Vision Transformer (ViT). However, ViT also introduces an over-smoothing problem. Recent research has made good progress in solving this issue, but when discussing CAM and its related segmentation predictions, it is easy to overlook their intrinsic information and the interrelationships between them. In this paper, we propose a Multi-Scale Feature Correspondence (MSFC) method. Our MSFC can obtain the feature correspondence of CAM and segmentation predictions at different scales, re-extract useful semantic information from them, enhancing the network's learning of feature information and improving the quality of CAM. Moreover, to further improve the segmentation precision, we design a Pseudo Label Retraining Strategy (PLRS). This strategy refines the accuracy in local regions, elevates the quality of pseudo labels, and aims to enhance segmentation precision. Experimental results on the PASCAL VOC 2012 and MS COCO 2014 datasets show that our method achieves impressive performance among end-to-end WSSS methods.

Image-level supervision and self-training for transformer-based cross-modality tumor segmentation

Deep neural networks are commonly used for automated medical image segmentation, but models will frequently struggle to generalize well across different imaging modalities. This issue is particularly problematic due to the limited availability of annotated data, both in the target as well as the source modality, making it difficult to deploy these models on a larger scale. To overcome these challenges, we propose a new semi-supervised training strategy called MoDATTS. Our approach is designed for accurate cross-modality 3D tumor segmentation on unpaired bi-modal datasets. An image-to-image translation strategy between modalities is used to produce synthetic but annotated images and labels in the desired modality and improve generalization to the unannotated target modality. We also use powerful vision transformer architectures for both image translation (TransUNet) and segmentation (Medformer) tasks and introduce an iterative self-training procedure in the later task to further close the domain gap between modalities, thus also training on unlabeled images in the target modality. MoDATTS additionally allows the possibility to exploit image-level labels with a semi-supervised objective that encourages the model to disentangle tumors from the background. This semi-supervised methodology helps in particular to maintain downstream segmentation performance when pixel-level label scarcity is also present in the source modality dataset, or when the source dataset contains healthy controls. The proposed model achieves superior performance compared to other methods from participating teams in the CrossMoDA 2022 vestibular schwannoma (VS) segmentation challenge, as evidenced by its reported top Dice score of 0.87±0.04 for the VS segmentation. MoDATTS also yields consistent improvements in Dice scores over baselines on a cross-modality adult brain gliomas segmentation task composed of four different contrasts from the BraTS 2020 challenge dataset, where 95% of a target supervised model performance is reached when no target modality annotations are available. We report that 99% and 100% of this maximum performance can be attained if 20% and 50% of the target data is additionally annotated, which further demonstrates that MoDATTS can be leveraged to reduce the annotation burden.

From image-level to pixel-level labeling: A weakly-supervised learning method for identifying aquaculture ponds using iterative anti-adversarial attacks guided by aquaculture features

Aquaculture mapping is essential for monitoring and managing aquaculture resources. However, accurately geotargeting individual aquaculture ponds from medium-resolution remote sensing imagery remains challenging, and convolutional deep learning methods for identifying aquaculture ponds require labor-intensive pixel-level annotations. This paper presents a novel weakly-supervised learning method to derive pixel-level labels from image-level annotations for aquaculture ponds. Our approach uses iterative anti-adversarial attacks to refine localization results from multi-scale class activation maps (CAMs). The improved method integrates two regularization methods guided by aquaculture features to form a joint loss function for adversarial samples: discriminative water region suppression and non-aquaculture water class suppression. We also propose an aquaculture feature termed CFNDWI to constrain the localization results and generate high-quality pseudo-labels. As a result, the pseudo-labels are used to train semantic segmentation networks for accurately identifying aquaculture ponds. We evaluated the performance of our method using commonly-used backbones on 10 m Sentinel-2 imagery. Our method achieves Intersection over Union (IoU) values of 0.618–0.655 for pseudo-label generation, and IoU values of 0.664–0.708 for semantic segmentation, outperforming state-of-the-art weakly-supervised methods and public datasets. The effectiveness of each module of our method was also testified through ablation experiments. Our method leverages knowledge-driven aquaculture features to guide the data-driven adversarial learning process, addressing the lack of high-quality aquaculture datasets for model training. The code for implementing our method will be accessible at https://github.com/designer1024/WSLM-AQ.

Better performance of deep learning pulmonary nodule detection using chest radiography with pixel level labels in reference to computed tomography: data quality matters

Labeling errors can significantly impact the performance of deep learning models used for screening chest radiographs. The deep learning model for detecting pulmonary nodules is particularly vulnerable to such errors, mainly because normal chest radiographs and those with nodules obscured by ribs appear similar. Thus, high-quality datasets referred to chest computed tomography (CT) are required to prevent the misclassification of nodular chest radiographs as normal. From this perspective, a deep learning strategy employing chest radiography data with pixel-level annotations referencing chest CT scans may improve nodule detection and localization compared to image-level labels. We trained models using a National Institute of Health chest radiograph-based labeling dataset and an AI-HUB CT-based labeling dataset, employing DenseNet architecture with squeeze-and-excitation blocks. We developed four models to assess whether CT versus chest radiography and pixel-level versus image-level labeling would improve the deep learning model’s performance to detect nodules. The models' performance was evaluated using two external validation datasets. The AI-HUB dataset with image-level labeling outperformed the NIH dataset (AUC 0.88 vs 0.71 and 0.78 vs.0.73 in two external datasets, respectively; both p < 0.001). However, the AI-HUB data annotated at the pixel level produced the best model (AUC 0.91 and 0.86 in external datasets), and in terms of nodule localization, it significantly outperformed models trained with image-level annotation data, with a Dice coefficient ranging from 0.36 to 0.58. Our findings underscore the importance of accurately labeled data in developing reliable deep learning algorithms for nodule detection in chest radiography.

Deep Learning-Based Localization and Detection of Malpositioned Nasogastric Tubes on Portable Supine Chest X-Rays in Intensive Care and Emergency Medicine: A Multi-center Retrospective Study.

Malposition of a nasogastric tube (NGT) can lead to severe complications. We aimed to develop a computer-aided detection (CAD) system to localize NGTs and detect NGT malposition on portable chest X-rays (CXRs). A total of 7378 portable CXRs were retrospectively retrieved from two hospitals between 2015 and 2020. All CXRs were annotated with pixel-level labels for NGT localization and image-level labels for NGT presence and malposition. In the CAD system, DeepLabv3 + with backbone ResNeSt50 and DenseNet121 served as the model architecture for segmentation and classification models, respectively. The CAD system was tested on images from chronologically different datasets (National Taiwan University Hospital (National Taiwan University Hospital)-20), geographically different datasets (National Taiwan University Hospital-Yunlin Branch (YB)), and the public CLiP dataset. For the segmentation model, the Dice coefficients indicated accurate delineation of the NGT course (National Taiwan University Hospital-20: 0.665, 95% confidence interval (CI) 0.630-0.696; National Taiwan University Hospital-Yunlin Branch: 0.646, 95% CI 0.614-0.678). The distance between the predicted and ground-truth NGT tips suggested accurate tip localization (National Taiwan University Hospital-20: 1.64 cm, 95% CI 0.99-2.41; National Taiwan University Hospital-Yunlin Branch: 2.83 cm, 95% CI 1.94-3.76). For the classification model, NGT presence was detected with high accuracy (area under the receiver operating characteristic curve (AUC): National Taiwan University Hospital-20: 0.998, 95% CI 0.995-1.000; National Taiwan University Hospital-Yunlin Branch: 0.998, 95% CI 0.995-1.000; CLiP dataset: 0.991, 95% CI 0.990-0.992). The CAD system also detected NGT malposition with high accuracy (AUC: National Taiwan University Hospital-20: 0.964, 95% CI 0.917-1.000; National Taiwan University Hospital-Yunlin Branch: 0.991, 95% CI 0.970-1.000) and detected abnormal nasoenteric tube positions with favorable performance (AUC: 0.839, 95% CI 0.807-0.869). The CAD system accurately localized NGTs and detected NGT malposition, demonstrating excellent potential for external generalizability.

Detection Method of Manipulator Grasp Pose Based on RGB-D Image

In order to better solve the visual detection problem of manipulator grasping non-cooperative targets, we propose a method of grasp pose detection based on pixel point and feature fusion. By using the improved U2net network as the backbone for feature extraction and feature fusion of the input image, and the grasp prediction layer detects the grasp pose on each pixel. In order to adapt the U2net to grasp pose detection and improve its detection performance, we improve detection speed and control sampling depth by simplifying its network structure, while retaining some shallow features in feature fusion to enhance its feature extraction capability. We introduce depthwise separable convolution in the grasp prediction layer, further fusing the features extracted from the backbone to obtain predictive feature maps with stronger feature expressiveness. FocalLoss is selected as the loss function to solve the problem of unbalanced positive and negative samples in network training. We use the Cornell dataset for training and testing, perform pixel-level labeling on the image, and replace the labels that are not conducive to the actual grasping. This adaptation helps the dataset better suit the network training and testing while meeting the real-world grasping requirements of the manipulator. The evaluation results on image-wise and object-wise are 95.65% and 91.20% respectively, and the detection speed is 0.007 s/frame. We also used the method for actual manipulator grasping experiments. The results show that our method has improved accuracy and speed compared with previous methods, and has strong generalization ability and portability.

Spatial and geometric learning for classification of breast tumors from multi-center ultrasound images: a hybrid learning approach

BackgroundBreast cancer is the most common cancer among women, and ultrasound is a usual tool for early screening. Nowadays, deep learning technique is applied as an auxiliary tool to provide the predictive results for doctors to decide whether to make further examinations or treatments. This study aimed to develop a hybrid learning approach for breast ultrasound classification by extracting more potential features from local and multi-center ultrasound data.MethodsWe proposed a hybrid learning approach to classify the breast tumors into benign and malignant. Three multi-center datasets (BUSI, BUS, OASBUD) were used to pretrain a model by federated learning, then every dataset was fine-tuned at local. The proposed model consisted of a convolutional neural network (CNN) and a graph neural network (GNN), aiming to extract features from images at a spatial level and from graphs at a geometric level. The input images are small-sized and free from pixel-level labels, and the input graphs are generated automatically in an unsupervised manner, which saves the costs of labor and memory space.ResultsThe classification AUCROC of our proposed method is 0.911, 0.871 and 0.767 for BUSI, BUS and OASBUD. The balanced accuracy is 87.6%, 85.2% and 61.4% respectively. The results show that our method outperforms conventional methods.ConclusionsOur hybrid approach can learn the inter-feature among multi-center data and the intra-feature of local data. It shows potential in aiding doctors for breast tumor classification in ultrasound at an early stage.

Unified weakly and semi-supervised crack segmentation framework using limited coarse labels

Obtaining extensive, high-quality datasets for crack segmentation with pixel-level labels is expensive and labor-intensive. The Unified Weakly and Semi-supervised Crack Segmentation (UWSCS) framework addresses this challenge by leveraging a limited number of images with coarse labels and a larger set of unlabeled images. Two label correction modules, based on super-pixel segmentation and a shrink module, are incorporated in the model training to improve crack label accuracy and optimize edge refinement. UWSCS employs a dual-encoder fusion network, combining transformers and convolutional neural networks, to enhance crack segmentation in complex backgrounds. An enhanced algorithm using the medial axis transform is proposed for accurately quantifying crack length and width. Extensive experiments were conducted on both synthetic and real crack datasets to validate the superior performance of UWSCS. The results underscore the significant impact of label quality and quantity used in training on model prediction accuracy. Trained on a concrete crack dataset with limited coarse labels, UWSCS achieves an Intersection of Union (IoU) of 77.53%, surpassing the fully supervised model using the same number of coarse labels by 28.64%. It closely approaches the performance of a fully supervised model with the same number of fine labels (IoU of 80.21%). UWSCS outperforms other advanced networks and semi-supervised/weakly supervised algorithms when trained with a limited set of more cost-effective manually labeled coarse labels. Integrated with the crack segmentation network, super-pixel segmentation, and shrink modules during training, UWSCS with limited coarse labels performs similarly to a fully supervised model using fine labels, thereby reducing manual labeling costs by over 90% and enhancing detection efficiency in practical engineering.

P D N: A Priori Dictionary Network for Fashion Parsing

The task of fashion parsing aims to assign pixel-level labels to clothing targets; thereby, parsing models are required to have good contextual recognition ability. However, the shapes of clothing components are complex, and the types are difficult to distinguish. Recent solutions focus on improving datasets and supplying abundant priori information, but the utilization of features by more efficient methods is rarely explored. In this paper, we propose a multi-scale fashion parsing model called the Priori Dictionary Network (PDN), which includes a priori attention module and a multi-scale backbone. The priori attention module extracts high dimensional features from our designed clothing average template as a priori information dictionary (priori dictionary, PD), and the PD is utilized to activate the feature maps of a CNN from a multi-scale attention mechanism. The backbone is derived from classical models, and five side paths are designed to leverage the richer features of local and global contextual representations. To measure the performance of our method, we evaluated the model on four public datasets, the CFPD, UTFR-SBD3, ModaNet and LIP, and the experimental results show that our model stands out from other State of the Art in all four datasets. This method can assist with the labeling problem of clothing datasets.

Enhancing point annotations with superpixel and confident learning guided for improving semi-supervised OCT fluid segmentation

Automatic segmentation of fluid in Optical Coherence Tomography (OCT) images is beneficial for ophthalmologists to make an accurate diagnosis. Although semi-supervised OCT fluid segmentation networks enhance their performance by introducing additional unlabeled data, the performance enhancement is limited. To address this, we propose Superpixel and Confident Learning Guide Point Annotations Network (SCLGPA-Net) based on the teacher-student architecture, which can learn OCT fluid segmentation from limited fully-annotated data and abundant point-annotated data. Specifically, we use points to annotate fluid regions in unlabeled OCT images and the Superpixel-Guided Pseudo-Label Generation (SGPLG) module generates pseudo-labels and pixel-level label trust maps from the point annotations. The label trust maps provide an indication of the reliability of the pseudo-labels. Furthermore, we propose the Confident Learning Guided Label Refinement (CLGLR) module identifies error information in the pseudo-labels and leads to further refinement. Experiments on the RETOUCH dataset show that we are able to reduce the need for fully-annotated data by 94.22%, closing the gap with the best fully supervised baselines to a mean IoU of only 2%. Furthermore, we constructed a private 2D OCT fluid segmentation dataset for evaluation. Compared with other methods, comprehensive experimental results demonstrate that the proposed method can achieve excellent performance in OCT fluid segmentation.

Lightweight convolutional neural networks for real-time lane segmentation

Autonomous driving depends on reliable perception systems that involve various perception modules and advanced computer vision techniques. One crucial component of these systems is lane detection, which traditional methods often rely on basic features like color or edges that are sensitive to lighting and perspective changes. Recently, convolutional neural networks (CNNs), have revolutionized lane detection. Nevertheless, existing methods still have some limitations, such as the need for pixel-level labeling and computational inefficiency for real-time applications. To address these challenges, this work leverages PP-LiteSeg for real-time semantic segmentation. PP-LiteSegs key elements are its Simple Pyramid Pooling Module (SPPM), Unified Attention Fusion Module (UAFM), and Flexible and Lightweight Decoder (FLD), which optimize lane detection efficiency. The FLD flexibly adjusts computational costs between the encoder and decoder, balancing efficiency and accuracy. The UAFM enhances feature representations using attention mechanisms, increasing segmentation accuracy. The SPPM efficiently aggregates contextual information while reducing computational complexity. The comprehensive method for lane segmentation achieves competitive results on popular lane detection datasets. The proposed model can adapt to different computational capabilities and significantly enhances lane detection efficiency for real-time applications.

Self-supervised pre-training for joint optic disc and cup segmentation via attention-aware network

Image segmentation is a fundamental task in deep learning, which is able to analyse the essence of the images for further development. However, for the supervised learning segmentation method, collecting pixel-level labels is very time-consuming and labour-intensive. In the medical image processing area for optic disc and cup segmentation, we consider there are two challenging problems that remain unsolved. One is how to design an efficient network to capture the global field of the medical image and execute fast in real applications. The other is how to train the deep segmentation network using a few training data due to some medical privacy issues. In this paper, to conquer such issues, we first design a novel attention-aware segmentation model equipped with the multi-scale attention module in the pyramid structure-like encoder-decoder network, which can efficiently learn the global semantics and the long-range dependencies of the input images. Furthermore, we also inject the prior knowledge that the optic cup lies inside the optic disc by a novel loss function. Then, we propose a self-supervised contrastive learning method for optic disc and cup segmentation. The unsupervised feature representation is learned by matching an encoded query to a dictionary of encoded keys using a contrastive technique. Finetuning the pre-trained model using the proposed loss function can help achieve good performance for the task. To validate the effectiveness of the proposed method, extensive systemic evaluations on different public challenging optic disc and cup benchmarks, including DRISHTI-GS and REFUGE datasets demonstrate the superiority of the proposed method, which can achieve new state-of-the-art performance approaching 0.9801 and 0.9087 F1 score respectively while gaining 0.9657 DCdisc\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$DC_{disc}$$\\end{document} and 0.8976 DCcup\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$DC_{cup}$$\\end{document}. The code will be made publicly available.

Automatic Pavement Crack Detection in Multisource Fusion Images Using Similarity and Difference Features

In urban pavement crack detection, the single detection platform with visible light sensors is mostly used for detection. Visible light sensors are highly susceptible to lighting conditions. In low-light environments or areas with shadows over the cracks, this may result in false positives or missed detections of pavement cracks. The paper utilizes an image acquisition system that integrates multiple detection platforms, including visible light and infrared sensors, to enhance both the efficiency and accuracy of pavement crack detection. We propose an automatic pavement crack detection approach that uses the similarities and differences between visible light images and infrared image features (MSFSD) to achieve all-weather, real-time detection of pavement cracks. MSFSD mainly includes multi-source image fusion and pixel-level crack detection. In multi-source image fusion, a generative adversarial network (C-GAN) with skip connections and dilated convolutional blocks (CRes2Net blocks) is constructed. The structural similarity index (SSIM) function and the sum of the correlations of differences (SCD) function are used as the loss function of the generator, allowing the fusion image to retain more details and edge information in the source image. In pixel-level crack detection, a crack detection model is constructed, and CRes2Net is used as the core network to obtain pixel-level labels and perform semantic segmentation. It is verified by experiments that MSFSD achieves 85.1% precision and 88.4% recall in different types of crack detection. The MIoU and detection efficiency of MSFSD are 9% and 48% higher than other methods, respectively.

A Novel Hybrid Approach for Concrete Crack Segmentation Based on Deformable Oriented-YOLOv4 and Image Processing Techniques

Regular crack inspection plays a significant role in the maintenance of concrete structures. However, most deep-learning-based methods suffer from the heavy workload of pixel-level labeling and the poor performance of crack segmentation with the presence of background interferences. To address these problems, the Deformable Oriented YOLOv4 (DO-YOLOv4) is first developed for crack detection based on the traditional YOLOv4, in which crack features can be effectively extracted by deformable convolutional layers, and the crack regions can be tightly enclosed by a series of oriented bounding boxes. Then, the proposed DO-YOLOv4 is further utilized in combination with the image processing techniques (IPTs), leading to a novel hybrid approach, termed DO-YOLOv4-IPTs, for crack segmentation. The experimental results show that, owing to the high precision of DO-YOLOv4 for crack detection under background noise, the present hybrid approach DO-YOLOv4-IPTs outperforms the widely used Convolutional Neural Network (CNN)-based crack segmentation methods with less labeling work and superior segmentation accuracy.

HSG-MGAF Net: Heterogeneous subgraph-guided multiscale graph attention fusion network for interpretable prediction of whole-slide image

Background and objectivePathological whole slide image (WSI) prediction and region of interest (ROI) localization are important issues in computer-aided diagnosis and postoperative analysis in clinical applications. Existing computer-aided methods for predicting WSI are mainly based on multiple instance learning (MIL) and its variants. However, most of the methods are based on instance independence and identical distribution assumption and performed at a single scale, which not fully exploit the hierarchical multiscale heterogeneous information contained in WSI. MethodsHeterogeneous Subgraph-Guided Multiscale Graph Attention Fusion Network (HSG-MGAF Net) is proposed to build the topology of critical image patches at two scales for adaptive WSI prediction and lesion localization. The HSG-MGAF Net simulates the hierarchical heterogeneous information of WSI through graph and hypergraph at two scales, respectively. This framework not only fully exploits the low-order and potential high-order correlations of image patches at each scale, but also leverages the heterogeneous information of the two scales for adaptive WSI prediction. ResultsWe validate the superiority of the proposed method on the CAMELYON16 and the TCGA- NSCLC, and the results show that HSG-MGAF Net outperforms the state-of-the-art method on both datasets. The average ACC, AUC and F1 score of HSG-MGAF Net can reach 92.7 %/0.951/0.892 and 92.2 %/0.957/0.919, respectively. The obtained heatmaps can also localize the positive regions more accurately, which have great consistency with the pixel-level labels. ConclusionsThe results demonstrate that HSG-MGAF Net outperforms existing weakly supervised learning methods by introducing critical heterogeneous information between the two scales. This approach paves the way for further research on light weighted heterogeneous graph-based WSI prediction and ROI localization.