Class Activation Mapping Research Articles

Keratoconus Progression Determined at the First Visit: A Deep Learning Approach With Fusion of Imaging and Numerical Clinical Data.

Multiple clinical visits are necessary to determine progression of keratoconus before offering corneal cross-linking. The purpose of this study was to develop a neural network that can potentially predict progression during the initial visit using tomography images and other clinical risk factors. The neural network's development depended on data from 570 keratoconus eyes. During the initial visit, numerical risk factors and posterior elevation maps from Scheimpflug imaging were collected. Increase of steepest keratometry of 1 diopter during follow-up was used as the progression criterion. The data were partitioned into training, validation, and test sets. The first two were used for training, and the latter for performance statistics. The impact of individual risk factors and images was assessed using ablation studies and class activation maps. The most accurate prediction of progression during the initial visit was obtained by using a combination of MobileNet and a multilayer perceptron with an accuracy of 0.83. Using numerical risk factors alone resulted in an accuracy of 0.82. The use of only images had an accuracy of 0.77. The most influential risk factors in the ablation study were age and posterior elevation. The greatest activation in the class activation maps was seen at the highest posterior elevation where there was significant deviation from the best fit sphere. The neural network has exhibited good performance in predicting potential future progression during the initial visit. The developed neural network could be of clinical significance for keratoconus patients by identifying individuals at risk of progression.

Weakly supervised instance segmentation via class double-activation maps and boundary localization

Weakly supervised instance segmentation based on image-level class labels has recently gained much attention, in which the primary key step is to generate the pseudo labels based on class activation maps (CAMs). Most methods adopt binary cross-entropy (BCE) loss to train the classification model. However, since BCE loss is not class mutually exclusive, activations among classes occur independently. Thus, not only do foreground classes are wrongly activated as background, but also incorrect activations among confusing classes are occurred in the foreground. To solve this problem, we propose the Class Double-Activation Map, called Double-CAM. Firstly, the vanilla CAM is extracted from the multi-label classifier and then fused with the output feature map of backbone. The enhanced feature map of each class is fed into the single-label classification branch with softmax cross-entropy (SCE) loss and entropy minimization module, from which the more accurate Double-CAM is extracted. It refines the vanilla CAM to improve the quality of pseudo labels. Secondly, to mine object edge cues from Double-CAM, we propose the Boundary Localization (BL) module to synthesize boundary annotations, so as to provide constraints for label propagation more explicitly without adding additional supervision. The quality of pseudo masks is also improved substantially with the addition of BL module. Finally, the generated pseudo labels are used to train fully supervised instance segmentation networks. The evaluations on VOC and COCO datasets show that our method achieves excellent performance, outperforming mainstream weakly supervised segmentation methods at the same supervisory level, even those that depend on stronger supervision.

Retinal disease diagnosis with unsupervised Grad-CAM guided contrastive learning

To alleviate the reliance on expensive annotations, contrastive learning techniques have been applied to diagnose diseases from various types of medical images. However, popular contrastive learning methods, which generate positive pairs by random cropping, face challenges in diagnosing retinal diseases, due to the retinal disease’s characteristic that retinal lesions are tiny and randomly distributed in abnormal fundus images. These lesions may be missed after random cropping, resulting in semantically inconsistent positive pairs that hinder the effectiveness of contrastive learning for retinal disease diagnosis. To address this issue, we propose a novel unsupervised gradient-weighted class activation mapping (Grad-CAM) strategy to roughly locate lesions, thereby suppressing or even eliminating semantically inconsistent positive pairs. Specifically, we develop a gradient-weighted Class Activation Map guided Contrastive Learning (CAMCL) method with two branches for the Grad-CAM based instance discrimination task and the k-nearest neighbors (KNN)-based cluster-wise discrimination task, respectively. By minimizing the KNN loss, the cluster-wise discrimination branch learns high-level representations containing class semantic information. This is then followed by gradient back-propagation to generate Grad-CAM heatmaps from unlabeled data. The generated heatmaps can highlight class-discriminative regions in abnormal fundus images (e.g., retinal lesions) to identify semantically consistent positive pairs while suppressing inconsistent ones. The semantically consistent positive pairs are then input to the instance discrimination task for contrastive learning. In this manner, the semantic inconsistency problem is relieved, and the improved contrastive learning pipeline can be effectively used for retinal disease diagnosis. Experimental results on five retinal disease classification datasets show that our model surpasses other contrastive learning methods, indicating a promising approach for clinical application.

Exploring the influence of attention for whole-image mammogram classification

Attention is an important component of modern Convolutional Neural Networks (CNNs) that has been shown to improve baseline model performance for a wide variety of tasks. Attention has shown specific promise in the classification and segmentation of mammograms, but we have a limited understanding of why attention improves performance in these domains. In this paper, we present a robust comparison of different combinations of baseline models and attention methods at two resolutions for whole mammogram classification of masses and calcifications. We find that attention generally helps to improve baseline model performance. However, the extent of improvement is governed by a combination of model architecture and the statistical characteristics of the data. Specifically, we show that high amounts of pooling and model complexity may result in decreased performance for data with high variability. To better understand the effect of attention on mammogram classification, we used LayerCAM, a hierarchical Class Activation Map (CAM) approach, to visualize where the network pays attention in the input image. This research provides statistical evidence that attention can improve the correlation between model performance and LayerCAM activation in the region of interest (ROI). However, these correlations are weak and variable, indicating that improvements in model performance due to attention are not necessarily caused by increased model activation near the ROI. Overall, our work provides novel insights to help guide future efforts in incorporating attention-based mechanisms for mammogram classification.

Robust deep learning method for fruit decay detection and plant identification: enhancing food security and quality control.

This paper presents a robust deep learning method for fruit decay detection and plant identification. By addressing the limitations of previous studies that primarily focused on model accuracy, our approach aims to provide a more comprehensive solution that considers the challenges of robustness and limited data scenarios. The proposed method achieves exceptional accuracy of 99.93%, surpassing established models. In addition to its exceptional accuracy, the proposed method highlights the significance of robustness and adaptability in limited data scenarios. The proposed model exhibits strong performance even under the challenging conditions, such as intense lighting variations and partial image obstructions. Extensive evaluations demonstrate its robust performance, generalization ability, and minimal misclassifications. The inclusion of Class Activation Maps enhances the model's capability to identify distinguishing features between fresh and rotten fruits. This research has significant implications for fruit quality control, economic loss reduction, and applications in agriculture, transportation, and scientific research. The proposed method serves as a valuable resource for fruit and plant-related industries. It offers precise adaptation to specific data, customization of the network architecture, and effective training even with limited data. Overall, this research contributes to fruit quality control, economic loss reduction, and waste minimization.

A novel approach of brain-computer interfacing (BCI) and Grad-CAM based explainable artificial intelligence: Use case scenario for smart healthcare

BackgroundIn order to push the frontiers of brain-computer interface (BCI) and neuron-electronics, this research presents a novel framework that combines cutting-edge technologies for improved brain-related diagnostics in smart healthcare. This research offers a ground-breaking application of transparent strategies to BCI, promoting openness and confidence in brain-computer interactions and taking inspiration from Grad-CAM (Gradient-weighted Class Activation Mapping) based Explainable Artificial Intelligence (XAI) methodology. The landscape of healthcare diagnostics is about to be redefined by the integration of various technologies, especially when it comes to illnesses related to the brain. New methodA novel approach has been proposed in this study comprising of Xception architecture which is trained on imagenet database following transfer learning process for extraction of significant features from magnetic resonance imaging dataset acquired from publicly available distinct sources as an input and linear support vector machine has been used for distinguishing distinct classes.Afterwards, gradient-weighted class activation mapping has been deployed as the foundation for explainable artificial intelligence (XAI) for generating informative heatmaps, representing spatial localization of features which were focused to achieve model’s predictions. ResultsThus, the proposed model not only provides accurate outcomes but also provides transparency for the predictions generated by the Xception network to diagnose presence of abnormal tissues and avoids overfitting issues. Hyperparameters along with performance-metrics are also obtained while validating the proposed network on unseen brain MRI scans to ensure effectiveness of the proposed network. Comparison with existing methods and conclusionsThe integration of Grad-CAM based explainable artificial intelligence with deep neural network namely Xception offers a significant impact in diagnosing brain tumor disease while highlighting the specific regions of input brain MRI images responsible for making predictions. In this study, the proposed network results in 98.92% accuracy, 98.15% precision, 99.09% sensitivity, 98.18% specificity and 98.91% dice-coefficient while identifying presence of abnormal tissues in the brain. Thus, Xception model trained on distinct dataset following transfer learning process offers remarkable diagnostic accuracy and linear support vector act as a classifier to provide efficient classification among distinct classes. In addition, the deployed explainable artificial intelligence approach helps in revealing the reasoning behind predictions made by deep neural network having black-box nature and provides a clear perspective to assist medical experts in achieving trustworthiness and transparency while diagnosing brain tumor disease in the smart healthcare

A deep learning approach for Maize Lethal Necrosis and Maize Streak Virus disease detection

Maize is an important crop cultivated in Sub-Saharan Africa, essential for food security. However, its cultivation faces significant challenges due to debilitating diseases such as Maize Lethal Necrosis (MLN) and Maize Streak Virus (MSV), which can lead to severe yield losses. Traditional plant disease diagnosis methods are often time-consuming and prone to errors, necessitating more efficient approaches. This study explores the application of deep learning, specifically Convolutional Neural Networks (CNNs), in the automatic detection and classification of maize diseases. We investigate six architectures: Basic CNN, EfficientNet V2 B0 and B1, LeNet-5, VGG-16, and ResNet50, using a dataset of 15344 images comprising MSV, MLN, and healthy maize leaves. Additionally, We performed hyperparameter tuning to improve the performance of the models and Gradient-weighted Class Activation Mapping (Grad-CAM) for model interpretability. Our results show that the EfficientNet V2 B0 model demonstrated an accuracy of 99.99% in distinguishing between healthy and disease-infected plants. The results of this study contribute to the advancement of AI applications in agriculture, particularly in diagnosing maize diseases within Sub-Saharan Africa.

Methodology for Evaluating the Generalization of ResNet

Convolutional neural networks (CNNs) have achieved promising results in many tasks, and evaluating the model’s generalization ability based on the trained model and training data is paramount for practical applications. Although many measures for evaluating the generalization of CNN models have been proposed, the existing works are limited to small-scale or simplified model sets, which would result in poor accuracy and applicability of the derived methods. This study addresses these limitations by leveraging ResNet models as a case study to evaluate the model’s generalization ability. We utilized Intersection over Union (IoU) as a method to quantify the ratio of task-relevant features to assess model generalization. Class activation maps (CAMs) were used as a representation of the distribution of features learned by the model. To systematically investigate the generalization ability, we constructed a diverse model set based on the ResNet architecture. A total of 2000 CNN models were trained on the ImageNet subset by systematically changing commonly used hyperparameters. The results of our experiments revealed a strong correlation between the IoU-based evaluation method and the model’s generalization performance (Pearson correlation coefficient more than 0.8). We also performed extensive experiments to demonstrate the feasibility and robustness of the evaluation methods.

Explainable Artificial Intelligence in Medical Imaging: A Case Study on Enhancing Lung Cancer Detection through CT Images

This study tackles the pressing challenge of lung cancer detection, the foremost cause of cancer-related mortality worldwide, hindered by late detection and diagnostic limitations. Aiming to improve early detection rates and diagnostic reliability, we propose an approach integrating Deep Convolutional Neural Networks (DCNN) with Explainable Artificial Intelligence (XAI) techniques, specifically focusing on the Residual Network (ResNet) architecture and Gradient-weighted Class Activation Mapping (Grad-CAM). Utilizing a dataset of 1,000 CT scans, categorized into normal, non-cancerous, and three types of lung cancer images, we adapted the ResNet50 model through transfer learning and fine-tuning for enhanced specificity in lung cancer subtype detection. Our methodology demonstrated the modified ResNet50 model's effectiveness, significantly outperforming the original architecture in accuracy (91.11%), precision (91.66%), sensitivity (91.11%), specificity (96.63%), and F1-score (91.10%). The inclusion of Grad-CAM provided insightful visual explanations for the model's predictions, fostering transparency and trust in computer-assisted diagnostics. The study highlights the potential of combining DCNN with XAI to advance lung cancer detection, suggesting future research should expand dataset diversity and explore multimodal data integration for broader applicability and improved diagnostic capabilities.

Toward explainable AI in radiology: Ensemble-CAM for effective thoracic disease localization in chest X-ray images using weak supervised learning.

Chest X-ray (CXR) imaging is widely employed by radiologists to diagnose thoracic diseases. Recently, many deep learning techniques have been proposed as computer-aided diagnostic (CAD) tools to assist radiologists in minimizing the risk of incorrect diagnosis. From an application perspective, these models have exhibited two major challenges: (1) They require large volumes of annotated data at the training stage and (2) They lack explainable factors to justify their outcomes at the prediction stage. In the present study, we developed a class activation mapping (CAM)-based ensemble model, called Ensemble-CAM, to address both of these challenges via weakly supervised learning by employing explainable AI (XAI) functions. Ensemble-CAM utilizes class labels to predict the location of disease in association with interpretable features. The proposed work leverages ensemble and transfer learning with class activation functions to achieve three objectives: (1) minimizing the dependency on strongly annotated data when locating thoracic diseases, (2) enhancing confidence in predicted outcomes by visualizing their interpretable features, and (3) optimizing cumulative performance via fusion functions. Ensemble-CAM was trained on three CXR image datasets and evaluated through qualitative and quantitative measures via heatmaps and Jaccard indices. The results reflect the enhanced performance and reliability in comparison to existing standalone and ensembled models.

Where is my attention? An explainable AI exploration in water detection from SAR imagery

Attention mechanisms have found extensive application in Deep Neural Networks (DNNs), with numerous experiments over time showcasing their efficacy in improving the overall performance of DNNs. Considering the black-box nature of DNNs, it is unclear how attention mechanisms affect model decision-making processes. Therefore, it is of great significance to explore the internal relationship between the focused area of attention mechanisms and the prediction results. For the first time, an explainable framework for the attention mechanism of Synthetic Aperture Radar (SAR) image analytics is proposed, integrating DNN and eXplainable Artificial Intelligence (XAI), which realizes the effective explanation for attention mechanisms across different locations. The framework consists of three parts: the water extraction network which contains the attention mechanism at different locations; the proposed Attention- Gradient Class Activation Mapping (A-GCAM) method for attribution analysis of attention mechanisms; and we invent Category-specific channel Score of Confidence Mapping (CSCM) to perform geo-visualization for the output features of attention mechanisms. Experiments are conducted with the widely used Sentinel-1 system for water detection, in which three attention mechanisms with different characteristics are added to the encoder and decoder in DNNs. The results show that the framework can make the decision-making process of attention mechanisms transparent, thus improving their comprehensiveness and trustworthiness in various tasks and providing a reliable approach to selecting suitable attention mechanisms for the given task.

Weakly supervised semantic segmentation based on superpixel affinity

Weakly supervised semantic segmentation (WSSS) usually employs the method of modifying and extending class activation map (CAM) seeds to achieve semantic segmentation. However, the down-sampling of the network weakens the edge awareness of CAMs, leading to under- or over-activation problems, which affects the segmentation quality. Considering the excellent contour attachment property of superpixels and the high semantic similarity between pixels within the same superpixel, we propose a superpixel affinity-based method that uses multi-scale features to aggregate superpixels with the same semantics, providing complete localization supervision for the generation of CAMs. In order to improve the accuracy of semantic labels for superpixels, we utilize a method of deep feature reorganization to improve the quality of network-generated CAM seeds. The experimental results indicate that the proposed method has achieved satisfactory performance on PASCAL VOC 2012 and MS COCO 2014 datasets.

On the evaluation of deep learning interpretability methods for medical images under the scope of faithfulness

Background and Objective:Evaluating the interpretability of Deep Learning models is crucial for building trust and gaining insights into their decision-making processes. In this work, we employ class activation map based attribution methods in a setting where only High-Resolution Class Activation Mapping (HiResCAM) is known to produce faithful explanations. The objective is to evaluate the quality of the attribution maps using quantitative metrics and investigate whether faithfulness aligns with the metrics results. Methods:We fine-tune pre-trained deep learning architectures over four medical image datasets in order to calculate attribution maps. The maps are evaluated on a threefold metrics basis utilizing well-established evaluation scores. Results:Our experimental findings suggest that the Area Over Perturbation Curve (AOPC) and Max-Sensitivity scores favor the HiResCAM maps. On the other hand, the Heatmap Assisted Accuracy Score (HAAS) does not provide insights to our comparison as it evaluates almost all maps as inaccurate. To this purpose we further compare our calculated values against values obtained over a diverse group of models which are trained on non-medical benchmark datasets, to eventually achieve more responsive results. Conclusion:This study develops a series of experiments to discuss the connection between faithfulness and quantitative metrics over medical attribution maps. HiResCAM preserves the gradient effect on a pixel level ultimately producing high-resolution, informative and resilient mappings. In turn, this is depicted in the results of AOPC and Max-Sensitivity metrics, successfully identifying the faithful algorithm. In regards to HAAS, our experiments yield that it is sensitive over complex medical patterns, commonly characterized by strong color dependency and multiple attention areas.

XAI-FusionNet: Diabetic foot ulcer detection based on multi-scale feature fusion with explainable artificial intelligence

Diabetic foot ulcer (DFU) poses a significant threat to individuals affected by diabetes, often leading to limb amputation. Early detection of DFU can greatly improve the chances of survival for diabetic patients. This work introduces FusionNet, a novel multi-scale feature fusion network designed to accurately differentiate DFU skin from healthy skin using multiple pre-trained convolutional neural network (CNN) algorithms. A dataset comprising 6963 skin images (3574 healthy and 3389 ulcer) from various patients was divided into training (6080 images), validation (672 images), and testing (211 images) sets. Initially, three image preprocessing techniques - Gaussian filter, median filter, and motion blur estimation - were applied to eliminate irrelevant, noisy, and blurry data. Subsequently, three pre-trained CNN algorithms -DenseNet201, VGG19, and NASNetMobile - were utilized to extract high-frequency features from the input images. These features were then inputted into a meta-tuner module to predict DFU by selecting the most discriminative features. Statistical tests, including Friedman and analysis of variance (ANOVA), were employed to identify significant differences between FusionNet and other sub-networks. Finally, three eXplainable Artificial Intelligence (XAI) algorithms - SHAP (SHapley Additive exPlanations), LIME (Local Interpretable Model-agnostic Explanations), and Grad-CAM (Gradient-weighted Class Activation Mapping) - were integrated into FusionNet to enhance transparency and explainability. The FusionNet classifier achieved exceptional classification results with 99.05 % accuracy, 98.18 % recall, 100.00 % precision, 99.09 % AUC, and 99.08 % F1 score. We believe that our proposed FusionNet will be a valuable tool in the medical field to distinguish DFU from healthy skin.

Fusion of satellite and street view data for urban traffic accident hotspot identification

As the number of vehicles and the volume of traffic swell in urban centers, cities have experienced a concomitant increase in traffic accidents. Proactively identifying accident-prone hotspots in urban environments holds the promise of preventing traffic mishaps, thereby curtailing the incidence of accidents and reducing property damage. This research introduces the Two-Branch Contextual Feature-Guided Converged Network (TCFGC-Net) utilizing multimodal satellite and street view data. Designed to extract global structural features from satellite imagery and dynamic continuous features from street view imagery, the model aims to improve the accuracy of detecting urban accident hotspots. For the satellite imagery branch, we propose the Contextual Feature Coupled Convolutional Neural Network (Trans-CFCCNN) designed to extract global spatial features and discern feature correlations across adjacent regions. For the street view imagery branch, we develop the Sequential Feature Recurrent Attention Network (SFRAN) to assimilate and integrate dynamic scene features captured from successive street view images. We designed the Multi-Branch Feature Adaptive Fusion Structure (MBFAF) to aggregate different branch features for accurate identification of accident hotspots. Experimental results show that the model performs well, with an overall accuracy of 93.7 %. Ablation studies confirm that relative to standalone street view and satellite branch analyses, implementing multimodal fusion enhances the model's accuracy by 12.05 % and 17.86 %, respectively. The innovative fusion structure proposed herein garners a 4.22 % increase in model accuracy, outpacing conventional feature concatenation techniques. Furthermore, the model outperforms existing deep learning models in terms of overall efficacy. Additionally, to showcase the efficacy of the proposed model structure, we utilize Class Activation Maps (CAM) to provide visual interpretability for the model. These results suggest that the dual-branch fusion model effectively decreases false alarm occurrences and directs the model's focus toward regions more pertinent to accident hotspots. Finally, the code and model used for identifying hotspots of urban traffic accidents in this study are available for access: https://github.com/gwt-ZJU/TCFGC-Net.

Tactile texture recognition of multi-modal bionic finger based on multi-modal CBAM-CNN interpretable method

To solve the problem of lacking surface information while using a single-modal sensor to scan fine textures, a multi-modal bionic finger tactile sensor is used to perceive the information contained in textures, and a multi-modal tactile texture recognition method is proposed to address the issue of poor interpretability for tactile texture recognition models. First, the experimental platform of multi-modal bionic finger tactile texture information acquisition system is built, which can simultaneously collect 14 channels of tactile electrode signals, one channel of pressure signals and one channel of temperature signals. Second, the multi-modal tactile signals can be fused at the data level after signal processing, and then the wavelet transform method is carried out on the basis of data fusion. Third, a multi-modal CBAM-CNN network (Convolutional Block Attention Module, Convolutional Neural Network) is established to extract the multi-modal tactile features after the wavelet transform and the multi-modal tactile features are fused at feature level. To verify the performance of multi-modal CBAM-CNN model, 27(3*9) kinds of millimeter-level fine textures of 3 datasets are tested and the recognition accuracy of each dataset is 98.47%, 97.63%, 99.16%. And it can also be seen that the 14-channel modal data contributes the most to the performance of multi-modal CBAM-CNN model for various textures. The results of ablation experiments prove that the identification accuracy of the fused multi-modal CBAM-CNN is higher than the single-modal CBAM-CNN no matter in time domain or time–frequency domain. Moreover, the accuracy of the weighted fusion method based on correlation function is higher than horizontal fusion method in 14 channels of tactile electrode signals modal. Finally, to interpret the classification decision of the multi-modal CBAM-CNN network in each modal intuitively, the improved multi-modal CAM (Class Activation Map) algorithm is adopted to visualize the interest regions of the samples in detail, and the activated areas in the map make the identification result of CBAM-CNN in each modal more transparent and interpretable.

A discrepancy-aware self-distillation method for multi-modal glioma grading

Gliomas are the most common intracranial primary tumors. Accurate grading of gliomas is crucial in determining treatment options and prognosis. Clinicians conventionally rely on multiple Magnetic Resonance Imaging (MRI) sequences for accurate glioma assessment. Traditional deep learning methods typically utilize the individual MRI sequence with the annotations of Region of Interest (ROIs), incurring substantial manual effort while losing the complementary information provided by other sequences. This task is inherently a standard multi-modal problem. However, existing methods often adopt complex multi-stream networks for feature extraction and fusion, leading to high resource demands and potential feature redundancy. To address the above challenge, a discrepancy-aware self-distillation method is proposed for multi-modal glioma grading, which requires only a single-stream network to concurrently analyze multiple MRI sequences without auxiliary ROIs. The first Modality Discrepancy-aware Fusion (MDF) module considers the MRI imaging differentiation and widens the inter-modal contrasts, allowing the modality-specific features to be highlighted and the modality-invariant features to be diminished. Furthermore, the proposed Class Activation Self-Distillation (CASD) strategy leverages the generated Class Activation Maps (CAMs) as dark knowledge for the distillation process. This guides shallow layers to focus on the category discriminative features specifically within the lesion region. Extensive experiments were conducted on the BraTS2018 and BraTS2019 datasets to evaluate the effectiveness of our method. The results show that our method outperforms other relevant glioma grading and multi-modal fusion methods on both datasets.

PipeTransUNet: CNN and Transformer fusion network for semantic segmentation and severity quantification of multiple sewer pipe defects

With the continuous development of urbanization, the service life of sewer pipes is gradually approaching a critical threshold. Defects within pipe networks can significantly affect the municipality operations and residents' quality of life. Towards efficient and automatic sewer pipeline inspection, an integrated framework was proposed for semantic segmentation and severity quantification of multiple sewer pipe defects using the PipeTransUNet model that fuses convolutional neural networks with Transformer. The capability of the fusion network to extract and localize sewer defects was further enhanced by incorporating the convolutional block attention module and improving the activation function. In extensive experiments, PipeTransUNet shows enough competitiveness after hyperparameter selection, architecture tweaks, and comparison with other state-of-the-art models. Specifically, PipeTransUNet outperformed other models in terms of both quantitative and visual evaluations, with mean intersection over union, the mean of pixel accuracy, mean pixel accuracy, mean recall, mean F1-score, mean specificity, mean Kappa, and frequency-weighted intersection over union values reaching 71.92%, 84.90%, 80.74%, 85.93%, 83.05%, 84.44%, 55.55%, and 91.32%, respectively. A severity level assessment method for different sewer defects was developed based on PipeTransUNet and compared with expert reviews' results to demonstrate its feasibility and effectiveness. Moreover, the in-depth image features extracted from the segmentation head of our proposed model were visually evaluated and interpreted using pixel-level gradient-weighted class activation mapping. In summary, PipeTransUNet canrecognize complex defects well and provide a solid foundation for inspecting and maintaining sewer pipe networks.

W-DRAG: A joint framework of WGAN with data random augmentation optimized for generative networks for bone marrow edema detection in dual energy CT

Dual-energy computed tomography (CT) is an excellent substitute for identifying bone marrow edema in magnetic resonance imaging. However, it is rarely used in practice owing to its low contrast. To overcome this problem, we constructed a framework based on deep learning techniques to screen for diseases using axial bone images and to identify the local positions of bone lesions. To address the limited availability of labeled samples, we developed a new generative adversarial network (GAN) that extends expressions beyond conventional augmentation (CA) methods based on geometric transformations. We theoretically and experimentally determined that combining the concepts of data augmentation optimized for GAN training (DAG) and Wasserstein GAN yields a considerably stable generation of synthetic images and effectively aligns their distribution with that of real images, thereby achieving a high degree of similarity. The classification model was trained using real and synthetic samples. Consequently, the GAN technique used in the diagnostic test had an improved F1 score of approximately 7.8% compared with CA. The final F1 score was 80.24%, and the recall and precision were 84.3% and 88.7%, respectively. The results obtained using the augmented samples outperformed those obtained using pure real samples without augmentation. In addition, we adopted explainable AI techniques that leverage a class activation map (CAM) and principal component analysis to facilitate visual analysis of the network’s results. The framework was designed to suggest an attention map and scattering plot to visually explain the disease predictions of the network.

Investigating the use of deep learning models for land cover classification from street‐level imagery

AbstractLand cover classification mapping is the process of assigning labels to different types of land surfaces based on overhead imagery. However, acquiring reference samples through fieldwork for ground truth can be costly and time‐intensive. Additionally, annotating high‐resolution satellite images poses challenges, as certain land cover types are difficult to discern solely from nadir images. To address these challenges, this study examined the feasibility of using street‐level imagery to support the collection of reference samples and identify land cover. We utilized 18,022 images captured in Japan, with 14 different land cover classes. Our approach involved using convolutional neural networks based on Inception‐v4 and DenseNet, as well as Transformer‐based Vision and Swin Transformers, both with and without pre‐trained weights and fine‐tuning techniques. Additionally, we explored explainability through Gradient‐Weighted Class Activation Mapping (Grad‐CAM). Our results indicate that using a Vision Transformer was the most effective method, achieving an overall accuracy of 86.12% and allowing for full explainability of land cover targets within an image. This paper proposes a promising solution for land cover classification from street‐level imagery, which can be used for semi‐automatic reference sample collection from geo‐tagged street‐level photos.