Class Activation Mapping Research Articles

Insights into Predicting Tooth Extraction from Panoramic Dental Images: Artificial Intelligence vs. Dentists

ObjectivesTooth extraction is one of the most frequently performed medical procedures. The indication is based on the combination of clinical and radiological examination and individual patient parameters and should be made with great care. However, determining whether a tooth should be extracted is not always a straightforward decision. Moreover, visual and cognitive pitfalls in the analysis of radiographs may lead to incorrect decisions. Artificial intelligence (AI) could be used as a decision support tool to provide a score of tooth extractability.Material and methodsUsing 26,956 single teeth images from 1,184 panoramic radiographs (PANs), we trained a ResNet50 network to classify teeth as either extraction-worthy or preservable. For this purpose, teeth were cropped with different margins from PANs and annotated. The usefulness of the AI-based classification as well that of dentists was evaluated on a test dataset. In addition, the explainability of the best AI model was visualized via a class activation mapping using CAMERAS.ResultsThe ROC-AUC for the best AI model to discriminate teeth worthy of preservation was 0.901 with 2% margin on dental images. In contrast, the average ROC-AUC for dentists was only 0.797. With a 19.1% tooth extractions prevalence, the AI model's PR-AUC was 0.749, while the dentist evaluation only reached 0.589.ConclusionAI models outperform dentists/specialists in predicting tooth extraction based solely on X-ray images, while the AI performance improves with increasing contextual information.Clinical relevanceAI could help monitor at-risk teeth and reduce errors in indications for extractions.

CAManim: Animating end-to-end network activation maps.

Deep neural networks have been widely adopted in numerous domains due to their high performance and accessibility to developers and application-specific end-users. Fundamental to image-based applications is the development of Convolutional Neural Networks (CNNs), which possess the ability to automatically extract features from data. However, comprehending these complex models and their learned representations, which typically comprise millions of parameters and numerous layers, remains a challenge for both developers and end-users. This challenge arises due to the absence of interpretable and transparent tools to make sense of black-box models. There exists a growing body of Explainable Artificial Intelligence (XAI) literature, including a collection of methods denoted Class Activation Maps (CAMs), that seek to demystify what representations the model learns from the data, how it informs a given prediction, and why it, at times, performs poorly in certain tasks. We propose a novel XAI visualization method denoted CAManim that seeks to simultaneously broaden and focus end-user understanding of CNN predictions by animating the CAM-based network activation maps through all layers, effectively depicting from end-to-end how a model progressively arrives at the final layer activation. Herein, we demonstrate that CAManim works with any CAM-based method and various CNN architectures. Beyond qualitative model assessments, we additionally propose a novel quantitative assessment that expands upon the Remove and Debias (ROAD) metric, pairing the qualitative end-to-end network visual explanations assessment with our novel quantitative "yellow brick ROAD" assessment (ybROAD). This builds upon prior research to address the increasing demand for interpretable, robust, and transparent model assessment methodology, ultimately improving an end-user's trust in a given model's predictions. Examples and source code can be found at: https://omni-ml.github.io/pytorch-grad-cam-anim/.

Deep learning based identification and interpretability research of traditional village heritage value elements: a case study in Hubei Province

The preservation and transmission of traditional villages is crucial to the prosperity and development of ethnic cultures. However, current traditional village surveys usually require a large number of experts and scholars to conduct field research, which is an expensive and time-consuming method, especially for large-scale tasks. Therefore, this study proposes an automatic classification method based on deep learning (DL) for the identification of traditional village heritage value elements (TVHVE). The study evaluates four selected convolutional neural network (CNN) frames using traditional villages in Hubei Province as a sample dataset. The results show that Residual Network152 (ResNet152) is the most suitable CNN frame for identifying TVHVE in Hubei. The stability and consistency of various TVHVE present in the ResNet152 model were evaluated using Area Under Curve (AUC) and Precision Recall Curve (PRC), which indicated satisfactory prediction performance for most elements, except for specific elements such as tombstones and stone carvings, which showed lower accuracy. In addition, the study sheds light on the areas of concern of the model with respect to different TVHVE images and elucidates the reasons behind the confusion between elements through semantic clustering based on image classification and interpretability analysis using the Gradient-Weighted Class Activation Mapping (Grad-CAM) heat map. By using an automated classification method based on DL, this study significantly reduces the cost and effort associated with traditional surveys. At the same time, insight into areas of concern and confusion in the model improves guidance for conservation efforts and provides valuable references for subsequent research.

A versatile automated pipeline for quantifying virus infectivity by label-free light microscopy and artificial intelligence

Virus infectivity is traditionally determined by endpoint titration in cell cultures, and requires complex processing steps and human annotation. Here we developed an artificial intelligence (AI)-powered automated framework for ready detection of virus-induced cytopathic effect (DVICE). DVICE uses the convolutional neural network EfficientNet-B0 and transmitted light microscopy images of infected cell cultures, including coronavirus, influenza virus, rhinovirus, herpes simplex virus, vaccinia virus, and adenovirus. DVICE robustly measures virus-induced cytopathic effects (CPE), as shown by class activation mapping. Leave-one-out cross-validation in different cell types demonstrates high accuracy for different viruses, including SARS-CoV-2 in human saliva. Strikingly, DVICE exhibits virus class specificity, as shown with adenovirus, herpesvirus, rhinovirus, vaccinia virus, and SARS-CoV-2. In sum, DVICE provides unbiased infectivity scores of infectious agents causing CPE, and can be adapted to laboratory diagnostics, drug screening, serum neutralization or clinical samples.

A Predictive Model for Intraoperative Cerebrospinal Fluid Leak During Endonasal Pituitary Adenoma Resection Using a Convolutional Neural Network

Cerebrospinal fluid (CSF) leak during endoscopic endonasal transsphenoidal surgery can lead to postoperative complications. The clinical and anatomic risk factors of intraoperative CSF leak are not well defined. We applied a two-dimensional (2D) convolutional neural network (CNN) machine learning model to identify risk factors from preoperative magnetic resonance imaging. All adults who underwent endoscopic endonasal transsphenoidal surgery at our institution from January 2007 to March 2023 who had accessible preoperative stereotactic magnetic resonance imaging were included. A retrospective classic statistical analysis was performed to identify demographic, clinical, and anatomic risk factors of intraoperative CSF leak. Stereotactic T2-weighted brain magnetic resonance imaging scans were used to train and test a 2D CNN model. Of 220 included patients, 81 (36.8%) experienced intraoperative CSF leak. Among all preoperative variables, visual disturbance was the only statistically significant identified risk factor (P= 0.008). The trained 2D CNN model predicted CSF leak with 92% accuracy and area under receiver operating characteristic curve of 0.90 (sensitivity of 86% and specificity of 93%). Class activation mapping of this model revealed that anatomic regions of CSF flow were most important in predicting CSF leak. Further review of the class activation mapping gradients revealed regions of the diaphragma sellae, clinoid processes, temporal horns, and optic nerves to have anatomic correlation to intraoperative CSF leak risk. Additionally, visual disturbances from anatomic compression of the optic chiasm were the only identified clinical risk factor. Our 2D CNN model can help a treating team to better anticipate and prepare for intraoperative CSF leak.

Leveraging Transfer Learning for Efficient Diagnosis of COPD Using CXR Images and Explainable AI Techniques

Chronic Obstructive Pulmonary Disease (COPD) is a predominant global health concern, ranking third in mortality rates, yet frequently remains undiagnosed until its advanced stages. Given its prevalence, the need for innovative and widely accessible diagnostic tools has never been more paramount. While spirometry tests serve as conventional diagnostic benchmarks, their reach remains limited, especially in regions with constrained medical resources. The presented research harnesses deep learning algorithms to facilitate early-stage COPD detection, specifically targeting Chest X-rays (CXRs). The clinically annotated VinDR-CXR dataset provides the primary foundation for model training, complemented by incorporating the ChestX-ray14 dataset for initial model pre-training. Such a dual-dataset strategy augments model generalization and adaptability. Among several explored Convolutional Neural Network (CNN) architectures, the Xception model emerges as a frontrunner. Through transfer learning methodologies, this model produces a noteworthy recall rate of 98.2%, markedly surpassing the metrics of the ResNet50 model. Recognizing the imperative for transparency in AI applications in medical imaging, the research integrates essential explainability approaches viz: Gradient Class Activation Mapping (Grad-CAM) and SHapley Additive exPlanations (SHAP). These techniques elucidate the AI’s decision-making process, offering invaluable visual and analytical insights for fostering trust among medical professionals. In essence, this study not only underscores the potential of integrating AI with medical imaging for COPD detection but also accentuates the pivotal role of transparency in AI-driven medical interventions.

Partial class activation mapping guided graph convolution cascaded U-Net for retinal vessel segmentation

Accurate segmentation of retinal vessels in fundus images is of great importance for the diagnosis of numerous ocular diseases. However, due to the complex characteristics of fundus images, such as various lesions, image noise and complex background, the pixel features of some vessels have significant differences, which makes it easy for the segmentation networks to misjudge these vessels as noise, thus affecting the accuracy of the overall segmentation. Therefore, accurately segment retinal vessels in complex situations is still a great challenge. To address the problem, a partial class activation mapping guided graph convolution cascaded U-Net for retinal vessel segmentation is proposed. The core idea of the proposed network is first to use the partial class activation mapping guided graph convolutional network to eliminate the differences of local vessels and generate feature maps with global consistency, and subsequently these feature maps are further refined by segmentation network U-Net to achieve better segmentation results. Specifically, a new neural network block, called EdgeConv, is stacked multiple layers to form a graph convolutional network to realize the transfer an update of information from local to global, so as gradually enhance the feature consistency of graph nodes. Simultaneously, in an effort to suppress the noise information that may be transferred in graph convolution and thus reduce adverse effects of noise on segmentation results, the partial class activation mapping is introduced. The partial class activation mapping can guide the information transmission between graph nodes and effectively activate vessel feature through classification labels, thereby improving the accuracy of segmentation. The performance of the proposed method is validated on four different fundus image datasets. Compared with existing state-of-the-art methods, the proposed method can improve the integrity of vessel to a certain extent when the pixel features of local vessels are significantly different, caused by objective factors such as inappropriate illumination and exudates. Moreover, the proposed method shows robustness when segmenting complex retinal vessels.

An interpretable Bayesian deep learning-based approach for sustainable clean energy

Sustainable Development Goal 7 is dedicated to ensuring access to clean and affordable energy that can be utilized in various applications. Solar panels (SP) are utilized to convert sunlight into electricity, acting as a renewable energy source. It is important to keep SP clean to obtain the required performance, as the accumulation of snow and dust on SP greatly affects the amount of electricity generated. On the other hand, excessive cleaning has some detrimental effects on the SP, therefore cleaning should only be done when necessary and not on a regular basis. Consequently, it is critical to determine whether the cleaning procedure is necessary by automatically detecting the presence of dust or snow on the panels while avoiding inaccurate predictions. Research efforts have been made to detect the presence of dust and snow on SP, but most of the proposed methods do not guarantee accurate detection results. This paper proposes an accurate, reliable, and interpretable approach called Solar-OBNet. The proposed Solar-OBNet can detect dusty SP and snow-covered SP very efficiently and be used in conjunction with the methods used to clean SP. The proposed Solar-OBNet is based on a Bayesian convolutional neural network, which enables it to express the amount of confidence in its predictions. Two measurements are used to estimate the uncertainty in the outcomes of the proposed Solar-OBNet, namely predictive entropy and standard deviation. The proposed Solar-OBNet can express confidence in the correct predictions by showing low values for predictive entropy and standard deviation. The proposed Solar-OBNet can also give an uncertainty warning in the case of erroneous predictions by showing high values of predictive entropy and standard deviation. The proposed Solar-OBNet’s efficacy was verified by interpreting its results using a method called Weighted Gradient-Directed Class Activation Mapping (Grad-CAM). The proposed Solar-OBNet has achieved a balanced accuracy of 94.07% and an average specificity 95.83%, outperforming other comparable methods.

Deep learning models for interpretation of point of care ultrasound in military working dogs.

Military working dogs (MWDs) are essential for military operations in a wide range of missions. With this pivotal role, MWDs can become casualties requiring specialized veterinary care that may not always be available far forward on the battlefield. Some injuries such as pneumothorax, hemothorax, or abdominal hemorrhage can be diagnosed using point of care ultrasound (POCUS) such as the Global FAST® exam. This presents a unique opportunity for artificial intelligence (AI) to aid in the interpretation of ultrasound images. In this article, deep learning classification neural networks were developed for POCUS assessment in MWDs. Images were collected in five MWDs under general anesthesia or deep sedation for all scan points in the Global FAST® exam. For representative injuries, a cadaver model was used from which positive and negative injury images were captured. A total of 327 ultrasound clips were captured and split across scan points for training three different AI network architectures: MobileNetV2, DarkNet-19, and ShrapML. Gradient class activation mapping (GradCAM) overlays were generated for representative images to better explain AI predictions. Performance of AI models reached over 82% accuracy for all scan points. The model with the highest performance was trained with the MobileNetV2 network for the cystocolic scan point achieving 99.8% accuracy. Across all trained networks the diaphragmatic hepatorenal scan point had the best overall performance. However, GradCAM overlays showed that the models with highest accuracy, like MobileNetV2, were not always identifying relevant features. Conversely, the GradCAM heatmaps for ShrapML show general agreement with regions most indicative of fluid accumulation. Overall, the AI models developed can automate POCUS predictions in MWDs. Preliminarily, ShrapML had the strongest performance and prediction rate paired with accurately tracking fluid accumulation sites, making it the most suitable option for eventual real-time deployment with ultrasound systems. Further integration of this technology with imaging technologies will expand use of POCUS-based triage of MWDs.

A multi-modal joint attention network for vibro-acoustic fusion diagnosis of engines

Abstract Deep learning has proven to be effective in diagnosing faults in power machinery and its diagnosis performance relies on a sufficient data set. In practice, a well-labeled data set with sufficient samples is very rare, especially for those machinery running in varying loading cases. The situation is particularly pronounced for multi-cylinder internal combustion engines, where the excitations from cylinders interact with significant background noise, and different data distributions are complicated. To tackle these issues, we propose a novelty multi-modal joint attention network (MJA-Net) for fusing the vibration and acoustic signals for diagnosing multiple faults. In MJA-Net, feature maps from both modalities are input separately into the convolutional module to learn independent features, and joint attention module (JAM) is utilized to enhance the vibro-acoustic information interaction and distribution consistency across modalities. The analysis of multiple loads vibro-acoustic experimental data shows that MJA-Net has a superior classification performance in limited sample tasks, compared to the single-modal methods. Furthermore, MJA-Net outperforms other fusion methods on average accuracy at 97.65%, as well as feature representativeness, and vibro-acoustic feature consistency across loads. JAM has superior diagnosis performance to other alternative modules. The class activation maps (CAM) generated by the Layer CAM highlight the key impact components related to the engine working mechanisms, providing valuable insight into MJA-Net’s interpretation for multi-fault recognition.

Interaction between clinicians and artificial intelligence to detect fetal atrioventricular septal defects on ultrasound: how can we optimize collaborative performance?

Artificial intelligence (AI) has shown promise in improving the performance of fetal ultrasound screening in detecting congenital heart disease (CHD). The effect of giving AI advice to human operators has not been studied in this context. Giving additional information about AI model workings, such as confidence scores for AI predictions, may be a way of further improving performance. Our aims were to investigate whether AI advice improved overall diagnostic accuracy (using a single CHD lesion as an exemplar), and to determine what, if any, additional information given to clinicians optimized the overall performance of the clinician-AI team. An AI model was trained to classify a single fetal CHD lesion (atrioventricular septal defect (AVSD)), using a retrospective cohort of 121 130 cardiac four-chamber images extracted from 173 ultrasound scan videos (98 with normal hearts, 75 with AVSD); a ResNet50 model architecture was used. Temperature scaling of model prediction probability was performed on a validation set, and gradient-weighted class activation maps (grad-CAMs) produced. Ten clinicians (two consultant fetal cardiologists, three trainees in pediatric cardiology and five fetal cardiac sonographers) were recruited from a center of fetal cardiology to participate. Each participant was shown 2000 fetal four-chamber images in a random order (1000 normal and 1000 AVSD). The dataset comprised 500 images, each shown in four conditions: (1) image alone without AI output; (2) image with binary AI classification; (3) image with AI model confidence; and (4) image with grad-CAM image overlays. The clinicians were asked to classify each image as normal or AVSD. A total of 20 000 image classifications were recorded from 10 clinicians. The AI model alone achieved an accuracy of 0.798 (95% CI, 0.760-0.832), a sensitivity of 0.868 (95% CI, 0.834-0.902) and a specificity of 0.728 (95% CI, 0.702-0.754), and the clinicians without AI achieved an accuracy of 0.844 (95% CI, 0.834-0.854), a sensitivity of 0.827 (95% CI, 0.795-0.858) and a specificity of 0.861 (95% CI, 0.828-0.895). Showing a binary (normal or AVSD) AI model output resulted in significant improvement in accuracy to 0.865 (P < 0.001). This effect was seen in both experienced and less-experienced participants. Giving incorrect AI advice resulted in a significant deterioration in overall accuracy, from 0.761 to 0.693 (P < 0.001), which was driven by an increase in both Type-I and Type-II errors by the clinicians. This effect was worsened by showing model confidence (accuracy, 0.649; P < 0.001) or grad-CAM (accuracy, 0.644; P < 0.001). AI has the potential to improve performance when used in collaboration with clinicians, even if the model performance does not reach expert level. Giving additional information about model workings such as model confidence and class activation map image overlays did not improve overall performance, and actually worsened performance for images for which the AI model was incorrect. © 2024 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

Assessing the Efficacy of Synthetic Optic Disc Images for Detecting Glaucomatous Optic Neuropathy Using Deep Learning.

Deep learning architectures can automatically learn complex features and patterns associated with glaucomatous optic neuropathy (GON). However, developing robust algorithms requires a large number of data sets. We sought to train an adversarial model for generating high-quality optic disc images from a large, diverse data set and then assessed the performance of models on generated synthetic images for detecting GON. A total of 17,060 (6874 glaucomatous and 10,186 healthy) fundus images were used to train deep convolutional generative adversarial networks (DCGANs) for synthesizing disc images for both classes. We then trained two models to detect GON, one solely on these synthetic images and another on a mixed data set (synthetic and real clinical images). Both the models were externally validated on a data set not used for training. The multiple classification metrics were evaluated with 95% confidence intervals. Models' decision-making processes were assessed using gradient-weighted class activation mapping (Grad-CAM) techniques. Following receiver operating characteristic curve analysis, an optimal cup-to-disc ratio threshold for detecting GON from the training data was found to be 0.619. DCGANs generated high-quality synthetic disc images for healthy and glaucomatous eyes. When trained on a mixed data set, the model's area under the receiver operating characteristic curve attained 99.85% on internal validation and 86.45% on external validation. Grad-CAM saliency maps were primarily centered on the optic nerve head, indicating a more precise and clinically relevant attention area of the fundus image. Although our model performed well on synthetic data, training on a mixed data set demonstrated better performance and generalization. Integrating synthetic and real clinical images can optimize the performance of a deep learning model in glaucoma detection. Optimizing deep learning models for glaucoma detection through integrating DCGAN-generated synthetic and real-world clinical data can be improved and generalized in clinical practice.

LA-ViT: A Network With Transformers Constrained by Learned-Parameter-Free Attention for Interpretable Grading in a New Laryngeal Histopathology Image Dataset.

Grading laryngeal squamous cell carcinoma (LSCC) based on histopathological images is a clinically significant yet challenging task. However, more low-effect background semantic information appeared in the feature maps, feature channels, and class activation maps, which caused a serious impact on the accuracy and interpretability of LSCC grading. While the traditional transformer block makes extensive use of parameter attention, the model overlearns the low-effect background semantic information, resulting in ineffectively reducing the proportion of background semantics. Therefore, we propose an end-to-end network with transformers constrained by learned-parameter-free attention (LA-ViT), which improve the ability to learn high-effect target semantic information and reduce the proportion of background semantics. Firstly, according to generalized linear model and probabilistic, we demonstrate that learned-parameter-free attention (LA) has a stronger ability to learn highly effective target semantic information than parameter attention. Secondly, the first-type LA transformer block of LA-ViT utilizes the feature map position subspace to realize the query. Then, it uses the feature channel subspace to realize the key, and adopts the average convergence to obtain a value. And those construct the LA mechanism. Thus, it reduces the proportion of background semantics in the feature maps and feature channels. Thirdly, the second-type LA transformer block of LA-ViT uses the model probability matrix information and decision level weight information to realize key and query, respectively. And those realize the LA mechanism. So, it reduces the proportion of background semantics in class activation maps. Finally, we build a new complex semantic LSCC pathology image dataset to address the problem, which is less research on LSCC grading models because of lacking clinically meaningful datasets. After extensive experiments, the whole metrics of LA-ViT outperform those of other state-of-the-art methods, and the visualization maps match better with the regions of interest in the pathologists' decision-making. Moreover, the experimental results conducted on a public LSCC pathology image dataset show that LA-ViT has superior generalization performance to that of other state-of-the-art methods.

Interpretable artificial intelligence-based analysis for morphologic classification of neuroblastic tumors.

10023 Background: Rapid and accurate identification of morphologic features of neuroblastic tumors (NTs) is critical for risk stratification and therapeutic decision making. The prognostic value of features like neuroblast differentiation, mitosis-karyorrhexis index (MKI), and Schwannian stromal presence is well established. Deep learning permits objective histopathological analysis, streamlining workflows for pathologists, notably in rare cancers. In rare cancers, our method minimizes bias and optimizes limited data using transfer and self-supervised learning (SSL) for feature extraction, with improved explainability. Here, we used an artificial intelligence-based model to morphologically classify NT tumors and MYCN-amplification. Methods: Annotated H&amp;E-stained slides of diagnostic NT tumor biopsies from the University of Chicago and the Children’s Oncology Group were digitalized. Pathologists defined three binarized measures including diagnostic category (ganglioneuroblastoma/neuroblastoma), grade (differentiating/poorly differentiating), and MKI (low and intermediate/high). MYCN status was abstracted from patient records (amplified/non-amplified). Using Slideflow, our open-source pipeline, we developed an attention-based multiple instance learning model with features extracted by CTransPath, a SSL model pretrained on pan-cancer images from The Cancer Genome Atlas. For each measure, model performance was evaluated using 5-fold cross validation by aggregating k-fold model predictions across multiple metrics. Patients were excluded from a model if the measure of interest was unknown. Feature significance was assessed visually using Class Activation Mapping (Grad-CAM). Results: The mean age of the study cohort (n = 172) was 3.66 years. Of patients with clinical information, 84 of 138 (60.2%) had metastatic disease and 94 of 133 (70.7%) were high-risk. Of the 148 tumors with a diagnostic category of neuroblastoma, 93.2% were poorly differentiated and 25% had high MKI. Of the 135 tumors with known MYCN status, 40 were amplified (29.6%). The final models excelled across all outcomes, performing best for diagnostic category, grade, and MYCN status (Table 1). Physician review of the attention-based heatmaps for all measures highlighted biologically relevant regions such as neuropil. Conclusions: We created a deep learning pipeline for auto-characterization of digitized H&amp;E-stained NT pathology slides. Our approach may also aid in identifying molecular features including MYCN-amplification. Review of heatmaps showed pertinent biological tissue, boosting model reliability.[Table: see text]

Breast cancer detection through attention based feature integration model

&lt;span lang="EN-US"&gt;Breast cancer is detected by screening mammography wherein X-rays are used to produce images of the breast. Mammograms for screening can detect breast cancer early. This research focuses on the challenges of using multi-view mammography to diagnose breast cancer. By examining numerous perspectives of an image, an attention-based feature-integration mechanism (AFIM) model that concentrates on local abnormal areas associated with cancer and displays the essential features considered for evaluation, analyzing cross-view data. This is segmented into two views the bi-lateral attention module (BAM) module integrates the left and right activation maps for a similar projection is used to create a spatial attention map that highlights the impact of asymmetries. Here the module's focus is on data gathering through medio-lateral oblique (MLO) and bilateral craniocaudal (CC) for each breast to develop an attention module. The proposed AFIM model generates using spatial attention maps obtained from the identical image through other breasts to identify bilaterally uneven areas and&lt;/span&gt;&lt;span lang="EN-US"&gt;class activation map (CAM) generated from two similar breast images to emphasize the feature channels connected to a single lesion in a breast. AFIM model may easily be included in ResNet-style architectures to develop multi-view classification models.&lt;/span&gt;

ScribFormer: Transformer Makes CNN Work Better for Scribble-Based Medical Image Segmentation.

Most recent scribble-supervised segmentation methods commonly adopt a CNN framework with an encoder-decoder architecture. Despite its multiple benefits, this framework generally can only capture small-range feature dependency for the convolutional layer with the local receptive field, which makes it difficult to learn global shape information from the limited information provided by scribble annotations. To address this issue, this paper proposes a new CNN-Transformer hybrid solution for scribble-supervised medical image segmentation called ScribFormer. The proposed ScribFormer model has a triple-branch structure, i.e., the hybrid of a CNN branch, a Transformer branch, and an attention-guided class activation map (ACAM) branch. Specifically, the CNN branch collaborates with the Transformer branch to fuse the local features learned from CNN with the global representations obtained from Transformer, which can effectively overcome limitations of existing scribble-supervised segmentation methods. Furthermore, the ACAM branch assists in unifying the shallow convolution features and the deep convolution features to improve model's performance further. Extensive experiments on two public datasets and one private dataset show that our ScribFormer has superior performance over the state-of-the-art scribble-supervised segmentation methods, and achieves even better results than the fully-supervised segmentation methods. The code is released at https://github.com/HUANGLIZI/ScribFormer.

Artificial Intelligence vs. Doctors: Diagnosing Necrotizing Enterocolitis on Abdominal Radiographs

BackgroundRadiographic diagnosis of necrotizing enterocolitis (NEC) is challenging. Deep learning models may improve accuracy by recognizing subtle imaging patterns. We hypothesized it would perform with comparable accuracy to that of senior surgical residents. MethodsThis cohort study compiled 494 anteroposterior neonatal abdominal radiographs (214 images NEC, 280 other) and randomly divided them into training, validation, and test sets. Transfer learning was utilized to fine-tune a ResNet-50 deep convolutional neural network (DCNN) pre-trained on ImageNet. Gradient-weighted Class Activation Mapping (Grad-CAM) heatmaps visualized image regions of greatest relevance to the pretrained neural network. Senior surgery residents at a single institution examined the test set. Resident and DCNN ability to identify pneumatosis on radiographic images were measured via area under the receiver operating curves (AUROC) and compared using DeLong's method. ResultsThe pretrained neural network achieved AUROC of 0.918 (95% CI, 0.837–0.978) with an accuracy of 87.8% with five false negative and one false positive prediction. Heatmaps confirmed appropriate image region emphasis by the pretrained neural network. Senior surgical residents had a median area under the receiver operating curve of 0.896, ranging from 0.778 (95% CI 0.615–0.941) to 0.991 (95% CI 0.971–0.999) with zero to five false negatives and one to eleven false positive predictions. The deep convolutional neural network performed comparably to each surgical resident's performance (p > 0.05 for all comparisons). ConclusionsA deep convolutional neural network trained to recognize pneumatosis can quickly and accurately assist clinicians in promptly identifying NEC in clinical practice. Level of EvidenceIII (study type: Study of Diagnostic Test, study of nonconsecutive patients without a universally applied “gold standard”)

A developed convolutional neural network model for accurately and stably predicting effective thermal conductivity of gradient porous ceramic materials

Accurate and stable prediction of the effective thermal conductivity (ETC) of porous ceramic materials is of great significance for their application in areas such as optimizing the design of thermal barrier coatings and improving energy conversion efficiency. Porous ceramic materials exhibit complex porous structures with gradient porosity distributions that pose a great challenge for effective medium theory (EMT) and conventional convolutional neural networks (CNN) in attempting to accurately and stably predict the ETC of porous media. In this study, a CNN model that integrates self-attention and a multiscale feature-fusion mechanism is proposed to predict the ETC of porous media with greater accuracy and stability. The integration of the self-attention and multiscale feature-fusion mechanisms enhances the CNN's ability to learn long-range dependencies and preserve detailed information. The optimization of the CNN's accuracy and stability is visually illustrated using gradient-weighted class activation mapping (Grad-CAM) for ETC. Additionally, by employing the proposed gradient quartet structure generation set (QSGS), a gradient porous ceramic media dataset comprising 10 000 images was built to train the CNN model. Finally, the prediction results and relative error distribution of the ETC were compared across different models. Our model demonstrated improvements in statistical metrics, including a 33.7 % decrease in mean error, 25.2 % decrease in median error, and 59.6 % decrease in maximum error. The decreases in these metrics and Grad-CAM for the ETC strongly demonstrates that the model proposed in this work greatly improved the accuracy and stability of predicting the ETC of ceramic materials with gradient porosity distributions.

DecomCAM: Advancing beyond saliency maps through decomposition and integration

Interpreting complex deep networks, notably pre-trained vision-language models (VLMs), is a formidable challenge. Current Class Activation Map (CAM) methods highlight regions revealing the model’s decision-making basis but lack clear saliency maps and detailed interpretability. To bridge this gap, we propose DecomCAM, a novel decomposition-and-integration method that distills shared patterns from channel activation maps. Utilizing singular value decomposition, DecomCAM decomposes class-discriminative activation maps into orthogonal sub-saliency maps (OSSMs), which are then integrated together based on their contribution to the target concept. Extensive experiments on six benchmarks reveal that DecomCAM not only excels in locating accuracy but also achieves an optimizing balance between interpretability and computational efficiency. Further analysis unveils that OSSMs correlate with discernible object components, facilitating a granular understanding of the model’s reasoning. This positions DecomCAM as a potential tool for fine-grained interpretation of advanced deep learning models. The code is avaible at https://github.com/CapricornGuang/DecomCAM.

Traffic Sign Recognition using HIICNN Stack Ensemble Method

Abstract: This project introduces a novel deep convolutional neural network (CNN) model aimed at enhancing traffic sign recognition for autonomous vehicle technology and road safety. Employing a stacking ensemble approach, we combine multiple CNNs with diverse improvement techniques, including preprocessing for adverse weather conditions and shooting errors, and data augmentation to handle class imbalance. Through adjustments in learning rates during model training, we mitigate overfitting. Our ensemble model achieves an exceptional 99.4 percent test accuracy on the German Traffic Sign Recognition Benchmark (GTSRB) dataset, surpassing prior studies. We utilize gradient-weighted Class Activation Mapping (Grad-CAM) for model explain- ability and an evidential deep learning approach to quantify classification uncertainty. Our framework underscores the efficacy of combining preprocessing, ensemble learning, and transfer learning for superior perfor- mance and reliability in traffic sign recognition systems, contributing significantly to autonomous driving and road safety.