Medical Image Datasets Research Articles

Rectify ViT Shortcut Learning by Visual Saliency.

Shortcut learning in deep learning models occurs when unintended features are prioritized, resulting in degenerated feature representations and reduced generalizability and interpretability. However, shortcut learning in the widely used vision transformer (ViT) framework is largely unknown. Meanwhile, introducing domain-specific knowledge is a major approach to rectifying the shortcuts that are predominated by background-related factors. For example, eye-gaze data from radiologists are effective human visual prior knowledge that has the great potential to guide the deep learning models to focus on meaningful foreground regions. However, obtaining eye-gaze data can still sometimes be time-consuming, labor-intensive, and even impractical. In this work, we propose a novel and effective saliency-guided ViT (SGT) model to rectify shortcut learning in ViT with the absence of eye-gaze data. Specifically, a computational visual saliency model (either pretrained or fine-tuned) is adopted to predict saliency maps for input image samples. Then, the saliency maps are used to filter the most informative image patches. Considering that this filter operation may lead to global information loss, we further introduce a residual connection that calculates the self-attention across all the image patches. The experiment results on natural and medical image datasets show that our SGT framework can effectively learn and leverage human prior knowledge without eye-gaze data and achieves much better performance than baselines. Meanwhile, it successfully rectifies the harmful shortcut learning and significantly improves the interpretability of the ViT model, demonstrating the promise of transferring human prior knowledge derived visual saliency in rectifying shortcut learning.

Predicting Radiologists' Gaze With Computational Saliency Models in Mammogram Reading

Previous studies have shown that there is a strong correlation between radiologists' diagnoses and their gaze when reading medical images. The extent to which gaze is attracted by content in a visual scene can be characterised as visual saliency. There is a potential for the use of visual saliency in computer-aided diagnosis in radiology. However, little is known about what methods are effective for diagnostic images, and how these methods could be adapted to address specific applications in diagnostic imaging. In this study, we investigate 20 state-of-the-art saliency models including 10 traditional models and 10 deep learning-based models in predicting radiologists' visual attention while reading 196 mammograms. We found that deep learning-based models represent the most effective type of methods for predicting radiologists' gaze in mammogram reading; and that the performance of these saliency models can be significantly improved by transfer learning. In particular, an enhanced model can be achieved by pre-training the model on a large-scale natural image saliency dataset and then fine-tuning it on the target medical image dataset. In addition, based on a systematic selection of backbone networks and network architectures, we proposed a parallel multi-stream encoded model which outperforms the state-of-the-art approaches for predicting saliency of mammograms.

Bilateral Context Modeling for Residual Coding in Lossless 3D Medical Image Compression.

Residual coding has gained prevalence in lossless compression, where a lossy layer is initially employed and the reconstruction errors (i.e., residues) are then losslessly compressed. The underlying principle of the residual coding revolves around the exploration of priors based on context modeling. Herein, we propose a residual coding framework for 3D medical images, involving the off-the-shelf video codec as the lossy layer and a Bilateral Context Modeling based Network (BCM-Net) as the residual layer. The BCM-Net is proposed to achieve efficient lossless compression of residues through exploring intra-slice and inter-slice bilateral contexts. In particular, a symmetry-based intra-slice context extraction (SICE) module is proposed to mine bilateral intra-slice correlations rooted in the inherent anatomical symmetry of 3D medical images. Moreover, a bi-directional inter-slice context extraction (BICE) module is designed to explore bilateral inter-slice correlations from bi-directional references, thereby yielding representative inter-slice context. Experiments on popular 3D medical image datasets demonstrate that the proposed method can outperform existing state-of-the-art methods owing to efficient redundancy reduction. Our code will be available on GitHub for future research.

Denoising diffusion probabilistic models for addressing data limitations in chest X-ray classification

Deep learning plays a crucial role in medical imaging analysis, particularly in tasks such as image classification and segmentation. However, learning from medical imaging datasets presents challenges, including scarcity of labeled examples, class imbalances, and inadequate representation of diverse patient populations. To address these challenges, there has been a growing interest in the use of deep generative models to create synthetic training data, with denoising diffusion probabilistic models (DDPMs) recently gaining attention for their ability to produce realistic and high-quality images. This study explores the potential of a DDPM to generate synthetic chest X-rays for multi-label classifier training. The results indicate that the use of a conditional DDPM has the potential to produce a realistic training set of synthetic chest X-rays. In addition, the study analyzes the impact on classification performance of addressing class imbalance. Balancing the synthetic training set increased the overall classification sensitivity from 0.02 to 0.59, but decreased the overall specificity from 0.99 to 0.71. Furthermore, we investigated the potential of unconditional pre-training to learn general representations, followed by conditional fine-tuning of the DDPM. The results indicate that this approach allows the amount of labeled training data to be reduced to 25% of the original set. Finally, we demonstrate that fidelity and classification metrics do not consistently exhibit the same trends. Integrating a DDPM into the classification pipeline underscores the benefits of having optimal control over the data and efficient use of available unlabeled data. Our research provides insights for making informed decisions about integrating generative models into medical image analysis.

Contrastive Registration for Unsupervised Medical Image Segmentation.

Medical image segmentation is an important task in medical imaging, as it serves as the first step for clinical diagnosis and treatment planning. While major success has been reported using deep learning supervised techniques, they assume a large and well-representative labeled set. This is a strong assumption in the medical domain where annotations are expensive, time-consuming, and inherent to human bias. To address this problem, unsupervised segmentation techniques have been proposed in the literature. Yet, none of the existing unsupervised segmentation techniques reach accuracies that come even near to the state-of-the-art of supervised segmentation methods. In this work, we present a novel optimization model framed in a new convolutional neural network (CNN)-based contrastive registration architecture for unsupervised medical image segmentation called CLMorph. The core idea of our approach is to exploit image-level registration and feature-level contrastive learning, to perform registration-based segmentation. First, we propose an architecture to capture the image-to-image transformation mapping via registration for unsupervised medical image segmentation. Second, we embed a contrastive learning mechanism in the registration architecture to enhance the discriminative capacity of the network at the feature level. We show that our proposed CLMorph technique mitigates the major drawbacks of existing unsupervised techniques. We demonstrate, through numerical and visual experiments, that our technique substantially outperforms the current state-of-the-art unsupervised segmentation methods on two major medical image datasets.

Pyramid Pixel Context Adaption Network for Medical Image Classification With Supervised Contrastive Learning.

Spatial attention (SA) mechanism has been widely incorporated into deep neural networks (DNNs), significantly lifting the performance in computer vision tasks via long-range dependency modeling. However, it may perform poorly in medical image analysis. Unfortunately, the existing efforts are often unaware that long-range dependency modeling has limitations in highlighting subtle lesion regions. To overcome this limitation, we propose a practical yet lightweight architectural unit, pyramid pixel context adaption (PPCA) module, which exploits multiscale pixel context information to recalibrate pixel position in a pixel-independent manner dynamically. PPCA first applies a well-designed cross-channel pyramid pooling (CCPP) to aggregate multiscale pixel context information, then eliminates the inconsistency among them by the well-designed pixel normalization (PN), and finally estimates per pixel attention weight via a pixel context integration. By embedding PPCA into a DNN with negligible overhead, the PPCA network (PPCANet) is developed for medical image classification. In addition, we introduce supervised contrastive learning to enhance feature representation by exploiting the potential of label information via supervised contrastive loss (CL). The extensive experiments on six medical image datasets show that the PPCANet outperforms state-of-the-art (SOTA) attention-based networks and recent DNNs. We also provide visual analysis and ablation study to explain the behavior of PPCANet in the decision-making process.

Transfer Learning Strategies for Fine-Tuning Pretrained Convolutional Neural Networks in Medical Imaging

In the area of medical imaging, transfer learning has become a potent technique that uses pretrained Convolutional Neural Networks (CNNs) to improve the performance of particular tasks. An overview of several transfer learning techniques used for optimising pretrained CNNs in the context of medical image analysis is given in this abstract. The size limitations of medical imaging datasets make it difficult to train deep learning models from scratch. Pre-trained CNNs are a good place to start, such as those that have been trained on huge natural picture datasets like ImageNet. When these pre-trained models are applied to medical imaging applications, fine-tuning is frequently used. One common method is feature extraction, where the bottom layers of the pretrained CNN are frozen and operate as feature extractors. Then, for the specific medical task at hand, these features are loaded into a bespoke classifier. The ability of the pretrained network to recognise subtle picture patterns is advantageous in this method. Another strategy is to optimise the CNN architecture as a whole, which enables the model to adjust to the features of medical images. Small learning rates are frequently used in transfer learning techniques to avoid overfitting during fine-tuning. Additionally, to further enhance model generalisation, domain-specific data augmentation is essential. The use of ensemble approaches, which combine several pretrained CNNs, is also investigated. These models are capable of offering various feature representations and improving classification precision. In order to bridge the domain gap between natural photos and medical images, domain adaption techniques are also used. One approach to align feature distributions is by adversarial training, while another is through domain-specific batch normalisation. The feature extraction, network fine-tuning, ensemble approaches, and domain adaptation are all part of transfer learning methodologies for optimising pretrained CNNs in medical imaging. Researchers have made great progress using these techniques in a number of medical image processing tasks, proving the value of transfer learning in this important area.

MSDANet: A multi-scale dilation attention network for medical image segmentation

Deep learning shows excellent performance in medical image segmentation. However, a pooling operation in its encoding stage leads to feature loss and the ability of multi-scale contextual information extraction is insufficient. In addition, there is a semantic gap between low-level features and high-level features in the encoder–decoder. Inspired by an U-shaped network (UNet), this work proposes a multi-scale dilation attention network (MSDANet) for medical image segmentation. Specifically, it is mainly composed of a parallel dilation pooling module (PD), a multi-scale channel attention mechanism (MSCA), a multilayer perceptron squeeze and excitation module (MSE), a big kernel convolution module (BC), and a skip feature pyramid module (SFP). The proposed PD reduces the loss of subtle features during downsampling. In order to improve the extraction of effective features, MSCA, MSE and BC are used in encoding and decoding stages. In the skip connection stage, SPF is used to reduce the difference between the low-level and high-level semantic information to accelerate the network learning. We validate the proposed model on two publicly available medical image datasets. The results show that MSDANet has superior performance on several evaluation metrics compared to state-of-the-art methods. Code is available at https://github.com/1999luan/MSDANet.

Distributed Federated Learning-Based Deep Learning Model for Privacy MRI Brain Tumor Detection

Distributed training can facilitate the processing of large medical image datasets, and improve the accuracy and efficiency of disease diagnosis while protecting patient privacy, which is crucial for achieving efficient medical image analysis and accelerating medical research progress. This paper presents an innovative approach to medical image classification, leveraging Federated Learning (FL) to address the dual challenges of data privacy and efficient disease diagnosis. Traditional Centralized Machine Learning models, despite their widespread use in medical imaging for tasks such as disease diagnosis, raise significant privacy concerns due to the sensitive nature of patient data. As an alternative, FL emerges as a promising solution by allowing the training of a collective global model across local clients without centralizing the data, thus preserving privacy. Focusing on the application of FL in Magnetic Resonance Imaging (MRI) brain tumor detection, this study demonstrates the effectiveness of the Federated Learning framework coupled with EfficientNet-B0 and the FedAvg algorithm in enhancing both privacy and diagnostic accuracy. Through a meticulous selection of preprocessing methods, algorithms, and hyperparameters, and a comparative analysis of various Convolutional Neural Network (CNN) architectures, the research uncovers optimal strategies for image classification. The experimental results reveal that EfficientNet-B0 outperforms other models like ResNet in handling data heterogeneity and achieving higher accuracy and lower loss, highlighting the potential of FL in overcoming the limitations of traditional models. The study underscores the significance of addressing data heterogeneity and proposes further research directions for broadening the applicability of FL in medical image analysis.

A Study On a Machine Learning Based Classification Approach in Identifying Heart Disease Within E-Healthcare

The heart, as the second most vital organ after the brain, is integral to maintaining bodily equilibrium, and disruptions to its function have profound health consequences. Heart disease, a leading global cause of mortality, often arises from cumulative daily physiological changes, emphasizing the importance of timely illness prediction. In healthcare, the fusion of data mining and machine learning, explored in this study using Support Vector Machine, Decision Tree, and Random Forest algorithms, addresses the challenges of diagnosing prevalent conditions like heart disease, particularly crucial in the field of cardiology. Our proposed machine learning-based approach for diagnosing cardiac disease employs a range of classification algorithms and advanced feature selection techniques, demonstrating superior accuracy in detecting heart diseases from extensive datasets of unprocessed medical images. This technological advancement holds the potential to significantly enhance patient care in various healthcare settings, showcasing the promising impact of artificial intelligence tools on improving the quality of life for billions worldwide.

Brain tumor grade classification using multi‐step pre‐training

AbstractMedical images offer a non‐invasive method to diagnose different diseases, but using them manually produces unreliable results. Modern deep learning architectures and techniques are computed and data‐intensive, making them difficult to use for relatively smaller datasets of medical images. Transfer learning has been used as a remedy for the problem mentioned above. However, the domain difference between the datasets used for pre‐training (e.g., ImageNet) and the target datasets, like medical images, negatively impacts the transfer learning results. Recently, many researchers have used additional pre‐training called domain‐adaptive pre‐training (DAPT) using the data from the target domain (e.g., medical images) before using the model on the target tasks to achieve superior performance. This study proposes a variant of DAPT by performing it on a subset of the architecture. It has achieved state‐of‐the‐art performance for brain tumor grading on the BraTS 2019 while being computationally efficient.

Classification of hyper-scale multimodal imaging datasets

Algorithms that classify hyper-scale multi-modal datasets, comprising of millions of images, into constituent modality types can help researchers quickly retrieve and classify diagnostic imaging data, accelerating clinical outcomes. This research aims to demonstrate that a deep neural network that is trained on a hyper-scale dataset (4.5 million images) composed of heterogeneous multi-modal data can be used to obtain significant modality classification accuracy (96%). By combining 102 medical imaging datasets, a dataset of 4.5 million images was created. A ResNet-50, ResNet-18, and VGG16 were trained to classify these images by the imaging modality used to capture them (Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Positron Emission Tomography (PET), and X-ray) across many body locations. The classification accuracy of the models was then tested on unseen data. The best performing model achieved classification accuracy of 96% on unseen data, which is on-par, or exceeds the accuracy of more complex implementations using EfficientNets or Vision Transformers (ViTs). The model achieved a balanced accuracy of 86%. This research shows it is possible to train Deep Learning (DL) Convolutional Neural Networks (CNNs) with hyper-scale multimodal datasets, composed of millions of images. Such models can find use in real-world applications with volumes of image data in the hyper-scale range, such as medical imaging repositories, or national healthcare institutions. Further research can expand this classification capability to include 3D-scans.

Analysis of Colorectal and Gastric Cancer Classification: A Mathematical Insight Utilizing Traditional Machine Learning Classifiers

Cancer remains a formidable global health challenge, claiming millions of lives annually. Timely and accurate cancer diagnosis is imperative. While numerous reviews have explored cancer classification using machine learning and deep learning techniques, scant literature focuses on traditional ML methods. In this manuscript, we undertake a comprehensive review of colorectal and gastric cancer detection specifically employing traditional ML classifiers. This review emphasizes the mathematical underpinnings of cancer detection, encompassing preprocessing techniques, feature extraction, machine learning classifiers, and performance assessment metrics. We provide mathematical formulations for these key components. Our analysis is limited to peer-reviewed articles published between 2017 and 2023, exclusively considering medical imaging datasets. Benchmark and publicly available imaging datasets for colorectal and gastric cancers are presented. This review synthesizes findings from 20 articles on colorectal cancer and 16 on gastric cancer, culminating in a total of 36 research articles. A significant focus is placed on mathematical formulations for commonly used preprocessing techniques, features, ML classifiers, and assessment metrics. Crucially, we introduce our optimized methodology for the detection of both colorectal and gastric cancers. Our performance metrics analysis reveals remarkable results: 100% accuracy in both cancer types, but with the lowest sensitivity recorded at 43.1% for gastric cancer.

A Recycling Training Strategy for Medical Image Segmentation with Diffusion Denoising Models

Denoising diffusion models have found applications in image segmentation by generating segmented masks conditioned on images. Existing studies predominantly focus on adjusting model architecture or improving inference, such as test-time sampling strategies. In this work, we focus on improving the training strategy and propose a novel recycling method. During each training step, a segmentation mask is first predicted given an image and a random noise. This predicted mask, which replaces the conventional ground truth mask, is used for denoising task during training. This approach can be interpreted as aligning the training strategy with inference by eliminating the dependence on ground truth masks for generating noisy samples. Our proposed method significantly outperforms standard diffusion training, self-conditioning, and existing recycling strategies across multiple medical imaging data sets: muscle ultrasound, abdominal CT, prostate MR, and brain MR. This holds for two widely adopted sampling strategies: denoising diffusion probabilistic model and denoising diffusion implicit model. Importantly, existing diffusion models often display a declining or unstable performance during inference, whereas our novel recycling consistently enhances or maintains performance. We show for the first time that, under a fair comparison with the same network architectures and computing budget, the proposed recycling-based diffusion models achieved on-par performance with non-diffusion-based supervised training. Furthermore, by ensembling the proposed diffusion model and the non-diffusion counterpart, significant improvements to the non-diffusion models have been observed across all applications, demonstrating the value of this novel training method. This paper summarizes these quantitative results and discusses their values, with a fully reproducible JAX-based implementation, released at <a href='https://github.com/mathpluscode/ImgX-DiffSeg'>https://github.com/mathpluscode/ImgX-DiffSeg</a>

A Review on Occluded Object Detection and Deep Learning-Based Approach in Medical Imaging-Related Research

Medical image occlusions can arise due to several factors, including anatomical features, imaging modality, and acquisition settings. Object occlusion occurrence has a very significant effect on detection accuracy in general and in medical cases, these effects hinder proper diagnosis and treatment plans that may be fatal for the patients. Thus, precise occluded object detection is imperative. This paper aims to review the various state-of-the-art models and approaches that had been proposed for the occluded object detection. The coverage of this paper includes occluded object detection models in other applications and medical imaging, its proposed Deep Learning implementations, hybrid Deep Learning, and statistical analysis that were integrated into Deep Learning models. It is found that, in overall, more annotated medical image datasets are required to reduce overfitting occurrence, numerous Deep Learning models and its hybrid combination’s applications yet to be tested of its limitations, and the extent of statistical analysis integration on Deep Learning models.

Comparing the Robustness of ResNet, Swin-Transformer, and MLP-Mixer under Unique Distribution Shifts in Fundus Images.

Diabetic retinopathy (DR) is the leading cause of visual impairment and blindness. Consequently, numerous deep learning models have been developed for the early detection of DR. Safety-critical applications employed in medical diagnosis must be robust to distribution shifts. Previous studies have focused on model performance under distribution shifts using natural image datasets such as ImageNet, CIFAR-10, and SVHN. However, there is a lack of research specifically investigating the performance using medical image datasets. To address this gap, we investigated trends under distribution shifts using fundus image datasets. We used the EyePACS dataset for DR diagnosis, introduced noise specific to fundus images, and evaluated the performance of ResNet, Swin-Transformer, and MLP-Mixer models under a distribution shift. The discriminative ability was evaluated using the Area Under the Receiver Operating Characteristic curve (ROC-AUC), while the calibration ability was evaluated using the monotonic sweep calibration error (ECE sweep). Swin-Transformer exhibited a higher ROC-AUC than ResNet under all types of noise and displayed a smaller reduction in the ROC-AUC due to noise. ECE sweep did not show a consistent trend across different model architectures. Swin-Transformer consistently demonstrated superior discrimination compared to ResNet. This trend persisted even under unique distribution shifts in the fundus images.

Covid-19 Classification using Convolutional Neural Networks Based on Adam, RMSP, and SGD Optimalization

In this comprehensive study, a meticulous analysis of the application of Convolutional Neural Network (CNN) methodologies in the classification of Covid-19 and non-Covid-19 cases was conducted. Leveraging diverse optimization techniques such as RMS, SGD, and Adam, the research systematically evaluated the performance of the CNN model in accurately discerning intricate patterns and distinct features associated with Covid-19 pathology. the implementation of the RMS and Adam optimization methods resulted in the highest accuracy levels, with both models achieving an impressive 98% accuracy in the classification of Covid-19 and non-Covid-19 cases. Leveraging the robust capabilities of these optimization techniques, the study successfully demonstrated the effectiveness of the RMS and Adam models in enhancing the precision and reliability of the Convolutional Neural Network (CNN) for the accurate identification and differentiation of Covid-19 patterns within the medical imaging datasets. The notable achievement of 98% accuracy further emphasizes the potential of these optimization methods in advancing the capabilities of CNN-based diagnostic tools, thus contributing significantly to the ongoing efforts in Covid-19 diagnosis and management.

Effect of Genetic Algorithm as a Feature Selection for Image Classification

Analysis of image content is important in the classification of images, identification, retrieval, and recognition processes. The medical image datasets for content-based medical image retrieval ( are large datasets that are limited by high computational costs and poor performance. The aim of the proposed method is to enhance this image retrieval and classification by using a genetic algorithm (GA) to choose the reduced features and dimensionality. This process was created in three stages. In the first stage, two algorithms are applied to extract the important features; the first algorithm is the Contrast Enhancement method and the second is a Discrete Cosine Transform algorithm. In the next stage, we used datasets of the medical images using GA-based feature selection to find feature vectors. Images from the datasets and images from the query are recognized using a correlation coefficient. The third stage of the proposed method used a diverse density algorithm feedback technique to enhance the performance of the . Images of breast cancer, brain cancer, lung cancer, thyroid cancer, etc., may be retrieved using the suggested procedure. By using a feature selection algorithm based on GA to determine the best subset of features, the challenge of system dimensionality is reduced. The suggested method has greater accuracy in precision, recall, and F-score than the other techniques.

Data Liberation and Crowdsourcing in Medical Research: The Intersection of Collective and Artificial Intelligence.

In spite of an exponential increase in the volume of medical data produced globally, much of these data are inaccessible to those who might best use them to develop improved health care solutions through the application of advanced analytics such as artificial intelligence. Data liberation and crowdsourcing represent two distinct but interrelated approaches to bridging existing data silos and accelerating the pace of innovation internationally. In this article, we examine these concepts in the context of medical artificial intelligence research, summarizing their potential benefits, identifying potential pitfalls, and ultimately making a case for their expanded use going forward. A practical example of a crowdsourced competition using an international medical imaging dataset is provided. Keywords: Artificial Intelligence, Data Liberation, Crowdsourcing © RSNA, 2023.

A Systematic Collection of Medical Image Datasets for Deep Learning

The astounding success made by artificial intelligence in healthcare and other fields proves that it can achieve human-like performance. However, success always comes with challenges. Deep learning algorithms are data dependent and require large datasets for training. Many junior researchers face a lack of data for a variety of reasons. Medical image acquisition, annotation, and analysis are costly, and their usage is constrained by ethical restrictions. They also require several other resources, such as professional equipment and expertise. That makes it difficult for novice and non-medical researchers to have access to medical data. Thus, as comprehensively as possible, this article provides a collection of medical image datasets with their associated challenges for deep learning research. We have collected the information of approximately 300 datasets and challenges mainly reported between 2007 and 2020 and categorized them into four categories: head and neck, chest and abdomen, pathology and blood, and others. The purpose of our work is to provide a list, as up-to-date and complete as possible, that can be used as a reference to easily find the datasets for medical image analysis and the information related to these datasets.