Medical Datasets Research Articles

A novel brain EEG clustering based on Minkowski distance to improve intelligent epilepsy diagnosis

This paper introduces a novel clustering approach based on Minkowski's mathematical similarity to improve EEG feature selection for classification and have efficient Particle Swarm Optimization (PSO) in the context of machine learning. Given the intricacy of high-dimensional medical datasets, feature selection plays a critical role in preventing disease and promoting public health. By employing Minkowski clustering, the objective is to group dataset records into two clusters exhibiting high feature coherence, thereby improving accuracy by applying optimization techniques like PSO to select the most optimal features. Furthermore, the proposed model can be extended to intelligent datasets, including EEG and others. As fewer features are needed for precise categorization, intelligent feature selection is an advanced step of machine learning. This paper investigates the key factors influencing feature selection in the EEG Bonn University dataset. The proposed system is compared against various optimization and feature selection methods, demonstrating superior performance in analyzing and classifying EEG signals based on accuracy measures. The experimental results have confirmed the effectiveness of the suggested model as a valuable tool for EEG data classification, achieving up to 100% accuracy. The outcomes of this research have the potential to benefit medical experts in related specialties by streamlining the process of identifying and diagnosing brain disorders. Technically, the machine learning algorithms RF, KNN, SVM, NB, and DT are employed to classify the selected features.

面向机器阅读理解的医学域数据集MedicalQA

Machine reading comprehension aims to make the computer understand the paragraph semantics and answer the questions raised by users using algorithms. The quality of the dataset used in this task can directly affect the experimental results of the model. In order to enrich the medical domain dataset of machine reading comprehension, this paper constructs MedicalQA, a medical domain dataset for machine reading comprehension, employing a combination of web crawlers and manual annotation techniques. The dataset takes two medical platforms (i.e. Xunyiwenyao Network and 39 Health Network) as main data sources, and includes 19,502 paragraphs and Q & A pairs, covering 9 medical departments, such as internal medicine, surgery, obstetrics and gynecology. The dataset is formatted as an Excel file, organized with 5e columns. The first column denotes the paragraph ID; the second column indicates the department to which the paragraph belongs; the third column contains the paragraph content; the fourth column lists the questions, and the fifth column provides corresponding answers to the questions. The construction of this dataset is conducive to the establishment of machine reading comprehension models in the medical domain, and can also promote the sharing of medical datasets in the field of machine reading comprehension.

Utilizing Nearest-Neighbor Clustering for Addressing Imbalanced Datasets in Bioengineering.

Imbalance classification is common in scenarios like fault diagnosis, intrusion detection, and medical diagnosis, where obtaining abnormal data is difficult. This article addresses a one-class problem, implementing and refining the One-Class Nearest-Neighbor (OCNN) algorithm. The original inter-quartile range mechanism is replaced with the K-means with outlier removal (KMOR) algorithm for efficient outlier identification in the target class. Parameters are optimized by treating these outliers as non-target-class samples. A new algorithm, the Location-based Nearest-Neighbor (LBNN) algorithm, clusters one-class training data using KMOR and calculates the farthest distance and percentile for each test data point to determine if it belongs to the target class. Experiments cover parameter studies, validation on eight standard imbalanced datasets from KEEL, and three applications on real medical imbalanced datasets. Results show superior performance in precision, recall, and G-means compared to traditional classification models, making it effective for handling imbalanced data challenges.

Advancements and Challenges of Machine Learning and Deep Learning in Medical Data Analysis: A Review of the Past Five Years and Future Directions

This literature review explores machine learning (ML) and deep learning (DL) methods applied to medical datasets over the past five years. The paper discusses the advancements, challenges, and future directions in utilizing ML and DL techniques for medical data analysis. It synthesizes recent research findings, highlighting key methodologies, datasets, and outcomes.

COVID-19 Image Classification: A Comparative Performance Analysis of Hand-Crafted vs. Deep Features

This study investigates techniques for medical image classification, specifically focusing on COVID-19 scans obtained through computer tomography (CT). Firstly, handcrafted methods based on feature engineering are explored due to their suitability for training traditional machine learning (TML) classifiers (e.g., Support Vector Machine (SVM)) when faced with limited medical image datasets. In this context, I comprehensively evaluate and compare 27 descriptor sets. More recently, deep learning (DL) models have successfully analyzed and classified natural and medical images. However, the scarcity of well-annotated medical images, particularly those related to COVID-19, presents challenges for training DL models from scratch. Consequently, I leverage deep features extracted from 12 pre-trained DL models for classification tasks. This work presents a comprehensive comparative analysis between TML and DL approaches in COVID-19 image classification.

Performance Evaluation of Machine Learning Model in Predicting Heart Disease Prevalent

Heart disease stands as the foremost contributor to global mortality in the present era. While a range of medical tests aids in diagnosing various manifestations of heart disease, anticipating its occurrence without resorting to these tests presents a formidable challenge. Machine learning has revealed a potential path for comprehending extensive medical datasets, unearthing concealed insights that elude human observation. Random Forest, Logistic Regression, XG Boost, k-nearest neighbor, and Gradient boosting was used to predict instances of heart diseases based on age and gender in this study. Regardless of the dataset's generalization or gender-specific characteristics, we observed that Random Forest, XG Boost, and K-Nearest Neighbors models exhibit high robustness and minimal sensitivity to hyperparameters. While the prediction accuracy remains consistent among Random Forest, Xgboost, and K-Nearest Neighbors, the crucial features responsible for achieving that accuracy vary among them. This observation indicates that it is important to test different machine learning algorithms when the interest is in prediction and understanding the most important feature(s). Keywords: Machine Learning, Heart Disease, Diagnosis, Predictions, Accuracy CISDI Journal Reference Format Abiola O. A. Adebisi R. O. Adeyemo A. B. (2024): Performance Evaluation of Machine Learning Model in Predicting Heart Disease Prevalent. Computing, Information Systems, Development Informatics & Allied Research Journal. Vol 15 No 1, Pp 63-72... dx.doi.org/10.22624/AIMS/CISDI/V15N1P5. Available online at www.isteams.net/cisdijournal

Stock market and the psychological health of investors

AbstractUtilizing a national individual‐level medical dataset and the home bias phenomenon in investment, our study shows a strong and robust link between declines in local stock returns and increased antidepressant consumption among investors. This effect intensifies in areas with higher per capita dividend income, suggesting a direct relationship between higher stock ownership and stronger responses. We confirm that portfolio losses, not local economic conditions, are responsible for increased antidepressant usage during market downturns. Using the frequency of psychotherapy sessions yields similar findings. Moreover, our study supports the loss aversion hypothesis as we find positive stock returns do not influence antidepressant usage.

Prediction model for diabetes mellitus using machine learning algorithms for enhanced diagnosis and prognosis in healthcare

<p>Diabetes mellitus (DM) affects the hormone insulin, which causes improper glucose metabolism and raises the body’s blood sugar levels. With 4.2 million fatalities in 2019, DM is one of the top 10 global causes of mortality. Early detection of DM will aid in its treatment and avert complications. There must be a quick and simple technique to diagnose it. Such diseases can be managed and human lives can be saved with early diagnosis. Smart prediction techniques like Machine Learning (ML) have produced encouraging outcomes in predictive classifications. There has been a lot of interest in ML-based decision-support platforms for the prediction of chronic illnesses to provide improved diagnosis and prognosis help to medical professionals and the general population. By building predictive models using diagnostic medical datasets gathered from DM patients, ML algorithms efficiently extract knowledge that helps predict diabetic individuals. The association between DM and a healthy lifestyle is used in the model. In this study, the NHANES (National Health and Nutrition Examination Survey) data set is utilized, and five ML methods such as Artificial Neural Networks (ANN), CATBoost, XGBoost, XGBoost-histogram, and Light GBM to predict DM. The results of the experiment demonstrate that the XGB-h model outperformed other ML methods regarding area under the receiver operating characteristic curve (AUC-ROC), and accuracy. The most effective XGB-h framework can be used in a mobile app and a website to rapidly forecast DM. Real-time prediction using details delivered by the model at runtime can be developed as a whole bundle as a product. Clinicians can quickly determine who is likely to get diabetes using the proposed strategy, which will facilitate prompt intervention and caring.</p>

MFHOD: Multi-modal image fusion method based on the higher-order degradation model

The task of multimodal image fusion aims to preserve the respective advantages of each modality, such as the detailed texture information from visible light images and the salient target information from infrared images. However, images in real environments are often not perfect high-quality images but are affected by various degradation factors such as noise, blur, and glare. Existing unsupervised multimodal image fusion algorithms use undegraded source images as input to fusion networks and generate high-quality fusion images through complex fusion strategies. This single linear learning approach from high-quality to high-quality cannot be applied to the fusion task of degraded multimodal images in real environments. To address this issue, this paper proposes an end-to-end multimodal image fusion network based on a high-order local degradation model (MFHOD). Firstly, inspired by the idea of probabilistic degradation, we propose a high-order local random degradation model (HODM), which inputs the source multimodal images into the degradation model to obtain degraded images before feeding them into the network. Secondly, we design a simple and efficient dual-branch feature extraction encoder to extract deep features from images. Then, from the perspectives of image pixels, brightness, and gradients, we propose an improved composite loss composed of multiple loss functions to constrain network training. Finally, we propose a L2-norm fusion strategy for preserving brightness information in low-light nighttime images. Our MFHOD demonstrates good performance on infrared and visible light image datasets as well as medical image datasets. Experimental results show that MFHOD can effectively suppress the effects of glare, noise, and smoke in adverse environments, and also improve the quality of fusion images in low-light and nighttime environments.

Convolutional Autoencoder-Based medical image compression using a novel annotated medical X-ray imaging dataset

With the ongoing development of machine learning techniques, it is now necessary to train and evaluate these algorithms to have access to high-quality medical X-ray datasets. This study unfolds on two critical axes within the realm of medical imaging. We introduce the proposed Medical X-ray Imaging Dataset (MXID), a meticulously curated resource featuring images spanning 18 body parts. It has been refined to include a classification based on body type within each gender category. This dataset addresses the limitations of existing datasets by offering comprehensive coverage, precise annotations, and optimal image quality. While numerous datasets often fall short in terms of encompassing a broad range of anatomical regions, particularly those that are designed for multi-body component analysis. Our primary contribution lies in bridging the gap in available resources, providing a foundation of developing robust machine learning algorithms in medical imaging. Moreover, we extend our investigation to the integration of the MXID dataset into the domain of medical image compression, evaluating the performance of techniques such as Principal Component Analysis (PCA), K-means clustering, Convolutional Neural Network (CNN), Deep Convolutional Autoencoders (DCAEs), Autoencoders (AEs), and Variational Autoencoder (VAE), using two scenarios, one is for single image compression, and the other is for MXID dataset, which offer a unique perspective that enables a comprehensive assessment of these techniques across diverse anatomical regions. Our study contributes to advancing machine learning and deep learning applications in medical imaging, emphasizing the creation of a dataset and its integration into image compression.

UDT: U‐shaped deformable transformer for subarachnoid haemorrhage image segmentation

AbstractSubarachnoid haemorrhage (SAH), mostly caused by the rupture of intracranial aneurysm, is a common disease with a high fatality rate. SAH lesions are generally diffusely distributed, showing a variety of scales with irregular edges. The complex characteristics of lesions make SAH segmentation a challenging task. To cope with these difficulties, a u‐shaped deformable transformer (UDT) is proposed for SAH segmentation. Specifically, first, a multi‐scale deformable attention (MSDA) module is exploited to model the diffuseness and scale‐variant characteristics of SAH lesions, where the MSDA module can fuse features in different scales and adjust the attention field of each element dynamically to generate discriminative multi‐scale features. Second, the cross deformable attention‐based skip connection (CDASC) module is designed to model the irregular edge characteristic of SAH lesions, where the CDASC module can utilise the spatial details from encoder features to refine the spatial information of decoder features. Third, the MSDA and CDASC modules are embedded into the backbone Res‐UNet to construct the proposed UDT. Extensive experiments are conducted on the self‐built SAH‐CT dataset and two public medical datasets (GlaS and MoNuSeg). Experimental results show that the presented UDT achieves the state‐of‐the‐art performance.

FedA3I: Annotation Quality-Aware Aggregation for Federated Medical Image Segmentation against Heterogeneous Annotation Noise

Federated learning (FL) has emerged as a promising paradigm for training segmentation models on decentralized medical data, owing to its privacy-preserving property. However, existing research overlooks the prevalent annotation noise encountered in real-world medical datasets, which limits the performance ceilings of FL. In this paper, we, for the first time, identify and tackle this problem. For problem formulation, we propose a contour evolution for modeling non-independent and identically distributed (Non-IID) noise across pixels within each client and then extend it to the case of multi-source data to form a heterogeneous noise model (i.e., Non-IID annotation noise across clients). For robust learning from annotations with such two-level Non-IID noise, we emphasize the importance of data quality in model aggregation, allowing high-quality clients to have a greater impact on FL. To achieve this, we propose Federated learning with Annotation quAlity-aware AggregatIon, named FedA3I, by introducing a quality factor based on client-wise noise estimation. Specifically, noise estimation at each client is accomplished through the Gaussian mixture model and then incorporated into model aggregation in a layer-wise manner to up-weight high-quality clients. Extensive experiments on two real-world medical image segmentation datasets demonstrate the superior performance of FedA3I against the state-of-the-art approaches in dealing with cross-client annotation noise. The code is available at https://github.com/wnn2000/FedAAAI.

FedCD: Federated Semi-Supervised Learning with Class Awareness Balance via Dual Teachers

Recent advancements in deep learning have greatly improved the efficiency of auxiliary medical diagnostics. However, concerns over patient privacy and data annotation costs restrict the viability of centralized training models. In response, federated semi-supervised learning has garnered substantial attention from medical institutions. However, it faces challenges arising from knowledge discrepancies among local clients and class imbalance in non-independent and identically distributed data. Existing methods like class balance adaptation for addressing class imbalance often overlook low-confidence yet valuable rare samples in unlabeled data and may compromise client privacy. To address these issues, we propose a novel framework with class awareness balance and dual teacher distillation called FedCD. FedCD introduces a global-local framework to balance and purify global and local knowledge. Additionally, we introduce a novel class awareness balance module to effectively explore potential rare classes and encourage balanced learning in unlabeled clients. Importantly, our approach prioritizes privacy protection by only exchanging network parameters during communication. Experimental results on two medical datasets under various settings demonstrate the effectiveness of FedCD. The code is available at https://github.com/YunzZ-Liu/FedCD.

DDViT: Double-Level Fusion Domain Adapter Vision Transformer (Student Abstract)

With the help of Vision transformers (ViTs), medical image segmentation was able to achieve outstanding performance. In particular, they overcome the limitation of convolutional neural networks (CNNs) which rely on local receptive fields. ViTs use self-attention mechanisms to consider relationships between all image pixels or patches simultaneously. However, they require large datasets for training and did not perform well on capturing low-level features. To that end, we propose DDViT, a novel ViT model that unites a CNN to alleviate data-hunger for medical image segmentation with two multi-scale feature representations. Significantly, our approach incorporates a ViT with a plug-in domain adapter (DA) with Double-Level Fusion (DLF) technique, complemented by a mutual knowledge distillation paradigm, facilitating the seamless exchange of knowledge between a universal network and specialized domain-specific network branches. The DLF framework plays a pivotal role in our encoder-decoder architecture, combining the innovation of the TransFuse module with a robust CNN-based encoder. Extensive experimentation across diverse medical image segmentation datasets underscores the remarkable efficacy of DDViT when compared to alternative approaches based on CNNs and Transformer-based models.

Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-World Multi-Turn Dialogue

Recent advances in Large Language Models (LLMs) have achieved remarkable breakthroughs in understanding and responding to user intents. However, their performance lag behind general use cases in some expertise domains, such as Chinese medicine. Existing efforts to incorporate Chinese medicine into LLMs rely on Supervised Fine-Tuning (SFT) with single-turn and distilled dialogue data. These models lack the ability for doctor-like proactive inquiry and multi-turn comprehension and cannot align responses with experts' intentions. In this work, we introduce Zhongjing, the first Chinese medical LLaMA-based LLM that implements an entire training pipeline from continuous pre-training, SFT, to Reinforcement Learning from Human Feedback (RLHF). Additionally, we construct a Chinese multi-turn medical dialogue dataset of 70,000 authentic doctor-patient dialogues, CMtMedQA, which significantly enhances the model's capability for complex dialogue and proactive inquiry initiation. We also define a refined annotation rule and evaluation criteria given the unique characteristics of the biomedical domain. Extensive experimental results show that Zhongjing outperforms baselines in various capacities and matches the performance of ChatGPT in some abilities, despite the 100x parameters. Ablation studies also demonstrate the contributions of each component: pre-training enhances medical knowledge, and RLHF further improves instruction-following ability and safety. Our code, datasets, and models are available at https://github.com/SupritYoung/Zhongjing.

Variable Importance in High-Dimensional Settings Requires Grouping

Explaining the decision process of machine learning algorithms is nowadays crucial for both model’s performance enhancement and human comprehension. This can be achieved by assessing the variable importance of single variables, even for high-capacity non-linear methods, e.g. Deep Neural Networks (DNNs). While only removal-based approaches, such as Permutation Importance (PI), can bring statistical validity, they return misleading results when variables are correlated. Conditional Permutation Importance (CPI) bypasses PI’s limitations in such cases. However, in high-dimensional settings, where high correlations between the variables cancel their conditional importance, the use of CPI as well as other methods leads to unreliable results, besides prohibitive computation costs. Grouping variables statistically via clustering or some prior knowledge gains some power back and leads to better interpretations. In this work, we introduce BCPI (Block-Based Conditional Permutation Importance), a new generic framework for variable importance computation with statistical guarantees handling both single and group cases. Furthermore, as handling groups with high cardinality (such as a set of observations of a given modality) are both time-consuming and resource-intensive, we also introduce a new stacking approach extending the DNN architecture with sub-linear layers adapted to the group structure. We show that the ensuing approach extended with stacking controls the type-I error even with highly-correlated groups and shows top accuracy across benchmarks. Furthermore, we perform a real-world data analysis in a large-scale medical dataset where we aim to show the consistency between our results and the literature for a biomarker prediction.

An enhanced machine learning approach for effective prediction of IgA nephropathy patients with severe proteinuria based on clinical data

IgA Nephropathy (IgAN) is a disease of the glomeruli that may eventually lead to chronic kidney disease or kidney failure. The signs and symptoms of IgAN nephropathy are usually not specific enough and are similar to those of other glomerular or inflammatory diseases. This makes a correct diagnosis more difficult. This study collected data from a sample of adult patients diagnosed with primary IgAN at the First Affiliated Hospital of Wenzhou Medical University, with proteinuria ≥1 g/d at the time of diagnosis. Based on these samples, we propose a machine learning framework based on weIghted meaN oF vectOrs (INFO). An enhanced COINFO algorithm is proposed by merging INFO, Cauchy Mutation (CM) and Oppositional-based Learning (OBL) strategies. At the same time, COINFO and Support Vector Machine (SVM) were integrated to construct the BCOINFO-SVM framework for IgAN diagnosis and prediction. Initially, the proposed enhanced COINFO is evaluated using the IEEE CEC2017 benchmark problems, with the outcomes demonstrating its efficient optimization capability and accuracy in convergence. Furthermore, the feature selection capability of the proposed method is verified on the public medical datasets. Finally, the auxiliary diagnostic experiment was carried out through IgAN real sample data. The results demonstrate that the proposed BCOINFO-SVM can screen out essential features such as High-Density Lipoprotein (HDL), Uric Acid (UA), Cardiovascular Disease (CVD), Hypertension and Diabetes. Simultaneously, the BCOINFO-SVM model achieves an accuracy of 98.56%, with sensitivity at 96.08% and specificity at 97.73%, making it a potential auxiliary diagnostic model for IgAN.

Comparative performance analysis of Boruta, SHAP, and Borutashap for disease diagnosis: A study with multiple machine learning algorithms

ABSTRACT Interpretable machine learning models are instrumental in disease diagnosis and clinical decision-making, shedding light on relevant features. Notably, Boruta, SHAP (SHapley Additive exPlanations), and BorutaShap were employed for feature selection, each contributing to the identification of crucial features. These selected features were then utilized to train six machine learning algorithms, including LR, SVM, ETC, AdaBoost, RF, and LR, using diverse medical datasets obtained from public sources after rigorous preprocessing. The performance of each feature selection technique was evaluated across multiple ML models, assessing accuracy, precision, recall, and F1-score metrics. Among these, SHAP showcased superior performance, achieving average accuracies of 80.17%, 85.13%, 90.00%, and 99.55% across diabetes, cardiovascular, statlog, and thyroid disease datasets, respectively. Notably, the LGBM emerged as the most effective algorithm, boasting an average accuracy of 91.00% for most disease states. Moreover, SHAP enhanced the interpretability of the models, providing valuable insights into the underlying mechanisms driving disease diagnosis. This comprehensive study contributes significant insights into feature selection techniques and machine learning algorithms for disease diagnosis, benefiting researchers and practitioners in the medical field. Further exploration of feature selection methods and algorithms holds promise for advancing disease diagnosis methodologies, paving the way for more accurate and interpretable diagnostic models.

Generative Adversarial Networks-based solution for improving medical data quality and insufficiency

As big data brings intelligent solutions and innovations to various fields, the goal of this research is to solve the problem of poor-quality and insufficient datasets in the medical field, to help poor areas can access to high quality and rich medical datasets as well. This study focuses on solving the current problem by utilizing variants of generative adversarial network, Super Resolution Generative Adversarial Network (SRGAN) and Deep Convolutional Generative Adversarial Network (DCGAN). In this study, OpenCV is employed to introduce fuzziness to the Brain Tumor MRI Dataset, resulting in a blurred dataset. Subsequently, the research utilizes both the unaltered and blurred datasets to train the SRGAN model, which is then applied to enhance the low-quality dataset through inpainting. Moving forward, the original dataset, low-quality dataset, and the improved dataset are each used independently to train the DCGAN model. In order to compare the difference between the produced image datasets and the real dataset, the FID Score is separately computed. The results of the study found that by training DCGAN with SRGAN repaired medical dataset, the naked eye can observe that the medical image dataset is significantly clearer and there is a reduction in Frchet Inception Distance (FID) Score. Therefore, by using SRGAN and DCGAN the current problem of low quality and small quantity of datasets in the medical field can be solved, which increase the potential possibilities of big data in artificial intelligence filed of medicine.

SAM-IE: SAM-based image enhancement for facilitating medical image diagnosis with segmentation foundation model

The Segment Anything Model (SAM) is a large-scale model developed for general segmentation tasks in computer vision. Trained on a substantial dataset, SAM can accurately segment various objects in natural scene images. However, due to significant semantic differences between medical and natural images, directly applying SAM to medical image segmentation does not yield optimal results. Therefore, effectively utilizing such a comprehensive foundation model for medical image analysis is an emerging research topic. Despite SAM’s current suboptimal performance in medical image segmentation, it shows preliminary recognition and localization of tissues and lesions that radiologists focus on in medical images. This implies that SAM’s generated masks, features, and stability scores hold potential value for medical image diagnosis. Therefore, based on the model output of SAM, this study introduces a SAM-based Image Enhancement (SAM-IE) method for disease diagnosis. Targeting popular medical image classification models (e.g., ResNet50 and Swin Transformer), SAM-IE is proposed to enhance image inputs by combining the binary mask and contour mask generated by SAM with the original image to create attention maps, thereby improving diagnostic performance. To validate the effectiveness of SAM-IE for diagnosis, experiments were conducted on four medical image datasets for eight classification tasks. The results demonstrate the effectiveness of our proposed SAM-IE model, showcasing SAM’s potential value in medical image classification. This study provides a feasible approach for integrating SAM into disease diagnosis.