Accuracy Metrics Research Articles

Machine learning based system for early heart disease detection and classification using audio signal processing approach

Cardiovascular diseases (CVDs) remain a leading cause of mortality globally, accounting for approximately 17.9 million deaths annually. Traditional diagnostic methods, though useful, have limitations such as invasiveness, high cost, and delays in detecting early-stage heart conditions. This study presents a novel machine learning-based system for early heart disease detection using audio signal processing, specifically leveraging phonocardiogram (PCG) data. Features were extracted using Mel-Frequency Cepstral Coefficients (MFCCs), Delta MFCCs, and Delta-Delta MFCCs, followed by dimensionality reduction via Principal Component Analysis (PCA). Support Vector Machine (SVM) and XGBoost classifiers were used to analyze the extracted features, and their performance was optimized through an ensemble model using the Moth Flame Optimization (MFO) algorithm. The model was rigorously evaluated using accuracy, precision, recall, and F1-score metrics. The ensemble model achieved an accuracy of 99.13%, precision of 98.94%, recall of 95.05%, and an F1-score of 97.46%. The application of SMOTE for data augmentation significantly improved classification performance, highlighting its effectiveness in addressing class imbalance. The proposed system provides a non-invasive, cost-effective solution for heart disease detection and holds potential for improving diagnostic access, particularly in resource-limited settings.

Advanced Image Classification Using Convolutional Neural Networks

Deep learning is essential for computer vision and object detection. This project explores the use of Convolutional Neural Networks (CNNs) for image classification using the CIFAR-10 dataset, which is widely used and provides a robust benchmark for evaluating image classification models. The CNN model, comprising various convolutional, pooling, and fully connected layers, is trained on a portion of the dataset and tested on another portion to evaluate its performance. Techniques such as data augmentation, dropout, and specific activation functions are employed to enhance the model’s efficacy. The study details the CNN architecture and reports the results, including accuracy metrics, to demonstrate the model’s effectiveness.

Deep learning-based algorithm for staging secondary caries in bitewings.

Despite the notable progress in developing artificial intelligence (AI)-based tools for caries detection in bitewings, limited research has addressed the detection and staging of secondary caries. Therefore, we aimed to develop a Convolutional neural network (CNN)-based algorithm for these purposes using a novel approach for determining lesion severity. We used a dataset from a Dutch dental practice-based research network containing 2,612 restored teeth in 413 bitewings from 383 patients aged 15 to 88 years and trained the Mask R-CNN architecture with a Swin Transformer backbone. Two-stage training fine-tuned caries detection accuracy and severity assessment. Annotations of caries around restorations were made by two evaluators and checked by two other experts. Aggregated accuracy metrics (mean ± Standard Deviation - SD) in detecting teeth with secondary caries were calculated considering two thresholds: detecting all lesions and dentine lesions. The correlation between the lesion severity scores obtained with the algorithm and the annotators' consensus was determined using the Pearson correlation coefficient and Bland-Altman plots. Our refined algorithm showed high specificity in detecting all lesions (0.966 ± 0.025) and dentine lesions (0.964 ± 0.019). Sensitivity values were lower: 0.737 ± 0.079 for all lesions and 0.808 ± 0.083 for dentine lesions. The areas under ROC curves (SD) were 0.940 (0.025) for all lesions and 0.946 (0.023) for dentine lesions. The correlation coefficient for severity scores was 0.802. We developed an improved algorithm to support clinicians in detecting and staging secondary caries in bitewing, incorporating an innovative approach for annotation, considering the lesion severity as a continuous outcome.

Assessment of Artificial Intelligence Chatbot Responses to Common Patient Questions on Bone Sarcoma.

The potential impacts of artificial intelligence (AI) chatbots on care for patients with bone sarcoma is poorly understood. Elucidating potential risks and benefits would allow surgeons to define appropriate roles for these tools in clinical care. Eleven questions on bone sarcoma diagnosis, treatment, and recovery were inputted into three AI chatbots. Answers were assessed on a 5-point Likert scale for five clinical accuracy metrics: relevance to the question, balance and lack of bias, basis on established data, factual accuracy, and completeness in scope. Responses were quantitatively assessed for empathy and readability. The Patient Education Materials Assessment Tool (PEMAT) was assessed for understandability and actionability. Chatbots scored highly on relevance (4.24) and balance/lack of bias (4.09) but lower on basing responses on established data (3.77), completeness (3.68), and factual accuracy (3.66). Responses generally scored well on understandability (84.30%), while actionability scores were low for questions on treatment (64.58%) and recovery (60.64%). GPT-4 exhibited the highest empathy (4.12). Readability scores averaged between 10.28 for diagnosis questions to 11.65 for recovery questions. While AI chatbots are promising tools, current limitations in factual accuracy and completeness, as well as concerns of inaccessibility to populations with lower health literacy, may significantly limit their clinical utility.

Using deep learning to detect atherosclerotic plaques on carotid ultrasound images in the UK Biobank

Abstract Background Atherosclerosis is the main underlying cause of cardiovascular disease (CVD). Existing CVD risk assessment tools do not consider the burden of subclinical atherosclerosis. The presence of carotid plaques on carotid ultrasound is a well-known marker of subclinical atherosclerosis. The accumulation of population-scale data on the presence of atherosclerotic plaques, along with deep phenotyping, can allow not only to address the effectiveness of carotid ultrasound in routine clinical practice, but to shed light on the biology of atherosclerosis development. Purpose To develop an effective deep learning model for plaque detection in carotid ultrasound images in the UK Biobank. Methods We used 680 carotid ultrasound images with manually annotated plaques to train a deep learning model employing the YOLOv8 architecture. Different augmentation techniques were used to increase the generalizability of the model. The developed model was applied to automatically detect plaques in raw ultrasound images from 19,507 UK Biobank participants. Logistic and Cox regression were used to explore the associations of plaque presence and number as predicted by the model with conventional CVD risk factors and the risk of future CVD events over follow-up. To explore the genetic architecture of subclinical atherosclerosis, we conducted a genome-wide association study (GWAS) on plaque presence, followed by meta-analysis with data from the CHARGE Consortium. Results Our plaque detection model achieved high classification metrics of accuracy, sensitivity, and specificity (89.3%, 89.5%, and 89.2%, respectively) and detected atherosclerotic plaques in 44% of UK Biobank participants. As expected, plaques were more common among men than women and their prevalence increased linearly with age. Both plaque presence and number of plaques were correlated with conventional CVD risk factors including diabetes, hypertension, and hyperlipidemia, and showed strong associations with future risk of incident CVD events (Hazard Ratio for plaque presence: 1.48 [95%CI: 1.21-1.82], for 2 plaques or more: 1.65, [95% CI: 1.28-2.13]). Incorporating plaque-derived phenotypes minimally altered the C-index of the time-to-event model. GWAS meta-analysis of carotid plaque presence revealed 5 previously known loci, as well as a significant locus including the LPA gene that had not previously been associated with carotid plaque. Conclusion We have developed and implemented an efficient plaque detection model to data from the UK Biobank, which holds significant promise for studying atherosclerosis at a population-wide scale through integration with multiomics data and electronic health records.

Evaluating Machine Learning Models for Stroke Prognosis and Prediction in Atrial Fibrillation Patients: A Comprehensive Meta-Analysis

Background/Objective: Atrial fibrillation (AF) complicates the management of acute ischemic stroke (AIS), necessitating precise predictive models to enhance clinical outcomes. This meta-analysis evaluates the efficacy of machine learning (ML) models in three key areas: stroke prognosis in AF patients, stroke prediction in AF patients, and AF prediction in stroke patients. The study aims to assess the accuracy and variability of ML models in forecasting AIS outcomes and detecting AF in stroke patients, while exploring the clinical benefits and limitations of integrating these models into practice. Methods: We conducted a systematic search of PubMed, Embase, and Cochrane databases up to June 2024, selecting studies that evaluated ML accuracy in stroke prognosis and prediction in AF patients and AF prediction in stroke patients. Data extraction and quality assessment were performed independently by two reviewers, with random-effects modeling applied to estimate pooled accuracy metrics. Results: The meta-analysis included twenty-four studies comprising 7,391,645 patients, categorized into groups for stroke prognosis in AF patients (eight studies), stroke prediction in AF patients (thirteen studies), and AF prediction in stroke patients (three studies). The pooled AUROC was 0.79 for stroke prognosis and 0.68 for stroke prediction in AF, with higher accuracy noted in short-term predictions. The mean AUROC across studies was 0.75, with models such as Extreme Gradient Boosting (XGB) and Random Forest (RF) showing superior performance. For stroke prognosis in AF, the mean AUROC was 0.78, whereas stroke prediction yielded a mean AUROC of 0.73. AF prediction post-stroke had an average AUROC of 0.75. These findings indicate moderate predictive capability of ML models, underscoring the need for further refinement and standardization. The absence of comprehensive sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) metrics limited the ability to conduct full meta-analytic modeling. Conclusions: While ML models demonstrate potential for enhancing stroke prognosis and AF prediction, they have yet to meet the clinical standards required for widespread adoption. Future efforts should focus on refining these models and validating them across diverse populations to improve their clinical utility.

Generation Method for HVAC Systems Design Schemes in Office Buildings Based on Deep Graph Generative Models

The design process of heating, ventilation, and air conditioning (HVAC) systems is complex and time consuming due to the need to follow design codes. Since the design standards are not fixed, the final outcome often depends on the designer’s experience. The development of building information modeling (BIM) technology has made information throughout the building lifecycle more integrated. BIM-based forward design is now widely used, providing a data foundation for combining HVAC system design with machine learning. This paper proposes an unsupervised learning method based on deep graph generative models to uncover hidden design patterns and optimization strategies from the design results. We trained and validated four deep graph generative models—GAE, GNF, GAN, and diffusion—using HVAC system terminal pipeline layout data. Accuracy and precision metrics were used to compare the generated designs with automated forward design solutions, assessing the models’ ability to capture both local variations and broader changes in design logic. A graph-neural-network-based evaluation method was employed to measure the models’ capacity to detect changes. The results indicate that all four models achieved prediction accuracies exceeding 90% and precision rates above 75%. The models effectively captured both local modifications made by designers and global design changes, showing greater sensitivity to global layout adjustments than to local updates. When comparing the results generated by deep graph generative models and the actual design, it is obvious that the accuracy of the predictions varies significantly due to the complexity of the test buildings.

Visible feature engineering to detect fraud in black and red peppers

Visible imaging is a fast, cheap, and accurate technique in the assessment of food quality and safety. The technique was used in the present research to detect sea foam adulterant levels in black and red peppers. The fraud levels included 0, 5, 15, 30, and 50%. Sample preparation, image acquisition and preprocessing, and feature engineering (feature extraction, selection, and classification) were the conducted steps in the present research. The efficient features were classified using artificial neural networks and support vector machine methods. The classifiers were evaluated using the specificity, sensitivity, precision, and accuracy metrics. The artificial neural networks had better results than the support vector machine method for the classification of different adulterant levels in black pepper with the metrics’ values of 98.89, 95.67, 95.56, and 98.22%, respectively. Reversely, the support vector machine method had higher metrics’ values (99.46, 98.00, 97.78, and 99.11%, respectively) for red pepper. The results showed the ability of visible imaging and machine learning methods to detect fraud levels in black and red pepper.

Oral screening of dental calculus, gingivitis and dental caries through segmentation on intraoral photographic images using deep learning

ObjectiveIntraoral photographic images are instrumental in the early screening and clinical diagnosis of oral diseases. In addition, people have been trying to apply artificial intelligence to these images. The purpose of this study is to investigate and evaluate a deep learning system designed to segment intraoral photographic images for the detection of dental caries, dental calculus, and gingivitis, and to assess the degree of dental calculus based on the overall features of the tooth surface and gingival margin.Material and methodsThis cross-sectional study collected 3,365 oral endoscopic images, randomly distributed in training datasets (2,019 images), validation dataset (673 images), and test dataset (673 images). The training set and verification set images are manually labeled. An oral endoscopic image segmentation method based on Mamba (Oral-Mamba) and an intelligent evaluation model of dental calculus degree were proposed, achieving the segmentation of two types of oral diseases, namely gingivitis and dental caries, as well as the segmentation of dental calculus regions, and the intelligent evaluation of the degree of dental calculus.ResultsOral-Mamba demonstrated high accuracy in segmentation, with accuracy rates for gingivitis, dental caries, and dental calculus at 0.83, 0.83, and 0.81, respectively. In particular, these rates surpassed those of the U-Net model in IoU, accuracy, and recall metrics. Furthermore, Oral-Mamba runs 25% faster than U-Net.The accuracy of degree classification in the intelligent evaluation model of dental calculus degree is 85%.ConclusionThe proposed deep learning system is expected to be used for the detection of two types of oral diseases and dental calculus, and the degree judgment of photographic images from an intraoral camera. This system offers a practical method to assist in the oral screening of dental caries, dental calculus, and gingivitis, providing benefits such as intuitive use, time efficiency, cost-effectiveness, and ease of deployment.

Implementation of Chaotic Neural Key Generation Algorithm For IoT Devices

This paper presents a new method for generating encryption keys for Internet of Things (IoT) devices. This method combines chaos theory with neural networks to create secure and efficient keys. We use the Lorenz chaotic system to generate complex patterns and a Convolutional Neural Network (CNN) to learn and predict these patterns. This approach is designed to address the unique security challenges of IoT devices, which often have limited computing power. We tested our method with different key sizes and evaluated its performance using accuracy, loss, entropy, and correlation metrics. The results show that key sizes between 256 and 512 bits offer the best balance between model performance and security for IoT devices. We also conducted Diehard statistical tests, which our key generation method passed successfully, demonstrating its ability to produce high-quality random keys.

Machine learning for pacemaker implantation prediction after TAVI using multimodal imaging data

Pacemaker implantation (PMI) after transcatheter aortic valve implantation (TAVI) is a common complication. While computed tomography (CT) scan data are known predictors of PMI, no machine learning (ML) model integrating CT with clinical, ECG, and transthoracic echocardiography (TTE) data has been proposed. This study investigates the contribution of ML methods to predict PMI after TAVI, with a focus on the role of CT imaging data. A retrospective analysis was conducted on a cohort of 520 patients who underwent TAVI. Recursive feature elimination with SHAP values was used to select key variables from clinical, ECG, TTE, and CT data. Six ML models, including Support Vector Machines (SVM), were trained using these selected variables. The model’s performance was evaluated using AUC-ROC, F1 score, and accuracy metrics. The PMI rate was 18.8%. The best-performing model achieved an AUC-ROC of 92.1% ± 4.7, an F1 score of 71.8% ± 9.9, and an accuracy of 87.9% ± 4.7 using 22 variables, 9 of which were CT-based. Membranous septum measurements and their dynamic variations were critical predictors. Our ML model provides robust PMI predictions, enabling personalized risk assessments. The model is implemented online for broad clinical use.

A novel approach to detecting epileptic patients: complex network-based EEG classification

Abstract Detection of epileptic seizures is important for early diagnosis and treatment. It is known that the behavioral patterns of the brain in electroencephalogram (EEG) signals have huge and complex fluctuations. Diagnosing epilepsy by analyzing signals are costly process. Various methods are used to classify epileptic seizures. However, the inadequacy of these approaches in classifying signals makes it difficult to diagnose epilepsy. Complex network science produces effective solutions for analyzing interrelated structures. Using methods based on complex network analysis, it is possible to EEG signals analyze the relationship between signals and perform a classification process. In this study proposes a novel approach for classifying epileptic seizures by utilizing complex network science. In addition, unlike the studies in the literature, classification processes were carried out with lower dimensional signals by using 1-s EEG signals instead of 23.6-s full-size EEG signals. Using the topological properties of the EEG signal converted into a complex network, the classification process has been performed with the Jaccard Index method. The success of the classification process with the Jaccard Index was evaluated using Accuracy, F1 Score, Recall, and K-Fold metrics. In the results obtained, the signals of individuals with epileptic seizures were separated with an accuracy rate of 98.15%.

ViT-HHO: Optimized vision transformer for diabetic retinopathy detection using Harris Hawk optimization

Diabetic retinopathy (DR) is a significant cause of vision impairment globally, emphasizing the importance of timely and precise detection to prevent severe consequences. This study presents an optimized Vision Transformer (ViT) model that incorporates Harris Hawk Optimization (HHO) to improve the automated detection of diabetic retinopathy (DR). The ViT architecture utilizes self-attention mechanisms to capture local and global features in retinal images. Additionally, HHO optimizes key hyperparameters to maximize the performance of the model. The proposed ViT-HHO model achieved exceptional performance on the APTOS-2019 and IDRiD datasets. Specifically, it achieved 99.83 % accuracy, 99.78 % sensitivity, 99.85 % specificity, and 99.80 % AUC-ROC on the APTOS-2019 dataset, surpassing traditional CNNs and alternative optimization techniques. The model exhibited strong generalization on the IDRiID dataset, achieving an accuracy of 99.11 % and an AUC-ROC of 99.12 %. The ViT-HHO model demonstrates the potential for enhancing the clinical detection of diabetic retinopathy (DR), providing high precision and reliability.•An optimized Vision Transformer (ViT) model was developed using HHO for improved detection of Diabetic Retinopathy (DR).•The model was validated on the APTOS-2019 and IDRiID datasets, demonstrating superior accuracy and AUC-ROC metrics.•The model's generalization and robustness were demonstrated through comprehensive performance evaluations.

An Echo State Network-Based Light Framework for Online Anomaly Detection: An Approach to Using AI at the Edge

Production efficiency is used to determine the best conditions for manufacturing goods at the lowest possible unit cost. When achieved, production efficiency leads to increased revenues for the manufacturer, enhanced employee safety, and a satisfied customer base. Production efficiency not only measures the amount of resources that are needed for production but also considers the productivity levels and the state of the production lines. In this context, online anomaly detection (AD) is an important tool for maintaining the reliability of the production ecosystem. With advancements in artificial intelligence and the growing significance of identifying and mitigating anomalies across different fields, approaches based on artificial neural networks facilitate the recognition of intricate types of anomalies by taking into account both temporal and contextual attributes. In this paper, a lightweight framework based on the Echo State Network (ESN) model running at the edge is introduced for online AD. Compared to other AD methods, such as Long Short-Term Memory (LSTM), it achieves superior precision, accuracy, and recall metrics while reducing training time, CO2 emissions, and the need for high computational resources. The preliminary evaluation of the proposed solution was conducted using a low-resource computing device at the edge of the real production machine through an Industrial Internet of Things (IIoT) smart meter module. The machine used to test the proposed solution was provided by the Italian company SIFIM Srl, which manufactures filter mats for industrial kitchens. Experimental results demonstrate the feasibility of developing an AD method that achieves high accuracy, with the ESN-based framework reaching 85% compared to 80.88% for the LSTM-based model. Furthermore, this method requires minimal hardware resources, with a training time of 9.5 s compared to 2.100 s for the other model.

A Labeling Intercomparison of Retrogressive Thaw Slumps by a Diverse Group of Domain Experts

ABSTRACTDeep‐learning (DL) models have become increasingly beneficial for the detection of retrogressive thaw slumps (RTS) in the permafrost domain. However, comparing accuracy metrics is challenging due to unstandardized labeling guidelines. To address this, we conducted an experiment with 12 international domain experts from a broad range of scientific backgrounds. Using 3 m PlanetScope multispectral imagery, they digitized RTS footprints in two sites. We evaluated label uncertainty by comparing manually outlined RTS labels using Intersection‐over‐Union (IoU) and F1 metrics. At the Canadian Peel Plateau site, we see good agreement, particularly in the active parts of RTS. Differences were observed in the interpretation of the debris tongue and the stable vegetated sections of RTS. At the Russian Bykovsky site, we observed a larger mismatch. Here, the same differences were documented, but several participants mistakenly identified non‐RTS features. This emphasizes the importance of site‐specific knowledge for reliable label creation. The experiment highlights the need for standardized labeling procedures and definition of their scientific purpose. The most similar expert labels outperformed the accuracy metrics reported in the literature, highlighting human labeling capabilities with proper training, site knowledge, and clear guidelines. These findings lay the groundwork for DL‐based RTS monitoring in the pan‐Arctic.

Analyzing Unimproved Drinking Water Sources and Their Determinants Using Supervised Machine Learning: Evidence from the Somaliland Demographic Health Survey 2020

Access to clean and safe drinking water is a fundamental human right. Despite global efforts, including the UN’s “Water for Life” program, a significant portion of the population in developing countries, including Somaliland, continues to rely on unimproved water sources. These unimproved sources contribute to poor health outcomes, particularly for children. This study aimed to investigate the factors associated with the use of unimproved drinking water sources in Somaliland by employing supervised machine learning models to predict patterns and determinants based on data from the 2020 Somaliland Demographic and Health Survey (SHDS). Secondary data from SHDS 2020 were used, encompassing 8384 households across Somaliland. A multilevel logistic regression model was applied to analyze the individual- and community-level factors influencing the use of unimproved water sources. In addition, machine learning models, including logistic regression, decision tree, random forest, support vector machine (SVM), and K-nearest neighbor (KNN), were compared in terms of accuracy, sensitivity, specificity, and other metrics using cross-validation techniques. This study uses supervised machine learning models to analyze unimproved drinking water sources in Somaliland, providing data-driven insights into the complex determinants of water access. This enhances predictive accuracy and informs targeted interventions, offering a robust framework for addressing water-related public health issues in Somaliland. The analysis identified key determinants of unimproved water source usage, including socioeconomic status, education, region, and household characteristics. The random forest model performed the best with an accuracy of 93.57% and an area under the curve (AUC) score of 98%. Decision tree and KNN also exhibited strong performance, while SVM had the lowest predictive accuracy. This study highlights the role of socioeconomic and community factors in determining access to clean drinking water in Somali Land. Factors such as age, education, gender, household wealth, media access, urban or rural residence, poverty level, and literacy level significantly influenced access. Local policies and resource availability also contribute to variations in access. These findings suggest that targeted interventions aimed at improving education, infrastructure, and community water management practices can significantly reduce reliance on unimproved water sources and improve the overall public health.

OSAIRIS: Lessons Learned From the Hospital-Based Implementation and Evaluation of an Open-Source Deep-Learning Model for Radiotherapy Image Segmentation

Several studies report the benefits and accuracy of using autosegmentation for organ at risk (OAR) outlining in radiotherapy treatment planning. Typically, evaluations focus on accuracy metrics, and other parameters such as perceived utility and safety are routinely ignored. Here, we report our finding from the implementation and clinical evaluation of OSAIRIS, an open-source AI model for radiotherapy image segmentation that was carried out as part of its development into a medical device. The device contours OARs in the head and neck and male pelvis (referred to as the prostate model), and is designed to be used as a time-saving workflow device, alongside a clinician. Unlike standard evaluation processes, which heavily rely on accuracy metrics alone, our evaluation sought to demonstrate the tangible benefits, quantify utility and assess risk within a specific clinical workflow. We evaluated the time-saving benefit this device affords to clinicians, and how this time-saving might be linked to accuracy metrics, as well as the clinicians' assessment of the usability of the OSAIRIS contours in comparison to their colleagues' contours and those from other commercial AI contouring devices. Our safety evaluation focused on whether clinicians can notice and correct any errors should they be included in the output of the device.We found that OSAIRIS affords a significant time-saving of 36% (5.4 ± 2.1 minutes) when used for prostate contouring and 67% (30.3 ± 8.7 minutes) for head and neck contouring. Combining editing time data with accuracy metrics, we found the Hausdorff distance best correlated with editing-time, outperforming dice, the industry-standard, with a Spearman correlation coefficient of 0.70, and a Kendall coefficient of 0.52. Our safety and risk-mitigation exercise showed that anchoring bias is present when clinicians edit AI-generated contours, with the effect seemingly more pronounced for some structures over others. Most errors, however, were corrected by clinicians, with 72% of the head and neck errors 81% of the prostate errors removed in the editing step. Notably, our blinded clinician contour rating exercise showed that gold standard clinician contours are not rated more highly than the AI-generated contours.We conclude that evaluations of AI in a clinical setting must consider the clinical workflow in which the device will be used, and not rely on accuracy metrics alone, in order to reliably assess the benefits, utility and safety of the device. The effects of human-AI inter-operation must be evaluated to accurately assess the practical usability and potential uptake of the technology, as demonstrated in our blinded clinical utility review. The clinical risks posed by the use of the device must be studied and mitigated as far as possible, and our ‘Mystery Shopping’ experiment provides a template for future such assessments.

Fine-Tuned Bidirectional Encoder Representations From Transformers Versus ChatGPT for Text-Based Outpatient Department Recommendation: Comparative Study.

Patients often struggle with determining which outpatient specialist to consult based on their symptoms. Natural language processing models in health care offer the potential to assist patients in making these decisions before visiting a hospital. This study aimed to evaluate the performance of ChatGPT in recommending medical specialties for medical questions. We used a dataset of 31,482 medical questions, each answered by doctors and labeled with the appropriate medical specialty from the health consultation board of NAVER (NAVER Corp), a major Korean portal. This dataset includes 27 distinct medical specialty labels. We compared the performance of the fine-tuned Korean Medical bidirectional encoder representations from transformers (KM-BERT) and ChatGPT models by analyzing their ability to accurately recommend medical specialties. We categorized responses from ChatGPT into those matching the 27 predefined specialties and those that did not. Both models were evaluated using performance metrics of accuracy, precision, recall, and F1-score. ChatGPT demonstrated an answer avoidance rate of 6.2% but provided accurate medical specialty recommendations with explanations that elucidated the underlying pathophysiology of the patient's symptoms. It achieved an accuracy of 0.939, precision of 0.219, recall of 0.168, and an F1-score of 0.134. In contrast, the KM-BERT model, fine-tuned for the same task, outperformed ChatGPT with an accuracy of 0.977, precision of 0.570, recall of 0.652, and an F1-score of 0.587. Although ChatGPT did not surpass the fine-tuned KM-BERT model in recommending the correct medical specialties, it showcased notable advantages as a conversational artificial intelligence model. By providing detailed, contextually appropriate explanations, ChatGPT has the potential to significantly enhance patient comprehension of medical information, thereby improving the medical referral process.

Experience in using artificial intelligence services for diagnosing compression fracture of vertebral body based on computed tomography – from testing to trials

Backgraund:Osteoporosis (OP) is often diagnosed at the stage of complications. Compression fractures (CF) of the vertebral bodies, complications of OP, and predictors of subsequent fractures are often asymptomatic. CF can be detected on CT scan performed for other indications. We have analyzed approaches to using artificial intelligence services (AI services) generated for diagnosing CF of the vertebral bodies. Aims:Testing AI services to identify CF based on chest CT and to assess the possibility of their implementation into the practice of medical facilities of the Moscow Healthcare Department. Materials and methods:For setting a clinical task for AI services, basic diagnostic requirements in the direction of “Osteoporosis” are formed. AI services go through the following stages: self-testing, functional and calibration testing, trial and trial operation. The first three stages of testing are carried out on previously generated datasets. At the stages of trial and trial operation, AI services analyze CT examinations performed in medical facilities. An expert group of radiologists evaluates the diagnostic accuracy and functional capacity of AI services at all stages. The obtained quantitative metrics of the accuracy of AI services are compared with the target values ​​specified in the requirements. Results:From June 2021 to June 2022, two AI services (AI service №1 and AI service №2) which used different methods for identifying a presence of CF were tested. AI services successfully passed the self-testing stage (6), functional (5) and calibration (100) testing. The ROC AUC for AI service №1 was 0.99 and for AI service №2–0.91. The AI service №1 passed the trial stage without any significant remarks, while AI service №2 was sent for revision. After the trial operation stage, the following accuracy metrics were obtained: the ROC AUC for AI service №1 was 0.93; for AI service №2–0.92. At all stages, both AI services showed sufficient metrics for clinical validation. Conclusions:A testing of the performance of AI services in the direction of “Osteoporosis” was carried out. A high quality of diagnosis was demonstrated. AI services can act as an auxiliary tool in the medical decision support system.

Variational benchmarks for quantum many-body problems.

The continued development of computational approaches to many-body ground-state problems in physics and chemistry calls for a consistent way to assess its overall progress. In this work, we introduce a metric of variational accuracy, the V-score, obtained from the variational energy and its variance. We provide an extensive curated dataset of variational calculations of many-body quantum systems, identifying cases where state-of-the-art numerical approaches show limited accuracy and future algorithms or computational platforms, such as quantum computing, could provide improved accuracy. The V-score can be used as a metric to assess the progress of quantum variational methods toward a quantum advantage for ground-state problems, especially in regimes where classical verifiability is impossible.