AI-Driven Three-Dimensional Movement Analysis for Objective Assessment of Surgical Expertise in Suturing.

BackgroundAdverse events refer to incidents with potential or actual harm to patients in hospitals. These events are typically documented through patient safety event (PSE) reports, which consist of detailed narratives providing contextual information on the occurrences. Accurate classification of PSE reports is crucial for patient safety monitoring. However, this process faces challenges due to inconsistencies in classifications and the sheer volume of reports. Recent advancements in text representation, particularly contextual text representation derived from transformer-based language models, offer a promising solution for more precise PSE report classification. Integrating the machine learning (ML) classifier necessitates a balance between human expertise and artificial intelligence (AI). Central to this integration is the concept of explainability, which is crucial for building trust and ensuring effective human-AI collaboration.ObjectiveThis study aims to investigate the efficacy of ML classifiers trained using contextual text representation in automatically classifying PSE reports. Furthermore, the study presents an interface that integrates the ML classifier with the explainability technique to facilitate human-AI collaboration for PSE report classification.MethodsThis study used a data set of 861 PSE reports from a large academic hospital’s maternity units in the Southeastern United States. Various ML classifiers were trained with both static and contextual text representations of PSE reports. The trained ML classifiers were evaluated with multiclass classification metrics and the confusion matrix. The local interpretable model-agnostic explanations (LIME) technique was used to provide the rationale for the ML classifier’s predictions. An interface that integrates the ML classifier with the LIME technique was designed for incident reporting systems.ResultsThe top-performing classifier using contextual representation was able to obtain an accuracy of 75.4% (95/126) compared to an accuracy of 66.7% (84/126) by the top-performing classifier trained using static text representation. A PSE reporting interface has been designed to facilitate human-AI collaboration in PSE report classification. In this design, the ML classifier recommends the top 2 most probable event types, along with the explanations for the prediction, enabling PSE reporters and patient safety analysts to choose the most suitable one. The LIME technique showed that the classifier occasionally relies on arbitrary words for classification, emphasizing the necessity of human oversight.ConclusionsThis study demonstrates that training ML classifiers with contextual text representations can significantly enhance the accuracy of PSE report classification. The interface designed in this study lays the foundation for human-AI collaboration in the classification of PSE reports. The insights gained from this research enhance the decision-making process in PSE report classification, enabling hospitals to more efficiently identify potential risks and hazards and enabling patient safety analysts to take timely actions to prevent patient harm.

Machine learning has achieved remarkable success across diverse domains. Nevertheless, concerns about interpretability in black-box models, especially within Deep Neural Networks (DNNs), have become pronounced in safety-critical fields like healthcare and finance. Classical machine learning (ML) classifiers, known for their higher interpretability, are preferred in these domains. Similar to DNNs, classical ML classifiers can exhibit bugs that could lead to severe consequences in practice. Test input prioritization has emerged as a promising approach to ensure the quality of an ML system, which prioritizes potentially misclassified tests so that such tests can be identified earlier with limited manual labeling costs. However, when applying to classical ML classifiers, existing DNN test prioritization methods are constrained from three perspectives: 1) Coverage-based methods are inefficient and time-consuming; 2) Mutation-based methods cannot be adapted to classical ML models due to mismatched model mutation rules; 3) Confidence-based methods are restricted to a single dimension when applying to binary ML classifiers, solely depending on the model’s prediction probability for one class. To overcome the challenges, we propose MLPrior, a test prioritization approach specifically tailored for classical ML models. MLPrior leverages the characteristics of classical ML classifiers (i.e., interpretable models and carefully engineered attribute features) to prioritize test inputs. The foundational principles are: 1) tests more sensitive to mutations are more likely to be misclassified, and 2) tests closer to the model’s decision boundary are more likely to be misclassified. Building on the first concept, we design mutation rules to generate two types of mutation features (i.e., <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">model mutation features</b> and <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">input mutation features</b> ) for each test. Drawing from the second notion, MLPrior generates <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">attribute features</b> of each test based on its attribute values, which can indirectly reveal the proximity between the test and the decision boundary. For each test, MLPrior combines all three types of features of it into a final vector. Subsequently, MLPrior employs a pre-trained ranking model to predict the misclassification probability of each test based on its final vector and ranks tests accordingly. We conducted an extensive study to evaluate MLPrior based on 185 subjects, encompassing natural datasets, mixed noisy datasets, and fairness datasets. The results demonstrate that MLPrior outperforms all the compared test prioritization approaches, with an average improvement of 14.74%∼66.93% on natural datasets, 18.55%∼67.73% on mixed noisy datasets, and 15.34%∼62.72% on fairness datasets.

BackgroundMovement patterns, activity levels and circadian rhythm are altered in Alzheimer’s disease (AD) and can be assessed by actigraphy using wearable sensors. We aimed to determine the diagnostic performance of actigraphy in AD in a memory clinic population by using a machine-learning classifier.MethodsIn our single-center cross-sectional study, 70 patients with AD (MCI-moderate dementia), dementia with Lewy bodies (DLB) (N = 29) and cerebrovascular disease (CVD) (N = 23), and 48 elderly healthy controls were included. Participants underwent actigraphy at home using two body-worn sensors (SENS Motion®) for 1 week. We derived movement patterns (walking, running, resting, etc.) from raw accelerometry data using a proprietary algorithm. By evaluating the movement patterns during day and nighttime, we calculated 510 activity-related features, including robustness and fragmentation of the circadian rhythm. These features were used to train a machine learning (ML) classifier using logistic regression. We evaluated the performance of our classifier by assessing the accuracy and precision of predictions.ResultsWe found that movement patterns as well as the robustness and fragmentation of the circadian rhythm differed significantly between groups. During the daytime, patients with AD performed less moderate activity and walked less than the healthy group. While we achieved a modest accuracy of 68.8% for differentiating AD and healthy, the performance was highest (accuracy: 80−89%; precision: 69−84%) when ML was applied to actigraphy data to differentiate dementia etiologies (AD vs. DLB + AD vs. CVD).ConclusionActigraphy accurately identifies different dementia etiologies and could serve as a supplement to diagnostic investigations in patients with suspected AD for differential diagnostic purposes.

A comparative study of supervised Machine Learning classifiers for Intrusion Detection in Internet of Things

Objectives: To determine the most effective traditional method for feature extraction and Machine Learning (ML) classifier for classification of segmented brain MRI images. Methods: The dataset of brain MRI images are segmented using the Morphological Active Contour Boltzmann Monte Carlo Method (MACB) model from our previous work. The feature extraction analysis of the segmented brain MRI images is done using the traditional feature extraction methods viz. shape, Gray Level Co-occurrence Matrix (GLCM), wavelet, Local Binary Pattern (LBP) and intensity. After feature extraction, the classification of brain tumor using the ML classifiers such as Support Vector Machine, Logistic Regression, Linear Discriminant Analysis, Decision Tree (DT), Random Forest, AdaBoost, K-Nearest Neighbours and Naive-Bayes are carried out in the segmented brain MRI images. An extensive classification analysis is made to find the best feature extraction method and best ML classifier for segmented brain MRI images. The public multi-class dataset of brain MRI images available in Kaggle website are used in this work, it contains 7023 images (5712 training and 1311 testing images). Findings: The 31 features, extracted using various combinations of 5 traditional feature extraction methods, are inputted to 8 ML classifiers and performance analysis is done. It is found from the analysis that the DT classifier achieves the highest classification accuracy of 96.05%. Novelty: The key novelty of the proposed approach depends on the tumor segmentation technique using our MACB model for segmenting brain tumors in MRI images, since the MACB model has itself demonstrated better performance in providing better quality tumor delineation, as compared to traditional segmentation methods. From the dataset of MACB segmented brain MRI images, 31 features, extracted using various combinations of 5 traditional feature extraction methods, are inputted to 8 ML classifiers and a performance analysis is done. It is concluded that the DT classifier achieves higher accuracy. Keywords: Image segmentation, Feature extraction, Machine Learning classifiers, Brain tumor classification

Background: Screening and early detection play a key role in cervical cancer prevention. The present study predicts the outcome of various diagnostic tests used to diagnose cervical cancer using machine learning algorithms. Methods: The present study ran various cervical cancer risk factors on a machine learning (ML) classifier to predict outcomes of Hinselmann, Schiller, cytology and biopsy. The dataset is publicly available on the Machine Learning Repository website of the University of California Irvine. The imbalanced dataset was pre-processed using oversampling methods. The significantly varied features between the two levels of a response variable were used to train the machine learning classifiers on MATLAB. The classifiers used were Decision Trees, Support Vector Machine, K-Nearest Neighbours and Ensemble learning classifiers. The performance metrics of the classifiers were expressed as accuracy, the area under the receiver operator characteristic (AU-ROC) curve, sensitivity and specificity. Results: The Fine Gaussian SVM classifier was the best to classify Hinselmann, cytology and biopsy with the accuracy of 97.5 %, 62.5 % and 98 %, respectively. However, Boosted trees performed best in the classification of Schiller with 81.3 % accuracy. Conclusion: The present study selected optimised features among multiple risk factors to train various ML classifiers to predict cervical cancer.

We study the performance of classical and quantum machine learning (ML) models in predicting outcomes of physical experiments. The experiments depend on an input parameter x and involve execution of a (possibly unknown) quantum process E. Our figure of merit is the number of runs of E required to achieve a desired prediction performance. We consider classical ML models that perform a measurement and record the classical outcome after each run of E, and quantum ML models that can access E coherently to acquire quantum data; the classical or quantum data are then used to predict the outcomes of future experiments. We prove that for any input distribution D(x), a classical ML model can provide accurate predictions on average by accessing E a number of times comparable to the optimal quantum ML model. In contrast, for achieving an accurate prediction on all inputs, we prove that the exponential quantum advantage is possible. For example, to predict the expectations of all Pauli observables in an n-qubit system ρ, classical ML models require 2^{Ω(n)} copies of ρ, but we present a quantum ML model using only O(n) copies. Our results clarify where the quantum advantage is possible and highlight the potential for classical ML models to address challenging quantum problems in physics and chemistry.

The primary objective of this study is to segment the uterine fibroids (leiomyoma) from the ultrasound images of the uterus through semantic segmentation, followed by second-order statistical feature extraction using the Gray-level Co-occurrence Matrix (GLCM). The next objective of the study is to compare the performance of the state-of-the-art method namely Vision Transformer (ViT) with three different machine learning (ML) classifiers such as the Support Vector Machine (SVM), Logistic Regression (LR) and [Formula: see text]-Nearest Neighbor ([Formula: see text]-NN) to classify the images into uterine fibroid and normal. The dataset consists of 50 ultrasound images of uterine fibroids and 50 normal images. Then the images are segmented using region-growing-based semantic segmentation followed by feature extraction and classification using the ML and deep learning (DL) classifiers. Among the ML classifiers, SVM produced a good accuracy of 93.1% compared to the other classifiers. ViT produced an excellent classification accuracy of 97.5%. Hence, ViT outperformed compared to the ML classifiers in uterine fibroid detection. These findings have important implications for clinical practice, as they could help physicians to diagnose and treat uterine fibroids more effectively.

Measurement of health-related social needs (HRSNs) is complex. We sought to develop and validate computable phenotypes (CPs) using structured electronic health record (EHR) data for food insecurity, housing instability, financial insecurity, transportation barriers, and a composite-type measure of these, using human-defined rule-based and machine learning (ML) classifier approaches. We collected HRSN surveys as the reference standard and obtained EHR data from 1550 patients in 3 health systems from 2 states. We followed a Delphi-like approach to develop the human-defined rule-based CP. For the ML classifier approach, we trained supervised ML (XGBoost) models using 78 features. Using surveys as the reference standard, we calculated sensitivity, specificity, positive predictive values, and area under the curve (AUC). We compared AUCs using the Delong test and other performance measures using McNemar's test, and checked for differential performance. Most patients (63%) reported at least one HRSN on the reference standard survey. Human-defined rule-based CPs exhibited poor performance (AUCs=.52 to .68). ML classifier CPs performed significantly better, but still poor-to-fair (AUCs = .68 to .75). Significant differences for race/ethnicity were found for ML classifier CPs (higher AUCs for White non-Hispanic patients). Important features included number of encounters and Medicaid insurance. Using a supervised ML classifier approach, HRSN CPs approached thresholds of fair performance, but exhibited differential performance by race/ethnicity. CPs may help to identify patients who may benefit from additional social needs screening. Future work should explore the use of area-level features via geospatial data and natural language processing to improve model performance.

Early gearbox defect detection is imperative for reducing unplanned downtime, ensuring reliability and efficiency, and minimizing maintenance expenses. In recent years, with the rise of Artificial Intelligence (AI) and digital transformation, gearbox defect detection using AI has gained popularity. Machine learning (ML) classifiers are very popular and transform gearbox condition monitoring from manual to automatic monitoring systems. This work proposes a moving window-based method for extracting statistical features from recorded vibration signals from the gearbox. The extracted features were used to train traditional ML classifiers. Moving window sizes of 300, 400, 500, 600, 700, and 800 were applied to extract statistical features from the publicly available benchmark dataset. The six different moving window sizes caused six types of datasets, each one corresponding to the moving window size. The generated datasets were partitioned using the K-fold cross-validation method to train and test ML models. This study explored and evaluated seven prominent ML classifiers: Decision Tree, Random Forest, Support Vector Machine (SVM), Naïve Bayes, K-Nearest Neighbor (KNN), Gradient Boosting Classifier (GBC), and Logistic Regression. The experimental results demonstrated that SVM, Logistic Regression, and GBC can outperform other ML classifiers. The experimental results in terms of accuracy, precision, and recall revealed that the ML classifier’s performance improves as the size of the moving window used for feature extraction increases.

Radiomic machine learning for pretreatment assessment of prognostic risk factors for endometrial cancer and its effects on radiologists' decisions of deep myometrial invasion

Coast type based accuracy assessment for coastline extraction from satellite image with machine learning classifiers

Glioma grading transformed in World Health Organization (WHO) 2021 CNS tumor classification, integrating molecular markers. However, the impact of this change on radiomics-based machine learning (ML) classifiers remains unexplored. To assess the performance of ML in classifying glioma tumor grades based on various WHO criteria. Retrospective. A neuropathologist regraded gliomas of 237 patients into WHO 2016 and 2021 from 2007 criteria. Multicentric 0.5 to 3 Tesla; pre- and post-contrast T1-weighted, T2-weighted, and fluid-attenuated inversion recovery. Radiomic features were selected using random forest-recursive feature elimination. The synthetic minority over-sampling technique (SMOTE) was implemented for data augmentation. Stratified 10-fold cross-validation with and without SMOTE was used to evaluate 11 classifiers for 3-grade (2, 3, and 4; WHO 2016 and 2021) and 2-grade (low and high grade; WHO 2007 and 2021) classification. Additionally, we developed the models on data randomly divided into training and test sets (mixed-data analysis), or data divided based on the centers (independent-data analysis). We assessed ML classifiers using sensitivity, specificity, accuracy, and the area under the receiver operating characteristic curve (AUC). Top performances were compared with a t-test and categorical data with the chi-square test using a significance level of P < 0.05. In the mixed-data analysis, Stacking Classifier without SMOTE achieved the highest accuracy (0.86) and AUC (0.92) in 3-grade WHO 2021 grouping. The results of WHO 2021 were significantly better than WHO 2016 (P-value<0.0001). In the 2-grade analysis, ML achieved 1.00 in all metrics. In the independent-data analysis, ML classifiers showed strong discrimination between grade 2 and 4, despite lower performance metrics than the mixed analysis. ML algorithms performed better in glioma tumor grading based on WHO 2021 criteria. Nonetheless, the clinical use of ML classifiers needs further investigation. 3 TECHNICAL EFFICACY: Stage 2.

One of the oldest and largest biodiversity-related citizen science (CS) projects is eBird (https://ebird.org/home), developed by the Cornell Lab of Ornithology. It provides a mobile application for birdwatchers to record checklists of when, where, and how they have seen or heard birds. The Cornell Lab has also developed a mobile application, Merlin, which uses a deep convolutional neural network to help users automatically identify bird species from photos, sounds (converted to spectrograms), or descriptions. This research investigates how the use of machine learning (ML) classification models affects the learning of novice birders. Our participants (computer science students with no previous background in ornithology) were randomly divided into three groups: one using the eBird application and identifying bird species themselves; one using the Merlin application, which uses ML to automatically identify birds from photos or sounds; and a control group. Participants were tested on their knowledge of birds before and after participating in the project to see how using the ML classification model affected their learning. We also interviewed selected participants after the post-test to understand what they had done and what might explain the results. Our results show that novice participants who participate in a CS project for even a short time significantly improve their content knowledge of familiar birds in their neighbourhood, and that eBird users outperform Merlin users on the knowledge post-test. Although AI may improve volunteer productivity and retention, there is a risk that it may reduce their learning. Further research with different participant profiles and project designs is needed to understand how to optimise volunteer productivity, retention, and learning in AI-assisted CS projects.

Machine learning (ML) classification models are increasingly being used in a wide range of applications where it is important that predictions are accompanied by uncertainties, including in climate and earth observation, medical diagnosis and bioaerosol monitoring. The output of an ML classification model is a type of categorical variable known as a nominal property in the International Vocabulary of Metrology (VIM). However, concepts related to uncertainty evaluation for nominal properties are not defined in the VIM, nor is such evaluation addressed by the Guide to the Expression of Uncertainty in Measurement (GUM). In this paper we propose a metrological conceptual uncertainty evaluation framework for nominal properties. This framework is based on probability mass functions and summary statistics thereof, and it is applicable to ML classification. We also illustrate its use in the context of two applications that exemplify the issues and have significant societal impact, namely, climate and earth observation and medical diagnosis. Our framework would enable an extension of the GUM to uncertainty for nominal properties, which would make both applicable to ML classification models.