Variation In Model Performance Research Articles

From citizen science to AI models: Advancing cetacean vocalization automatic detection through multi-annotator campaigns

Continuous underwater Passive Acoustic Monitoring (PAM) has emerged as a strong tool for cetacean research. To handle the vast volume of collected data, it is essential to employ automated detection and classification methods. The recent advancement of deep learning, involving model training and testing, requires a large amount of labeled data. These labels are derived through the manual annotation of audio files often reliant on human experts. Based on an annotation campaign focusing on blue whale calls in the Indian Ocean involving 19 novice annotators and one expert in bioacoustics, this study explores the integration of novice annotators in marine bioacoustics research, through citizen science programs, which could drastically increase the size of labeled datasets and enhance the performance of detection and classification models. The analysis reveals distinctive annotation profiles influenced by the complexity of vocalizations and the annotators' strategies, ranging from conservative to permissive. To address the challenges of annotation discrepancies, Convolutional Neural Networks (CNNs) are trained on annotations from both novices and the expert. The results show variations in model performance. Our work highlights the importance of annotation guidelines encouraging a more conservative approach to improve overall annotation quality. In an effort to optimize the potential of multi-annotation and mitigate the presence of noisy labels, two annotation aggregation methods (majority voting and soft labeling) are proposed and tested. The results demonstrate that both methods, particularly when a sufficient number of annotators are involved, significantly improve model performance and reduce variability: the standard deviation of the area under PR and ROC curves fall under 0.02 for both vocalizations with 13 aggregated annotators, while it was at 0.17 and 0.21 for the Blue Whale Dcalls and 0.05 and 0.04 for the SEIO PBW vocalizations with all annotators separately. Moreover, these aggregation methods enable the training of models using non-expert annotations that achieve performance of models trained with expert annotations. These findings suggest that crowdsourced annotations from novice annotators can be a viable alternative to expert annotations.

Artificial Intelligence Support for Informal Patient Caregivers: A Systematic Review.

This study aims to explore how artificial intelligence can help ease the burden on caregivers, filling a gap in current research and healthcare practices due to the growing challenge of an aging population and increased reliance on informal caregivers. We conducted a search with Google Scholar, PubMed, Scopus, IEEE Xplore, and Web of Science, focusing on AI and caregiving. Our inclusion criteria were studies where AI supports informal caregivers, excluding those solely for data collection. Adhering to PRISMA 2020 guidelines, we eliminated duplicates and screened for relevance. From 947 initially identified articles, 10 met our criteria, focusing on AI's role in aiding informal caregivers. These studies, conducted between 2012 and 2023, were globally distributed, with 80% employing machine learning. Validation methods varied, with Hold-Out being the most frequent. Metrics across studies revealed accuracies ranging from 71.60% to 99.33%. Specific methods, like SCUT in conjunction with NNs and LibSVM, showcased accuracy between 93.42% and 95.36% as well as F-measures spanning 93.30% to 95.41%. AUC values indicated model performance variability, ranging from 0.50 to 0.85 in select models. Our review highlights AI's role in aiding informal caregivers, showing promising results despite different approaches. AI tools provide smart, adaptive support, improving caregivers' effectiveness and well-being.

Modeling weed seedling emergence for time-specific weed management: a systematic review

Abstract Understanding the timing of weed emergence is crucial to effective management. Management practices implemented too early may fail to completely control late-emerging seedlings, whereas management practices implemented too late will suffer from low efficacy. Weed emergence times reflect biological factors, such as seed dormancy and germination requirements, as well as environmental conditions. We conducted a systematic review of studies that developed models to predict weed emergence temporal patterns. We screened 1,854 studies, 98 of which were included in the final dataset. Most of the studies included were conducted in North America (51%) or Europe (30%). A wide variety of weed species (102) and families (21) were included, and many studies modeled several weeds. Grass weeds (Poaceae) were modeled most frequently (83 instances). Most weeds (40%) had base temperature ${T_{\rm{b}}}$ values between 0 and 5 C, and 38% had base water potential ${{\rm{\psi }}_{\rm{b}}}$ ranging from −1.0 to −0.5 MPa. Most studies used empirical parametric models, such as Weibull (40%) or Gompertz (30%) models. Nonparametric and mechanistic models were also represented. Models varied in their biological and environmental data requirements. In general, empirical parametric models based on hydrothermal time (i.e., time above base temperature and water potential thresholds) represented a good balance between ease of use and prediction accuracy. Soft computing approaches such as artificial neural networks demonstrated substantial potential in situations with complex emergence patterns and limited data availability, although they (soft computing approaches) can be susceptible to overfitting. Our study also demonstrated variability in model performance and limited generalizability across species and regions. This finding underscores the need for context-specific and well-validated weed emergence models to inform management, especially in the context of climate change.

Estimating Obesity Levels Using Decision Trees and K-Fold Cross-Validation: A Study on Eating Habits and Physical Conditions

This study harnesses the predictive capabilities of machine learning to explore the determinants of obesity within populations from Mexico, Peru, and Colombia, using a Decision Tree algorithm bolstered by 5-fold cross-validation. Our comprehensive analysis of 2111 individuals' lifestyle and physical condition data yielded accuracy, precision, recall, and F1-scores that notably peaked in the third and fifth folds. The findings affirmed the significance of dietary habits and physical activity as substantial predictors of obesity levels. The variability in model performance across the folds underscored the importance of robust cross-validation in enhancing the model's generalizability. This research contributes to the burgeoning field of data science in public health by providing a viable model for obesity prediction and laying the groundwork for targeted health interventions. Our study's insights are pivotal for public health officials and policymakers, serving as a stepping stone towards more sophisticated, data-driven approaches to combating obesity. The study, however, recognizes the inherent limitations of self-reported data and the need for broader datasets that encompass more diverse variables. Future research directions include the analysis of longitudinal data to establish causal relationships and the comparison of various machine learning models to optimize predictive performance

Identifying low acuity Emergency Department visits with a machine learning approach: The low acuity visit algorithms (LAVA).

To improve the performance of International Classification of Disease (ICD) code rule-based algorithms for identifying low acuity Emergency Department (ED) visits by using machine learning methods and additional covariates. We used secondary data on ED visits from the National Hospital Ambulatory Medical Survey (NHAMCS), from 2016 to 2020. We established baseline performance metrics with seven published algorithms consisting of International Classification of Disease, Tenth Revision codes used to identify low acuity ED visits. We then trained logistic regression, random forest, and gradient boosting (XGBoost) models to predict low acuity ED visits. Each model was trained on five different covariate sets of demographic and clinical data. Model performance was compared using a separate validation dataset. The primary performance metric was the probability that a visit identified by an algorithm as low acuity did not experience significant testing, treatment, or disposition (positive predictive value, PPV). Subgroup analyses assessed model performance across age, sex, and race/ethnicity. We used 2016-2019 NHAMCS data as the training set and 2020 NHAMCS data for validation. The training and validation data consisted of 53,074 and 9542 observations, respectively. Among seven rule-based algorithms, the highest-performing had a PPV of 0.35 (95% CI [0.33, 0.36]). All model-based algorithms outperformed existing algorithms, with the least effective-random forest using only age and sex-improving PPV by 26% (up to 0.44; 95% CI [0.40, 0.48]). Logistic regression and XGBoost trained on all variables improved PPV by 83% (to 0.64; 95% CI [0.62, 0.66]). Multivariable models also demonstrated higher PPV across all three demographic subgroups. Machine learning models substantially outperform existing algorithms based on ICD codes in predicting low acuity ED visits. Variations in model performance across demographic groups highlight the need for further research to ensure their applicability and fairness across diverse populations.

Prognostic risk models for incident hypertension: A PRISMA systematic review and meta-analysis.

Our goal was to review the available literature on prognostic risk prediction for incident hypertension, synthesize performance, and provide suggestions for future work on the topic. A systematic search on PUBMED and Web of Science databases was conducted for studies on prognostic risk prediction models for incident hypertension in generally healthy individuals. Study-quality was assessed using the Prediction model Risk of Bias Assessment Tool (PROBAST) checklist. Three-level meta-analyses were used to obtain pooled AUC/C-statistic estimates. Heterogeneity was explored using study and cohort characteristics in meta-regressions. From 5090 hits, we found 53 eligible studies, and included 47 in meta-analyses. Only four studies were assessed to have results with low risk of bias. Few models had been externally validated, with only the Framingham risk model validated more than thrice. The pooled AUC/C-statistics were 0.82 (0.77-0.86) for machine learning models and 0.78 (0.76-0.80) for traditional models, with high heterogeneity in both groups (I2 > 99%). Intra-class correlations within studies were 60% and 90%, respectively. Follow-up time (P = 0.0405) was significant for ML models and age (P = 0.0271) for traditional models in explaining heterogeneity. Validations of the Framingham risk model had high heterogeneity (I2 > 99%). Overall, the quality of included studies was assessed as poor. AUC/C-statistic were mostly acceptable or good, and higher for ML models than traditional models. High heterogeneity implies large variability in the performance of new risk models. Further, large heterogeneity in validations of the Framingham risk model indicate variability in model performance on new populations. To enable researchers to assess hypertension risk models, we encourage adherence to existing guidelines for reporting and developing risk models, specifically reporting appropriate performance measures. Further, we recommend a stronger focus on validation of models by considering reasonable baseline models and performing external validations of existing models. Hence, developed risk models must be made available for external researchers.

Evaluation of a single-layer urban energy balance model using measured energy fluxes by scaled outdoor experiments in humid subtropical climate

Urban energy balance models can simulate the localized climate and fluxes in urban areas. They are often evaluated using real urban site observations. However, uncertainties in anthropogenic emission, geometry and materials can hinder interpretation. These uncertainties can be eliminated by scaled outdoor models. We assess the performance of a single-layer urban canopy model (SLUCM) offline in predicting surface energy balance (SEB) fluxes within a street canyon-like urban configuration using field measurements from a scaled outdoor setup in Guangzhou, China. We utilize observations of SEB fluxes and surface temperature from a homogenous scaled urban site characterized by street canyons. The scaled observations in June–December 2020 in a dense urban environment with a height-to-width ratio (H/W) of 2 are employed to assess seasonal variations in model performance. Additionally, the December data from two distinct urban scenarios (H/W = 2(2020) and 0.5(2019)) are utilized to compare model performance in deep and shallow street canyon settings. Results show that QS(R2 = ∼0.47, RMSE = ∼46.0) is the most challenging to predict, followed by QH (R2 = ∼0.92, RMSE = ∼65.5), Q*(R2 = ∼0.99, RMSE = ∼39.5), L↑(R2 = ∼0.95, RMSE = ∼33.7), and K↑(R2 = ∼0.99, RMSE = ∼6.2). SLUCM faces difficulties simulating L↑ particularly during nighttime and cloudy conditions. It also tends to underestimate daytime net radiation (Q*), especially during hot, humid months. Moreover, H/W ratio slightly affects SLUCM's performance in simulating SEB fluxes under dry cool conditions denser urban settings (H/W = 2) show better model performance in K↑, QH, and QS but worse performance in L↑ and Q*.

Transferability of Temperature Evolution of Dissimilar Wire-Arc Additively Manufactured Components by Machine Learning.

Wire-arc additive manufacturing (WAAM) is a promising industrial production technique. Without optimization, inherent temperature gradients can cause powerful residual stresses and microstructural defects. There is therefore a need for data-driven methods allowing real-time process optimization for WAAM. This study focuses on machine learning (ML)-based prediction of temperature history for WAAM-produced aluminum bars with different geometries and process parameters, including bar length, number of deposition layers, and heat source movement speed. Finite element (FE) simulations are used to provide training and prediction data. The ML models are based on a simple multilayer perceptron (MLP) and performed well during baseline training and testing, giving a testing mean absolute percentage error (MAPE) of less than 0.7% with an 80/20 train-test split, with low variation in model performance. When using the trained models to predict results from FE simulations with greater length or number of layers, the MAPE increased to an average of 3.22% or less, with greater variability. In the cases of greatest difference, some models still returned a MAPE of less than 1%. For different scanning speeds, the performance was worse, with some outlier models giving a MAPE of up to 14.91%. This study demonstrates the transferability of temperature history for WAAM with a simple MLP approach.

A systematic review of hyperparameter tuning techniques for software quality prediction models

BACKGROUND: Software quality prediction models play a crucial role in identifying vulnerable software components during early stages of development, and thereby optimizing the resource allocation and enhancing the overall software quality. While various classification algorithms have been employed for developing these prediction models, most studies have relied on default hyperparameter settings, leading to significant variability in model performance. Tuning the hyperparameters of classification algorithms can enhance the predictive capability of quality models by identifying optimal settings for improved accuracy and effectiveness. METHOD: This systematic review examines studies that have utilized hyperparameter tuning techniques to develop prediction models in software quality domain. The review focused on diverse areas such as defect prediction, maintenance estimation, change impact prediction, reliability prediction, and effort estimation, as these domains demonstrate the wide applicability of common learning algorithms. RESULTS: This review identified 31 primary studies on hyperparameter tuning for software quality prediction models. The results demonstrate that tuning the parameters of classification algorithms enhances the performance of prediction models. Additionally, the study found that certain classification algorithms exhibit high sensitivity to their parameter settings, achieving optimal performance when tuned appropriately. Conversely, certain classification algorithms exhibit low sensitivity to their parameter settings, making tuning unnecessary in such instances. CONCLUSION: Based on the findings of this review, the study conclude that the predictive capability of software quality prediction models can be significantly improved by tuning their hyperparameters. To facilitate effective hyperparameter tuning, we provide practical guidelines derived from the insights obtained through this study.

Integrating EEG and Machine Learning to Analyze Brain Changes during the Rehabilitation of Broca's Aphasia.

The fusion of electroencephalography (EEG) with machine learning is transforming rehabilitation. Our study introduces a neural network model proficient in distinguishing pre- and post-rehabilitation states in patients with Broca's aphasia, based on brain connectivity metrics derived from EEG recordings during verbal and spatial working memory tasks. The Granger causality (GC), phase-locking value (PLV), weighted phase-lag index (wPLI), mutual information (MI), and complex Pearson correlation coefficient (CPCC) across the delta, theta, and low- and high-gamma bands were used (excluding GC, which spanned the entire frequency spectrum). Across eight participants, employing leave-one-out validation for each, we evaluated the intersubject prediction accuracy across all connectivity methods and frequency bands. GC, MI theta, and PLV low-gamma emerged as the top performers, achieving 89.4%, 85.8%, and 82.7% accuracy in classifying verbal working memory task data. Intriguingly, measures designed to eliminate volume conduction exhibited the poorest performance in predicting rehabilitation-induced brain changes. This observation, coupled with variations in model performance across frequency bands, implies that different connectivity measures capture distinct brain processes involved in rehabilitation. The results of this paper contribute to current knowledge by presenting a clear strategy of utilizing limited data to achieve valid and meaningful results of machine learning on post-stroke rehabilitation EEG data, and they show that the differences in classification accuracy likely reflect distinct brain processes underlying rehabilitation after stroke.

A simulation approach to investigate factors influencing the cost of omitted objectives in multiattribute models

AbstractEmpirical evidence suggests that decision‐makers are ill‐equipped to identify all relevant objectives in a decision problem. We examine the effect of an incomplete set of objectives using a Monte Carlo simulation to compare a baseline model to a reduced model incorporating only a subset of objectives. We assess the performance of reduced models varying in the number of objectives, the number of alternatives, the correlations among objectives, and attribute weights. Results suggest that missing objectives will most impact multiattribute models with negative correlations between objectives; similarly, models with equally weighted objectives suffer more than models with unequal weights. Decision problems with more objectives tend to be less impacted by missing objectives, given the same proportion of missing objectives. In contrast, decision problems with more alternatives are more impacted for some performance measures but less on others. However, the variation in model performance due to the number of objectives and alternatives is relatively minor compared to the variation due to the nature of the correlation between objectives.

Artificial Intelligence in Laboratory Technologies for Early Detection and Prognostication of Sepsis: A Systematic Review

Background: Sepsis: a complex clinical syndrome--represents life-threatening organ dysfunction instigated by an infection's dysregulated host response. Early detection and accurate prognostication of sepsis are crucial; they pave the way for timely intervention, ultimately enhancing patient outcomes. The rise in interest towards Artificial Intelligence (AI) applications within laboratory technologies is directly related to its potential for improving early detection and prognosis forecasting in sepsis cases; this interest comes as AI continues its advancement. Methods: We conducted a systematic review of studies utilizing AI algorithms in laboratory settings for early sepsis detection and prognostication: our methods entailed searching relevant databases for research published until October 2023. Our inclusion criteria spanned original articles; these applied machine learning (ML) and deep learning (DL) techniques to laboratory data--with the aim being sepsis prediction. We assessed the quality of the studies, extracted and synthesized data on AI model performance metrics - including: area under receiver operating characteristic curve (AUROC), sensitivity, specificity, and accuracy. Results: The review encompassed eight studies meeting the inclusion criteria; AI models showcased exceptional predictive capabilities--evidenced by a range of AUROC values from 0.799 to 0.9213, signifying noticeably acceptable performance. However, there was wide variation in sensitivity and specificity among these analyses: an indicator of heterogeneity in model performance. Superior prognostic accuracy and potential for real-time monitoring of patients' early sepsis signs emerged in several models: notably, within the first 12 hours of patient admission - their highest predictive period. The models frequently outperformed traditional scoring systems. Conclusion: Laboratory technology's AI applications significantly promise sepsis' early detection and prognostication. Reviewed studies suggest AI models may surpass traditional methods, offering potential integration into clinical workflows for rapid sepsis identification aid. Nevertheless, we also acknowledged both the variability in model performance and necessity of additional validation across diverse clinical settings. Future research: it must concentrate on two key aspects--the refinement of AI algorithms to enhance sensitivity and precision; furthermore, it should delve into evaluating the clinical impact of tools for sepsis prediction that are assisted by AI.

Benchmarking Perception to Streaming Inputs in Vision-Centric Autonomous Driving

In recent years, vision-centric perception has played a crucial role in autonomous driving tasks, encompassing functions such as 3D detection, map construction, and motion forecasting. However, the deployment of vision-centric approaches in practical scenarios is hindered by substantial latency, often deviating significantly from the outcomes achieved through offline training. This disparity arises from the fact that conventional benchmarks for autonomous driving perception predominantly conduct offline evaluations, thereby largely overlooking the latency concerns prevalent in real-world deployment. Although a few benchmarks have been proposed to address this limitation by introducing effective evaluation methods for online perception, they do not adequately consider the intricacies introduced by the complexity of input information streams. To address this gap, we propose the Autonomous driving Streaming I/O (ASIO) benchmark, aiming to assess the streaming input characteristics and online performance of vision-centric perception in autonomous driving. To facilitate this evaluation across diverse streaming inputs, we initially establish a dataset based on the CARLA Leaderboard. In alignment with real-world deployment considerations, we further develop evaluation metrics based on information complexity specifically tailored for streaming inputs and streaming performance. Experimental results indicate significant variations in model performance and ranking under different major camera deployments, underscoring the necessity of thoroughly accounting for the influences of model latency and streaming input characteristics during real-world deployment. To enhance streaming performance consistently across distinct streaming input features, we introduce a backbone switcher based on the identified streaming input characteristics. Experimental validation demonstrates its efficacy in perpetually improving streaming performance across varying streaming input features.

Examination of alternative eGFR definitions on the performance of deep learning models for detection of chronic kidney disease from fundus photographs.

Deep learning (DL) models have shown promise in detecting chronic kidney disease (CKD) from fundus photographs. However, previous studies have utilized a serum creatinine-only estimated glomerular rate (eGFR) equation to measure kidney function despite the development of more up-to-date methods. In this study, we developed two sets of DL models using fundus images from the UK Biobank to ascertain the effects of using a creatinine and cystatin-C eGFR equation over the baseline creatinine-only eGFR equation on fundus image-based DL CKD predictors. Our results show that a creatinine and cystatin-C eGFR significantly improved classification performance over the baseline creatinine-only eGFR when the models were evaluated conventionally. However, these differences were no longer significant when the models were assessed on clinical labels based on ICD10. Furthermore, we also observed variations in model performance and systemic condition incidence between our study and the ones conducted previously. We hypothesize that limitations in existing eGFR equations and the paucity of retinal features uniquely indicative of CKD may contribute to these inconsistencies. These findings emphasize the need for developing more transparent models to facilitate a better understanding of the mechanisms underpinning the ability of DL models to detect CKD from fundus images.

Classifying a Highly Polymorphic Tree Species across Landscapes Using Airborne Imaging Spectroscopy

Vegetation classifications on large geographic scales are necessary to inform conservation decisions and monitor keystone, invasive, and endangered species. These classifications are often effectively achieved by applying models to imaging spectroscopy, a type of remote sensing data, but such undertakings are often limited in spatial extent. Here we provide accurate, high-resolution spatial data on the keystone species Metrosideros polymorpha, a highly polymorphic tree species distributed across bioclimatic zones and environmental gradients on Hawai’i Island using airborne imaging spectroscopy and LiDAR. We compare two tree species classification techniques, the support vector machine (SVM) and spectral mixture analysis (SMA), to assess their ability to map M. polymorpha over 28,000 square kilometers where differences in topography, background vegetation, sun angle relative to the aircraft, and day of data collection, among others, challenge accurate classification. To capture spatial variability in model performance, we applied Gaussian process classification (GPC) to estimate the spatial probability density of M. polymorpha occurrence using only training sample locations. We found that while SVM and SMA models exhibit similar raw score accuracy over the test set (96.0% and 93.4%, respectively), SVM better reproduces the spatial distribution of M. polymorpha than SMA. We developed a final 2 m × 2 m M. polymorpha presence dataset and a 30 m × 30 m M. polymorpha density dataset using SVM classifications that have been made publicly available for use in conservation applications. Accurate, large-scale species classifications are achievable, but metrics for model performance assessments must account for spatial variation of model accuracy.

Performance Analysis and Development of QnA Chatbot Model Using LSTM in Answering Questions

This research aims to evaluate the performance of a Long Short-Term Memory (LSTM) based chatbot in answering questions (QnA). LSTM is a type of Recurrent Neural Network (RNN) architecture specifically designed to overcome vanishing gradient problems and can store long-term information. The method used is 5-fold cross-validation to train the chatbot model with 15 epochs at each fold using the dataset provided. The results showed variations in model performance at each fold. At the 5th fold, there was a decrease in performance with 84.63% accuracy, 96.36% precision, 64.9% recall, and 69.84% loss value. This finding shows that there is variability in the performance of the QnA chatbot model at each fold. In conclusion, the LSTM chatbot model can provide good answers with high accuracy and precision. Still, performance variations need to be considered in the use of this chatbot.

A framework for population inference: Combining machine learning, network analysis, and non-probability road sensor data

An increasing number of technologies is recording information about human traces, offering enormous potential for analyzing human behavior. Sensor data is typically non-probability-based because not every element in the target population has a positive and known probability of being recorded. Accordingly, using such data as a primary data source for population inference is currently an active field of research. In this paper, an algorithmic population inference framework using network analysis and non-probability data is developed. This approach is demonstrated using road sensor data as the primary source to infer the Dutch freight traffic across the state road network. We interpret the Dutch state road network as a graph, with traffic junctions as vertices and state roads as edges. Road sensors are installed on a non-probability sample of edges detecting passing transport vehicles. Photographs of the license plates allow for the rare opportunity of linking sensor data with population registers. Extreme gradient boosting is applied to learn the probability of vehicle detection by a sensor from features about time, edge, vehicle and vehicle owner. Population inference is made using the learned relationship to predict the probability of detection on each day of the year, along each edge in the network for each vehicle in the population. Different data scenarios were designed to simulate the effects of the non-probability nature of the data and the extreme class imbalance. Furthermore, several performance metrics were applied. With about 27 million records and over 100 features trained and tested on an imbalanced non-probability sample, substantial variation in model performance across test sets was found. Promising results were achieved using a balanced probability sample as a control: the model performed about halfway between random guessing and perfect prediction. These results are of high practical importance because combining non-probability with administrative data is currently considered one of the most promising across several disciplines.

Soft computing for nonlinear risk assessment of complex socio-technical systems

Work in socio-technical systems (STS) exhibits dynamic and complex behaviors, becoming difficult to model, evaluate and predict. This study develops an integrated soft computing approach for nonlinear risk assessment in STS: the functional resonance analysis method (FRAM) has been integrated with fuzzy sets. While FRAM is helpful to model performance variability in qualitative terms, the assessments are usually subjected to a high degree of uncertainty. This novel approach is meant to overcome the subjectivity associated with the qualitative analyses performed by experts’ judgments required by FRAM. For demonstration purposes, the approach has been applied to model a waste recycling process for construction materials. The results show how the approach allows assessing and ranking critical activities in STS operations.

The influence of atrial flutter in automated detection of atrial arrhythmias - are we ready to go into clinical practice?”

ObjectiveTo investigate the impact of atrial flutter (Afl) in the atrial arrhythmias classification task. We additionally advocate the use of a subject-based split for future studies in the field in order to avoid within-subject correlation which may lead to over-optimistic inferences. Finally, we demonstrate the effectiveness of the classifiers outside of the initially studied circumstances, by performing an inter-dataset model evaluation of the classifiers in data from different sources. MethodsECG signals of two private and three public (two MIT-BIH and Chapman ecgdb) databases were preprocessed and divided into 10s segments which were then subject to feature extraction. The created datasets were divided into a training and test set in two ways, based on a random split and a patient split. Classification was performed using the XGBoost classifier, as well as two benchmark classification models using both data splits. The trained models were then used to make predictions on the test data of the remaining datasets. ResultsThe XGBoost model yielded the best performance across all datasets compared to the remaining benchmark models, however variability in model performance was seen across datasets, with accuracy ranging from 70.6% to 89.4%, sensitivity ranging from 61.4% to 76.8%, and specificity ranging from 87.3% to 95.5%. When comparing the results between the patient and the random split, no significant difference was seen in the two private datasets and the Chapman dataset, where the number of samples per patient is low. Nonetheless, in the MIT-BIH dataset, where the average number of samples per patient is approximately 1300, a noticeable disparity was identified. The accuracy, sensitivity, and specificity of the random split in this dataset of 93.6%, 86.4%, and 95.9% respectively, were decreased to 88%, 61.4%, and 89.8% in the patient split, with the largest drop being in Afl sensitivity, from 71% to 5.4%. The inter-dataset scores were also significantly lower than their intra-dataset counterparts across all datasets. ConclusionsCAD systems have great potential in the assistance of physicians in reliable, precise and efficient detection of arrhythmias. However, although compelling research has been done in the field, yielding models with excellent performances on their datasets, we show that these results may be over-optimistic. In our study, we give insight into the difficulty of detection of Afl on several datasets and show the need for a higher representation of Afl in public datasets. Furthermore, we show the necessity of a more structured evaluation of model performance through the use of a patient-based split and inter-dataset testing scheme to avoid the problem of within-subject correlation which may lead to misleadingly high scores. Finally, we stress the need for the creation and use of datasets with a higher number of patients and a more balanced representation of classes if we are to progress in this mission.

Predicting Tree Mortality Using Spectral Indices Derived from Multispectral UAV Imagery

Past research has shown that remotely sensed spectral information can be used to predict tree health and vitality. Recent developments in unmanned aerial vehicles (UAVs) have now made it possible to derive such information at the tree and stand scale from high-resolution imagery. We used visible and multispectral bands from UAV imagery to calculate a set of spectral indices for 52,845 individual tree crowns within 38 forest stands in western Canada. We then used those indices to predict the mortality of these canopy trees over the following year. We evaluated whether including multispectral indices leads to more accurate predictions than indices derived from visible wavelengths alone and how the performance varies among three different tree species (Picea glauca, Pinus contorta, Populus tremuloides). Our results show that spectral information can be effectively used to predict tree mortality, with a random forest model producing a mean area under the receiver operating characteristic curve (AUC) of 89.8% and a balanced accuracy of 83.3%. The exclusion of multispectral indices worsened the model performance, but only slightly (AUC = 87.9%, balanced accuracy = 81.8%). We found variation in model performance among species, with higher accuracy for the broadleaf species (balanced accuracy = 85.2%) than the two conifer species (balanced accuracy = 73.3% and 77.8%). However, all models overpredicted tree mortality by a major degree, which limits the use for tree mortality predictions on an individual level. Further improvements such as long-term monitoring, the use of hyperspectral data and cost-sensitive learning algorithms, and training the model with a larger and more balanced data set are necessary. Nevertheless, our results demonstrate that imagery from UAVs has strong potential for predicting annual mortality for individual canopy trees.