Classification Metrics Research Articles

Optimizing poultry audio signal classification with deep learning and burn layer fusion

This study introduces a novel deep learning-based approach for classifying poultry audio signals, incorporating a custom Burn Layer to enhance model robustness. The methodology integrates digital audio signal processing, convolutional neural networks (CNNs), and the innovative Burn Layer, which injects controlled random noise during training to reinforce the model's resilience to input signal variations. The proposed architecture is streamlined, with convolutional blocks, densely connected layers, dropout, and an additional Burn Layer to fortify robustness. The model demonstrates efficiency by reducing trainable parameters to 191,235, compared to traditional architectures with over 1.7 million parameters. The proposed model utilizes a Burn Layer with burn intensity as a parameter and an Adamax optimizer to optimize and address the overfitting problem. Thorough evaluation using six standard classification metrics showcases the model's superior performance, achieving exceptional sensitivity (96.77%), specificity (100.00%), precision (100.00%), negative predictive value (NPV) (95.00%), accuracy (98.55%), F1 score (98.36%), and Matthew’s correlation coefficient (MCC) (95.88%). This research contributes valuable insights into the fields of audio signal processing, animal health monitoring, and robust deep-learning classification systems. The proposed model presents a systematic approach for developing and evaluating a deep learning-based poultry audio classification system. It processes raw audio data and labels to generate digital representations, utilizes a Burn Layer for training variability, and constructs a CNN model with convolutional blocks, pooling, and dense layers. The model is optimized using the Adamax algorithm and trained with data augmentation and early-stopping techniques. Rigorous assessment on a test dataset using standard metrics demonstrates the model's robustness and efficiency, with the potential to significantly advance animal health monitoring and disease detection through audio signal analysis.

Bayesian optimized multimodal deep hybrid learning approach for tomato leaf disease classification

Manual identification of tomato leaf diseases is a time-consuming and laborious process that may lead to inaccurate results without professional assistance. Therefore, an automated, early, and precise leaf disease recognition system is essential for farmers to ensure the quality and quantity of tomato production by providing timely interventions to mitigate disease spread. In this study, we have proposed seven robust Bayesian optimized deep hybrid learning models leveraging the synergy between deep learning and machine learning for the automated classification of ten types of tomato leaves (nine diseased and one healthy). We customized the popular Convolutional Neural Network (CNN) algorithm for automatic feature extraction due to its ability to capture spatial hierarchies of features directly from raw data and classical machine learning techniques [Random Forest (RF), XGBoost, GaussianNB (GNB), Support Vector Machines (SVM), Multinomial Logistic Regression (MLR), K-Nearest Neighbor (KNN)], and stacking for classifications. Additionally, the study incorported a Boruta feature filtering layer to capture the statistically significant features. The standard, research-oriented PlantVillage dataset was used for the performance testing, which facilitates benchmarking against prior research and enables meaningful comparisons of classification performance across different approaches. We utilized a variety of statistical classification metrics to demonstrate the robustness of our models. Using the CNN-Stacking model, this study achieved the highest classification performance among the seven hybrid models. On an unseen dataset, this model achieved average precision, recall, f1-score, mcc, and accuracy values of 98.527%, 98.533%, 98.527%, 98.525%, and 98.268%, respectively. Our study requires only 0.174 s of testing time to correctly identify noisy, blurry, and transformed images. This indicates our approach's time efficiency and generalizability in images captured under challenging lighting conditions and with complex backgrounds. Based on the comparative analysis, our approach is superior and computationally inexpensive compared to the existing studies. This work will aid in developing a smartphone app to offer farmers a real-time disease diagnosis tool and management strategies.

Are light curve classification metrics good proxies for SN Ia cosmological constraining power?

When selecting a lc classifier for use as part of a photometric supernova Ia (SN Ia) cosmological analysis, it is common to make decisions based on metrics of classification performance, such as the contamination within the photometrically classified SN Ia sample, rather than a measure of cosmological constraining power. If the former is an appropriate proxy for the latter, this practice would save those designing an analysis pipeline from eliminate the computational expense of a full cosmology forecast in the analysis pipeline design process This study tests the assumption that lc classification metrics are an appropriate proxy for cosmology metrics. We emulated photometric SN Ia cosmology lc samples with controlled contamination rates of individual contaminant classes and evaluated each of them under a set of classification metrics. We then derived cosmological parameter constraints from all samples under two common analysis approaches and quantified the impact of contamination by each contaminant class on the resulting cosmological parameter estimates. We observe that cosmology metrics are sensitive to both the contamination rate and the class of the contaminating population, whereas the classification metrics are shown to be insensitive to the latter. Based on these findings, we discourage any exclusive reliance on classification-based metrics for analysis design decisions, which (counterintuitively) include but are not limited to the classifier choice. Instead, we recommend optimising science analysis pipeline design choices using a metric of the information gained about the physical parameters of interest.

Development of Machine Learning Models for Predicting the 1-Year Risk of Reoperation After Lower Limb Oncological Resection and Endoprosthetic Reconstruction Based on Data From the PARITY Trial.

Oncological resection and reconstruction involving the lower extremities commonly lead to reoperations that impact patient outcomes and healthcare resources. This study aimed to develop a machine learning (ML) model to predict this reoperation risk. This study was conducted according to TRIPOD + AI. Data from the PARITY trial was used to develop ML models to predict the 1-year reoperation risk following lower extremity oncological resection and reconstruction. Six ML algorithms were tuned and calibrated based on fivefold cross-validation. The best-performing model was identified using classification and calibration metrics. The polynomial support vector machine (SVM) model was chosen as the best-performing model. During internal validation, the SVM exhibited an AUC-ROC of 0.73 and a Brier score of 0.17. Using an optimal threshold that balances all quadrants of the confusion matrix, the SVM exhibited a sensitivity of 0.45 and a specificity of 0.81. Using a high-sensitivity threshold, the SVM exhibited a sensitivity of 0.68 and a specificity of 0.68. Total operative time was the most important feature for reoperation risk prediction. The models may facilitate reoperation risk stratification, allowing for better patient counseling and for physicians to implement measures that reduce surgical risks.

Comparison of a machine learning model with a conventional rule-based selective dry cow therapy algorithm for detection of intramammary infections

We trained machine learning models to identify intramammary infections (IMI) in late lactation cows at dry-off to guide antibiotic treatment, and compared their performance to a rule-based algorithm that is currently used on dairy farms in the US. We conducted an observational test-characteristics study using a data set of 3,645 cows approaching dry-off from 68 US dairy herds. The outcome variables of interest were cow-level IMI caused by all pathogens, major pathogens, and Streptococcus and Strep-like organisms (SSLO), which were determined using aerobic culture of aseptic quarter-milk samples and identification of isolates using MALDI-TOF. Individual cow records were extracted from the farm software to create 53 feature variables at the cow and 39 at the herd-level which were derived from cow-level descriptive data, records of clinical mastitis events, results from routine testing of milk for volume and concentrations of somatic cell count (SCC), fat, and protein. ML algorithms evaluated were logistic regression, decision tree, random forest, light gradient-boosting machine, naïve bayes, and neural networks. For comparison, cows were also classified according to a conventional rule-based algorithm that considered a cow as high risk for IMI if she had at one or more high SCC (>200,000 cells/ml) tests or ≥2 cases of clinical mastitis during the lactation of enrollment. Area under the curve (AUC) and Youden's index were used to compare models, in addition to binary classification metrics, including sensitivity, specificity, and predictive values. ML models had slightly higher AUC and Youden's index values than the rule-based algorithm for all IMI outcomes of interest. However, these improvements in prediction accuracy were substantially less than what we had considered necessary for the technology to be a worthwhile alternative to the rule-based algorithm. Therefore, evidence is lacking to support the wholesale use of ML-guided selective dry cow therapy at the moment. We recommend that producers wanting to implement algorithm-guided SDCT use a rule-based method.

Comparison of blood-based liver fibrosis scores in the Mount Sinai Health System, MASLD Registry, and NHANES 2017-2020 study.

Liver fibrosis is a critical public health concern, necessitating early detection to prevent progression. This study evaluates the recently developed LiverRisk score and steatosis-associated Fibrosis Estimator (SAFE) score against established indices for prognostication and/or fibrosis prediction in 4diverse cohorts, including participants with metabolic dysfunction-associated steatotic liver disease (MASLD). We used data from the Mount Sinai Data Warehouse (32,828 participants without liver disease diagnoses), the Mount Sinai MASLD/MASH Longitudinal Registry (422 participants with MASLD), and National Health and Nutrition Examination Survey 2017-2020 (4133 participants representing the general population) to compare LiverRisk score, FIB-4 index, APRI, and SAFE score. Analyses included Cox proportional hazards regressions, Kaplan-Meier estimates, and classification metrics to evaluate performance in prognostication and fibrosis prediction. In Mount Sinai Data Warehouse, LiverRisk score was significantly associated with future liver-related outcomes but did not significantly outperform FIB-4 or APRI for predicting any of the outcomes. In the general population, LiverRisk score and SAFE score outperformed FIB-4 and APRI in identifying fibrosis, but LiverRisk score underperformed among participants who were non-White or had type 2 diabetes. Among participants with MASLD, SAFE score outperformed FIB-4 and APRI in 1 of 2 cohorts, but there were generally few significant performance differences between all 4 scores. LiverRisk score does not consistently outperform existing predictors in diverse populations, and further validation is needed before adoption in settings with significant differences from the original derivation cohorts. It remains necessary to replicate the ability of these scores to predict liver-specific mortality, as well as to develop diagnostic tools for liver fibrosis that are accessible and substantially better than current scores, especially among patients with MASLD and other chronic liver conditions.

MEMPSEP‐I. Forecasting the Probability of Solar Energetic Particle Event Occurrence Using a Multivariate Ensemble of Convolutional Neural Networks

AbstractThe Sun continuously affects the interplanetary environment through a host of interconnected and dynamic physical processes. Solar flares, Coronal Mass Ejections (CMEs), and Solar Energetic Particles (SEPs) are among the key drivers of space weather in the near‐Earth environment and beyond. While some CMEs and flares are associated with intense SEPs, some show little to no SEP association. To date, robust long‐term (hours‐days) forecasting of SEP occurrence and associated properties (e.g., onset, peak intensities) does not effectively exist and the search for such development continues. Through an Operations‐2‐Research support, we developed a self‐contained model that utilizes a comprehensive data set and provides a probabilistic forecast for SEP event occurrence and its properties. The model is named Multivariate Ensemble of Models for Probabilistic Forecast of Solar Energetic Particles (MEMPSEP). MEMPSEP workhorse is an ensemble of Convolutional Neural Networks that ingests a comprehensive data set (MEMPSEP‐III by Moreland et al. (2024, https://doi.org/10.1029/2023SW003765)) of full‐disc magnetogram‐sequences and in situ data from different sources to forecast the occurrence (MEMPSEP‐I—this work) and properties (MEMPSEP‐II by Dayeh et al. (2024, https://doi.org/10.1029/2023SW003697)) of a SEP event. This work focuses on estimating true SEP occurrence probabilities achieving a 2.5% improvement in reliability and a Brier score of 0.14. The outcome provides flexibility for the end‐users to determine their own acceptable level of risk, rather than imposing a detection threshold that optimizes an arbitrary binary classification metric. Furthermore, the model‐ensemble, trained to utilize the large class‐imbalance between events and non‐events, provides a clear measure of uncertainty in our forecast.

Two-step deep learning models for detection and identification of the manufacturers and types of dental implants on panoramic radiographs.

The purpose of this study is to develop two-step deep learning models that can automatically detect implant regions on panoramic radiographs and identify several types of implants. A total of 1,574 panoramic radiographs containing 3675 implants were included. The implant manufacturers were Kyocera, Dentsply Sirona, Straumann, and Nobel Biocare. Model A was created to detect oral implants and identify the manufacturers using You Only Look Once (YOLO) v7. After preparing the image patches that cropped the implant regions detected by model A, model B was created to identify the implant types per manufacturer using EfficientNet. Model A achieved very high performance, with recall of 1.000, precision of 0.979, and F1 score of 0.989. It also had accuracy, recall, precision, and F1 score of 0.98 or higher for the classification of the manufacturers. Model B had high classification metrics above 0.92, exception for Nobel's class 2 (Parallel). In this study, two-step deep learning models were built to detect implant regions, identify four manufacturers, and identify implant types per manufacturer.

Leakage Identification of Underground Structures Using Classification Deep Neural Networks and Transfer Learning.

Water leakage defects often occur in underground structures, leading to accelerated structural aging and threatening structural safety. Leakage identification can detect early diseases of underground structures and provide important guidance for reinforcement and maintenance. Deep learning-based computer vision methods have been rapidly developed and widely used in many fields. However, establishing a deep learning model for underground structure leakage identification usually requires a lot of training data on leakage defects, which is very expensive. To overcome the data shortage, a deep neural network method for leakage identification is developed based on transfer learning in this paper. For comparison, four famous classification models, including VGG16, AlexNet, SqueezeNet, and ResNet18, are constructed. To train the classification models, a transfer learning strategy is developed, and a dataset of underground structure leakage is created. Finally, the classification performance on the leakage dataset of different deep learning models is comparatively studied under different sizes of training data. The results showed that the VGG16, AlexNet, and SqueezeNet models with transfer learning can overall provide higher and more stable classification performance on the leakage dataset than those without transfer learning. The ResNet18 model with transfer learning can overall provide a similar value of classification performance on the leakage dataset than that without transfer learning, but its classification performance is more stable than that without transfer learning. In addition, the SqueezeNet model obtains an overall higher and more stable performance than the comparative models on the leakage dataset for all classification metrics.

Optimizing machine learning algorithms for diabetes data: A metaheuristic approach to balancing and tuning classifiers parameters

Diabetes mellitus poses a global health concern, prompting the development of machine learning algorithms designed to construct a model for the accurate classification of patients, enabling precise diagnoses and early-stage treatment. However, the efficacy of classifying diabetes patients through machine learning relies on datasets, often plagued by imbalance, leading to biased classification and inaccurate diagnoses. Previous research attempts, employing techniques like random sampling (under-sampling and oversampling) and the Synthetic Minority Oversampling Technique (SMOTE), have struggled to achieve optimally balanced datasets. Additionally, setting the best parameters for machine learning classifiers remains a challenging task. To address these issues, this research focuses on devising a methodological metaheuristic optimization, a machine learning algorithm tailored for diabetes data balancing, and classifier hyperparameter tuning. Leveraging Particle Swarm Optimization (PSO) algorithm for diabetes data balancing and a genetic algorithm to select the optimal architecture for various machine learning classifiers. The study compares the performance of the proposed metaheuristic data balancer and classifier architecture parameter tuner using classification metrics (F1 score, Average Precision–Recall (APR), AUC, and accuracy). The PSO balanced dataset emerges as the most effective in classifying diabetes, with an Average Percentage Improvement (API) in classification performance metrics: 20.78% accuracy, 16.79% area under the curve for receiver operating characteristics, and a significant 32.78% enhancement in APR. Moreover, the XGBOOST classifier trained with a genetic algorithm demonstrates minimal computational training time for the Centre for Disease Control and Prevention (CDC) diabetes dataset compared to the artificial neural network and random forest classifier. Notably, the imbalanced CDC diabetes dataset exhibits the least APR compared to random under-sampling and the PSO data balancing technique.

Transformer-based deep learning networks for fault detection, classification, and location prediction in transmission lines

ABSTRACT Fault detection, classification, and location prediction are crucial for maintaining the stability and reliability of modern power systems, reducing economic losses, and enhancing system protection sensitivity. This paper presents a novel Hierarchical Deep Learning Approach (HDLA) for accurate and efficient fault diagnosis in transmission lines. HDLA leverages two-stage transformer-based classification and regression models to perform Fault Detection (FD), Fault Type Classification (FTC), and Fault Location Prediction (FLP) directly from synchronized raw three-phase current and voltage samples. By bypassing the need for feature extraction, HDLA significantly reduces computational complexity while achieving superior performance compared to existing deep learning methods. The efficacy of HDLA is validated on a comprehensive dataset encompassing various fault scenarios with diverse types, locations, resistances, inception angles, and noise levels. The results demonstrate significant improvements in accuracy, recall, precision, and F1-score metrics for classification, and Mean Absolute Errors (MAEs) and Root Mean Square Errors (RMSEs) for prediction, showcasing the effectiveness of HDLA for real-time fault diagnosis in power systems.

Performance of evaluation metrics for classification in imbalanced data

Performance of evaluation metrics for classification in imbalanced data

Comparing discrete choice and machine learning models in predicting destination choice

Destination choice modeling has long been dominated by theory-based discrete choice models. Simultaneously, machine learning has demonstrated improved predictive performance to other fields of discrete choice modeling. The objective of this research was to compare machine learning models and a multinomial logit model in predicting destination choice. The models were assessed on their predictive performance using metrics for both binary classification and probabilistic classification. The results indicate that machine learning models, especially a random forest model, could bring improvements in prediction accuracy. The more data was used in training the models, the better the machine learning models tended to perform compared to the multinomial logit model. With less data, the multinomial logit model performed comparatively well. The findings are relevant for the field of destination choice modeling, where evidence on the use of machine learning models is very limited. In addition, the unbalanced choice sets of destination choice models with multiple non-chosen alternatives increases the need for further research in model fit and parameter tuning.

Identifying arguments within a text: Categorizing errors and their impact in arguments' relation prediction

The automatic identification of argument units within a text is a crucial task, as it is the first step that should be performed by an end-to-end argument mining system. In this work, we propose an approach for categorizing errors in predicted argument units, which allows the evaluation of segmentation models from an argumentative perspective. We assess the ability of several models to generalize knowledge across different text domains and, through the proposed categorization, we show differences in their behavior that may not be noticeable using standard classification metrics. Furthermore, we assess how the errors in predicted argument units impact on a task that rely on accurate unit identification, an aspect that has not been studied in previous research, and that helps to evaluate the usability of an imperfect segmentation model beyond the segmentation task itself.

A new data complexity measure for multi-class imbalanced classification tasks

The skewed class distribution and data complexity may severely affect the imbalanced classification results. The cost of classification can be significantly reduced if these data complexity are measured and pre-processed prior to training, particularly when dealing with large-scale and high-dimensional datasets. Although many methods have been proposed to evaluate data complexity, most of them fail to fully consider the interaction among different data characteristics, or the connection between class imbalance and these characteristics, thus posing a serious challenge to effectively evaluate the difficulty of classification. This paper presents a new data complexity measure MFII (multi-factor imbalance index), which measures the combined effects of the skewed class distribution and data characteristics on classification difficulty. In particular, it further comprehensively investigates the impact of overlap size, confusion degree, and sub-cluster structure. VoR (value of resolution) and DoC (degree of consistency) are also proposed to evaluate the resolution and interpretability of complexity measures. The experimental results demonstrate that MFII has lower VoR and a stronger correlation with classification metrics, which indicates that MFII can more accurately evaluate the difficulty of multi-class imbalanced classification tasks.

Значение видео-уродинамического исследования в выборе метода лечения детей с нейрогенными дисфункциями мочевого пузыря

Video urodynamics is the standard for assessment of the lower urinary tract in children with neurogenic urinary disorders aiming to provide more information about the condition of the lower urinary tract v. retrograde urethrocystography. Video urodynamics parameters are used as possible indicators for choosing various surgical treatment options, however, all work is focused on adult patients, and data for the pediatric cohort were not available as yet. The purpose of this research was the evaluation of the role of video urodynamics and retrograde urethrocystography in the selection of effective conservative and surgical treatment of neurogenic urinary disorders in children. Materials and methods used: 112 patients aged 1,8 to 18 y/o with neurogenic urinary disorders were included in the study, of whom 101 were those who had been proven effectively treated with conservative M-cholinoblockers or had successfully undergone minimally invasive surgery as follows: intradetrusor injection of botulinum toxin (n=21), transurethral correction of VUR (n=17), simultaneous execution of two of these interventions (n=22). Based on the results of video urodynamic and cystographic examination of these patients with the CHAID technique, the classification trees were built in order to evaluate the effective treatment (conservative/surgical). Results: the median age of patients was 9,0 years [7,8-12,0]. The first classification tree containing the parameter “presence of VUR according to the results of retrograde cystography.” The sensitivity of the obtained model for the effectiveness of surgical treatment was 83,3% v. 73,2% for conservative treatment. The accuracy was 79,2%. The ROC-AUC value was 0,783. The second classification tree contained the maximum Pves value according to the results of the video urodynamics, presence of hyperactivity, presence of VUR. The sensitivity of the obtained model for the effectiveness of surgical treatment was 93,3% v. 97,6% for conservative treatment. The accuracy was 95,0%. The ROC-AUC value was 0,972. Comparison of ROC-AUC values by the Delong test revealed statistically significant differences between these binary classification metrics (p<0.001). Conclusion: the parameters of video urodynamics have greater diagnostic accuracy in effective prediction of both conservative and surgical treatment methods compared to the retrograde urethrocystography.

Radiomics signature for automatic hydronephrosis detection in unenhanced Low-Dose CT

PurposeTo investigate the diagnostic performance of an automatic pipeline for detection of hydronephrosis on kidney's parenchyma on unenhanced low-dose CT of the abdomen. MethodsThis retrospective study included 95 patients with confirmed unilateral hydronephrosis in an unenhanced low-dose CT of the abdomen. Data were split into training (n = 67) and test (n = 28) cohorts. Both kidneys for each case were included in further analyses, whereas the kidney without hydronephrosis was used as control. Using the training cohort, we developed a pipeline consisting of a deep-learning model for automatic segmentation (a Convolutional Neural Network based on nnU-Net architecture) of the kidney's parenchyma and a radiomics classifier to detect hydronephrosis. The models were assessed using standard classification metrics, such as area under the ROC curve (AUC), sensitivity and specificity, as well as semantic segmentation metrics, including Dice coefficient and Jaccard index. ResultsUsing manual segmentation of the kidney’s parenchyma, hydronephrosis can be detected with an AUC of 0.84, a sensitivity of 75% and a specificity of 82%, a PPV of 81% and a NPV of 77%. Automatic kidney segmentation achieved a mean Dice score of 0.87 and 0.91 for the right and left kidney, respectively. Additionally, automatic segmentation achieved an AUC of 0.83, a sensitivity of 86%, specificity of 64%, PPV of 71%, and NPV of 82%. ConclusionOur proposed radiomics signature using automatic kidney's parenchyma segmentation allows for accurate hydronephrosis detection on unenhanced low-dose CT scans of the abdomen independently of widened renal pelvis. This method could be used in clinical routine to highlight hydronephrosis to radiologists as well as clinicians, especially in patients with concurrent parapelvic cysts and might reduce time and costs associated with diagnosing hydronephrosis.

Opti-CAM: Optimizing saliency maps for interpretability

Methods based on class activation maps (CAM) provide a simple mechanism to interpret predictions of convolutional neural networks by using linear combinations of feature maps as saliency maps. By contrast, masking-based methods optimize a saliency map directly in the image space or learn it by training another network on additional data. In this work we introduce Opti-CAM, combining ideas from CAM-based and masking-based approaches. Our saliency map is a linear combination of feature maps, where weights are optimized per image such that the logit of the masked image for a given class is maximized. We also fix a fundamental flaw in two of the most common evaluation metrics of attribution methods. On several datasets, Opti-CAM largely outperforms other CAM-based approaches according to the most relevant classification metrics. We provide empirical evidence supporting that localization and classifier interpretability are not necessarily aligned.

RailTrack-DaViT: A Vision Transformer-Based Approach for Automated Railway Track Defect Detection.

Railway track defects pose significant safety risks and can lead to accidents, economic losses, and loss of life. Traditional manual inspection methods are either time-consuming, costly, or prone to human error. This paper proposes RailTrack-DaViT, a novel vision transformer-based approach for railway track defect classification. By leveraging the Dual Attention Vision Transformer (DaViT) architecture, RailTrack-DaViT effectively captures both global and local information, enabling accurate defect detection. The model is trained and evaluated on multiple datasets including rail, fastener and fishplate, multi-faults, and ThaiRailTrack. A comprehensive analysis of the model's performance is provided including confusion matrices, training visualizations, and classification metrics. RailTrack-DaViT demonstrates superior performance compared to state-of-the-art CNN-based methods, achieving the highest accuracies: 96.9% on the rail dataset, 98.9% on the fastener and fishplate dataset, and 98.8% on the multi-faults dataset. Moreover, RailTrack-DaViT outperforms baselines on the ThaiRailTrack dataset with 99.2% accuracy, quickly adapts to unseen images, and shows better model stability during fine-tuning. This capability can significantly reduce time consumption when applying the model to novel datasets in practical applications.

“Dead or Alive?” Assessment of the Binary End-of-Event Outcome Indicator for the NEMSIS Public Research Dataset

Objectives The National Emergency Medical Services Information System (NEMSIS) provides a robust set of data to evaluate prehospital care. However, a major limitation is that the vast majority of the records lack a definitive outcome. This study aimed to evaluate the performance of a recently proposed method (“MLB” method) to impute missing end-of-EMS-event outcomes (“dead” or “alive”) for patient care reports in the NEMSIS public research dataset. Methods This study reproduced the recently published method for patient outcome imputation in the NEMSIS database and replicated the results for years 2017 through 2022 (n = 686,075). We performed statistical analyses leveraging an array of established performance metrics for binary classification from the machine learning literature. Evaluation metrics included overall accuracy, true positive rate, true negative rate, balanced accuracy, precision, F1 score, Cohen’s Kappa coefficient, Matthews’ coefficient, Hamming loss, the Jaccard similarity score, and the receiver operating characteristic/area under the curve. Results Extended metrics show consistently good imputation performance from year-to-year but reveal weakness in accurately indicating the minority class: e.g., after adjustments for conflicting labels, “dead” prediction accuracy is 77.7% for 2018 and 61.8% over the six-year NEMSIS sub-sample, even though overall accuracy is 98.8%. Slight over-fitting is also present. Conclusions This study found that the recently published MLB method produced reasonably good “dead” or “alive” indicators. We recommend reporting of True Positive Rate (“dead” prediction accuracy) and True Negative Rate (“alive” prediction accuracy) when applying the imputation method for analyses of NEMSIS data. More attention by EMS clinicians to complete documentation of target NEMSIS elements can further improve the method’s performance.