Poor Model Performance Research Articles

Barriers and facilitators for implementation of AI algorithms for ECG analysis and triage in patients with chest pain

Abstract Introduction Numerous artificial intelligence (AI) algorithms have been developed to assist in clinical decision-making, with a focus on areas like electrocardiogram (ECG) analysis due to the signal complexity and the varying levels of expertise among healthcare professionals. However, ensuring acceptance of these algorithms by end-users is paramount for successful implementation. Purpose This study aimed to identify barriers and facilitators related to implementation of an AI algorithm for ECG analysis and triage in patients with chest pain. Methods A three-round modified electronic Delphi study was conducted among Dutch potential end-users of an ECG-AI algorithm, including physicians, nurses and ambulance professionals. During the first round, participants brainstormed on barriers and facilitators, which were subsequently mapped to the Consolidated Framework for Implementation Research (CFIR). In the following two rounds, participants reviewed the identified barriers and facilitators for relevance and ranked them according to their importance. Results Twenty-five out of the initial forty respondents completed all three rounds, resulting in a dropout rate of 38%. Four barriers and twelve facilitators were identified. The most important facilitator was "Faster recognition of subtle ECG abnormalities", while "Poor model performance" emerged as the most important barrier. The most frequently mentioned corresponding CFIR domains were the innovation and inner setting domains. Conclusion The identification of barriers and facilitators provides valuable insights into the acceptance and utilisation of ECG-AI algorithms by end-users. By considering the CFIR domains, including innovation and inner setting, this study offers a comprehensive understanding of the contextual factors that influence the successful implementation of AI algorithms in healthcare settings. Addressing these barriers and facilitators will be crucial to facilitating future implementation processes.Barriers and facilitators

Bayesian optimized deep Q-network for diagnosing mine ventilation systems windage alteration fault targeting imbalanced data

Fault diagnosis of mine ventilation system is of great significance for mine safety production. Traditional machine learning algorithms have been widely applied in the field of mine ventilation systems windage alteration faults (WAFs) diagnosis, but these algorithms have poor intelligence and weak generalization ability. Therefore, this research constructs a WAFs diagnosis model (WAFs-BODQN) based on Bayesian optimized (BO) deep Q-network (DQN). Meanwhile, in order to solve the problem of poor performance of intelligent models when the dataset is imbalanced, a reinforcement learning reward function (KMeans-SDF) was designed, which combines the KMeans++ algorithm and spatial distance function (SDF). The WAFs-BODQN model with KMeans-SDF reward function has fault location diagnosis accuracies of 98.75 %, 97.50 %, and 95.00 % for mild, moderate, and extreme imbalanced datasets UB1, UB2, and UB3, respectively. This effectively avoids the "eccentricity" phenomenon of the intelligent model on majority class and achieves accurate prediction of fault locations. The accuracy of fault location diagnosis using the WAFs-BODQN model in simulation experiments and on-site empirical applications is higher than 95 %, demonstrating the intelligence, generalization, and feasibility of WAFs-BODQN model in the field of WAFs diagnosis in mine ventilation systems. By comparing the results with traditional machine learning and deep learning fault diagnosis models, it has been proven that the WAFs-BODQN model has advantages in intelligence and generalization ability. It is expected to establish a universal architecture for diagnosing WAFs, providing theoretical guidance and technical support for achieving intelligent mine ventilation.

Identifying the Key Factors of Intermodal Travel Using Interpretative Ensemble Learning

Intermodal travel is considered an effective method for achieving sustainable urban transportation. Understanding the factors influencing intermodal travel is crucial. Due to the relatively small proportion of intermodal trips within cities, datasets are significantly imbalanced, leading to poor performance of traditional logit models. In this paper, we develop a novel interpretable ensemble learning (IEL) model to identify key factors through voting by five types of machine learning models. We test our model on two datasets with different numbers of features. The results show that travel duration, travel distance, vehicle ownership, and distance to the nearest metro station are the key factors influencing intermodal travel, cumulatively contributing nearly 70% in the JDS2021 dataset with 14 features and nearly 80% in the SHS2019 dataset with 8 features. Furthermore, we analyze the interpretability of our model and compare it with the logit model. Our model enriches the methodology for modeling intermodal travel behavior.

Current predictors for morbidity regarding choice of birth after a previous caesarean section, show poor predictive value in prediction modelling

IntroductionAfter a previous caesarean section, morbidity in the subsequent delivery in general is considered to depend on the probability of a vaginal birth after caesarean. However counselling could be improved by adding individualized probability of serious morbidity following either trial of labour or elective repeat caesarean section. The objective of this study was to develop prediction models for morbidity for both a repeat caesarean section and a trial of labor for a Dutch population. Material and methodsIn this cohort study, data were joined from three previous studies (SIMPLE 1, SIMPLE 2 and SIMPLE 2-implementation study). A cohort of 2592 women with one previous caesarean section and a singleton pregnancy who delivered ≥37 weeks, without a contraindication for vaginal delivery was formed. Maternal morbidity was defined as postpartum hemorrhage, blood transfusion, uterine rupture, ICU admittance or death. Neonatal morbidity was defined as asphyxia, NICU-admittance or death. Potential predictors for morbidity were chosen based on literature and expert opinion. Logistic regression was used to develop the models. Internal validation was intended using bootstrapping techniques. Main outcome measures were predictors for morbidity and for validation of the model we used the area under the receiver operating characteristic curve for discriminative capacity and calibration for accuracy. ResultsIn 324 out of the 2592 cases (12.7 %) maternal or fetal complications occurred. In general total morbidity was higher in women choosing TOL as compared to ERCS (p < 0.001). The performance of the several developed models was insufficient, the area under the receiver operating characteristic curve did not rise above 0.6. Due to poor model performance, before correction for overfitting, interval validation was not conducted. ConclusionIn this large cohort, developing a Dutch population based prediction model that aimed to improve counselling on the mode of delivery, by predicting individual chances of morbidity for different delivery modes was not possible, due to lack of performance. Further study could be directed to cut off on VBAC success rates to a more general advice regarding the safest mode of delivery.

SimpleTrackV2: Rethinking the Timing Characteristics for Multi-Object Tracking.

Multi-object tracking tasks aim to assign unique trajectory codes to targets in video frames. Most detection-based tracking methods use Kalman filtering algorithms for trajectory prediction, directly utilizing associated target features for trajectory updates. However, this approach often fails, with camera jitter and transient target loss in real-world scenarios. This paper rethinks state prediction and fusion based on target temporal features to address these issues and proposes the SimpleTrackV2 algorithm, building on the previously designed SimpleTrack. Firstly, to address the poor prediction performance of linear motion models in complex scenes, we designed a target state prediction algorithm called LSTM-MP, based on long short-term memory (LSTM). This algorithm encodes the target's historical motion information using LSTM and decodes it with a multilayer perceptron (MLP) to achieve target state prediction. Secondly, to mitigate the effect of occlusion on target state saliency, we designed a spatiotemporal attention-based target appearance feature fusion (TSA-FF) target state fusion algorithm based on the attention mechanism. TSA-FF calculates adaptive fusion coefficients to enhance target state fusion, thereby improving the accuracy of subsequent data association. To demonstrate the effectiveness of the proposed method, we compared SimpleTrackV2 with the baseline model SimpleTrack on the MOT17 dataset. We also conducted ablation experiments on TSA-FF and LSTM-MP for SimpleTrackV2, exploring the optimal number of fusion frames and the impact of different loss functions on model performance. The experimental results show that SimpleTrackV2 handles camera jitter and target occlusion better, achieving improvements of 1.6%, 3.2%, and 6.1% in MOTA, IDF1, and HOTA, respectively, compared to the SimpleTrack algorithm.

Multifunctional Image Enhancement Tool with Image Processing Methods

The success of image-based machine learning and deep learning algorithms depends on the amount and variety of data sets. The richness and heterogeneity of the dataset has a direct impact on the capacity of the model to generalize to new, previously unknown data. Obtaining a large number and variety of images is a very difficult and laborious process, especially for practical applications to be developed in areas such as health, industry and agriculture. If this difficulty cannot be overcome, the performance of machine learning models may decrease or may not reach the desired level at all. This problem may lead to problems such as poor performance or overfitting of the developed models. To solve this problem, data augmentation techniques are used to increase the amount and variety of data sets. These strategies help to provide the necessary data set for the model to learn more successfully. In this publication, we present an image improvement tool that is specifically developed for academics working with limited data sets. This tool uses a variety of data augmentation techniques, including rotation, zooming, panning, noise addition, and blurring, to expand the scale and diversity of available data. This application offers a realistic method for dealing with the issues of limited data availability in machine learning research.

Mapping soil thickness by accounting for right‐censored data with survival probabilities and machine learning

AbstractIn digital soil mapping, modelling soil thickness poses a challenge due to the prevalent issue of right‐censored data. This means that the true soil thickness exceeds the depth of sampling, and neglecting to account for the censored nature of the data can lead to poor model performance and underestimation of the true soil thickness. Survival analysis is a well‐established domain of statistical modelling that can deal with censored data. The random survival forest is a notable example of a survival‐related machine learning approach used to address right‐censored soil property data in digital soil mapping. Previous studies that employed this model either focused on mapping the probability of soil thickness exceeding certain depths, and thereby not mapping soil thickness itself, or dismissed it due to perceived poor performance. In this study, we propose an alternative survival model to map soil thickness that is based on the inverse probability of censoring weighting. In this approach, calibration data are weighted by the inverse of the probability that soil thickness exceeds a certain depth, that is, a survival probability. These weights can then be used with most machine learning models. We used the weights with a regular random forest, and compared it with a random survival forest, and other strategies for handling right‐censored data, through a comprehensive synthetic simulation study and two real‐world case studies. The results suggest that the weighted random forest model produces competitive predictions, establishing it as a viable option for mapping right‐censored soil property data.

Attention-based fuzzy neural networks designed for early warning of financial crises of listed companies

Developing an early warning model for company financial crises holds critical significance in robust risk management and ensuring the enduring stability of the capital market. Although the existing research has achieved rich results, the disadvantages of insufficient text information mining and poor model performance still exist. To alleviate the problem of insufficient text information mining, we collect related financial and annual report data from 820 listed companies in mainland China from 2018 to 2023 by using sophisticated web crawlers and advanced text sentiment analysis technologies and using missing value interpolation, standardization, and data balancing to build multi-source datasets of companies. Ranking the feature importance of multi-source data promotes understanding the formation of financial crises for companies. In the meantime, a novel Attention-based Fuzzy Neural Network (AFNN) was proposed to parse multi-source data to forecast financial crises among listed companies. Experimental results indicate that AFNN exhibits significantly improved performance compared to other advanced methods.

Integrated ECOD-KNN Algorithm for Missing Values Imputation in Datasets: Outlier Removal

Missing data cause the incompleteness of data sets and can lead to poor performance of models which also can result in poor decisions, despite using the best handling methods. When there is a presence of outliers in the data, using KNN algorithm for missing values imputation produce less accurate results. Outliers are anomalies from the observations and removing outliers is one of the most important pre-processing step in all data analysis models. KNN algorithms are able to adapt to missing value imputation even though they are sensitive to outliers, which might end up affecting the quality of the imputation results. KNN is mainly used among other machine learning algorithms because it is simple to implement and have a relatively high accuracy. In the literature, various studies have explored the application of KNN in different domains, however failing to address the issue of how sensitive it is to outliers. In the proposed model, outliers are identified using a combination of the Empirical- Cumulative-distribution-based Outlier Detection (ECOD), Local Outlier Factor (LOF) and isolation forest (IForest). The outliers are substituted using the median of the non- outlier data and the imputation of missing values is done using the k-nearest neighbors algorithm. For the evaluation of the model, different metrics were used such as the Root Mean Square Error (RMSE), (MSE), R2 squared (R2 ) and Mean Absolute Error (MAE). It clearly indicated that dealing with outliers first before imputing missing values produces better imputation results than just using the traditional KNN technique which is sensitive to outliers.

Explaining poor performance of text-based machine learning models for vulnerability detection

Explaining poor performance of text-based machine learning models for vulnerability detection

A cross-dataset study on automatic detection of autism spectrum disorder from text data.

The goals of this article are as follows. First, to investigate the possibility of detecting autism spectrum disorder (ASD) from text data using the latest generation of machine learning tools. Second, to compare model performance on two datasets of transcribed statements, collected using two different diagnostic tools. Third, to investigate the feasibility of knowledge transfer between models trained on both datasets and check if data augmentation can help alleviate the problem of a small number of observations. We explore two techniques to detect ASD. The first one is based on fine-tuning HerBERT, a BERT-based, monolingual deep transformer neural network. The second one uses the newest, multipurpose text embeddings from OpenAI and a classifier. We apply the methods to two separate datasets of transcribed statements, collected using two different diagnostic tools: thought, language, and communication (TLC) and autism diagnosis observation schedule-2 (ADOS-2). We conducted several cross-dataset experiments in both a zero-shot setting and a setting where models are pretrained on one dataset and then training continues on another to test the possibility of knowledge transfer. Unlike previous studies, the models we tested obtained average results on ADOS-2 data but reached very good performance of the models in TLC. We did not observe any benefits from knowledge transfer between datasets. We observed relatively poor performance of models trained on augmented data and hypothesize that data augmentation by back translation obfuscates autism-specific signals. The quality of machine learning models that detect ASD from text data is improving, but model results are dependent on the type of input data or diagnostic tool.

Optimising synthetic datasets for machine learning-based prediction of building damage due to tunnelling

Assessment of tunnelling-induced building damage is a complex Soil-Structure Interaction (SSI) probelm, influenced by numerous geometric and material parameters of both the soil and structures, and is characterised by strong non-linear behaviour. Currently, there is a trend towards developing data-driven models using Machine Learning (ML) to capture this complex behaviour. Given the scarcity of real data, which typically comes from specific case studies, many researchers have turned to creating extensive synthetic datasets via sophisticated and validated numerical models like Finite Element Method (FEM). However, the development of these datasets and the training of advanced ML algorithms present significant challenges. poses significant challenges. Reliance solely on parameter domains and ranges derived from case studies can lead to imbalanced data distributions and subsequently poor performance of models in less populated regions. In this paper, we introduce a strategy for designing optimal high-confidence datasets through an iterative procedure. This process begins with a systematic literature review to determine the importance of parameters, their ranges, and dependencies as they pertain to building damage induced by SSI. Starting with several hundred FEM simulations, we generate an initial dataset and assess its quality and impact through Sensitivity Analysis (SA) studies, statistical modelling, and re-sampling in statistically significant regions. This evaluation allows us to refine the model’s input space, seeking scenarios that mitigate output distribution imbalances. The procedure is repeated until the datasets achieve a satisfactory balance for training metamodels, minimising bias effectively. Our findings highlight the success of this approach in identifying an optimal and feasible input space that significantly reduces imbalanced distributions of output features. This approach not only proves effective in our study but also offers a versatile methodology that could be adapted to other disciplines aiming to generate high-quality synthetic datasets.

A Novel Adversarial Deep Learning Method for Substation Defect Image Generation.

The presence of defects in substation equipment is a major factor affecting the safety of power transmission. Therefore, timely and accurate detection of these defects is crucial. As intelligent inspection robots advance, using mainstream object detection models to diagnose surface defects in substation equipment has become a focal point of current research. However, the lack of defect image data is one of the main factors affecting the accuracy of supervised deep learning-based defect detection models. To address the issue of insufficient training data for defect images with complex backgrounds, such as rust and surface oil leakage in substation equipment, which leads to the poor performance of detection models, this paper proposes a novel adversarial deep learning model for substation defect image generation: the Abnormal Defect Detection Generative Adversarial Network (ADD-GAN). Unlike existing generative adversarial networks, this model generates defect images based on effectively segmented local areas of substation equipment images, avoiding image distortion caused by global style changes. Additionally, the model uses a joint discriminator for both overall images and defect images to address the issue of low attention to local defect areas, thereby reducing the loss of image features. This approach enhances the overall quality of generated images as well as locally generated defect images, ultimately improving image realism. Experimental results demonstrate that the YOLOV7 object detection model trained on the dataset generated using the ADD-GAN method achieves a mean average precision (mAP) of 81.5% on the test dataset, and outperforms other image data augmentation and generation methods. This confirms that the ADD-GAN method can generate a high-fidelity image dataset of substation equipment defects.

Improving support vector machine and backpropagation performance for diabetes mellitus classification

Diabetes mellitus (DM) is a glucose disorder disease in the human body that contributes significantly to the high mortality rate. Various studies on early detection and classification have been conducted as a DM prevention effort by applying a machine learning model. The problems that may occur are weak model performance and misclassification caused by imbalanced data. The existence of dominating (majority) data causes poor model performance in identifying minority data. This paper proposed handling the problem of imbalanced data by performing the synthetic minority oversampling technique (SMOTE) and observing its effect on the classification performance of the support vector machine (SVM) and Backpropagation artificial neural network (ANN) methods. The experiment showed that the SVM method and imbalanced data achieved 94.31% accuracy, and the Backpropagation ANN achieved 91.56% accuracy. At the same time, the SVM method and balanced data produced an accuracy of 98.85%, while the Backpropagation ANN method and balanced data produced an accuracy of 94.90%. The results show that oversampling techniques can improve the performance of the classification model for each data class.

Improving support vector machine and backpropagation performance for diabetes mellitus classification

Diabetes mellitus is a glucose disorder disease in the human body that contributes significantly to the high mortality rate. Various studies on early detection and classification have been conducted as a diabetes mellitus prevention effort by applying a machine learning model. The problems that may occur are weak model performance and misclassification caused by imbalanced data. The existence of dominating (majority) data causes poor model performance in identifying minority data. This paper proposed handling the problem of imbalanced data by performing the synthetic minority oversampling technique (SMOTE) and observing its effect on the classification performance of the support vector machine (SVM) and Backpropagation artificial neural network (ANN) methods. The experiment showed that the SVM method and imbalanced data achieved 94.31% accuracy, and the Backpropagation ANN achieved 91.56% accuracy. At the same time, the SVM method and balanced data produced an accuracy of 98.85%, while the Backpropagation ANN method and balanced data produced an accuracy of 94.90%. The results show that oversampling techniques can improve the performance of the classification model for each data class.

DynamicWeighted Particle Swarm Optimization - Support Vector Machine Optimization in Recursive Feature Elimination Feature Selection

Feature Selection is a crucial step in data preprocessing to enhance machine learning efficiency, reduce computational complexity, and improve classification accuracy. The main challenge in feature selection for classification is identifying the most relevant and informative subset to enhance prediction accuracy. Previous studies often resulted in suboptimal subsets, leading to poor model performance and low accuracy. This research aims to enhance classification accuracy by utilizing Recursive Feature Elimination (RFE) combined with Dynamic Weighted Particle Swarm Optimization (DWPSO) and Support Vector Machine (SVM) algorithms. The research method involves the utilization of 12 datasets from the University of California, Irvine (UCI) repository, where features are selected via RFE and applied to the DWPSO-SVM algorithm. RFE iteratively removes the weakest features, constructing a model with the most relevant features to enhance accuracy. The research findings indicate that DWPSO-SVM with RFE significantly improves classification accuracy. For example, accuracy on the Breast Cancer dataset increased from 58% to 76%, and on the Heart dataset from 80% to 97%. The highest accuracy achieved was 100% on the Iris dataset. The conclusion of these findings that RFE in DWPSO-SVM offers consistent and balanced results in True Positive Rate (TPR) and True Negative Rate (TNR), providing reliable and accurate predictions for various applications.

Improved Intrusion Detection Based on Hybrid Deep Learning Models and Federated Learning.

The security of the Industrial Internet of Things (IIoT) is of vital importance, and the Network Intrusion Detection System (NIDS) plays an indispensable role in this. Although there is an increasing number of studies on the use of deep learning technology to achieve network intrusion detection, the limited local data of the device may lead to poor model performance because deep learning requires large-scale datasets for training. Some solutions propose to centralize the local datasets of devices for deep learning training, but this may involve user privacy issues. To address these challenges, this study proposes a novel federated learning (FL)-based approach aimed at improving the accuracy of network intrusion detection while ensuring data privacy protection. This research combines convolutional neural networks with attention mechanisms to develop a new deep learning intrusion detection model specifically designed for the IIoT. Additionally, variational autoencoders are incorporated to enhance data privacy protection. Furthermore, an FL framework enables multiple IIoT clients to jointly train a shared intrusion detection model without sharing their raw data. This strategy significantly improves the model's detection capability while effectively addressing data privacy and security issues. To validate the effectiveness of the proposed method, a series of experiments were conducted on a real-world Internet of Things (IoT) network intrusion dataset. The experimental results demonstrate that our model and FL approach significantly improve key performance metrics such as detection accuracy, precision, and false-positive rate (FPR) compared to traditional local training methods and existing models.

Comprehensive Analysis of Influencing Factors on Building Energy Performance and Strategic Insights for Sustainable Development: A Systematic Literature Review

A prerequisite for decreasing the intensification of energy in buildings is to evaluate and understand the influencing factors of building energy performance (BEP). These factors include building envelope features and outdoor climactic conditions, among others. Based on the importance of the influencing factors in the development of the building energy prediction model, various researchers are continuously employing different types of factors based on their popularity in academic literature, without a proper investigation of the most relevant factors, which, in some cases, potentially leads to poor model performance. However, this can be due to the absence of an adequate comprehensive analysis or review of all factors influencing BEP ubiquitously. Therefore, this paper conducts a holistic and comprehensive review of studies that have explored the various factors influencing energy use in residential and commercial buildings. In total, 74 research articles were systematically selected from the Scopus, ScienceDirect, and Institute of Electrical Electronics Engineers (IEEE) databases. Subsequently, by means of a systematic and bibliometric analysis, this paper comprehensively analyzed several important factors influencing BEP. The results reveals the important factors (such as windows and roofs) and engendered or shed light on the application of some energy-efficient strategies such as the utilization of a green roof and photovoltaic (PV) window, among others.

Poor transferability of richness models: Predicting plot-scale plant diversity in the Waterberg, South Africa

Modelling measures of biodiversity for understudied taxa or regions is one method to address taxonomic and geographic biases in biodiversity data. However, modelling biodiversity metrics, such as species richness, to unsampled areas is only useful if predictions are reliable. As a result, testing the transferability of richness models is necessary for assessing the potential for models to predict to unsampled areas. Here we test the transferability of plant richness models between two reserves to examine if the richness-environment relationship from one reserve can accurately estimate richness patterns in the other reserve, using the vascular plant species richness of the Waterberg region (savanna biome; northern South Africa) as a model system. Six richness response variables (total species, grass species, herb species, woody species, genus, and family richness) and a set of 16 predictor variables were analysed with three modelling approaches and two statistical techniques to build: (1) models comprising all available predictor variables, (2) models using a subset of predictor variables chosen based on model performance, and (3) models using a subset of predictor variables that reduce the difference in the environmental conditions between the two reserves. The models’ performance in the training area varied considerably, but soil variables were consistently the most important predictors of plant richness. However, the transferability of all the models was consistently poor across all modelling approaches and both techniques, possibly reflecting the degree to which each reserve contains novel environments absent from the other reserve (despite being separated by only c. 60 km and sharing vegetation types). Due to the poor performance of these richness models, they are currently not useful for predicting richness to other areas in the vicinity of the reserves or in the broader region. However, in areas, like the Waterberg region, that have high plant diversity and are poorly sampled, there may be value in continued development of richness models to address biodiversity gaps, thereby providing better data to inform conservation decisions.

Enhancing Histopathological Image Classification Performance through Synthetic Data Generation with Generative Adversarial Networks.

Breast cancer is the second most common cancer worldwide, primarily affecting women, while histopathological image analysis is one of the possibile methods used to determine tumor malignancy. Regarding image analysis, the application of deep learning has become increasingly prevalent in recent years. However, a significant issue is the unbalanced nature of available datasets, with some classes having more images than others, which may impact the performance of the models due to poorer generalizability. A possible strategy to avoid this problem is downsampling the class with the most images to create a balanced dataset. Nevertheless, this approach is not recommended for small datasets as it can lead to poor model performance. Instead, techniques such as data augmentation are traditionally used to address this issue. These techniques apply simple transformations such as translation or rotation to the images to increase variability in the dataset. Another possibility is using generative adversarial networks (GANs), which can generate images from a relatively small training set. This work aims to enhance model performance in classifying histopathological images by applying data augmentation using GANs instead of traditional techniques.