Evaluation Metrics Research Articles

Robustness in deep learning models for medical diagnostics: security and adversarial challenges towards robust AI applications

The current study investigates the robustness of deep learning models for accurate medical diagnosis systems with a specific focus on their ability to maintain performance in the presence of adversarial or noisy inputs. We examine factors that may influence model reliability, including model complexity, training data quality, and hyperparameters; we also examine security concerns related to adversarial attacks that aim to deceive models along with privacy attacks that seek to extract sensitive information. Researchers have discussed various defenses to these attacks to enhance model robustness, such as adversarial training and input preprocessing, along with mechanisms like data augmentation and uncertainty estimation. Tools and packages that extend the reliability features of deep learning frameworks such as TensorFlow and PyTorch are also being explored and evaluated. Existing evaluation metrics for robustness are additionally being discussed and evaluated. This paper concludes by discussing limitations in the existing literature and possible future research directions to continue enhancing the status of this research topic, particularly in the medical domain, with the aim of ensuring that AI systems are trustworthy, reliable, and stable.

Patch Correctness Assessment: A Survey

Most automated program repair methods rely on test cases to determine the correctness of the generated patches. However, due to the incompleteness of available test suites, some patches that pass all the test cases may still be incorrect. This issue is known as the patch overfitting problem. Overfitting problem is a longstanding problem in automated program repair. Due to overfitting patches, the patches obtained by automated program repair tools require further validation to determine their correctness. Researchers have proposed many methods to automatically assess the correctness of patches, but no systematic review provides a detailed introduction to this problem, the existing solutions, and the challenges. To address this deficiency, we systematically review the existing approaches to patch correctness assessment. We first offer a few examples of overfitting patches to acquire a more detailed understanding of this problem. We then propose a comprehensive categorization of publicly available techniques and datasets, examine the commonly used evaluation metrics, and perform an in-depth analysis of the effectiveness of the existing models in addressing the challenge of overfitting. Based on our analysis, we provided the difficulties encountered by current methodologies, alongside the possible avenues for future research exploration.

A novel MissForest-based missing values imputation approach with recursive feature elimination in medical applications

BackgroundMissing values in datasets present significant challenges for data analysis, particularly in the medical field where data accuracy is crucial for patient diagnosis and treatment. Although MissForest (MF) has demonstrated efficacy in imputation research and recursive feature elimination (RFE) has proven effective in feature selection, the potential for enhancing MF through RFE integration remains unexplored.MethodsThis study introduces a novel imputation method, “recursive feature elimination-MissForest” (RFE-MF), designed to enhance imputation quality by reducing the impact of irrelevant features. A comparative analysis is conducted between RFE-MF and four classical imputation methods: mean/mode, k-nearest neighbors (kNN), multiple imputation by chained equations (MICE), and MF. The comparison is carried out across ten medical datasets containing both numerical and mixed data types. Different missing data rates, ranging from 10 to 50%, are evaluated under the missing completely at random (MCAR) mechanism. The performance of each method is assessed using two evaluation metrics: normalized root mean squared error (NRMSE) and predictive fidelity criterion (PFC). Additionally, paired samples t-tests are employed to analyze the statistical significance of differences among the outcomes.ResultsThe findings indicate that RFE-MF demonstrates superior performance across the majority of datasets when compared to four classical imputation methods (mean/mode, kNN, MICE, and MF). Notably, RFE-MF consistently outperforms the original MF, irrespective of variable type (numerical or categorical). Mean/mode imputation exhibits consistent performance across various scenarios. Conversely, the efficacy of kNN imputation fluctuates in relation to varying missing data rates.ConclusionThis study demonstrates that RFE-MF holds promise as an effective imputation method for medical datasets, providing a novel approach to addressing missing data challenges in medical applications.

Enhancing Black-and-White Image Colorization Using Cycle GAN: A Study on Unpaired Image Translation

Abstract. This paper explores the application of Cycle-Consistent Generative Adversarial Network (Cycle GAN) for the colorization of black-and-white images, addressing the challenges in image to the image translation and it was not requiring paired datasets. Cycle GAN, a generative adversarial network, uses cycle consistency loss to map grayscale images to color, effectively preserving structural integrity. The model is trained on unpaired datasets containing black-and-white and color images, enabling it to generate colorized versions from grayscale inputs. Performance evaluation is conducted using Two significant evaluation metrics: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). Results demonstrate that Cycle GAN effectively preserves image structure and detail, achieving high PSNR and SSIM values, particularly in scenes with distinct color boundaries. However, challenges arise in complex and low-resolution scenes where color accuracy and detail reconstruction may degrade. Additionally, this paper assesses the impact of different loss functions, learning rate schedules, and training strategies on the models output quality. Future work will focus on addressing these limitations by incorporating attention mechanisms and other advanced techniques to improve Cycle GAN's ability to handle intricate image features and complex scenes.

Analyzing Collaborative Challenges and Needs of UX Practitioners when Designing with AI/ML

UX designers and researchers who work with AI/ML face different kinds of challenges throughout the design process. Though close collaborations with AI/ML developers and data scientists could address some of these challenges, such interdisciplinary collaborations are non-routine and hard to realize. In this work, we investigate barriers for effective collaboration with ML practitioners, how they affect UX practice of AI/ML applications, and what UX practitioners need to overcome these challenges. We conducted a qualitative study with 14 UX practitioners who are working on AI/ML products as designers or researchers. Our findings show that UX practitioners face challenges in communication, understanding the model and model development processes, establishing ways to collaborate, and reconciling model-centric metrics of evaluation with user-centric outcomes. They described various needs in terms of more visibility into model development processes, access to comprehensible and contextual model information, and hypothetical tools that can potentially support collaboration with ML practitioners and enhance UX design processes. We discuss implications of this research for designing collaborative tools and empowering UX practitioners.

Rumor detection model with weighted GraphSAGE focusing on node location

While social media platforms promote people’s information exchange and dissemination, they also make rumors spread rapidly on online platforms. Therefore, how to detect rumors quickly, timely and accurately has become a hot topic for scholars in related fields. Traditional deep learning models ignore the relationship and topology between nodes in the rumor detection task and use fixed weights or mean aggregation strategies in the feature aggregation process, which fail to capture the complex interactions between nodes and the dynamics of information propagation, limiting the accuracy and robustness of the rumor detection model. To address these problems, we propose a location-aware weighted GraphSAGE rumor detection model GSMA. We first introduce an attention mechanism that dynamically assigns different attention weights to different neighboring nodes for different degrees of aggregation, improving GraphSAGE’s strategy of using mean-value aggregation for all neighboring nodes during the aggregation process; second, we introduce a modulated position encoding into the model and encode the position information of nodes into the features to improve the model’s ability to perceive the relative position and order of nodes; finally, the post text sentiment is incorporated into the features to provide additional semantic information for the model as a way to achieve rumor detection in microblogging platforms. Experiments show that the accuracy of the GSMA model on Ma-Weibo and Weibo23 reaches 97.43% and 97.55%, which is an improvement of 1.11% and 0.77% compared to the benchmark GraphSAGE, and all the evaluation metrics are also improved compared to other optimal rumor detection models.

Boosting Financial Market Prediction Accuracy With Deep Learning and Big Data

The accuracy of financial market trend prediction directly impacts investors' decisions and financial institutions' risk management, making it a focal point of research in the financial domain. While traditional statistical methods lay the foundation for market forecasting, they still face limitations in handling complex, nonlinear, and non-stationary financial time series data with accuracy and adaptability. Time series analysis plays a pivotal role in financial market prediction, revealing historical patterns and future trends within the data. Addressing the limitations of existing methods, this study proposes a deep learning-based CEEMDAN-CNN-LSTM model. Leveraging the CEEMDAN decomposition method to capture the multi-frequency components of time series, CNN extracts spatial features, and LSTM learns temporal dependencies. Experimental results demonstrate that the performance of the CEEMDAN-CNN-LSTM model surpasses traditional models on standardized evaluation metrics such as RMSE, MAE, and MAPE across four major financial datasets.

Data-Driven Analysis of Ocean Fronts’ Impact on Acoustic Propagation: Process Understanding and Machine Learning Applications, Focusing on the Kuroshio Extension Front

Ocean fronts, widespread across the global ocean, cause abrupt shifts in physical properties such as temperature, salinity, and sound speed, significantly affecting underwater acoustic communication and detection. While past research has concentrated on qualitative analysis and small-scale research on ocean front sections, a comprehensive analysis of ocean fronts’ characteristics and their impact on underwater acoustics is lacking. This study employs high-resolution reanalysis data and in situ observations to accurately identify ocean fronts, sound speed structures, and acoustic propagation features from over six hundred thousand Kuroshio Extension Front (KEF) sections. Utilizing marine big data statistics and machine learning evaluation metrics such as out-of-bag (OOB) error and Shapley values, this study quantitatively assesses the variations in sound speed structures across the KEF and their effects on acoustic propagation shifts. This study’s key findings reveal that differences in sound speed structure are significantly correlated with KEF strength, with the channel axis depth and conjugate depth increasing with front strength, while the thermocline intensity and depth excess decrease. Acoustic propagation features in the KEF environment exhibit notable seasonal variations.

Review of Ni-Based Materials for Industrial Alkaline Hydrogen Production.

Hydrogen has been recognized as a green energy carrier, which can relieve energy shortage and environmental pollution. Currently, alkaline water electrolysis (AWE) driven by renewable energy to produce large-scale green hydrogen is a mainstream technology. However, tardy cathodic hydrogen evolution reaction (HER) and stability issue of catalysts make it challenging to meet the industrial requirements. Ni-based materials have attracted wide attention, thanks to their low cost and rich tuning possibilities, and many efforts have focused on their activity and stability. However, due to the significant discrepancy between laboratory and industrial conditions, these catalysts have not been widely deployed in industrial AWE. In this review, we first introduce the differences between laboratory and industrial stage, especially concerning equipment, protocols and evaluation metrics. To shorten these gaps, some strategies are proposed to improve the activity and stability of the Ni-based catalysts. Besides, some key issues related to the catalysts in industrial AWE device are also emphasized, including reverse-current and foreign ions in the electrolyte. Finally, the challenges and outlooks on the industrial alkaline AWE are discussed.

Разработка предметных графов знаний на основе семантического аннотирования табличных данных

The article outlines an approach and software tool for the automated enrichment of domain-oriented knowledge graphs with new facts derived from semantically annotated tabular data. For semantic annotation of table columns, a combination of three heuristic methods is proposed, leveraging named entity recognition in cells, lexical matching, and feature grouping. This approach is implemented as a dedicated handler within the Talisman software platform. An example and experimental evaluation of the approach during the semantic annotation of columns are provided using a test set of tabular data across six thematic categories: "organization employees," "open vacancies," "car model market," "famous scientists," "book sales," and "tennis player rankings." Evaluation metrics included precision, recall, and F-measure, with final results across all six categories as follows: precision - 79%, recall- 63%, F-measure - 70%. These results highlight the potential of the developed approach for enriching domain-oriented knowledge graphs with new facts from semantically annotated tabular data. The limitations of the proposed approach are also discussedfrom semantically annotated tabular data. The paper also provides a number of limitations of the proposed approach.

Inconsistency among evaluation metrics in link prediction

Abstract Link prediction is a paradigmatic and challenging problem in network science, which aims to predict missing links, future links and temporal links based on known topology. Along with the increasing number of link prediction algorithms, a critical yet previously ignored risk is that the evaluation metrics for algorithm performance are usually chosen at will. This paper implements extensive experiments on hundreds of real networks and 26 well-known algorithms, revealing significant inconsistency among evaluation metrics, namely different metrics probably produce remarkably different rankings of algorithms. Therefore, we conclude that any single metric cannot comprehensively or credibly evaluate algorithm performance. In terms of information content, we suggest the usage of at least two metrics: one is the area under the receiver operating characteristic curve (AUC), and the other is one of the following three candidates, say the area under the precision-recall curve (AUPR), the area under the precision curve (AUC-Precision), and the normalized discounted cumulative gain (NDCG). When the data is imbalanced, say the number of negative samples significantly outweighs the number of positive samples, the area under the generalized ROC curve (AUC-gROC) should also be used. In addition, as we have proved the essential equivalence of threshold-dependent metrics, if in a link prediction task, some specific thresholds are meaningful, we can consider any one threshold-dependent metric with those thresholds. This work completes a missing part in the landscape of link prediction, and provides a starting point toward a well-accepted criterion or standard to select proper evaluation metrics for link prediction.

Research on the Forward Simulation and Intelligent Detection of Defects in Highways Using Ground-Penetrating Radar

The increasing variety and frequency of subgrade defects in operational highways have led to a rise in road safety incidents. This study employed ground-penetrating radar (GPR) detection and forward simulation to analyze the characteristic patterns of common subgrade defects, such as looseness, voids, and cavities. Through the integration of instantaneous feature information from different defect patterns with complex signal techniques, the boundary judgment of structural layers and anomalies in GPR images of various subgrade defects was improved. An intelligent recognition platform was established, and a radar image dataset was created and trained to evaluate the recognition performance of the You Only Look Once (YOLO) v3 and Single-Shot Multi-Box Detector (SSD) algorithms. Evaluation metrics such as precision, recall, F1-score, average precision (AP), and mean average precision (mAP) were used to assess the detection efficiency and accuracy for subgrade defect images. The results showed that YOLO v3 achieved an average detection accuracy of 76.69%, while the SSD achieved 75.07%. This study demonstrates that the reliability of the intelligent recognition and classification of highway subgrade defects can be enhanced by using GPR for non-destructive testing.

Origami-Inspired Collapsible Structures for Small Rotorcraft Collision Resilience and Landing

Abstract This article reports on the design, fabrication, and testing of flexible, collapsible structures inspired by origami and kirigami that can be used to protect small rotorcraft from collision impacts and to serve as landing gear. Twenty structures created from variations of Pako Pako, Magic Ball, and Herringbone fold patterns were laser cut from cardstock and evaluated experimentally. Some of the Pako Pako structures were designed to have graduated mechanical stiffness. An empirical procedure based on acceleration measurements has been developed that enables quantitative evaluation of each structure based on the characterization parameters of peak acceleration, peak velocity, translational kinetic energy, impact duration, mass, and mechanical stiffness. Structures were mounted onto the front of a radio-controlled (RC) car and driven on a linear test track into a rigid wall. Accelerations measured during the collisions were numerically integrated to determine velocities over the impact and collapse/compression of the structures. Flight tests conducted with small RC quadcopters demonstrated that the collapsible structures could successfully be used to mitigate collisions, enough to enable the quadcopters to continue flying after a direct impact and to land indoors or on outdoor terrain with varying slopes and surfaces. Based on trade-off comparisons between the evaluative metrics of the cases studied, conical Pako Pako structures with uniform stiffness are shown to be the most effective for collision resilience and landing.

Identifying Landslide Hotspots Using Unsupervised Clustering: A Case Study

Landslides pose significant threats to life, property, and infrastructure. This study explores applying unsupervised learning techniques to identify and understand landslide-prone areas. We analyzed topographic data by employing K-Means, Hierarchical Clustering, Spectral Clustering, Mean Shift Clustering, and DBSCAN to uncover hidden patterns in landslide occurrence. Evaluation metrics, including the Silhouette Score, Davies-Bouldin Index, and Calinski-Harabasz Index, were used to assess the performance of these algorithms. Hierarchical Clustering achieved the highest Silhouette Score of 0.635, indicating excellent cluster separation. However, Mean Shift Clustering outperformed the other methods with a superior Davies-Bouldin Index of 0.603 and the highest Calinski-Harabasz Index of 4121.75, demonstrating the best overall clustering performance. DBSCAN also performed well, with a Silhouette Score of 0.610 and 12 noise points identified. These findings contribute to a deeper understanding of landslide spatial distribution and can inform the development of effective early warning systems and mitigation strategies.

Artificial intelligence-assisted magnetic resonance imaging technology in the differential diagnosis and prognosis prediction of endometrial cancer

It aimed to analyze the value of deep learning algorithm combined with magnetic resonance imaging (MRI) in the risk diagnosis and prognosis of endometrial cancer (EC). Based on the deep learning convolutional neural network (CNN) architecture residual network with 101 layers (ResNet-101), spatial attention and channel attention modules were introduced to optimize the model. A retrospective collection of MRI image data from 210 EC patients was used for model segmentation and reconstruction, with 140 cases as the test set and 70 cases as the validation set. The performance was compared with traditional ResNet-101 model, ResNet-101 model based on spatial attention mechanism (SA-ResNet-101), and ResNet-101 model based on channel attention mechanism (CA-ResNet-101), using accuracy (AC), precision (PR), recall (RE), and F1 score as evaluation metrics. Among the 70 cases in the validation set, there were 45 cases of low-risk EC and 25 cases of high-risk EC. Using ROC curve analysis, it was found that the area under the curve (AUC) for the diagnosis of high-risk EC of the proposed model in this article (0.918) was visibly larger as against traditional ResNet-101 (0.613), SA-ResNet-101 (0.760), and CA-ResNet-101 models (0.758). The AC, PR, RE, and F1 values of the proposed model for the diagnosis of EC risk were visibly higher (P < 0.05). In the validation set, postoperative recurrence occurred in 13 cases and did not occur in 57 cases. Using ROC curve analysis, it was found that the AUC for postoperative recurrence prediction of the patients by the proposed model (0.926) was visibly larger as against traditional ResNet-101 (0.620), SA-ResNet-101 (0.729), and CA-ResNet-101 models (0.767). The AC, PR, RE, and F1 values of the proposed model for postoperative recurrence prediction were visibly higher (P < 0.05). The proposed model in this article, assisted by MRI, presented superior performance in diagnosing high-risk EC patients, with higher sensitivity (Sen) and specificity (Spe), and also demonstrated excellent predictive AC in postoperative recurrence prediction.

Enhanced convolutional neural network architecture optimized by improved chameleon swarm algorithm for melanoma detection using dermatological images.

Early detection and treatment of skin cancer are important for patient recovery and survival. Dermoscopy images can help clinicians for timely identification of cancer, but manual diagnosis is time-consuming, costly, and prone to human error. To conduct this, an innovative deep learning-based approach has been proposed for automatic melanoma detection. The proposed method involves preprocessing dermoscopy images to remove artifacts, enhance contrast, and cancel noise, followed by feeding them into an optimized Convolutional Neural Network (CNN). The CNN is trained using an innovative metaheuristic called the Improved Chameleon Swarm Algorithm (CSA) to optimize its performance. The approach has been validated using the SIIM-ISIC Melanoma dataset and the results have been confirmed through rigorous evaluation metrics. Simulation results demonstrate the efficacy of the proposed method in accurately diagnosing melanoma from dermoscopy images by highlighting its potential as a valuable tool for clinicians in early cancer detection.

NOVEL DEVICE FOR ENHANCING TUBERCULOSIS DIAGNOSIS FOR FASTER, MORE ACCURATE SCREENING RESULTS

Tuberculosis (TB) continues to be a leading global cause of illness and death, highlighting an urgent need for rapid, precise diagnostic solutions. Conventional methods, such as chest X-rays and sputum smear microscopy, often lack adequate sensitivity and specificity, resulting in treatment delays and increased transmission. This research introduces a novel device leveraging deep learning to enhance TB diagnosis, utilizing a convolutional neural network (CNN) model designed to automatically detect TB-related abnormalities in chest X-ray images. The device, powered by a large annotated dataset, incorporates transfer learning to fine-tune pre-trained models, maximizing diagnostic performance. Key evaluation metrics—accuracy, precision, recall, and F1-score—demonstrate the model’s efficacy compared to traditional diagnostics. Preliminary findings show that the AI-driven approach significantly enhances diagnostic accuracy, reducing false negatives and facilitating faster screenings. These results suggest that embedding AI technology in clinical workflows can enable earlier TB detection, optimize healthcare resources, and improve patient outcomes. Future directions include testing the device across diverse clinical environments and exploring integration with mobile health technologies to broaden access to rapid TB diagnostics.

A Comprehensive Review of Remaining Useful Life Estimation Approaches for Rotating Machinery

This review paper comprehensively analyzes the prognosis of rotating machines (RMs), focusing on mechanical-flaw and remaining-useful-life (RUL) estimation in industrial and renewable energy applications. It introduces common mechanical faults in rotating machinery, their causes, and their potential impacts on RM performance and longevity, particularly in wind, wave, and tidal energy systems, where reliability is crucial. The study outlines the primary procedures for RUL estimation, including data acquisition, health indicator (HI) construction, failure threshold (FT) determination, RUL estimation approaches, and evaluation metrics, through a detailed review of published work from the past six years. A detailed investigation of HI design using mechanical-signal-based, model-based, and artificial intelligence (AI)-based techniques is presented, emphasizing their relevance to condition monitoring and fault detection in offshore and hybrid renewable energy systems. The paper thoroughly explores the use of physics-based, data-driven, and hybrid models for prognosis. Additionally, the review delves into the application of advanced methods such as transfer learning and physics-informed neural networks for RUL estimation. The advantages and disadvantages of each method are discussed in detail, providing a foundation for optimizing condition-monitoring strategies. Finally, the paper identifies open challenges in prognostics of RMs and concludes with critical suggestions for future research to enhance the reliability of these technologies.

Multi-classification prediction of PM2.5 concentration based on improved adaptive boosting rotation forest

Developing efficient, stable, and user-friendly methods and technologies for predicting air quality has contributed to environmental research and management. Most traditional machine learning (ML) models often struggle to efficiently process extensive air quality data and grapple with the challenge of imbalanced data distributions. To this end, we introduced a novel multi-strategy collaborative approach that incorporates weighted feature selection, an adaptive enhanced rotation forest algorithm, and Bayesian Optimization for parameter tuning. Moreover, to improve the transparency in black box ML models, the novel Shapley Additive Explanations (SHAP) method was applied to interpret the outputs and analyze the importance of individual variables. Through experiments on real air quality datasets, we have verified the accuracy of our proposed method in classifying air quality levels. Notably, our model demonstrated an average test set accuracy improvement of approximately 10 % in cities like Beijing, Tianjin, Shijiazhuang, and Baoding after hyperparameter optimization using Bayesian methods. Furthermore, compared to alternative algorithms, our model showed an improvement of 1–2 % in evaluation metrics. By introducing novel methodologies and tools, our research contributes significantly to the advancement of air quality classification technology and holds profound implications for informing environmental policy decisions.

Vision-Based Drone Detection in Complex Environments: A Survey

The frequent illegal use of drones poses a serious threat to public security and property. Counter-drones are crucial tools. The prerequisite for an effective counter-drone is to detect drones accurately. With the rapid advancements in computer vision, vision-based drone detection methods have emerged as a hot topic of research. However, current reviews of vision-based drone detection are less focused on algorithmic summarization and analysis. For this reason, this survey aims to comprehensively review the latest methods for vision-based drone detection in complex environments, with the goal of providing a more thorough reference and guidance for related research. We first explore the imaging and drone characteristics in complex environments and summarize the main challenges of visual UAV detection. Then, we summarize the existing solutions for the main challenges. Finally, we systematically organize and introduce the commonly used datasets and evaluation metrics and conduct experiment comparisons based on the representative methods. We not only reveal the current development status of visual UAV detection but also analyze the deficiencies in current research. On this basis, we further look forward to future research directions and possible breakthroughs, with a view to providing useful insights for further research and development in related fields.