Reliable Performance Assessment Research Articles

Random Cross-Validation Produces Biased Assessment of Machine Learning Performance in Regional Landslide Susceptibility Prediction

Machine learning (ML) models are extensively used in spatial predictive modeling, including landslide susceptibility prediction. The performance statistics of these models are vital for assessing their reliability, which is typically obtained using the random cross-validation (R-CV) method. However, R-CV has a major drawback, i.e., it ignores the spatial autocorrelation (SAC) inherent in spatial datasets when partitioning the training and testing sets. We assessed the impact of SAC at three crucial phases of ML modeling: hyperparameter tuning, performance evaluation, and learning curve analysis. As an alternative to R-CV, we used spatial cross-validation (S-CV). This method considers SAC when partitioning the training and testing subsets. This experiment was conducted on regional landslide susceptibility prediction using different ML models: logistic regression (LR), k-nearest neighbor (KNN), linear discriminant analysis (LDA), artificial neural networks (ANN), support vector machine (SVM), random forest (RF), and C5.0. The experimental results showed that R-CV often produces optimistic performance estimates, e.g., 6–18% higher than those obtained using the S-CV. R-CV also occasionally fails to reveal the true importance of the hyperparameters of models such as SVM and ANN. Additionally, R-CV falsely portrays a considerable improvement in model performance as the number of variables increases. However, this was not the case when the models were evaluated using S-CV. The impact of SAC was more noticeable in complex models such as SVM, RF, and C5.0 (except for ANN) than in simple models such as LDA and LR (except for KNN). Overall, we recommend S-CV over R-CV for a reliable assessment of ML model performance in large-scale LSM.

Adaptive Performance Evaluation of Container Terminals Through Normalization and Parameter Analysis

Background: Container terminals are a pivotal part of global logistics networks, influencing supply chain reliability and port competitiveness. Traditional performance evaluation methods, such as KPI-based assessments or multi-criteria analyses, often fail in dynamic operational conditions with inherent uncertainty and variability. Methods: This study proposes a normalization-based framework to evaluate container terminal performance by standardizing operational parameters, including availability, non-productive operations, operation time, energy consumption, and throughput. The methodology involves parameter definition, normalization, weight assignment, index calculation, and performance classification. Results: The findings demonstrate that normalization ensures a transparent and adaptable evaluation framework. Sample calculations show how parameter weights influence terminal assessments across varied scenarios, confirming the robustness of the proposed method in capturing dynamic operational changes. Conclusions: Normalization offers a practical tool for enhancing container terminal efficiency and competitiveness. It enables decision-makers to adapt strategies to changing priorities, such as throughput maximization or energy efficiency, ensuring comprehensive and reliable performance assessments.

Which is the Superior Thoracolumbar Injury Classification Tool? TLICS Versus AOSpine 2013: A Systematic Review.

Systematic Literature Review. To address whether TLICS or AOSpine is best used in clinical practice through assessment of interobserver and intraobserver reliability, agreement, and imaging modality performance. This systematic literature review was reported in accordance with PRISMA 2020 guidelines. Articles were included based on meeting eligibility criteria: studies evaluating TLICS, AOSpine, and/or TL AOSIS through reliability, agreement, or imaging modality performance with adult patients (≥18) suffering from traumatic thoracolumbar fractures. Articles were acquired in April 2023 from Medline, CINAHL, and Scopus. Risk of bias was assessed through a modified COSMIN checklist. Tabulated results were separated by classification tool (TLICS or AOSpine/TL AOSIS) and reliability, agreement, or imaging modality results. Twenty-one studies were included in the final review. Interobserver and intraobserver AOSpine morphology reliability was on average superior to TLICS. Increased familiarity with the tool positively influenced both AOSpine and TLICS performance. For surgical treatment recommendation, AOSpine differentiated between stable and unstable burst fractures and guided clinician's more accurately than TLICS. Regarding conservative treatment, both TLICS and AOSpine reported similar clinical accuracy. TLICS performed significantly better when MRI was incorporated compared to CT alone. CT was sufficient as an imaging modality for AOSpine/TL AOSIS performance. AOSpine outperformed TLICS in surgical reliability, agreement and did not require additional MRI imaging to improve accuracy. Limitations of evidence include low quality of available studies and significant heterogeneity in patient and observer number. Future prospective multicentre research is recommended. This study was not funded and not registered on PROSPERO.

UNC Charlotte Autonomous Shuttle Pilot Study: An Assessment of Operational Performance, Reliability, and Challenges

This paper presents the findings from an autonomous shuttle pilot program conducted at the University of North Carolina at Charlotte between June and December 2023 as part of the North Carolina Department of Transportation’s Connected Autonomous Shuttle Supporting Innovation (CASSI) initiative. The shuttle completed 825 trips, transporting 565 passengers along a 2.2-mile mixed-traffic campus route. The study evaluates the shuttle’s operational performance, reliability, and challenges using data from onboard sensors, system logs, and operator reports. Key analyses are divided into four areas: service reliability, which assesses autonomy disengagements caused by signal loss, technical issues, and environmental factors; service robustness, focusing on the shuttle’s ability to maintain operations under adverse conditions; performance metrics, including average speed, autonomy percentage, and battery usage; and service usage, which examines the number of trips and passengers to gauge efficiency. Signal loss and battery-related issues were the primary causes of service interruptions, while environmental factors like weather and vegetation also affected shuttle performance. Recommendations include enhancing vehicle-to-infrastructure communication and optimizing battery management.

Urban biodiversity performance determining model (UrBioPDeM): The case of Isparta, Türkiye

Identifying, protecting, and developing biodiversity in urban environments contributes to ecosystem integrity. Although there are studies addressing biodiversity in natural areas in Türkiye, there has been no study specifically measuring the biodiversity of city centers. Adapting existing methods for measuring city biodiversity to Turkish cities is challenging due to their unique local dynamics. Therefore, there is a need to identify indicators that can be used to monitor and evaluate biodiversity performance for the protection and sustainability of urban areas in Türkiye and to develop a feasible, reliable, and measurable biodiversity performance assessment approach. The developed UrBioPDeM has 4 main factors and 32 sub-factors. Indicators for the assessment and management of urban biodiversity performance in Isparta, Türkiye, were identified and analyzed, and it was found that Isparta’s urban biodiversity has low potential. Therefore, suggestions were made to increase the urban biodiversity of Isparta. The developed approach is an appropriate and effective biodiversity performance model that can be applied in Turkish cities. The factors and sub-factors used in this method can be adjusted and revised according to the potential biodiversity characteristics of a city.

Junction Temperature Prediction and Lifetime Assessment in a PV Inverter Using a 10-year Mission profile

The expansion of great-scale photovoltaic (PV) power plants indicates the need for an accurate lifetime assessment of inverters to maintain energy supply availability. In this context, the study contributes in two ways. First, we use machine-learning (ML) models for junction temperature prediction. Second, we perform reliability assessments using a 10-year mission profile in three Brazilian cities. The thermal loadings are obtained through a look-up table approach. Although the ML models exhibit different performances in regression, other factors must be considered, such as easy-to-apply, interpretability, and generalization capability. The reliability assessment is typically based on an annual mission profile, assuming damage repeats until failure. However, only the historical series can confirm whether this choice was acceptable, pessimistic, or optimistic. For instance, in Campos do Jordão-SP, if the chosen mission profile is 2014, the expected failure of 10\% of inverter samples occurs three years earlier than suggested by the historical series. Regardless of the methodology used to estimate thermal loading or accumulated damage, the mission profile significantly influences photovoltaic inverter reliability, indicating that if more data is available, the chosen mission profile should align with the historical series.

Operational reliability and non-deterministic resilience estimation of active distribution network incorporating effect of real-time dynamic hosting capacity

Active distribution networks are increasingly recognized essential for achieving sustainable development goals. Traditionally, hosting capacity was considered as a static measure for planning distributed energy resources integration. This work introduces the concept of dynamic hosting capacity, which recurrently re-evaluates hosting capacity in response to erratic modern grid conditions. The introduction of dynamic hosting capacity facilitated testing variations of power injection from minimum to 100 %, sustaining power system governing parameter limits. This embarked the need of operational reliability assessment and enhancing situational awareness for optimum power injection and balance. To achieve operational reliability analysis based on dynamic hosting capacity, hybrid probability distribution function-based Monte Carlo simulation is proposed. This resulted in 85–90 %. improvisation of solar photovoltaic generation and load alignment, as this methodology provides comprehensive and accurate assessment of system performance under diverse uncertainties. The framework's validation includes projection of time-varying operational reliability indices, over time independent reliability indices i.e., dynamic loss of load probability, dynamic loss of load expectation, dynamic loss of load duration, dynamic loss of load frequency, dynamic grid margin, and dynamic grid dependency. This resulted in 30 % improvement in assessment of grid margin, facilitating reliable uncertainty handling competence. Additionally, expectation maximization algorithm is proposed to evaluate non-deterministic resilience due to ambiguities associated with solar photovoltaic distributed energy resources. The non-deterministic resilience assessment testified 80 % bounce-back rate, demonstrating better adaptability and robustness. The entire analysis is conducted in MATLAB, validated using Typhoon Hardware-in-Loop real-time platform, and compared with existing literatures to demonstrate its effectiveness.

Multi-Fidelity Aeromechanical Design Framework for High Flow Speed Multistage Axial Compressors

Abstract The development of novel engine architectures is vital in achieving the aviation sector’s net-zero carbon emission target by 2050. With today’s digital decade providing support for an accelerated technology maturation, the challenge for turbomachinery design remains to significantly push the limits of current performance within an ambitious development lead time. In this context, it is essential to adopt a design framework where the predictive models or simulations employed target a sufficiently reliable performance assessment. These models must be tailored to the dynamics of evolving industrial design processes, and therefore continuously balance required design flexibility, robust evaluation, appropriate fidelity (i.e. the level of detail and accuracy they provide), and resulting evaluation time. This paper discusses a framework for designing axial compressors and its application to the aeromechanical optimisation of a high-speed compressor rotor. The design environment integrates geometry parametrisation, modular evaluation with different levels of fidelity for the aerodynamic and structural models, and surrogate-based optimisation (SBO) capabilities. It is shown how the combination of a modular sequencing of the different models and the acceleration enabled by high-performance computing (HPC) and machine learning allows for a more advanced preliminary design. A significant gain in isentropic efficiency is attained while satisfying all structural constraints. At the same time, it is demonstrated that the framework is compatible with the characteristics of the preliminary design phase: both in its ability to adapt to cycle and design changes as well as regarding the turnaround time of the optimisation itself.

TPE-Optimized DNN with Attention Mechanism for Prediction of Tower Crane Payload Moving Conditions

Accurately predicting the payload movement and ensuring efficient control during dynamic tower crane operations are crucial for crane safety, including the ability to predict payload mass within a safe or normal range. This research utilizes deep learning to accurately predict the normal and abnormal payload movement of tower cranes. A scaled-down tower crane prototype with a systematic data acquisition system is built to perform experiments and data collection. The data related to 12 test case scenarios are gathered, and each test case represents a specific combination of hoisting and slewing motion and payload mass to counterweight ratio, defining tower crane operational variations. This comprehensive data is investigated using a novel attention-based deep neural network with Tree-Structured Parzen Estimator optimization (TPE-AttDNN). The proposed TPE-AttDNN achieved a prediction accuracy of 0.95 with a false positive rate of 0.08. These results clearly demonstrate the effectiveness of the proposed model in accurately predicting the tower crane payload moving condition. To ensure a more reliable performance assessment of the proposed AttDNN, we carried out ablation experiments that highlighted the significance of the model’s individual components.

Operational reliability assessment of complex mechanical systems with multiple failure modes: An adaptive decomposition-synchronous-coordination approach

To effectively perform Complex Mechanical Systems Operational Reliability Assessment (CSORA) with multiple failures, the Adaptive Decomposition-Synchronous-Coordination Approach (ADSCA) is proposed by integrating Decomposition-Synchronous-Coordination strategy, adaptive modeling technology, surrogate model and multi-objective reliability assessment theory. In which, the main problem is decomposed into related sub-problems using a Decomposition-Synchronization-Coordination strategy, with sub-models coordinated to achieve synchronous mapping between multiple variables, thereby enhancing efficiency in large multi-subsystem problems. The adaptive modeling technique is adopted to construct a series of interrelated sub models based on the surrogate model as the basis function, improving performance. A multi-objective reliability assessment theory is adopted to achieve CSORA under multiple fault modes, where multiple weak links are integrated to comprehensively evaluate the reliability of complex systems. The significance of the ADSCA lies in its ability to manage complexity, enhance scalability, and improve the accuracy and efficiency of reliability assessments in complex mechanical systems. To demonstrate the effectiveness of the proposed approach, two illustrative examples are used. This includes approximation and probability analysis of nonlinear functions that exhibit multiple responses, as well as reliability analysis of temperature in civil aircraft braking system. The study can provide theoretical reference for the reliability evaluation of multi-failure operation in complex mechanical systems.

A Semi-Markov Stochastic Model for Operational Reliability Assessment of Hybrid AC and LCC-VSC-Based DC System With Remote Wind Farms

A Semi-Markov Stochastic Model for Operational Reliability Assessment of Hybrid AC and LCC-VSC-Based DC System With Remote Wind Farms

Training framework for high-stakes OSCE: Experience from volunteer standardized patients' bank.

Training of standardized patients (SPs) for national high-stakes OSCE helps to ensure a reliable assessment of student performance in various clinical scenarios. However, workflow protocols to train SPs vary. Medical schools adopt specific measures to ensure standardization. We present a development workflow of the SPs' training framework for high-stakes OSCE using a volunteer SPs' bank. Our approach was guided by the social learning theory. Three educators and 17/20 (85%) members of our volunteer SPs' bank worked in a collaborative partnership on the construction of pedagogical content of the training framework comprising three 2-hour sessions. Since SPs have to demonstrate acquired behaviors, intended learning outcomes used the words "apply", "perform" and "participate." A principal part of the evaluation was the achievement of intended learning outcomes by the SPs during 3 formative OSCEs. Seventeen SPs, 356 fourth year medical students and 60 examiners participated. Quantitative and qualitative data were collected by post-session questionnaires and analyzed using descriptive statistics and thematic analysis. Twelve examiners evaluated a mean of 29.7+/-0.14 SD patient-student encounters. In total, 15/16 SPs (93.8%) considered the contact with students as easy and 4 SPs (31%) reported the experience as stressful. Two themes emerged from the free-text comments: "Gaining experience as SP" and "Concerns for evaluated students." The proposed SPs' training framework for high-stakes OSCE may be useful for other medical disciplines and health professions initiating SP-based assessment programs. The strategy of development and evaluation are outlined to guide a successful application of the curriculum standards.

Use of optical see-through head-mounted devices in dentistry - a scoping review.

The aim of this scoping review was to identify the scientific evidence related to the utilization of Optical See- Through Head-Mounted Display (OST-HMD) in dentistry, and to determine future research needs. The research question was formulated using the "Population" (P), "Concept" (Cpt), and "Context" (Cxt) framework for scoping reviews. Existing literature was designated as P, OST-HMD as Cpt, and Dentistry as Cxt. An electronic search was conducted in PubMed, Embase, Web of Science, and CENTRAL. Two authors independently screened titles and abstracts and performed the full-text analysis. The search identified 286 titles after removing duplicates. Nine studies, involving 138 participants and 1760 performed tests were included in this scoping review. Seven of the articles were preclinical studies, one was a survey, and one was a clinical trial. The included manuscripts covered various dental fields: three studies in orthodontics, two in oral surgery, two in conservative dentistry, one in general dentistry, and the remaining one in prosthodontics. Five articles focused on educational purposes. Two brands of OST-HMD were used: in eight studies HoloLens Microsoft was used, while Google Glass was utilized in one article. The overall number of included studies was low; therefore, the available data from this review cannot yet support an evidence-based recommendation for the clinical use of OST-HMDs. However, the existing preclinical data indicate a significant capacity for clinical and educational implementation. Further adoption of these devices will facilitate more reliable and objective quality and performance assessments, as well as more direct comparisons with conventional workflows. More clinical studies must be conducted to substantiate the potential benefits and reliability for patients and clinicians.

Proficiency testing for event-specific quantification of genetically modified maize MON87427: Performance assessment based on the metrologically traceable reference values as assigned values

The Asia Pacific Metrology Program and the Accreditation Cooperation joint Proficiency Testing (PT) program for the quantification of genetically modified maize MON87427 was organized by the National Institute of Metrology, China, to enhance the measurement accuracy and metrological traceability in the region. Certified reference materials were employed as test samples; metrologically traceable certified reference values served as PT reference values (PTRVs) for evaluating the participants results. The consensus values obtained from the participants were higher than the assigned values, potentially due to the systematic effects of DNA extraction process. The participants' relatively poor overall performance by the ζ-score compared with z-score demonstrates their need to thoroughly investigate quantification bias to elevate the measurement capability of genetically modified (GM) content and deepen their understanding of uncertainty estimation. This program confirmed the importance of using metrologically traceable reference values instead of consensus values as PTRV for reliable performance assessment.

Machine Learning Library in C++

This project aims to develop a comprehensive and versatile machine learning library in C++ tailored to address the diverse needs of developers and researchers in the field. The library encompasses a robust set of core machine learning algorithms, encompassing supervised, unsupervised, and reinforcement learning techniques. Additionally, it incorporates essential data preprocessing tools to streamline data manipulation and feature engineering tasks, along with model evaluation capabilities crucial for assessing algorithm performance. The library's primary focus is on providing a rich suite of machine learning algorithms, This empowers users to effectively prepare data for training machine learning models. Additionally, the library provides tools for data splitting, and model evaluation to ensure reliable and robust model performance assessment.

Operational Reliability Analysis of Turbine Blisk Using an Enhanced Moving Neural Network Framework

As one of the key components of an aeroengine, turbine blisk endures complex coupling loads under a harsh operational environment so that the reliability of turbine blisk directly influences the safe operation of aeroengine. It is urgent to precisely perform the reliability estimation of a complex blisk structure. To address this issue, an enhanced Moving Neural Network Framework (MNNF) is proposed by integrating compact support region theory, improve sooty tern optimization algorithm (ISTOA), and Bayesian regularization strategy into artificial neural network. The compact support region theory is applied to select the efficient samples for modeling from the training samples set, the ISTOA is to determine the optimal compact support region, and Bayesian regularization thought is utilized to improve the generalization ability of neural network model. The operational reliability assessment of aeroengine blisk is performed with the consideration of transient loads to verify the proposed MNNF method. It is shown that the reliability degree of turbine blisk stain is 0.9984 when the allowable value is 5.2862 × 10−3 m. In line with the comparison of methods, the developed MNNF approach has 0.99738 in root means square error, 3.1634 × 10−4 m in goodness of fit, 0.423 s in modeling time, 99.99% in simulation precision, and 0.496 s in simulation time under 10,000 simulations, which are superior to all other methods (i.e., 99.96%, 99.91%, 99.93%, 99.97%, and 99.97% in simulation precision and 16.27%, 4.82%, 30.07%, 39.87%, and 23.59% in simulation efficiency, for the response surface method (RSM), Kriging, support vector machine (SVM), back propagation-artificial neural network (BP-NN), and BP-NN based on particle swarm optimization (BP-PSO) methods, respectively). It is demonstrated that the MNNF method holds excellent modeling and simulation performances. The efforts of this study provide promising tools and insights into the reliability design of complex structures, and enrich and develop reliability theory.

An Analytical Reliability Performance Assessment of Cell-Free Massive MIMO Systems for 5G and Beyond Cellular Networks

In today‟s realm, the 5G cellular networks offer a new era of fully digital environment by connecting people anywhere anytime. This era requires inspiring demands to the networks in terms of spectral and energy efficiencies, low-latency and ultra-reliable communications. A major development to deal with these unparalleled demands is the introduction of Cell- Free Massive MIMO (CF mMIMO) systems to the cellular network. In recent years, mathematical foundations have been laid for a CF mMIMO communication system to evaluate the spectral and energy efficiencies given the number of access points. Various existing literatures have adopted mathematical models to formulate closed-form expressions for these efficiencies using uncorrelated Rayleigh fading for simple linear processing detectors. In this paper, we try to address a compelling and interesting question which has not received much attention lately. The question is how to evaluate the reliability of a connected user terminal in both a CF mMIMO and conventional mMIMO system? We answer this question by first analyzing the signal models for both systems assuming exact channel estimates and consider an uncorrelated Rayleigh fading channel model. The exact Bit Error Rate, as a reliability assessment, in closed-form are then derived based on strong mathematical foundations, which are simple to numerically evaluate in Matlab R2022, for MPSK and MQAM modulation schemes. The mathematical models presented in this work can be used in 5G and Beyond 5G networks to evaluate the distributions of error rates for a user given the number of Access points or Base Station antennas and modulation scheme. Machine Learning algorithms can be applied to the analytical expressions with a view to predict the error performance of a user terminal given the fading characteristic of the propagation environmen

A Review of Cross-Scale Theoretical Contact Models for Bolted Joints Interfaces

Bolted joints structures are critical fastening components widely used in mechanical equipment. Under long-term loading conditions, the bolted joints interface generates strong nonlinearities within the system. The nonlinear stiffness inside the bolt leads to changes in the stiffness of the whole system. This affects the dynamic characteristics of the whole system. It brings challenges and difficulties to the performance prediction and reliability assessment of the equipment. A cross-scale theoretical model study based on the microscopic contact mechanism can provide a more comprehensive understanding and cognition of the degradation behavior of bolted joints interfaces. The current development status and deformation process of asperity models are summarized. The research progress of statistical summation model and contact fractal model based on microscopic contact mechanism is analyzed. The experimental methods for parameter identification of connection interfaces are reviewed. The study of numerical modelling of bolted joints structures from the surface contact mechanism is briefly described. Future research directions for cross-scale modelling of bolted joints structures are outlined.

Electric current stressing enhanced damping properties in Sn5Sb solder

PurposeThe purpose of this paper is to investigate the effects of electric current stressing on damping properties of Sn5Sb solder.Design/methodology/approachUniformly shaped Sn5Sb solders were prepared as samples. The length, width and thickness of the samples were 60.0, 5.0 and 0.5 mm, respectively. The damping properties of the samples were tested by dynamic mechanical analyzer with a cooling system to control the test temperature in the range of −100 to 100°C. Simultaneously, electric current was imposed to the tested samples using a direct current supply. After tests, the samples were characterized using scanning electron microscope, electron backscatter diffraction and transmission electron microscope, which was aimed to figure out the damping mechanism in terms of electric current stressing induced microstructure evolution.FindingsIt is confirmed experimentally that the increase in damping properties is due to Joule heating and athermal effects of current stressing, in which Joule heating should make a higher contribution. G–L theory can be used to explain the damping properties of strain amplitude under current stressing by quantitative description of geometrically necessary dislocation density. While the critical strain amplitude and high temperature activation energy decrease with increasing electric current.Originality/valueThese results provide a new method for vibration reliability evaluation of high-temperature lead-free solders in serving electronics. Notably, this method should be also inspiring for the mechanical performance evaluation and reliability assessment of conductive materials and structures serving under electric current stressing.

Machine Learning-Based Framework for Failure Forecast and Shear Strength Estimation of Non-Conforming RC Shear Walls

ABSTRACT Earthquake reconnaissance after the recent Kahramanmaras, Turkey, earthquake sequence in February 6, 2023 has shown that the majority of the reinforced concrete (RC) structural elements do not comply with the Turkish Building Earthquake Code (TBEC 2018) requirements. Accurate estimation of the actual strength of existing non-compliant RC structural members is critical to accomplish reliable earthquake performance assessments. This study aims to propose a machine learning-based framework to establish a realistic failure forecast as well as an accurate shear strength estimation depending on the expected failure modes of such walls. To achieve this, shear wall design properties are assigned as inputs whereas failure mode and shear strength were designated as outputs for the corresponding classification and regression problems, respectively. Widely used machine learning methods, namely: Ridge Linear Regression, Logistic Regression, Multi-layer Perceptron, Support Vector Machine, Random Forest, XGBoost, LightGBM, and CatBoost are employed using a database consisting of 432 non-conforming shear walls. The data is randomly split to train and test sets (80% and 20%, respectively) to develop the predictive models and to evaluate model performances, respectively. Performance metrics (e.g. R 2 , RMSE) are evaluated over a hundred random splits to ensure robustness. The proposed CatBoost-based models achieve overall accuracies above 90% for both failure prediction and shear strength estimation. The proposed framework is believed to be a promising tool for earthquake engineers for practical and accurate failure mode forecast and shear strength estimation of non-compliant RC shear walls.