Reliable Performance Evaluation Research Articles

Operational reliability evaluation integrating prediction models for enhanced situational awareness in wind-integrated system

The extensive implementation of wind power generation (WPG) challenges power systems' flexibility and reliability during operations. Fluctuating wind speeds lead to variability in generation within wind energy systems, directly impacting operational reliability. Therefore, evaluating wind energy systems' situational awareness (SA) is imperative to forecast two-day-ahead operational reliability, facilitating effective power system planning. The foundation of SA for wind energy systems lies in monitoring the wind profile. This involves gaining insights into the system's state through perceiving wind speed and WPG, comprehending the dynamics of its state, and subsequently predicting operational reliability. To achieve this, a Simulink model of the wind system has been developed and integrated with dSPACE hardware through a real-time interface for real-time validation and implementation of a cloud-based IoT platform. This IoT platform aids in acquiring real-time data on wind profiles, enhancing the assessment of SA. The representation of the wind energy system utilizes a multi-state wind speed model, incorporating the inherent randomness of wind energy generation. A time-series-based non-linear autoregressive with exogenous input (NARX) artificial neural network (ANN) model is employed to predict wind speed and WPG for SA-based operational reliability. This model proves instrumental in forecasting the dynamic nature of wind speed and WPG. The proposed approach for operational planning is validated by implementing the wind energy system in real-time on the dSPACE platform and integrating it with IoT. This approach considers the uncertainties associated with wind generation, providing a comprehensive framework for assessing SA and optimizing power system operations.

Towards a framework for reliable performance evaluation in defect prediction

Enhancing software reliability, dependability, and security requires effective identification and mitigation of defects during early development stages. Software defect prediction (SDP) models have emerged as valuable tools for this purpose. However, there is currently a lack of consensus in evaluating the predictive performance of newly proposed models, which hinders accurate measurement of progress and can lead to misleading conclusions. To tackle this challenge, we present MATTER (a fraMework towArd a consisTenT pErformance compaRison), which aims to provide reliable and consistent performance comparisons for SDP models. MATTER incorporates three key considerations. First, it establishes a global reference point, ONE (glObal baseliNe modEl), which possesses the 3S properties (Simplicity in implementation, Strong predictive ability, and Stable prediction performance), to serve as the baseline for evaluating other models. Second, it proposes using the SQA-effort-aligned threshold setting to ensure fair performance comparisons. Third, it advocates for consistent performance evaluation by adopting a set of core performance indicators that reflect the practical value of prediction models in achieving tangible progress. Through the application of MATTER to the same benchmark data sets, researchers and practitioners can obtain more accurate and meaningful insights into the performance of defect prediction models, thereby facilitating informed decision-making and improving software quality. When evaluating representative SDP models from recent years using MATTER, we surprisingly observed that: none of these models demonstrated a notable enhancement in prediction performance compared to the simple baseline model ONE. In future studies, we strongly recommend the adoption of MATTER to assess the actual usefulness of newly proposed models, promoting reliable scientific progress in defect prediction.

Performance reliability evaluation of high‐pressure internal gear pump

AbstractWith the development of the high‐end equipment technology, the performance requirements of the internal gear pump (IGP) under high pressure are also increasing. However, the increase of working pressure will lead to the instability of gear pump performance in terms of volumetric efficiency, noise, reliability and so on, it is necessary to reasonably evaluate the reliability level of high‐pressure IGP. The reliability analysis of the high‐pressure IGP is carried out from the aspects of flow, noise, and gear strength in this paper. First, the output flow rate and far‐field flow‐induced noise of the high‐pressure IGP were obtained through fluid numerical simulation, and experimental verification was conducted. Then, based on the time‐varying meshing stiffness, backlash function and static transmission error of the gear pair, a nonlinear dynamic model of the internal meshing gear pair was established. The time‐varying meshing force was obtained through the dynamic model of the gear pair, and then the tooth contact stress and tooth root bending stress were obtained. Finally, considering the uncertain factors affecting the performance of the high‐pressure IGP, Latin hypercube sampling (LHS) combined with dendrite network (DD) was used for random response modeling. The performance reliability of the high‐pressure IGP, including output flow rate, far‐field flow‐induced noise, and the strength of gear pair, were estimated based on the fourth moment‐based saddlepoint approximation (FMSA). The reliability analysis results can provide a theoretical basis for the structural optimization design of the high‐pressure IGP.

A Model and Data Hybrid-Driven Method for Operational Reliability Evaluation of Power Systems Considering Endogenous Uncertainty

A Model and Data Hybrid-Driven Method for Operational Reliability Evaluation of Power Systems Considering Endogenous Uncertainty

An integrated development environment based situational awareness for operational reliability evaluation in wind energy systems incorporating uncertainties

Wind power fluctuations impact power system flexibility and reliability, emphasizing situational awareness (SA) in wind power integration. Monitoring wind speed and power generation forms the core of SA, enabling insight into wind system conditions. An Integrated Development Environment (IDE) becomes pivotal in assessing the SA of wind-integrated systems, which is vital for forecasting operational reliability in power system planning. The IDE facilitates Simulink-based wind model development and is linked to dSPACE hardware through a real-time interface (RTI) for validation. This IDE-driven integration utilizes cloud-based Internet of Things (IoT) capabilities, acquiring wind speed and power generation data for precise SA evaluations. A multi-state wind speed model is employed to incorporate energy generation fluctuations. To improve this, the artificial neural network (ANN) model, i.e., a time series-based non-linear autoregressive with exogenous input (NARX) ANN, is utilized to predict future wind speed and power generation, strengthening SA-based operational reliability. The study investigates the effectiveness of a suggested operational planning strategy under wind generation uncertainties. This assessment is carried out within a real-time deployment of wind energy systems seamlessly integrated with IoT on the dSPACE platform. The primary objective is to evaluate the system's ability to maintain SA effectively.

Long-term performance of basalt fiber reinforced composite under coupled effect of alkalinity and freeze-thaw cycles

Basalt fiber, a kind of natural fiber produced from the melt of basalt stones, has been regarded as a promising material in the manufacture of fiber reinforced composites considering the balance between mechanical properties and environmental costs compared with other non-renewable/petroleum materials. Moreover, the basalt fiber reinforced polmyers (BFRP) are capable to fabricate the energy harvesting devices in marine engineering when piezoelectric nanocomposites and piezoelectric polymer matrix are adopted to collect offshore renewable energy. However, the long-term performance of BFRP in concrete alkaline and seawater environment has not been comprehensively understood yet, and the lack of knowledge on the performance of BFRP reinforcement under coupled alkalinity and freeze-thaw cycles prevents further application of composite structure in the frigid areas. To address above questions, experimental investigations were carried out with respect to the mechanical properties and deterioration mechanism of BFRP under alkaline environment affected by seawater and temperature as well as under coupled alkaline and freeze-thaw cycles. The results showed that the strength retention of BFRP bars in seawater-sea sand concrete and immersed in seawater is higher than that in distilled water. As the erosion time increases and the temperature rises, severe resin damage occurs inside BFRP bars, and the weakening of connections between fibers and matrix becomes critical in the reduction of tensile performance. Moreover, it is found that the long-term performance of BFRP under coupled alkaline and freeze-thaw cycles is affected by deterioration of resin at the surface, while the majority of internal fibers are free from attack by corrosive agents. These findings will provide a scientific basis for the reliable performance evaluation and further development of high performance and durable multifunctional BFRP materials in marine engineering.

Performance evaluation approach for accelerating rate calorimeters by means of Joule heat

Currently, the commonly used performance evaluation approach for accelerating rate calorimeters (ARCs) is based on di‑tert‑butyl peroxide (DTBP). However, DTBP is not a certified reference material that contains untraceable and inaccurate thermodynamic and kinetic parameters. To address this issue, an evaluation approach using Joule heat was proposed. The key point of this approach is the use of Joule heat to simulate the exothermic process of a chemical reaction. First, an evaluation apparatus with reaction rate feedback control was developed. Subsequently, exothermic reaction experiments using the DTBP/toluene solution were simulated. The experimental results indicated that the thermodynamic and kinetic parameters obtained using this approach align with the preset values and exhibit good repeatability. This approach is traceable and can be traced back to the International System of Units, which facilitates reliable performance evaluation of commercial ARCs.

A Region-Based Approach for the Operational Reliability Evaluation of Power Systems With Renewable Energy Integration

Power system operational reliability evaluation is computationally time-consuming due to the need to analyze numerous system states resulting from uncertain renewable energies and loads as well as random outages of components. This paper presents a load feasible region (LFR) approach to accelerate the process of operational reliability evaluation. The LFR of each state, which is defined as a polyhedron in the load space, can be built in advance offline. The inner points of the LFR correspond to the loads that can be satisfied under the system operational constraints. Then, the preestablished LFR can be corrected with little computing effort considering the renewable energy fluctuation during the operation phase. The optimal load shedding under each state can be obtained directly by calculating the minimum Manhattan distance to the LFR's boundary. Multiple application examples are presented to demonstrate the efficiency and scalability of the proposed region-based approach.

Operational Reliability of Integrated Energy Systems Considering Gas Flow Dynamics and Demand-Side Flexibilities

The interdependency among the electricity, gas, heat, and cooling energy systems is ever-increasing. The flexible energy utilization patterns on the demand side and gas flow dynamics in the transmission system bring both opportunities and challenges to the reliable operation of integrated energy systems (IES). For example, if the electricity supply is interrupted, the gas system can ramp up the gas supply to the gas-fired units using linepacks. By this means, the reliability of the electricity system at this moment can be improved, while the gas system's capability of withstanding future risks may be undermined. Therefore, the operational reliability between different energy systems and time periods should be carefully balanced. This paper proposes an operational reliability evaluation framework for the IES considering flexibilities from both the demand side and transmission system. Firstly, the flexibilities of end-users and linepacks are explored based on the Energy Hub and gas flow dynamics models. Then, the reliability models of IES components are developed using the discretized-time Markov process to characterize the temporal state evolution in the operational horizon. A look-ahead contingency management scheme of the IES is then proposed to minimize the electricity and gas load curtailments. Taking account of all the possible system states, the operational reliabilities of the IES are evaluated using the time-sequential Monte Carlo simulation. Finally, the proposed method is validated by using the IEEE Reliability Test System and the practical Belgium gas transmission system.

Robust data envelopment analysis with variable budgeted uncertainty

Including uncertainty in data envelopment analysis (DEA) is vital to achieving stable and reliable performance evaluations for the field of operational research as business environments are becoming increasingly volatile and unpredictable. Robust DEA models with budgeted uncertainty have been drawing particular attention in the DEA literature for modelling uncertainties, aiming to obtain robust efficiency scores in a way that guarantees the feasibility of solutions. A concern with such robust DEA models – which has been largely ignored in the literature – is that incorporating high uncertainty levels might result in too conservative efficiency measures, possibly reducing the decision support value of such information. To address this concern, this paper tackles uncertainties by employing variable budgeted uncertainty, which is a generalisation of the budgeted uncertainty. We introduce a novel robust DEA model with variable budgeted uncertainty that is less conservative than extant robust DEA models. Furthermore, we suggest a solution for specifying the probabilistic bounds for constraint violations of the uncertain parameters in robust DEA models. A comparison of the introduced robust DEA model with existing robust DEA models based on a numerical example shows an average reduction in the price of robustness by approximately 20%. Finally, the usefulness and applicability of the suggested model are demonstrated by using a large-scale data set in the context of grocery retailing.

A Sensitivity-based Rolling Operational Reliability Evaluation that Follows the Changing Available Generation Capacity of Renewable Energies

A Sensitivity-based Rolling Operational Reliability Evaluation that Follows the Changing Available Generation Capacity of Renewable Energies

Improving the accuracy of QoT estimation with insertion loss distribution evaluation for C + L band transmission systems

The accuracy of quality of transmission (QoT) estimation is crucial for software-defined optical networks (SDONs), where network planning and system management of the centralized controller depend on reliable performance evaluation. The insertion loss (IL) commonly exists along the light path between the transmitter side (Tx) and the receiver side (Rx) in fiber optical transmission systems, and the IL values vary for each connector and pigtail. In the case of C+L band transmission systems that are about to be deployed on a large scale, in the presence of severe stimulated Raman scattering (SRS), IL further influences the launch power profile (LPP) as well as the noise spectrum, and the process in the QoT estimation will be disturbed without available IL values. Therefore, there is a great need for an accurate calculation method of IL distribution. In this paper, we propose a method to estimate IL distribution in fiber optical transmission links to improve the accuracy of QoT estimation based on the SRS effect. The results demonstrate that, compared with other common methods, our proposed method greatly improves the accuracy of QoT estimation for C+L band transmission systems with accurate IL distribution evaluation. The high accuracy, robustness, and ability to adapt dynamic conditions make our proposed method a promising QoT estimation scheme in practical C+L band fiber optical communication systems.

Estimation of photovoltaic models using an enhanced Henry gas solubility optimization algorithm with first-order adaptive damping Berndt-Hall-Hall-Hausman method

A reliable methodology is essential for accurately estimating the parameters of PV models, enabling reliable performance evaluations, effective control studies, accurate analysis of partial shading effects, and optimal optimization of Photovoltaic (PV) systems. It ensures that the obtained parameters reflect the true characteristics of the PV system, leading to more accurate and reliable results in various applications. The existing literature extensively explores the utilization of powerful Metaheuristic Algorithms (MAs) to address the complex constrained optimization problem in PV systems and achieve optimal solutions. However, it is important to note that a significant portion of these MAs primarily concentrates on the development of methodologies, often overlooking the design of the objective function tailored for PV systems. This oversight has created a theoretical gap in this research domain, underscoring the necessity for additional exploration and investigation to address this limitation. To address the existing theoretical gap, this study focused on developing an objective function that accurately estimates the initial root parameters of Photovoltaic (PV) models. This objective function was designed by incorporating the first-order Berndt-Hall-Hall-Hausman (BHHH) numerical method, along with the non-linear damping parameter of the Levenberg-Marquardt technique (LM). By implementing this approach, the study aimed to significantly improve the precision and reliability of estimating the initial root parameters in PV models, effectively filling the theoretical void in this specific research area. Then in terms of methodology, the Enhanced Henry Gas Solubility Optimization (EHGSO) algorithm is combined with the Sine-Cosine mutualism phase of Symbiotic Organisms Search (SOS) for efficiently estimating the unknown parameters of PV models. The keystone of EHGSO in terms of methodology enhances exploration at the beginning of optimization and intensifies exploitation in later iterations. The proposed EHGSO methodology based on the adaptive damping BHHH technique (EHGSOAdBHHH) is tested on Single Diode (SD), and Double Diode (DD) PV models using actual experimental data. EHGSOAdBHHH exhibits outstanding accordance with attained experimental data compared with other algorithms, and its superiority is validated using several statistical criteria.

Reliability Evaluation and Optimization Configuration Method of Network Distribution Side Energy Storage System Based on Machine Vision

To reduce the motor power allocation error and improve voltage weak node conditions, this paper designed a reliability evaluation and optimization configuration method for network distribution side energy storage system based on machine vision. Firstly, the abnormal operation parameters of the energy storage system are collected and transmitted on-site. The operational reliability monitoring data of the distribution network side serves as the foundation for designing the operational reliability evaluation algorithm for the energy storage system on the same distribution network side. This algorithm further enables the optimal configuration of the energy storage system. The experimental comparison results demonstrate that upon implementation of the methods of this paper, the motor power allocation error is less, and the lowest is only 0.17 kW. Meanwhile, the fluctuation is relatively flat, and the range is between 0 kW and 0.46 kW. The improvement of weak voltage nodes is good, the voltage load is significantly decreased, and the minimum voltage load is only 5 kW, which proves that the design method of this paper has a certain application value.

Research of methods for assessing the quality indicators of Web services

This article explores the research of methods for assessing the quality indicators of web services, shedding light on the critical importance of ensuring top-notch performance, reliability, availability, and security in today's digital landscape. Understanding the significance of quality indicators and their assessment methodologies is essential for organizations that rely on web services to deliver seamless user experiences. The article delves into various evaluation methods, including performance analysis, availability assessment, reliability and robustness evaluation, and security analysis. Additionally, challenges and considerations, such as data collection, scalability, and privacy, are addressed to provide a comprehensive view of the assessment process. Looking to the future, the article discusses emerging trends in web service quality assessment and the integration of machine learning and artificial intelligence. Continuous quality assessment is emphasized as a key practice in adapting to the evolving landscape of web services. This article serves as a valuable resource for professionals and organizations seeking to enhance the quality of their web services, enabling them to meet the increasing demands of users and remain competitive in an ever-changing digital world.

Performance degradation and reliability evaluation of crystalline silicon photovoltaic modules without and with considering measurement reproducibility: A case study in desert area

In this paper, performance degradation and reliability evaluation of crystalline silicon photovoltaic modules deployed in desert climate was investigated based on the more than 7-year tracking test. It is found that peak power standard deviation first increased with time, reaching the peak when modules operated for 3 years, and then continuously decreased. Then, relationship among power warranty, power degradation rate and tolerance on maximum power is revealed for the first time. If 1% of module failures during warranty period is allowable, the 30-year warranty can be realized when degradation rate is 0.4–0.6 %/year with tolerance 0 ∼ +5% and 0 ∼ +3%, or degradation rate is 0.5 %/year with tolerance ±5% and ±3%, or degradation rate is 0.4 %/year with tolerance ±10%, ±5% and ±3%. Finally, reproducibility of power measurement is introduced for more accurately evaluating module performance degradation and reliability. Meanwhile, determination procedure of reproducibility for module electrical performance is systematically set forth. It is found that power warranty period can be prolonged if reproducibility is considered. The results obtained in this paper can provide new insights into performance degradation, reliability assessment and power warranty for fielded crystalline silicon solar modules.

The Validity and Reliability Performance Evaluation of the Fama-French Six Risk Premium Factors Model: Evidence from the Capital Stock Market of Pakistan

The study goal, aims, purpose and objective are to examine the Validity, Reliability and Performance Evaluation of the Fama-French Six Risk Premium Factor’s Model: Evidence from Capital Stock Market of Pakistan. The sample used were the balance sheet and Income statements annual reports of an eligible Non-Financial Companies issued from a State Bank of Pakistan (SBP). However, the idea for portfolio creation and a breakpoint have taken from keneath french data side. The sample data is 130 Non-Financial Companies covering the actual analysis from July 2014 to June, 2019. The (2x2) = 4 sort approaches for emerging markets and returns are used to construct portfolio and breakpoints for the SMB, HML, RMW, CMA, and WML other than Mkt-Rft. A total set of 16 set of portfolios are designed from dual categorization (Bivariate) designed for separately couple of size and one additional factor. A whole set of (4x4= 16), portfolios shaped as a dependent factor typically recognized as left-hand-side portfolios (LHS) and the same for independent risk premium factors for right hand side (RHS). The study found and concluded regarding the intercepts that all 16 set of four portfolios are insignificant and there are no pricing errors in the time of cost determination and predictions. The other objectives regarding slopes are found and concluded that the slopes of all risk premium i.e., MKT, SMB, HML, RMW, CMA, and WML and along with error correction term ECT (-1) are all positives and negatives and also statistically significant and explaining and predicting the average portfolio returns for 16 set of portfolios excess returns. The third main objectives regarding the other performance evaluation indicators, proposed to FF6F model’s relatively greater descriptive and explanatory power in each group of portfolios. Further, the value of R-squared is less than Durbin Watson value, means there is no sign of spurious linear regression too. Finally, it is found and concluded that the quality criteria suggests that the small values of Avg|α| or Avg|α-α ̅|in all equations are as the better model. It is further found and concluded that the Avg|α| of all six risk premium factors asset pricing models are the lowest values and acceptable range. It also further found and concluded that the fGRS and pGRS display the alpha is insignificant for 16 set of portfolios The research identifies that the asset pricing models are relatively more effective for diversified portfolios than concentrated portfolios and it is challenging to find a model that is effective in explaining the variation in the returns of any portfolio, but there is the possibility of finding a portfolio which works for a model and the investors can maximize their returns with a carefully chosen portfolio based on particular characteristics with a fewer number of stocks. Study proposes that corporate managers, policymakers, financial experts, and individual investors should have a stronger understanding of how to forecast stock prices on the Pakistan Stock Market, since it will help them when they are making investment decisions.

The Management of Power System Reliability in the Offshore Oil and Gas Field

The reliability of electric power systems is needed in offshore oil and gas field operations because disruptions can have a direct impact on oil production, costs, and company profits.Objectives: to determine the evaluation of reliability performance and propose a model for managing the power system reliability of the PHE OSES offshore oil and gas field.Methodology: A case study with data collected through observation, interviews, and analysis of relevant documents.Finding: Reliability in 2015-2016 has not reached the target but from 2017-2021 has reached the target of 97.5%, as well as availability in 2015, 2018, 2019 & 2020 has not reached the target but in 2016, 2017 & 2021 has reached the target of 95%. This resulted in the highest production losses in 2015 at 266,965 and the lowest in 2021 at 21,388 barrels of oil. The model is proposed by combining elements in risk-based asset management method, RCM method, and redesign.Conclusion: In recent years, reliability has been on target but availability is volatile. The model is proposed to maintain reliability & availability on target so that production losses can be minimized.

Class imbalance should not throw you off balance: Choosing the right classifiers and performance metrics for brain decoding with imbalanced data

Machine learning (ML) is increasingly used in cognitive, computational and clinical neuroscience. The reliable and efficient application of ML requires a sound understanding of its subtleties and limitations. Training ML models on datasets with imbalanced classes is a particularly common problem, and it can have severe consequences if not adequately addressed. With the neuroscience ML user in mind, this paper provides a didactic assessment of the class imbalance problem and illustrates its impact through systematic manipulation of data imbalance ratios in (i) simulated data and (ii) brain data recorded with electroencephalography (EEG), magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI). Our results illustrate how the widely-used Accuracy (Acc) metric, which measures the overall proportion of successful predictions, yields misleadingly high performances, as class imbalance increases. Because Acc weights the per-class ratios of correct predictions proportionally to class size, it largely disregards the performance on the minority class. A binary classification model that learns to systematically vote for the majority class will yield an artificially high decoding accuracy that directly reflects the imbalance between the two classes, rather than any genuine generalizable ability to discriminate between them. We show that other evaluation metrics such as the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC), and the less common Balanced Accuracy (BAcc) metric - defined as the arithmetic mean between sensitivity and specificity, provide more reliable performance evaluations for imbalanced data. Our findings also highlight the robustness of Random Forest (RF), and the benefits of using stratified cross-validation and hyperprameter optimization to tackle data imbalance. Critically, for neuroscience ML applications that seek to minimize overall classification error, we recommend the routine use of BAcc, which in the specific case of balanced data is equivalent to using standard Acc, and readily extends to multi-class settings. Importantly, we present a list of recommendations for dealing with imbalanced data, as well as open-source code to allow the neuroscience community to replicate and extend our observations and explore alternative approaches to coping with imbalanced data.

Evaluation of the Polymer Modified Tack Coat on Aged Concrete Pavement: An Experimental Study on Adhesion Properties

This study addresses the challenges of overlaying old concrete pavement with asphalt by introducing a new trackless tack coat material containing polymer. The aim is to enhance the durability of asphalt concrete overlay pavement on old cement concrete pavement. It contributes to the development of improved construction techniques for pavement rehabilitation and highlights the need for reliable adhesion performance evaluation based on different spray amounts and surface conditions. Additionally, to evaluate the effect of the adhesion performance based on the spraying amount, a tensile adhesion test was conducted by applying spray amounts of 0.30, 0.45, and 0.60 l/m2 on different surface conditions. The basic and adhesion performances of the polymer-modified tack coat material are evaluated through direct tensile and shear bond strength tests. The test outcomes demonstrated that the newly developed polymer-modified tack coat material had considerably greater adhesion strength compared to the traditional rapid-setting products. Its adhesive strength was 1.68 times higher on concrete and 1.78 times higher on asphalt. The new trackless tack coat material exhibited an adhesion performance of 1.05 MPa in direct tensile strength at 0.45 l/m2, which was 1.21 times higher than the rapid-setting tack coat. Results also confirmed that the new tack coat material exhibits values 1.90 times greater than the conventional rapid-setting tack coat material in shear bond strength, respectively. By simulating the process of separation and re-adhesion of pavement layers caused, the new tack coat material shows a tensile adhesion strength of 63% of the original state, which is advantageous for securing the durability of the pavement. Overall, the newly developed polymer-modified trackless tack coat has been shown to effectively enhance the adhesion performance between pavement layers without process delay, highlighting the potential of the new tack coat material to enhance the durability of asphalt concrete overlay pavement on old cement concrete pavement.