Two-stage Evaluation Research Articles

A Two-Stage Evaluation of China’s New Energy Industrial Policy Package

Energy structural transformation plays a strategically important role in achieving the dual-carbon reduction goals. Among the various approaches to carbon reduction, the Chinese government regards the growth of the new energy industry as an essential means. Considering that the government policy support determines the long-term growth of the new energy industry, how to improve and optimize the policy support system has always been the core issue. Based on the fact that policy evaluation is a prerequisite, and the new energy industrial development requires the government to promote solutions in the form of a policy package rather than just individual policies, we investigate whether the implementation of the new energy industry policy package (NEIPP) is effective through an empirical case study of Shanghai. A two-stage evaluation method, which integrates the content analysis method (CAM) and synthetic control method (SCM), was used to empirically evaluate the actual effect of the NEIPP. At Stage One, four policy goals were summarized. SCM was used to identify the pure multi-effect of the NEIPP. The results showed that the NEIPP had a significant positive effect on green economic growth and industrial structure, while having a negative effect on carbon emissions. The NEIPP had no impact on the promotion of technological innovation. Several policy implications were drawn from this study.

Assessing the impact of COVID-19 non-pharmaceutical interventions and relaxation policies on Class B respiratory infectious diseases transmission in China

This study investigates the incidence of Class B respiratory infectious diseases (RIDs) in China under the Coronavirus disease 2019 (COVID-19) epidemic and examines variations post-epidemic, following the relaxation of non-pharmaceutical interventions (NPIs). Two-stage evaluation was used in our study. In the first stage evaluation, we established counterfactual models for the pre-COVID-19 period to estimate expected incidences of Class B RIDs without the onset of the epidemic. In the second stage evaluation, we constructed seasonal autoregressive integrated moving average intervention (SARIMA-Intervention) models to evaluate the impact on the Class B RIDs after NPIs aimed at COVID-19 pandemic were relaxed. The counterfactual model in the first stage evaluation suggested average annual increases of 10.015%, 78.019%, 70.439%, and 67.799% for tuberculosis, scarlet fever, measles, and pertussis respectively, had the epidemic not occurred. In the second stage evaluation, the total relative reduction in 2023 of tuberculosis, scarlet fever, measles and pertussis were − 35.209%, − 59.184%, − 4.481%, and − 9.943% respectively. The actual incidence declined significantly in the first stage evaluation. However, the results of the second stage evaluation indicated that a rebound occurred in four Class B RIDs after the relaxation of NPIs; all of these showed a negative total relative reduction rate.

Recursive two-stage evaluation model for dynamic decision making under ambiguity

In this paper, the two-stage-evaluation (TSE) model for decision making under ambiguity (He 2021) is extended to intertemporal setting in an axiomatic approach. The first set of axioms employed are commonly adopted for dynamic non-expected utility models in the literature. Besides these regular axioms, I also assume dynamic consistency and conditions which deliver a static TSE for consumption plans only pay non-zero consequences for one period. It is shown that these axioms hold if and only if these exists a recursively defined evaluation utility model representing decision maker (DM)’s preferences over consumption plans conditional on arriving at any node in an event tree. Such a recursive form implies that one can apply dynamic programming technique (rolling back the decision tree) to solve a dynamic decision making problem under TSE model. It can be shown that the solution for the recursively defined dynamic TSE model exists uniquely. Due to the “small domain” setup, the agent is short-sighted in the sense that they only process subjective probabilities over events defined on one period uncertainty over next period states, which differs from “far-sighted” assumption in most extant models that assume subjective probabilities exist over events defined on multiple periods. It is shown that under some extra conditions, our DM applies Bayes’ rule “updating’ her subjective beliefs.

Dynamic Robust Optimization Method Based on Two-Stage Evaluation and Its Application in Optimal Scheduling of Integrated Energy System

As an emerging energy allocation method, shared energy storage devices play an important role in modern power systems. At the same time, with the continuous improvement in renewable energy penetration, modern power systems are facing more uncertainties from the source side. Therefore, a robust optimization algorithm that considers both shared energy storage devices and source-side uncertainty is needed. Responding to the above issues, this paper first establishes an optimal model of a regional integrated energy system with shared energy storage. Secondly, the uncertainty problem is transformed into a dynamic optimization problem with time-varying parameters, and a modified robust optimization over time algorithm combined with scenario analysis is proposed to solve such optimization problems. Finally, an optimal scheduling objective function with the lowest operating cost of the system as the optimization objective is established. In the experimental part, this paper first establishes a dynamic benchmark test function to verify the validity of proposed method. Secondly, the multi-mode actual verification of the proposed algorithm is carried out through a regional integrated energy system. The simulation results show that the modified robust optimization over time (ROOT) algorithm could find solutions with better robustness in the same dynamic environment based on the two-stage evaluation strategy. Compared with the existing algorithms, the average fitness and survival time of the robust solution obtained by the modified ROOT algorithm are increased by 94.41% and 179.78%. At the same time, the operating cost of the system is reduced by 11.65% by using the combined optimization scheduling method proposed in this paper.

Improving the efficiency of research proposals evaluation: A two-stage procedure

Abstract An important share of research funding is allocated via competitive programs, which entail considerable direct and indirect costs, such as to develop and evaluate the proposals. The goal of this article is to explore whether adopting a two-stage evaluation procedure could improve the efficiency of the process. For this purpose, we study the evaluation system designed by the Foundation Dam (Stiftelsen Dam), one of the largest foundations in Norway supporting health research. In 2020, Foundation Dam adopted a new evaluation procedure consisting in a short proposal for a first stage of selection and, for those selected, a second-stage evaluation of a long proposal. We explore whether such a procedure reduces the evaluation costs and how the evaluation procedures compare in terms of reliability. Survey responses from 94 of the 594 applicants in the one-stage procedure (2018–19) and all the 668 applicants in the two-stage procedure (2020–21) show that the two-stage procedure reduced the average time that applicants spent in drafting the proposal(s) by 38%. According to the reviewers’ estimate, the two-stage procedure also reduced by 28% the time they spent on average to evaluate an applicant’s proposal(s). The analysis of the evaluation data of 594 long proposals in the one-stage procedure and 668 short proposals in the two-stage procedure shows that reviewers’ scores of short proposals display greater reliability and agreement than the reviewers’ scores of long proposals in the old one-stage procedure. Our findings suggest that a two-stage procedure can substantially improve the efficiency of grant writing and review, without harming the reliability of the evaluation.

A Two-Stage Sustainable Supplier Selection Model Considering Disruption Risk

The global spread of the pandemic has changed many aspects of life and placed the supply chain at risk of disruption. To solve the problem of supplier selection under the risk of supply chain disruption, in this paper, we propose a two-stage evaluation model to address the issue of supplier selection in the context where a pandemic requires a lockdown. First, we incorporate the lead time into the epidemic model that predicts the evolution of the pandemic to identify suppliers that have a high risk of disruption caused by the pandemic’s evolution. Second, we propose a best–worst method combined with regret theory to rank candidate suppliers. Our model provides a dynamic link between the pandemic’s evolution and supplier selection, and it allows selecting suppliers according to various criteria while avoiding supply chain disruptions due to inappropriate supplier selection. We validate the proposed model on a real case study with epidemic data from China. This paper is the first to consider the impact of lockdowns during the pandemic on supplier selection. We develop a novel MCDM model BWM-RT for supplier selection; our model can be an effective decision support approach to help decision makers better cope with the risk of supply chain disruptions.

Sentiment analysis of COP9-related tweets: a comparative study of pre-trained models and traditional techniques.

Sentiment analysis has become a crucial area of research in natural language processing in recent years. The study aims to compare the performance of various sentiment analysis techniques, including lexicon-based, machine learning, Bi-LSTM, BERT, and GPT-3 approaches, using two commonly used datasets, IMDB reviews and Sentiment140. The objective is to identify the best-performing technique for an exemplar dataset, tweets associated with the WHO Framework Convention on Tobacco Control Ninth Conference of the Parties in 2021 (COP9). A two-stage evaluation was conducted. In the first stage, various techniques were compared on standard sentiment analysis datasets using standard evaluation metrics such as accuracy, F1-score, and precision. In the second stage, the best-performing techniques from the first stage were applied to partially annotated COP9 conference-related tweets. In the first stage, BERT achieved the highest F1-scores (0.9380 for IMDB and 0.8114 for Sentiment 140), followed by GPT-3 (0.9119 and 0.7913) and Bi-LSTM (0.8971 and 0.7778). In the second stage, GPT-3 performed the best for sentiment analysis on partially annotated COP9 conference-related tweets, with an F1-score of 0.8812. The study demonstrates the effectiveness of pre-trained models like BERT and GPT-3 for sentiment analysis tasks, outperforming traditional techniques on standard datasets. Moreover, the better performance of GPT-3 on the partially annotated COP9 tweets highlights its ability to generalize well to domain-specific data with limited annotations. This provides researchers and practitioners with a viable option of using pre-trained models for sentiment analysis in scenarios with limited or no annotated data across different domains.

Defining Complex Adaptive Systems: An Algorithmic Approach

Despite a profusion of literature on complex adaptive system (CAS) definitions, it is still challenging to definitely answer whether a given system is or is not a CAS. The challenge generally lies in deciding where the boundaries lie between a complex system (CS) and a CAS. In this work, we propose a novel definition for CASs in the form of a concise, robust, and scientific algorithmic framework. The definition allows a two-stage evaluation of a system to first determine whether it meets complexity-related attributes before exploring a series of attributes related to adaptivity, including autonomy, memory, self-organisation, and emergence. We demonstrate the appropriateness of the definition by applying it to two case studies in the medical and supply chain domains. We envision that the proposed algorithmic approach can provide an efficient auditing tool to determine whether a system is a CAS, also providing insights for the relevant communities to optimise their processes and organisational structures.

C-XAI: A conceptual framework for designing XAI tools that support trust calibration

Recent advancements in machine learning have spurred an increased integration of AI in critical sectors such as healthcare and criminal justice. The ethical and legal concerns surrounding fully autonomous AI highlight the importance of combining human oversight with AI to elevate decision-making quality. However, trust calibration errors in human-AI collaboration, encompassing instances of over-trust or under-trust in AI recommendations, pose challenges to overall performance. Addressing trust calibration in the design process is essential, and eXplainable AI (XAI) emerges as a valuable tool by providing transparent AI explanations. This paper introduces Calibrated-XAI (C-XAI), a participatory design framework specifically crafted to tackle both technical and human factors in the creation of XAI interfaces geared towards trust calibration in Human-AI collaboration. The primary objective of the C-XAI framework is to assist designers of XAI interfaces in minimising trust calibration errors at the design level. This is achieved through the adoption of a participatory design approach, which includes providing templates, guidance, and involving diverse stakeholders in the design process. The efficacy of C-XAI is evaluated through a two-stage evaluation study, demonstrating its potential to aid designers in constructing user interfaces with trust calibration in mind. Through this work, we aspire to offer systematic guidance to practitioners, fostering a responsible approach to eXplainable AI at the user interface level.

Characterization of model uncertainty of the p-y method for reliability-based design of offshore wind turbines under the serviceability limit state

Recent studies show that the API p-y method may underestimate the pile-soil initial stiffness of offshore wind turbine (OWT) monopiles in clay while overestimate that in sand. To facilitate the reliability-based design of OWT monopiles with the API p-y design method, a two-stage evaluation method of the model bias factor for the p-y method (εpy) of monopiles under the serviceability limit state (SLS) was presented. The model bias factor εpy is derived by comparing field test data with high-fidelity 3D finite element model (FEM) results, and the p-y method predictions, addressing the issues of limited field test data for OWT monopiles. Results show that the mean of the model bias factor εpy depends on the characteristics of piles, the soil properties, and the composition of the layered ground. The systematic dependency is expressed by a trend function and the residual model bias factor follows a lognormal distribution with a mean of 1.01 and a COV of 0.14. The significance of the model uncertainty is illustrated using a case study. The results of the field example show that the design in clay sites may be considerably conservative and may be less conservative in sand sites if ignoring the model uncertainty. The result of this work can provide a basis for promoting reliability-based design for OWT monopiles.

What Does Men's Health Education Look Like in Australian University Health Curricula? A Formative Evaluation and Future Enhancement Opportunities.

Developing the capacity of the health system, and the practitioners within it, to provide quality gender responsive care to men and boys remains critical to advancing men's health, and reducing health inequities amongst men. The aim for this study was to undertake a formative evaluation of Australian university health curricula for men's health content and scope the opportunities for future enhancement. A two-stage evaluation first involved a review of online course information for a sample of medicine (n = 10), nursing (n = 10), pharmacy (n = 10), clinical psychology (n = 10), social work (n = 12) and public health (n = 15) university curricula for men's health and gender content and opportunities for curricula enhancement. Secondly, university staff completed a survey on the coverage of men's health in their course(s), and receptivity, barriers and facilitators to curricula enhancement. The curricula review found no dedicated men's health courses, and men's health was referenced in the information for 10 of 1246 courses (0.8%) in 8 of 67 curricula. Gender was rarely referenced in course information, particularly for the disciplines of medicine, nursing, pharmacy, and clinical psychology. There was an average of 16 enhancement opportunities per curricula with 40% relating to communicating and engaging with men within healthcare. Seventy staff from 25 universities and all target disciplines validated the curricula review findings of limited dedicated men's health content. Eighty-three percent were receptive to curricula enhancement, facilitated by content integration into existing courses. This review provides clear evidence that there are gaps, opportunities, and educator receptiveness for improving and implementing content regarding men's health education and gender responsive care in Australian university health curricula.

Bio Mathematical Analysis on the Evaluation Scheme for Large Scale Innovation Competition

Large-scale innovation contests are becoming increasingly popular globally, with many innovators and enterprises actively participating. Work allocation optimization and fair judging are two significant concerns in innovation contests. This academic paper aims to recommend an optimized” cross-assignment” program by leveraging the optimal objective function method to improve the comparability of scores given by various judges. We conducted data-based descriptive statistical analysis and concluded that the two-stage and weighted evaluation schemes are more beneficial than the traditional judging scheme. Nonetheless, there are still some shortcomings that require addressing. To enhance fairness, we propose an improved two-stage evaluation scheme. In the first stage, we normalize the scores with a normal distribution. In the second stage of the process, we implement a system utilizing the Borda sorting technique to categorize submissions into five distinct groups for judges to evaluate based on their perceptions. We also detail a method for weighting tied scores to determine the final rankings. Testing indicates that this approach yields a Normalized Discounted Cumulative Gain (NDCG) of 0.8667, implying greater fairness and precision in the assessment of submissions.

Cutting-edge multi-task model: unveiling COVID-19 through fusion of image processing algorithms

ABSTRACT The COVID-19 pandemic underscores the vital need for accurate lung infection diagnosis to guide effective medical interventions. In response, this research introduces a novel deep multi-task model that seamlessly integrates segmentation and classification tasks for the detection of COVID-19 in CT scan images. This innovative model leverages a shared encoder for feature extraction, a dedicated decoder for segmentation, and a multi-layer perceptron for classification. The primary objective of this model is to address the challenge of task imbalance introduced by the application of image processing algorithms in the multi-task models. Our study involves a two-stage evaluation. Initially, we apply the proposed multi-task model with image processing algorithms to highlight task imbalance. Subsequently, we balance tasks by combining binary image processing algorithms. Evaluation on four datasets shows impressive results with a Dice coefficient of 88.91 ± 0.01 for segmentation and 0.97 classification accuracy. In summary, this model advances medical image analysis for enhanced diagnostic precision in healthcare.

Recognition to weightlifting postures using convolutional neural networks with evaluation mechanism

For modern sport training, critical posture recognition of athletes can be helpful for athlete training. This paper proposes convolutional neural networks using a two-stage evaluation mechanism to recognize four critical postures of a weightlifter, that is, force releasing, knee flexion, knee extension and highest point. Using the proposed convolutional neural networks classify images and extract image features. Meanwhile, a two-stage evaluation mechanism is adopted to calculate the scores of image features, based on the calculated scores, the four critical postures can be accurately recognized. Experimental results show that the accuracy of our method is 92.85% in the recognition of the four critical postures, which defeats the competitive methods in critical posture recognition. Moreover, the training time of the proposed method linearly augments along with the increasing of data volume, that is, non-exponential growth, consequently, our method can be applied to large-scale image datasets. We demonstrate that the two-stage mechanism can calculate the scores of image features independently of specific scenarios, which assist neural networks improve classification capabilities. Moreover, using the two-stage mechanism can simplify the designed complexity of neural network architectures, thus reducing the training parameter of neural networks in the process of critical posture recognition.

Antecedents of Sales Performance in the Malaysian Commercial Banking Industry

Sales performance is important to every salesperson as a key performance indicator for their yearly performance. However, nearly half of them are still unable to fulfill it. Hence, this study aims to investigate the personal factors that influence the sales performance of salespersons in the banking industry. The study examines the direct relationship between improvisational behavior, adaptive selling, and self-efficacy in relation to sales performance among salespersons working in various local commercial banks in Malaysia. In these banks, despite the increasing importance of sales performance, almost half of the salespeople still fail to meet their annual performance target. A survey design with clustered random samples was utilised for this study. Questionnaires were used to collect data from salespersons in Malaysian commercial banks in the Klang Valley. The data was then analyzed using Structural Equation Modelling (SEM) with SmartPLS 4.0. In this study, three hypotheses were developed and tested through a two-stage evaluation. The results show that improvisational behavior is the most significant predictor of sales performance, followed by adaptive selling and self-efficacy. The results of the study provide managers with novel concepts to understand the factors that influence a salesperson's sales performance in the Malaysian banking sector.

A comparative analysis of the methods of evaluating dependability using Big Data of complex system operation

Aim. The paper aims to research the methods of evaluating dependability indicators based on the analysis of Big Data obtained in the course of monitoring the operation of technical systems and their components. A comparative analysis is performed of the efficiency of robust statistical methods of evaluating the dependability indicators of complex technical systems based on operation data.Methods. The paper uses methods of mathematical statistics, specifically robust methods of evaluating the translation parameter of a noisy sample and numerical methods of statistical modeling. The authors consider five methods of evaluating the translation parameter: sample mean as a nonrobust method used for comparison; sample median as the simplest robust method of evaluating the translation parameter; two-stage evaluation procedure with truncation of outliers according to the three sigma rule; two-stage evaluation procedure with truncation of outliers using Tukey’s box-and-whisker plot; Huber’s robust method. The comparative analysis of the methods of evaluating system dependability indicators was conducted by means of statistical modeling in the R statistical computation package. Five distribution laws for generating an element’s time-to-failure and recovery time samples were considered: exponential distribution, Weibull distribution, log-normal distribution, gamma distribution and uniform distribution.Results. Statistical analysis of Big Data associated with the operation of technical systems is complicated by the heterogeneity and noisiness of such data, as well as the presence of errors and outliers of varied nature. That is primarily due to the varied loads and operating conditions of each object. Herein this problem is examined as regards the problem of evaluating the dependability indicators of a structurally complex monotonic system with independent element recovery. The paper examines methods of rejecting anomalous data and robust evaluation of the sample position parameter and performs a comparative analysis of the efficiency of such methods for various distribution laws. It is shown that robust methods of evaluation enable significantly higher accuracy as compared to the standard sample mean. The two-stage procedure based on the truncation of outliers and Tukey’s box-and-whisker plot proved to be the most efficient.Conclusions. The paper’s findings allow improving the accuracy of evaluation of dependability indicators based on complex technical system operation data. They can be used in Big Data processing and complex system dependability theory.

Responsible Research Assessment

Felix Schönbrodt, Professor of Psychology at Ludwig-Maximilians-Universität (LMU) in Munich, tells about an initiative that he coordinates within the Deutsche Gesellschaft für Psychologie (German Psychological Society). Motivated by the Reproducibility Crisis and a rising frustration with the publishers of high-ranking journals, Schönbrodt has co-authored three position papers on the theme of responsible research assessment. The suggestion is to develop a two-stage evaluation system for hiring, the first of which will use responsible metrics with emphasis on open data, pre-registration and several aspects of reproducibility, whereas the second stage will focus on a qualitative (content-oriented) evaluation of selected candidates. The propositions of Schönbrodt’s group have so far led to published feedback from more than 40 different scholars. Besides his nation-wide work within the German Psychological Society, Schönbrodt is the managing director of LMU’s Open Science Centre, where scholars from different disciplines convene for workshops on various aspects of Open Science. Under the nickname «nicebread» (Schön = nice, Brodt = bread), he also runs a personal blog and a project webpage on GitHub.
 Interview recorded in Munich, October 2023. First published online: November 29, 2023.

Two-Stage Evaluation of the Pre-merger Potential Gains of Taiwan Financial Holding Companies: Dynamic Network Slack-Based Measure Analysis Approach

Two-Stage Evaluation of the Pre-merger Potential Gains of Taiwan Financial Holding Companies: Dynamic Network Slack-Based Measure Analysis Approach

Semi-supervised learning method incorporating structural optimization for shear-wall structure design using small and long-tailed datasets

Intelligent structural design based on machine learning represents a novel structural design paradigm and has received extensive attention in recent years. However, the performance of the machine learning models is heavily dependent on the quality and quantity of training data, as the underlying approaches are inherently data-driven. Well-recognized data issues ‒ particularly data insufficiencies and long-tailed data distributions ‒ have become critical impediments in this research area. To address these data issues, this study formulates a schematic structural design task as a semi-supervised learning problem. Specifically, a semi-supervised learning method using small, long-tailed datasets is proposed in which a structural optimization method is incorporated into a self-training framework. As a practical application of the proposed method, a shear-wall layout optimization procedure is devised, based on a two-stage evaluation strategy and the previously established empirical design rules. Results of the numerical experiments indicate that the proposed method can improve the design performance on the tail data by 11.6 % and reduce the performance difference between the head and tail data by 21.3 %, compared to the conventional supervised learning method. A typical case study shows that the shear-wall layout created by the proposed method can satisfactorily resemble that by design engineers whilst meeting key code-specified requirements.

Intergenerational mobility through inhabited meritocracy: Evidence from civil service examinations of the early- and mid-Ming dynasty.

The civil service examination system emerged to strengthen the emperor's power by recruiting political elites through open examinations. Did it, during the early- and mid-Ming dynasty, facilitate intergenerational mobility? Rather than oversimplifying it as a single-stage system of meritocracy, we propose a two-stage evaluation framework. In the first stage, the Metropolitan Exam featured merit-based evaluations and generated credentials necessary for becoming political elites. The subsequent non-eliminating Palace Exam then functioned to assess the students' organizational fit in line with an emperor's political calculations. In particular, those whose families served in the bureaucracy were favored, while those from affluent families were discriminated against. We test this two-stage framework using the records of 12,427 students who passed 46 exams between 1400 and 1580, a period characterizing the heyday of this system. Our empirical findings from the mixed-effect regression models confirm this argument and suggest promising directions for future research.