Systematic Evaluation Method Research Articles

Approach for Evaluating the Effectiveness of the Airport Roadway Traffic Sign System

The airport road traffic sign system is essential for drivers to find their destinations. However, no comprehensive design standards have been established for airport roadway traffic signs in China. An airport roadway traffic sign system was designed based on the Chinese Road Traffic Signs and Markings standard (GB5768-2009), called scheme 1. A second sign system was established using international standards and following drivers’ recognition, sign information hierarchy, called scheme 2. The Airport Roadway Traffic Sign System Evaluation Method (ARTSSEM) was established to evaluate the effectiveness of the airport roadway traffic sign system. This method includes a goal layer, a method layer, an indicator layer, and a comprehensive evaluation layer. We obtained drivers’ driving behavior data from the driving simulation. The subjects drove at a lower and more stable average speed in scheme 2 (63.71 ± 3.91) than in scheme 1 (65.89 ± 4.81). An ergonomic model that considers the sign system’s information demand, information comprehension, and information volume (DCI) was established and evaluated using subjective experiments. Scheme 2 provided a higher DCI score (0.46) than scheme 1 (0.17). The performance of the sign systems was comprehensively evaluated using the fuzzy evaluation method to evaluate the driving behavior, wayfinding performance, and ergonomics of the sign system. Scheme 2 (76.13) had a higher score than scheme 1 (74.13). The results revealed that scheme 2 performed better than scheme 1. The effectiveness of ARTSSEM was verified by the case study of the Daxing Airport signage system. This study provides a general approach to designing and evaluating other sign systems.

Bayesian-Based Traffic Safety Evaluation Study for Driverless Infiltration

Although driverless technology belongs to the frontier of science and technology, there is no sufficient actual data. From the lack of a comprehensive systematic evaluation method of traffic safety under driverless penetration, considering the impact of the three core systems of driverless perception, decision-making, control, and the complex road factors on the safety of driving, we review the main risk causal factors through the analysis of the accident causal model STAMP and put forward the fusion of the Leaky Noisy-OR Gate and Bayesian network model. The Bayesian network professional analysis tool GeNIe 2.0 was used to simulate, analyze, and evaluate the driverless traffic risk Bayesian network model, which accurately assessed the traffic safety risk under driverless penetration and diagnosed and identified the sensitive risk factors. The results of this study concluded that, in order to effectively deal with the future traffic safety risks of driverless vehicles, vehicle enterprises, research institutions, software and hardware suppliers in the field of driverless driving should strengthen the research and development and manufacturing of key components such as perception, and enhance the depth of research and development of AI decision-making software, which provides a new way of thinking about the management of the safety risk of driverless traffic and a theoretical basis for the development and implementation of risk control measures.

Understanding and Predicting User Satisfaction with Conversational Recommender Systems

User satisfaction depicts the effectiveness of a system from the user’s perspective. Understanding and predicting user satisfaction is vital for the design of user-oriented evaluation methods forconversational recommender systems (CRSs). Current approaches rely on turn-level satisfaction ratings to predict a user’s overall satisfaction with CRS. These methods assume that all users perceive satisfaction similarly, failing to capture the broader dialogue aspects that influence overall user satisfaction.We investigate the effect of several dialogue aspects on user satisfaction when interacting with a CRS. To this end, we annotate dialogues based on six aspects (i.e.,relevance,interestingness,understanding,task-completion,interest-arousal, andefficiency) at the turn and dialogue levels. We find that the concept of satisfaction varies per user. At the turn level, a system’s ability to make relevant recommendations is a significant factor in satisfaction. We adopt these aspects as features for predicting response quality and user satisfaction. We achieve an F1-score of 0.80 in classifying dissatisfactory dialogues, and a Pearson’srof 0.73 for turn-level response quality estimation, demonstrating the effectiveness of the proposed dialogue aspects in predicting user satisfaction and being able to identify dialogues where the system is failing.With this article, we release our annotated data.1

Research on the construction of weaponry indicator system and intelligent evaluation methods

To decrease subjective interference and improve the construction efficiency of the traditional weapon and equipment index system, an index system construction method based on target detection is proposed in combination with the equipment test video data. The three-level index system of combat effectiveness of a certain type of equipment is established, and various intelligent assessment methods are proposed. Firstly, an optimaized IPSO-BP network model is proposed, in which dynamic weights are set to improve the particle search network, and adaptive learning factors are introduced to optimize the update speed. Secondly, an improved DS evidence-parallel neural network assessment method is proposed, setting multiple parallel neural networks with different parameters, and improving the angle cosine to weaken the numerical nonlinear attributes in DS evidence fusion and increase the model's assessment operation stability. Thirdly, the three types of view features corresponding to the index item images are extracted to train the base classifiers. The integrated CNN network based multi-view feature integration assessment model is constructed and the improved residual network block is introduced to optimize the network gradient. Comparison with existing evaluation methods shows that the proposed methods achieve efficient and intelligent construction and evaluation of the indicator system and enrich the evaluation of indicator data.

An Angle Precision Evaluation Method of Rotary Laser Scanning Measurement Systems with a High-Precision Turntable

Rotary laser scanning measurement systems, such as the workshop measurement positioning system (wMPS), play critical roles in manufacturing industries. The wMPS realizes coordinate measurement through the intersection of multiple rotating fanned lasers. The measurement model of multi-laser plane intersection poses challenges in terms of accurately evaluating the system, making it difficult to establish a standardized evaluation method. The traditional evaluation method is based on horizontal and vertical angles derived from scanning angles, which are the direct observation of wMPS. However, the horizontal- and vertical-angle-based methods ignore the assembly errors of fanned laser devices and mechanical shafts. These errors introduce calculation errors and affect the accuracy of angle measurement evaluation. This work proposes a performance evaluation method for the scanning angle independent of the assembly errors above. The transmitter of the wMPS is installed on a high-precision turntable that provides the angle reference. The coordinates of enhanced reference points (ERP) distributed in the calibration space are measured by the laser tracker multilateration method. Then, the spatial relationship between the transmitter and the turntable is reconstructed based on the high-precision turntable and the good rotational repeatability of the transmitter. The simulation was carried out to validate the proposed method. We also studied the effect of fanned laser devices and shaft assembly errors on horizontal and vertical angles. Subsequently, the calibration results were validated by comparing the residuals with those derived from the space-resection method. Furthermore, the method was also validated by comparing the reference and scanning angles. The results show that the maximum angle measurement error was approximately 2.79″, while the average angle measurement error was approximately 1.26″. The uncertainty (k = 1) of the scanning angle was approximately 1.7″. Finally, the coordinate measurement test was carried out to verify the proposed method by laser tracker. The results show that the average re-scanning error was 2.17″.

Evaluation of Green and Low-Carbon Development Level of Chinese Provinces Based on Sustainable Development Goals

Green and low-carbon development are important initiatives to promote the realization of the Sustainable Development Goals (SDGs). In this study, a systematic evaluation method for regional green and low-carbon development levels was established by referring to the evaluation methods of other literature. The evaluation method includes assessing the overall green and low-carbon development levels of the evaluation objects, as well as analyzing the coupling coordination of the two subsystems of green development and low-carbon development. The results show that China’s green low-carbon development level shows a rising trend year by year from 2012 to 2021, but there is still much space for progress. As for 2021, 2 provinces are in the Fair stage, 25 provinces are in the Accepted stage, and 4 provinces are in the Good stage. Provinces with higher levels of green and low-carbon development are characterized by spatial clustering, and the level of green and low-carbon development in the southeast coastal provinces is significantly higher than that in the northwest inland provinces. This study measures the coupling coordination degree of the two subsystems of green development and low-carbon development in each province. The average coupling coordination degree of China’s green and low-carbon development shows a rising trend year by year from 2012 to 2021. By 2021, the coupling coordination degree of China’s 29 provinces belongs to the High level type, and only two provinces belong to the Low level type. This study provides a provincial map of China’s green and low-carbon development levels and a distribution map of the coupling coordination degree of the green and low-carbon development subsystems, which provides support for an in-depth understanding of the spatial–temporal distribution of and development trends in China’s green and low-carbon development level. This study provides support for a deeper understanding of the spatial and temporal distribution and development trend of China’s green and low-carbon development level. The study also provides data to support China’s efforts to promote synergies in reducing pollution and carbon emissions.

Research on Performance Evaluation Index System and Assessment Methods for Microgrid Operation in the Port Area

The vertical integration of ‘Generation-Grid-Load-Storage’ in microgrids for port areas is a prevailing trend. To comprehensively and accurately assess the operational efficiency of microgrids and develop an effective means for promoting the sustainable and scalable development of microgrids in port areas, an applicable evaluation index system and comprehensive evaluation method are essential. Addressing the issues of subjectivity in quantitative evaluation decision-making for different configuration and allocation schemes in port areas and the lack of a comprehensive evaluation system for microgrids in port areas, this study aims to scientifically and reasonably evaluate and select the most optimal configuration and energy dispatching schemes for microgrids in port areas. In this paper, the performance indicators of microgrids in port areas are hierarchically structured and classified into five dimensions: economic, energy efficiency, environmental, system reliability, and safety. A comprehensive evaluation index system for microgrid systems in port areas is constructed. Furthermore, an evaluation model for microgrid operational efficiency based on improved CRITIC-TOPSIS is proposed. The coefficient of variation is introduced to measure the relative strength and relative variation of indicators among dimensions, reducing the impact of correlation between indicators and improving the accuracy and objectivity of the evaluation results. Then, the TOPSIS method is used to calculate the comprehensive evaluation values and rank the optimal microgrid schemes in port areas. Finally, the feasibility of the evaluation index system and evaluation method for the operational efficiency of microgrids in port areas is verified through case analysis. The results indicate that the evaluation method based on improved CRITIC-TOPSIS can objectively and quantitatively evaluate the operational efficiency of microgrids in different port areas. The proposed method reasonably avoids the strong subjectivity in weight calculation by traditional expert judgment during the planning phase of microgrids in port areas, providing a new and scientifically effective engineering evaluation and analysis method for the evaluation of microgrid operational efficiency in port areas.

Video Turing Test: A first step towards human‐level AI

AbstractThe development of artificial intelligence (AI) agents capable of human‐level understanding of video content and conducting conversations with humans on this basis is a promising application that people expect. However, this is a challenging task that requires the holistic integration of multimodal information with temporal dependencies and reasoning, as well as social and physical commonsense. In addition, the development of appropriate systematic evaluation methods is essential. In this context, we introduce the Video Turing Test (VTT), a blind test used to evaluate human‐likeness in terms of video comprehension ability. Moreover, we propose Vincent as a video understanding AI. We explain the configuration of VTT, the architecture of Vincent to prepare for VTT and the proposed evaluation methods for video comprehension. We also estimate the current intelligence level of AI based on our results and discuss future research directions.

Signal-Vehicle Coordination Control Modeling and Roadside Unit Deployment Evaluation under V2I Communication Environment

Signal-vehicle coordinated control holds substantial promise for enhancing urban transportation efficiency. However, its development faces notable challenges: (1) most existing studies have been conducted based on the assumption of perfect communication conditions. This assumption overlooks the significant impact of vehicle-to-infrastructure (V2I) communication quality on control performance, which leads to poor applicability in practice. (2) The evaluation of roadside unit (RSU) deployment for optimizing signal-vehicle control has not been well studied. Hence, the modeling of signal-vehicle coordination control and RSU deployment evaluation under V2I environment are studied in this paper. First, we introduce a communication model that characterizes the imperfections in communication between RSUs and connected vehicles (CVs). Second, we propose a model for signal-vehicle coordination control within this connected environment. This model integrates strategies from both signal control optimization and the speed optimization of CV platoons. Finally, to assess the impact of the RSU deployment parameters on the performance of signal-vehicle coordination control, we introduce a systematic evaluation method. The reduction in vehicle delays is introduced as the evaluation indicator for control performance. Six other indicators—the number of vehicles in the RSU communication domain, connectivity probability between the CV and RSU, number of vehicles whose speeds are successfully optimized, number of speed adjustments, green extension time, and overlap rate of the communication domains of multiple RSUs—are introduced as the observation indicators. The simulation experiments verify the effectiveness of the proposed model in implementing signal-vehicle coordination control under imperfect communication and environments in low-traffic, medium-traffic, and high-traffic scenarios. Furthermore, these experiments show the quantitative impact of RSU deployment parameters (communication distance, command transmission cycle, installation position, and number of RSUs) on control performance.

ANALYSIS OF USABILITY USING HEURISTIC EVALUATION METHOD AND MEASUREMENT OF SUS ON PRICILIA APPLICATION

The Presence Digital Application (PRICILIA) is a presence application owned by PT. BGR Logistics Indonesia. However, until now, there has never been an evaluation of usability testing. Complaints from users regarding the PRICILIA application include menu displays that are less interactive, long loading times, and the unavailability of other alternatives besides GPS. Of course, this affects the level of user satisfaction with the application. Therefore, usability testing is needed to be able to measure the level of user comfort, application feasibility, and the application interface. In this study, the system evaluation method used is Heuristic Evaluation with measurement using the System Usability Scale (SUS). The results of this study indicate that aspects that need to be improved with high priority are Error Prevention (H5) and Recognition rather than Recall (H6) because they have a seriousness rating on a scale of 3, while the average score of the final seriousness rating obtained from a total of 10 heuristic aspects is 1, 72 which is then rounded off to a scale of 2. The SUS test results obtained an average final SUS score of 55.13. The results of the calculation of the SUS method are that the Acceptance Ranges have low marginal status, the Grade Scale is on a D scale, and the Adjective Twigs are at the OK level. This shows that the PRICILIA application still needs improvement. Therefore, 30 recommendations for improvement are proposed for future application development.

Research on energy flexibility demand analysis and evaluation method of power systems with a high proportion of new energy

With the increase of the scale of highly uncertain power supply in the system, the system needs more and more energy flexibility. Existing studies focus more on short-term energy demand volatility changes and lack quantitative means to analyze the system’s demand for energy flexibility at different time scales. Aiming at the above problems, this paper proposes an energy flexibility demand analysis and evaluation method for power systems with a high proportion of new energy. It constructs an energy flexibility demand evaluation index system at different time scales. Combined with the analysis of domestic provincial cases, the results verify the effectiveness of the proposed method.

Tai Ji on Cognitive Function Improvement in Parkinson's Disease: A Meta-Analysis.

Tai Ji (TJ) can improve cognitive function, which in turn brings life quality to Parkinson's disease (PD) patients. Cognitive function is thus important for PD patients. Analyzing the cognitive function and obtaining precise TJ exercise prescriptions are effective in treating PD patients. The purpose of this review was to discuss the current Randomized Controlled Trials (RCTs) of TJ and cognitive function improvement in PD. Databases, including PubMed, Web of Science, Scopus, Cochrane Library, EBSCOhost, Wan Fang, and CNKI, were searched. Cochrane systematic evaluation method was adopted wherein 6 RCTs met the final inclusion criteria. Among the 6 included RCTs, 3 were of high quality, and the rest were of medium quality. Quantitative analysis exhibited that TJ intervention improved the cognitive function of PD patients. TJ promoted the global cognitive function (p < 0.05) and the executive function (p = 0.09) compared with the control. However there was no significant improvement in cognitive motor. TJ training intervention doses of two times a week with 45-60 min sessions over ≤12 weeks could improve the cognitive function of PD patients. TJ affects the cognitive function of PD patients. However, this effect may have a reduced or no significant impact as the disease increases. PD patients can choose TJ as an intervention for 45-60 min twice a week for at least 12 weeks to accomplish maximum improvement in cognitive function.

Retracted: The Teaching Evaluation Index System and Intelligent Evaluation Methods of Vocational Undergraduate Pilot Colleges

Retracted: The Teaching Evaluation Index System and Intelligent Evaluation Methods of Vocational Undergraduate Pilot Colleges

Integrated energy system evaluation method based on dimensionality reduction and indexes updating with incomplete information

Integrated energy system evaluation method based on dimensionality reduction and indexes updating with incomplete information

Performance evaluation of non-repairable cyclic phased-mission systems using evidential reasoning rule and multi-valued decision diagrams

Phased-mission systems (PMS) often perform different missions that have multiple consecutive, non-overlapping phases. Many recent studies have focused on the reliability analysis for PMS. However, not much attention has been paid to the PMS performance evaluation and the phase cycle characteristic of PMS. In this paper, a performance evaluation method of non-repairable cyclic phased-mission systems (CPMS) is proposed. Considering the uncertainty caused by the change of phase duration and component performance degradation, the average duration of each phase and the fuzzy state probability vector of each component are obtained by the evidential reasoning rule (ER rule). Then, the multi-valued decision diagrams (MDD) manipulation rules suitable for cyclic phase conditions are proposed, and the component state composition path generation method based on cycle-end component states is developed. On the above basis, MDD models for CPMS (CPMS-MDD) in different states are established, and the system performance is evaluated by the continuous time Markov chains (CTMC). A phase duration adjustment strategy is further proposed based on the evaluation results. The effectiveness and correctness of the proposed method are verified by case studies on performance evaluation of satellite turntable system. Comparative study shows the concise and efficient of the proposed method.

Parameter Adjustment of the Autothrust Control Law by Flight Tests

Aiming at deriving precise automatic control of airspeed, a comprehensive flight test solution for autothrust control law parameter adjustment was proposed, providing a worthy reference for other aircraft types’ work, including the flight test points and methods, aircraft modification requirements for the test, the system performance analysis and evaluation methods, and the flight test risk analysis. A method invoking test scripts by a flight test interface unit and a control panel to configure parameters of autothrust control law and input excitation signals was proposed for the first time. The test scripts and the test devices cross-link can be fully verified in advance in both iron bird and real aircraft ground tests, which can greatly reduce unexpected risks of the test method based on the flight test engineers’ personal experience. According to the evaluation from three aspects of system response, robustness and throttle lever movement based on the test data, the optimal autothrust control law parameters configuration was given.

A Neural Network Method for Systematic Evaluation of Informatization Development Level in Smart Court Construction

&lt;p&gt;The systematic evaluation of informatization development level (IDL) is an evaluation that follows the development trend of digitization, networking and intelligence, and focuses on the formation of business capabilities of informatization. Research on informatization evaluation methods has been extensively studied by both domestic and international academics over the years. However, traditional evaluation methods suffer from flaws like complex mechanism design, unreliable metric conversion, difficulty obtaining the relative importance of indexes, complex evaluation process, and high computational volume. This paper attempts to introduce the neural network method into the information system evaluation, and uses the Extreme Learning Machine (ELM) algorithm to establish the evaluation model. The evaluation of the smart court system is used as an example to simulate and test the model, and the results show that the neural network-based evaluation model of informatization system is more applicable to large-scale evaluation indexes, and by continuously increasing the learning samples, it objectively improves the accuracy of evaluation, effectively avoids human subjective factors, and has the advantageous features of advanced, accurate and convenient.&lt;/p&gt; &lt;p&gt;&amp;nbsp;&lt;/p&gt;

Productivity Analysis and Evaluation of Fault-Fracture Zones Controlled by Complex Fracture Networks in Tight Reservoirs: A Case Study of Xujiahe Formation

The development of tight gas reservoirs presents a significant challenge for sustainable development, as it requires specialized techniques that can have adverse environmental and social impacts. To address these challenges, efficient development technologies, such as multistage hydraulic fracturing, have been adopted to enable access to previously inaccessible natural gas resources, increase energy efficiency and security, and minimizing environmental impacts. This paper proposes a novel evaluation method to analyze the post fracturing productivity controlled by complex fault fracture zones in tight reservoirs. In this article, a systematic method to evaluate stimulated reservoir volume (SRV) and fault-fracture zone complexity after stimulation was established, along with the analysis and prediction of productivity through coupled fall-off and well-test analyses. Focusing on the Xujiahe formation in the Tongnanba anticline of northeastern Sichuan Basin, a 3D geological model was developed to analyze planar heterogeneity. The fall-off analytical model, coupled with rock mechanical parameters and fracturing parameters such as injection rates, fracturing fluid viscosity, and the number of clusters within a single stage, was established to investigate the fracture geometric parameters and complexities of each stage. The trilinear flow model was used to solve the well-test analysis model of multi-stage fractured horizontal wells in tight sandstone gas reservoirs, and well-test curves of the heterogeneous tight sandstone gas fracture network model were obtained. The results show that hydraulic fractures connect the natural fractures in fault-fracture zones. An analysis of the relationship between the fracture geometric outcomes of each segment and the net pressure reveals that as the net pressure in the fracture increases, the area ratio of natural fractures to main fractures increases notably, whereas the half length of the main fracture exhibits a decreasing trend. The overall area of natural fractures following stimulation is 7.64 times greater than that of the main fractures and is mainly a result of the extensive development of natural fractures in the target interval. As the opening ratio of natural fractures increases, the length of the main fractures decreases accordingly. Therefore, increasing net pressure within fractures will significantly enhance the complexity of fracturing fractures in shale gas reservoirs. Furthermore, the initial production of Well X1–10, which is largely controlled by fault-fracture zones, and the cumulative gas production after one year, are estimated. The systematic evaluation method in this study proposed a new way to accurately measure fracturing in tight reservoirs, which is a critical and helpful component of sustainable development in the natural gas industry.

PRIORITY TOOLS FOR THE INTEGRATION OF UKRAINIAN ENTERPRISES INTO VALUE CHAINS

The essence of the concept of value-oriented management, the methodological principles of calculating the indicator of economic added value of the enterprise are considered. Tasks and sequence of analysis based on value-oriented models are formulated. Advantages and problematic aspects of the application of the value concept, determination of indicators and models of economic added value are highlighted.&#x0D; The purpose of the study is to systematize the methodological foundations of the analysis of economic added value, the main provisions of value-oriented management and the formation of a systematic method of cost evaluation of the activity of objects, the conditions of its use.&#x0D; The processes of formation of value-added chains by Ukrainian enterprises are studied, which affects the efficiency of the Ukrainian economy and opens the possibility of entering global value-added chains in the conditions of international integration.&#x0D; The tasks set and solved in the study are:&#x0D; - description of methodical aspects of using aggregated value added assessment models;&#x0D; - generalization of the practice of creating chains of added value by economic structures in Ukraine and disclosure of comparative advantages;&#x0D; - formation of directions for the integration of domestic enterprises into global chains of added value;&#x0D; - assessment of the impact of integration processes and the consequences of creating added value on the development of the economy of Ukraine.&#x0D; The results of the study include the following:&#x0D; consideration of individual components of the concept of "added value" is proposed;&#x0D; the historical stages of the development of added value, the reasons for implementation and use to determine the effectiveness of activities are analyzed;&#x0D; the main conceptual models for determining added value, which are used in international practice, are considered; indicated advantages and disadvantages of each of them and conditions for their use;&#x0D; the relevance of the implementation of value indicators in the activity of Ukrainian enterprises for the construction of an effective management system has been proven.

Effects of horticultural therapy on health in the elderly: A review and meta-analysis.

Given the current aging population, the demand on medical facilities, facilities for elderly care, and even their high prevalence, it is crucial to investigate the advantages of older people. The purpose of our study was to systematically review the existing literature on the health effects of horticultural therapy as a treatment option for the elderly. Article searches were conducted through five databases: Web of science, Science Direct, PubMed, EBSCO, and Google Scholar, according to the standard method of systematic evaluation and meta-analysis. Thirty-two published articles were included, and 27 relevant variables were meta-analyzed to assess the benefits of horticultural therapy in terms of physical and psychological functioning in the elderly. Results show that horticultural therapy may be helpful in helping seniors lose weight -0.195 (95% CI -0.507,0.117), reduce their waist circumference -0.327 (95% CI -0.637, -0.017), lower their stress -0.339 (95% CI -0.610, -0.069) and cortisol -0.902 (95% CI -0.728, -0.002) levels, improve their physical flexibility 0.302 (95% CI 0.036, 0.569), social interaction 0.370, (95% CI 0.115, 0.624), and daily vegetables and fruit consumption 0.688 (95% CI: 0.287, 1.089). Horticultural therapy may be a useful tool for enhancing the physical, mental and social aspects of the elderly. However, there is substantial heterogeneity and wide variation in the quality of the included studies. Further high-quality studies, rigorous controls and adjustments for significant confounding variables, and larger populations are needed in the future to further our understanding of the link between horticultural therapy and elder health. The online version contains supplementary material available at 10.1007/s10389-023-01938-w.