Formal Evaluation Methods Research Articles

An Overview of the Empirical Evaluation of Explainable AI (XAI): A Comprehensive Guideline for User-Centered Evaluation in XAI

Recent advances in technology have propelled Artificial Intelligence (AI) into a crucial role in everyday life, enhancing human performance through sophisticated models and algorithms. However, the focus on predictive accuracy has often resulted in opaque black-box models that lack transparency in decision-making. To address this issue, significant efforts have been made to develop explainable AI (XAI) systems that make outcomes comprehensible to users. Various approaches, including new concepts, models, and user interfaces, aim to improve explainability, build user trust, enhance satisfaction, and increase task performance. Evaluation research has emerged to define and measure the quality of these explanations, differentiating between formal evaluation methods and empirical approaches that utilize techniques from psychology and human–computer interaction. Despite the importance of empirical studies, evaluations remain underutilized, with literature reviews indicating a lack of rigorous evaluations from the user perspective. This review aims to guide researchers and practitioners in conducting effective empirical user-centered evaluations by analyzing several studies; categorizing their objectives, scope, and evaluation metrics; and offering an orientation map for research design and metric measurement.

A Conditional Privacy Preserving Generalized Ring Signcryption Scheme for Micro Aerial Vehicles.

Micro Aerial Vehicles (MAVs) are a type of UAV that are both small and fully autonomous, making them ideal for both civilian and military applications. Modern MAVs can hover and navigate while carrying several sensors, operate over long distances, and send data to a portable base station. Despite their many benefits, MAVs often encounter obstacles due to limitations in the embedded system (such as memory, processing power, energy, etc.). Due to these obstacles and the use of open wireless communication channels, MAVs are vulnerable to a variety of cyber-physical attacks. Consequently, MAVs cannot execute complex cryptographic algorithms due to their limited computing power. In light of these considerations, this article proposes a conditional privacy-preserving generalized ring signcryption scheme for MAVs using an identity-based cryptosystem. Elliptic Curve Cryptography (ECC), with a key size of 160 bits, is used in the proposed scheme. The proposed scheme's security robustness has been analyzed using the Random Oracle Model (ROM), a formal security evaluation method. The proposed scheme is also compared in terms of computation cost, communication cost and memory overhead against relevant existing schemes. The total computation cost of the proposed scheme is 7.76 ms, which is 8.14%, 5.20%, and 11.40% schemes. The results show that the proposed scheme is both efficient and secure, proving its viability.

A Formal Performance Evaluation Method for Customised Plug-and-Play Manufacturing Systems Using Coloured Petri Nets.

Recent technological advancements and the evolution of industrial manufacturing paradigms have substantially increased the complexity of product-specific production systems. To reduce the time cost of modelling and verification and to enhance the degree of uniformity in the modelling process of system components, this article presents a componentised framework for domain modelling and performance analysis based on the concept of “multi-granularity and multi-view” for a production line of personalised and customised products, for plug-and-play manufacturing processes to involving a large number of model input parameters. The coloured Petri net tool is utilised as a simulation tool for mapping domain models to computational models for simulation and performance evaluation. This paper presents a method for setting the input parameters of a production system when using WIP, through-put and cycle time as metrics. The results of the performance analysis demonstrate the applicability of the proposed framework and provide direction for the production line’s layout design and scheduling strategy.

Rigorously interpreted quotation analysis for evaluating causal loop diagrams in late‐stage conceptualization

AbstractAs a field, system dynamics has made more progress in developing formal model evaluation methods that use quantitative data than ones that use qualitative data. This article describes a formal method for evaluating a causal loop diagram (CLD) in late‐stage conceptualization – referred to as rigorously interpreted quotation analysis – and illustrates its application in a case study. The method uses a systematic and explicit interpretive process to confirm or disconfirm all diagram elements in a CLD by comparing it to stakeholders’ verbatim descriptions of their experiences in a complex dynamic situation. In so doing, this method enables the resolution of discrepancies between a CLD and qualitative data, building confidence in the structural aspects of a dynamic hypothesis. It does so via a process that is approachable for experts and stakeholders alike. © 2022 System Dynamics Society.

Policy Analysis of Sustainable Private Forest Partnership

Private forest partnership has been considered as a new strategy in dealing with conventional socio-economic model problem of the industrial wood raw material sustainability supply. As a new strategy to address the issue of sustainability of industrial wood supply, synergy is needed between forest farmer groups and timber-based industry through private forest partnership schemes. This research aims to analyze private forest partnership policy and its implementation in terms of cooperation aspects in order to formulate a sustainable private forest partnership policy. Content analysis was used to identify all policies related to private forestry partnerships, while policy implementation was analyzed by using formal evaluation method. The descriptive method was implemented to analyze standing stock managemet, and Analytical Hierarchy Process (AHP) as a method to formulate the concept of sustainable private forest partnership policy. The results showed that private forest partnership policy now is more focused on production facilities and infrastructure aspect. The level of policy implementation in research location is at a low grade, only reached 16.28%, while 43.75% of the private forest partnerships in the research area are not sustainable due to low potential of standing stock. This research proved that fostering the farmers is a key factor to develop sustainable private forest partnership policy.

A hierarchical formal method for performance evaluation of WSNs protocol

The design and the evaluation of communication protocols in WSNs is a crucial issue. Generally, researchers use simulation methods to evaluate them. However, formal modelling and analysis techniques are an efficient alternative to simulation methods. Indeed, these techniques allow performance evaluation and model verification. In this paper, a formal approach is proposed to modelling and to evaluating the Carrier Sense Multiple Access with Collision Avoidance (CSMA/CA) MAC protocol with a star topology. Moreover, the proposed approach deals with some properties that are not stated in most existing works. The approach uses Hierarchical Timed Coloured Petri Nets (HTCPNs) formalism to model the protocol and exploits the CPN-Tools to analyse the generated models. HTCPNs provide timed aspect which facilitates the consideration of time constraints inherent to the CSMA/CA protocol.

You Cannot Manage What You Cannot Measure—Developing Central Sterilizing Performance Metrics

Background: Metrics are critical to healthcare, particularly in areas that are highly process oriented. Our hospital’s central sterilizing department functions on standardized processes that have almost no industry specific measurements. Furthermore, the department had no formal internal method for process performance evaluation. A multiphase 6 Sigma DMAIC project was initiated in February 2019 in the central sterilizing department, with ‘phase 1’ being the development of performance metrics to evaluate current state and track future improvement. Methods: A DMAIC team was formed and completed a data inventory that included identifying all existing data sources pertaining to central sterilizing. Sources identified as reliable included count of daily surgical cases, counts of defects based on surgical services communication logs, number of surgical trays processed in the central sterilizing department, and the number of unsterile trays in the department at the start of each 7:00 a.m. shift. Results: A multidisciplinary team including surgeons and senior leadership formed the DMAIC team. Using identified data sources and input from frontline staff and leadership, official metric definitions were created. Weekly defect rates were chosen as a measurement and translated into the number of ready-for-use surgical trays at the time of case setup for every 100 surgical cases. Data from November 2018 through January 20, 2019, demonstrated the mean ready-for-use surgical trays at the time of case setup: for every 100 surgical cases, 96 the surgical trays was ready for use. Further analysis of the data over time demonstrated that variability was out of control. Defects were stratified into 12 categories. Moreover, 6 defects were creating 80% of the issues in trays not being ready: ‘hole in wrapper,’ ‘biohazardous debris,’ ‘nonbiohazardous debris,’ ‘missing instrument,’ ‘incorrect/mislabeled instrument trays,’ and ‘missing indicators.’ Results for unsterile trays at 7:00 a.m. showed that, on average, the central sterilizing department had 100 or more trays at the start of each 7:00 a.m. shift. Conclusions: Without a formal method for data collection, defined metrics, and ongoing analysis, evaluation of performance is based on anecdotal conclusions resulting in missed opportunities for improvement and, subsequently, opportunities for improved patient safety. Regularly published visible dashboards provide a true picture of performance, allowing staff to identify unwanted variation and to put interventions in place so that future variation is better controlled and or prevented. Additionally, having solid metrics, even when no industry ones exist, aid in measuring the effectiveness of improvement efforts, such as the DMAIC project, as they move into subsequent phases.Funding: NoneDisclosures: None

Novel Language Resources for Hindi: An Aesthetics Text Corpus and a Comprehensive Stop Lemma List

This paper is an effort to complement the contributions made by researchers working toward the inclusion of non-English languages in natural language processing studies. Two novel Hindi language resources have been created and released for public consumption. The first resource is a corpus consisting of nearly thousand pre-processed fictional and nonfictional texts spanning over hundred years. The second resource is an exhaustive list of stop lemmas created from 12 corpora across multiple domains, consisting of over 13 million words, from which more than 200,000 lemmas were generated, and 11 publicly available stop word lists comprising over 1000 words, from which nearly 400 unique lemmas were generated. This research lays emphasis on the use of stop lemmas instead of stop words owing to the presence of various, but not all morphological forms of a word in stop word lists, as opposed to the presence of only the root form of the word, from which variations could be derived if required. It was also observed that stop lemmas were more consistent across multiple sources as compared to stop words. In order to generate a stop lemma list, the parts of speech of the lemmas were investigated but rejected as it was found that there was no significant correlation between the rank of a word in the frequency list and its part of speech. The stop lemma list was assessed using a comparative method. A formal evaluation method is suggested as future work arising from this study.

Training for Cost-Effectiveness vs. Training as Tax Evasion Tool – A Study on Selected Private Companies in Bangladesh

Bangladeshi organizations consider employee training as an indispensable strategic tool for enhancing employee’s performance. Although training seems to be an expense, it is regarded as an investment. Every company is increasing the amount of the budget on training programs showing the reason that training brings the competitive edge. The study is focused on analyzing the tendency of increasing the training budget to know whether it is an investment or an evasion medium of tax in Bangladesh. A survey has been administered to find the degree of effectiveness of training programs, the utilization of budget on training, the cost-effectiveness of training programs and the possibility of training programs used as a tax evasion medium in Bangladesh. We have collected data through a semi-structured questionnaire from a sample of 200 employees working in 20 private organizations located in Dhaka city. We have observed from our survey that most of the time training programs held in different organizations in Bangladesh are not always effective. Trainees are not getting the most out of it. Contrary to the expectation, the analysis showed that the training budget is not sufficient. Although companies are increasing the amount of the training budget every year, it is not enough to give best results. And the budget is not utilized effectively in many cases. For this, many company’s performance in cost-effectiveness of training is not satisfactory. The formal evaluation method is not followed in every company, and the training programs are not improved always. Every organization doesn’t conduct cost and benefit analysis of training programs. The study has also found that training can be used as a tax evasion tool and in many organizations, it is somewhat being used. Training program opens up the opportunity to include false expenses which increases the total cost. The result of increased expense is a decrease in taxable profit. We have used mean, standard deviation, variance, percentage analysis, hypothesis testing, etc. for statistical analysis.

Improving the Writing Skills of English Learners: An Impact on Student Learning Analysis

In this action research project, English learners’ (ELs) progress is monitored with a variety of formal and informal evaluation methods through the Impact on Student Learning Analysis (ISLA). The purpose of this ISLA is to: a) determine the effect of instruction; b) use assessment methods to guide instruction; and c) communicate the results to multiple audiences. As evidence of this, lessons centered on writing conventions were implemented, with formative writing assessments to guide future lessons. Writing quizzes were given every week to determine the effect of the instruction. As a result of this analysis, it was deduced which formal and informal evaluation methods best worked with these ELs. By directly focusing on writing conventions, all ELs showed an increase in writing convention knowledge and overall writing skills.

Knowledge Management Framework for Emerging Infectious Diseases Preparedness and Response: Design and Development of Public Health Document Ontology.

BackgroundThere are increasing concerns about our preparedness and timely coordinated response across the globe to cope with emerging infectious diseases (EIDs). This poses practical challenges that require exploiting novel knowledge management approaches effectively.ObjectiveThis work aims to develop an ontology-driven knowledge management framework that addresses the existing challenges in sharing and reusing public health knowledge.MethodsWe propose a systems engineering-inspired ontology-driven knowledge management approach. It decomposes public health knowledge into concepts and relations and organizes the elements of knowledge based on the teleological functions. Both knowledge and semantic rules are stored in an ontology and retrieved to answer queries regarding EID preparedness and response.ResultsA hybrid concept extraction was implemented in this work. The quality of the ontology was evaluated using the formal evaluation method Ontology Quality Evaluation Framework.ConclusionsOur approach is a potentially effective methodology for managing public health knowledge. Accuracy and comprehensiveness of the ontology can be improved as more knowledge is stored. In the future, a survey will be conducted to collect queries from public health practitioners. The reasoning capacity of the ontology will be evaluated using the queries and hypothetical outbreaks. We suggest the importance of developing a knowledge sharing standard like the Gene Ontology for the public health domain.

Reporting Usability Defects: A Systematic Literature Review

Usability defects can be found either by formal usability evaluation methods or indirectly during system testing or usage. No matter how they are discovered, these defects must be tracked and reported. However, empirical studies indicate that usability defects are often not clearly and fully described. This study aims to identify the state of the art in reporting of usability defects in the software engineering and usability engineering literature. We conducted a systematic literature review of usability defect reporting drawing from both the usability and software engineering literature from January 2000 until March 2016. As a result, a total of 57 studies were identified, in which we classified the studies into three categories: reporting usability defect information, analysing usability defect data and key challenges. Out of these, 20 were software engineering studies and 37 were usability studies. The results of this systematic literature review show that usability defect reporting processes suffer from a number of limitations, including: mixed data, inconsistency of terms and values of usability defect data, and insufficient attributes to classify usability defects. We make a number of recommendations to improve usability defect reporting and management in software engineering.

Developing a Value Framework: The Need to Reflect the Opportunity Costs of Funding Decisions.

A growing number of health care systems internationally use formal economic evaluation methods to support health care funding decisions. Recently, a range of organizations have been advocating forms of analysis that have been termed "value frameworks." There has also been a push for analytical methods to reflect a fuller range of benefits of interventions through multicriteria decision analysis. A key principle that is invariably neglected in current and proposed frameworks is the need to reflect evidence on the opportunity costs that health systems face when making funding decisions. The mechanisms by which opportunity costs are realized vary depending on the system's financial arrangements, but they always mean that a decision to fund a specific intervention for a particular patient group has the potential to impose costs on others in terms of forgone benefits. These opportunity costs are rarely explicitly reflected in analysis to support decisions, but recent developments to quantify benefits forgone make more appropriate analyses feasible. Opportunity costs also need to be reflected in decisions if a broader range of attributes of benefit is considered, and opportunity costs are a key consideration in determining the appropriate level of total expenditure in a system. The principles by which opportunity costs can be reflected in analysis are illustrated in this article by using the example of the proposed methods for value-based pricing in the United Kingdom.

OP110 Survey Of Health Technology Assessment Evaluation Strategies For Patient And Public Involvement

INTRODUCTION:Although there is increased awareness of patient and public involvement (PPI) among Health Technology Assessment (HTA) organizations, evaluations of PPI initiatives are relatively scarce. Our objective as members of HTAi's Patient and Citizen Involvement Group (PCIG) was to advance understanding of the range of evaluation strategies adopted by HTA organizations and their potential usefulness.METHODS:In March 2016, a survey was sent to HTA organizations through the International Network of Agencies for Health Technology Assessment (INAHTA) and contacts of members of HTAi's PCIG. Respondents were asked about their organizational structure; how patients and members of the public are involved; whether and how PPI initiatives have been evaluated, and, if so, which facilitators and challenges to evaluation were found and how results were used and disseminated.RESULTS:Fifteen programs from twelve countries responded that involved patient (14/15) and members of the public (10/15) in HTA activities. Seven programs evaluated their PPI activities, including participant satisfaction (5/7), process evaluations (5/7) and impact evaluations (4/7). Evaluation results were used to improve PPI activities, identify education and training needs, and direct strategic priorities. Facilitators and challenges revolved around the need for stakeholder buy-in, sufficient resources, senior leadership, and including patients in evaluations. Participants also provided suggestions based on their experiences for others embarking on this work, for example including patients and members of the public in the process.CONCLUSIONS:We identified a small but diverse set of HTA organizations internationally that are evaluating their PPI activities. Our results add to the limited literature by documenting a range of evaluation strategies that reflect the range of rationales and approaches to PPI in HTA. It will be important for HTA organizations to draw on formal evaluation theories and methods when planning future evaluations, and to also share their approaches and experiences with evaluation.

LOOKING BACK ON 5 YEARS OF HORIZON SCANNING IN ONCOLOGY.

The regularly structured adaptation of health technology assessment (HTA) programs is of utmost importance to sustain the relevance of the products for stakeholders and to justify investment of scarce financial resources. This study describes internal adjustments and external measures taken to ensure the Horizon Scanning Programme in Oncology (HSO) is current. Formal evaluation methods comprising a survey, a download, an environmental analysis, and a Web site questionnaire were used to evaluate user satisfaction. The evaluation showed that users were satisfied with HSO outputs in terms of timeliness, topics selected, and depth of information provided. Discussion of these findings with an expert panel led to changes such as an improved dissemination strategy and the introduction of an additional output, that is, the publication of a league table of emerging oncology drugs. The rather high level of international usage and the environmental analysis highlighted a considerable overlap in topics assessed and, thus, the potential for international collaboration. As a consequence, thirteen reports were jointly published based on eleven "calls for collaboration." To further facilitate collaboration and the usability of reports for other agencies, HSO reports will be adjusted according to tools developed at a European level. Evaluation of the impact of HTA programs allows the tailoring of outputs to fit the needs of the target population. However, within a fast developing HTA community, estimates of impact will increasingly be determined by international collaborative efforts. Refined methods and a broader definition of impact are needed to ultimately capture the efficiency of national HTA programs.

Benchmarking reputation systems: A quantitative verification approach

Trust and reputation systems are classes of decision support tools which help detecting malicious behavior based on collecting ratings and opinions. Despite their advantages, these systems are vulnerable to some kinds of attacks in which the attacker can deceive the system using sequences of misleading behaviors. Robustness of reputation systems against these attacks are frequently investigated in the literature. However the existing works usually evaluate the robustness using a qualitative simulation method. Lack of a formal evaluation method and a quantitative measure of robustness make it hard to extend the results and to compare the systems precisely. This paper proposes a quantitative robustness measure for reputation systems based on a formal verification approach. Using the robustness measure and the verification method, a comprehensive benchmarking of a number of well-known reputation systems is presented which includes evaluation of the systems against basic and the worst case attacks. The results are used for ranking and classifying the systems. The studies show that robustness is not an absolute feature of a reputation model, but it also depends on the properties of the environment. The benchmarking results have been also used to indicate the proper environment for each class of systems/attacks.

Disintegration and Coordination of Forecast Parameters of Socio-Economic Development and the Fuel and Energy Balance of the Region

The scope of strategic planning documents for the constituent entities of the Russian Federation, the procedure for their development and coordination on interregional and federal levels, the requirements for their content and conjunction with other long- and medium-term territorial programs are now approved by statute. The article presents the theoretical and methodological problems of detailing and interrelation of the region’s socio-economic development scenarios with the forecast parameters of regional energy consumption based on the fuel and energy balance under conditions of incomplete retrospective information. This situation is typical of the market environment, and some restrictions on access to statistical data are irremovable. This fact reduces the opportunity to apply formal and rigorous evaluation methods and the objectivity level of not only the forecast indicators, but also of the current ones. The coordination of these documents is methodologically and practically relevant due to the relative isolation of their formation process, a different level of detailing of the forecast scenarios, and a lack of the required information. The author uses the energy saving and energy efficiency measurement technique that is based on the structural comparison of performance, current and estimated fuel and energy balances, consistent with the region’s socio-economic development forecast. The author is also concerned with the development of this technique for the purpose of a comparative regional energy consumption analysis in retrospective and predictive periods. Since 2007, the author has been involved in practical calculations within the framework of the state order of the Ministry of Energy, Housing and Utilities of the Sverdlovsk Region. The article describes methodological characteristics of the author’s approaches to the development of variants of the fuel and energy balance, taking into account maintaining the official scenarios of socio-economic development of the region, the errors and the incompleteness of statistical data, and the regulatory requirements pertaining to the quality of forecasts.

ИССЛЕДОВАНИЕ КАЧЕСТВА ОБСЛУЖИВАНИЯ ПОТРЕБИТЕЛЕЙ УСЛУГ: ПРОБЛЕМЫ И МЕТОДОЛОГИЯ

The article presents data from a study of the market of cellular communication of the Samara region and assess the quality of customer service by mobile operators for the period 2010-2012. Providing quality customer service is one of the major factors of competitiveness, enterprise services rendered. In a market economy, the problem of quality of service is the most important factor in improving the standard of living, economic, social and environmental security. The main problem in providing quality customer service system is that the formal criteria and evaluation methods used in the production sphere, do not apply to service businesses due to immateriality as a process of production and sales of services, and the service itself. The findings of the study conclusions and generalizations can be used in practice management companies providing public services provider. DOI: http://dx.doi.org/10.12731/2218-7405-2013-9-85

Workshop Report: Evaluation of Epidemiological Data Consistency for Application in Regulatory Risk Assessment

Epidemiological study results have a key role in the assessment of health risks associated with exposures to chemicals and pollutants, and often serve as the basis for the development of regulatory limits for environmental and occupational health. A key uncertainty in the application of epidemiological study results in risk assessments stems from variability in defining and operationalizing the concept of consistency of findings across studies, with assessments of consistency often a controversial component of risk assessments. Although assessment of consistency of findings across a diverse collection of epidemiological studies is central to evaluating that body of evidence for supporting causal inferences, the variability in definition and formal evaluation methods strongly suggest the need for constructive approaches to consistently and transparently evaluate data consistency. In response to the need to improve approaches to assessing consistency in epidemiological study results, the Johns Hopkins Risk Sciences and Public Policy Institute organized a workshop held in Baltimore, Maryland in September 2010 to identify and discuss key methodological issues, and to develop recommendations for qualitative and quantitative approaches to addressing those issues. A multi-disciplinary approach was utilized for the workshop, involving invited experts from a variety of fields, and the invited participants were drawn from academia, industry, government, and the public interest sectors. This report provides a summary of selected epidemiology methodological issues discussed by the workshop participants and provides the workshop’s key findings and recommendations for future approaches to addressing this issue.

A Formal Method for Evaluation of a Modeled System Architecture

Modeling is a powerful way to represent the desired organization and performance of a particular system and how it will meet the desired system objectives. There is a multitude of modeling methods, but determining whether the completed model effectively represents the desired system organization can be a challenge. System engineers can inspect the modeled system architecture to determine whether it is acceptable, but few formal methods exist to aid in the performance of this task. In practice, engineers apply heuristics and their experience to identify the characteristics of a “good” architecture. By formalizing these characteristics using logical notation, the quality attributes that constitute a “good” system architecture can be quantified and applied to determine the quality of an architectural model. Because these attributes are defined using a general notation, they can be instantiated using many system architecture tools and can be adapted to meet the needs of a specific architectural framework or particular project.