Framework For Systematic Evaluation Research Articles

A hybrid multi-attribute decision-making model for the systematic evaluation of exoticism-themed retail spaces from the perspective of consumer experience

Previous studies have provided abundant evidence confirming the significant impact of the physical environment in retail spaces themed with exoticism on brand establishment, consumer attraction, and stimulation of consumption behavior. However, there has been limited research employing systematic approaches to examine the influence of physical environmental elements on consumer experiences in Exoticism-themed retail spaces. Therefore, this study aims to propose a comprehensive and systematic evaluation framework from the perspective of consumer experience. It integrates the Caption Evaluation Method (CEM), Kano model, and Decision Making Trial and Evaluation Laboratory (DEMATEL) technique into a Multiple Attribute Decision Making (MADM) model. This study selects cases from China to demonstrate the superiority of the model. The research findings suggest that in the development and renovation process of integrating exoticism elements into retail environments, more attention should be paid to infrastructure such as street layout and facilities. Real plants, in particular, have the greatest primary influence on other elements. Theoretically, it offers a novel perspective on decision-making concerning the enhancement of the physical environment system in Exoticism-themed retail spaces. Practically, the analytical model developed in this study can provide decision-making insights and experiential references for similar themed retail spaces globally.

A low-carbon evaluation framework for regional power systems

With the escalating challenges of climate change and the imperative for carbon neutrality, evaluating the carbon footprint of regional power systems becomes crucial. To address this issue, a low-carbon evaluation framework for regional power systems is developed. First, this study identifies the fundamental requirements that the index system for the low-carbon power system should follow and specifies the general direction for the index system’s creation. Second, an evaluation model for carbon reduction ability is proposed by considering generation side, grid side and load side of the power system. Then, a comparative evaluation model for the carbon reduction capability of multi-regional power systems is proposed to spatiotemporally compare the carbon reduction capability of different power systems. Next, a quantification method for the weights of low-carbon evaluation indicators is established based on the power system simulation of multi-scenario scheduling and index sensitivity analysis method. Finally, a low-carbon comprehensive score for regional power systems is created by combining the evaluation results of the system’s carbon reduction capacity with index weights. The practical research results in a region of Zhejiang province in China demonstrate that the proposed model can provide a reasonable and feasible evaluation plan for the low-carbon construction of regional power systems.

Evaluation of McCauley et al.: A framework for systematic evaluation of tissue signaling responses by single-cell RNA-seq

One snapshot of the peer review process for "A map of signaling responses in the human airway epithelium" (McCauley etal., 2024).1.

Evaluation Framework for Multi-Modal Public Transport Systems Based on Connectivity and Transfers at Stop Level

Multi-modal public transport (PT) networks within metropolitan areas are often characterized by complexity resulting mainly from their infrastructure, design, operations, and demand. This complexity leads to a significant amount of effort on behalf of the transit agencies to properly evaluate their performance at certain locations and proceed with improvements. This study proposes a methodology based on clustering techniques that facilitates the evaluation of PT networks. The evaluation framework refers to the comparison between the levels of supply and demand at a certain stop. Service supply is quantified through an existing connectivity index, whereas demand is considered through the number of transfers that are performed at each stop. Transfers are critical within multi-modal mobility and often serve as a hindrance for choosing PT. The case study here is the Helsinki PT network in Finland. General Transit Feed Specification (GTFS) data are used for quantifying connectivity and a dataset deriving from smartphone ticketing application for quantifying transfers. Results include the evaluation for each PT mode and for the overall multi-modal PT network. Focusing on the evaluation of the overall multi-modal PT network, connectivity and transfers levels for 75.60% of stops are found to be well aligned. Therefore, these stops could be eliminated from the list of candidate stops for performing improvements. Of the remaining stops, 19.73% belongs to the case of higher connectivity than transfers and 4.67% to the case of lower connectivity than transfers. Stops included in these two cases require further attention and prioritization during planning processes.

A Practical Framework for Novel Electronic Nicotine Delivery System Evaluation: Chemical and Toxicological Characterization of JUUL2 Aerosol and Comparison with Reference Cigarettes.

Electronic nicotine delivery systems (ENDSs) are designed as a non-combustible alternative to cigarettes, aiming to deliver nicotine without the harmful byproducts of tobacco combustion. As the category evolves and new ENDS products emerge, it is important to continually assess the levels of toxicologically relevant chemicals in the aerosols and characterize any related toxicology. Herein, we present a proposed framework for characterizing novel ENDS products (i.e., devices and formulations) and determining the reduced risk potential utilizing analytical chemistry and in vitro toxicological studies with a qualitative risk assessment. To demonstrate this proposed framework, long-term stability studies (12 months) analyzing relevant toxicant emissions from six formulations of a next-generation product, JUUL2, were conducted and compared to reference combustible cigarette (CC) smoke under both non-intense and intense puffing regimes. In addition, in vitro cytotoxicity, mutagenicity, and genotoxicity assays were conducted on aerosol and smoke condensates. In all samples, relevant toxicants under both non-intense and intense puffing regimes were substantially lower than those observed in reference CC smoke. Furthermore, neither cytotoxicity, mutagenicity, nor genotoxicity was observed in aerosol condensates generated under both intense and non-intense puffing regimes, in contrast to results observed for reference cigarettes. Following the proposed framework, the results demonstrate that the ENDS products studied in this work generate significantly lower levels of toxicants relative to reference cigarettes and were not cytotoxic, mutagenic, or genotoxic under these in vitro assay conditions.

Evaluation framework for autonomous systems: the case of Programmable Electronic Medical Systems

Evaluation framework for autonomous systems: the case of Programmable Electronic Medical Systems

A qualitative analysis of the key contextual factors for the scaling-up of the ADLIFE toolbox in the Basque Health Service

Introduction: The translation of new digital solutions in different settings requires a preliminary analysis of contextual factors. The results might contribute to the development of strategies addressing barriers, for a successful implementation.
 ADLIFE is an EU funded project to improve the quality of life of older people with advanced chronic diseases by providing integrated personalised care via a toolbox including innovative digital solutions. The contextual factors of the participating organizations were evaluated prior to implementing this toolbox to identify the most relevant ones for the translation of this innovation action into routine practice.
 To structure this evaluation process, a bespoke framework for implementation assessment was developed, based on the Health Information System evaluation framework HOT-fit and the Consolidated Framework for Implementation Research (CFIR). These frameworks provide a structured and systematic way to identify constructs influencing the implementation of ADLIFE.
 This work presents the analysis of the contextual factors that would influence the uptake and effectiveness of ADLIFE tools in the Basque Health Service.
 Methods: Semi-structured interviews were conducted with different stakeholders before the implementation of the ADLIFE toolbox. A standardized set of questions was utilized, based on the main dimensions of the developed framework (human, organization, technology).
 These interviews were conducted online. They were recorded, transcribed, pseudonymized and subsequently analyzed using qualitative content analysis. These analyses were conducted using the software MAXQDA.
 Results: A total of 9 interviews were conducted: 2 primary care physicians, 2 specialized care physicians, 2 nurses, 2 IT staff and a manager.
 Through the implementation of the ADLIFE toolbox, interviewees expect improvements in communication between professionals and with patients, as well as facilitation of patient empowerment.
 Training about the new systems and time to get used to them were considered essential, which would help to overcome the digital gap that currently influences the lack of adherence to this type of intervention.
 Finally, direct linkage between the existing and the new systems were considered key to motivate professionals to participate and to facilitate the integration of patient care.
 Discussion and conclusions: The scaling up of the ADLIFE toolbox in the Basque Health Service requires it to be integrated into the digital systems currently used for the management of the Electronic Health Record. ADLIFE can serve to promote an improvement, not only in quality of life, but also in the quality of care, provided that professionals who care for these patients are involved.
 This knowledge will help to establish the most effective strategies to facilitate the intervention in the Basque Health Service.
 Acknowledgements
 We are grateful for the participation of the professionals who were interviewed.
 ADLIFE (GA 875209) received funding from the EU Horizon 2020. The authors thank all partners within ADLIFE.

Evaluation of UN SDG-related formal learning activities in a university common core curriculum

Purpose Higher education plays an essential role in achieving the United Nations sustainable development goals (SDGs). However, there are only scattered studies on monitoring how universities promote SDGs through their curriculum. The purpose of this study is to investigate the connection of existing common core courses in a university to SDG education. In particular, this study wanted to know how common core courses can be classified by machine-learning approach according to SDGs. Design/methodology/approach In this report, the authors used machine learning techniques to tag the 166 common core courses in a university with SDGs and then analyzed the results based on visualizations. The training data set comes from the OSDG public community data set which the community had verified. Meanwhile, key descriptions of common core courses had been used for the classification. The study used the multinomial logistic regression algorithm for the classification. Descriptive analysis at course-level, theme-level and curriculum-level had been included to illustrate the proposed approach’s functions. Findings The results indicate that the machine-learning classification approach can significantly accelerate the SDG classification of courses. However, currently, it cannot replace human classification due to the complexity of the problem and the lack of relevant training data. Research limitations/implications The study can achieve a more accurate model training through adopting advanced machine learning algorithms (e.g. deep learning, multioutput multiclass machine learning algorithms); developing a more effective test data set by extracting more relevant information from syllabus and learning materials; expanding the training data set of SDGs that currently have insufficient records (e.g. SDG 12); and replacing the existing training data set from OSDG by authentic education-related documents (such as course syllabus) with SDG classifications. The performance of the algorithm should also be compared to other computer-based and human-based SDG classification approaches for cross-checking the results, with a systematic evaluation framework. Furthermore, the study can be analyzed by circulating results to students and understanding how they would interpret and use the results for choosing courses for studying. Furthermore, the study mainly focused on the classification of topics that are taught in courses but cannot measure the effectiveness of adopted pedagogies, assessment strategies and competency development strategies in courses. The study can also conduct analysis based on assessment tasks and rubrics of courses to see whether the assessment tasks can help students understand and take action on SDGs. Originality/value The proposed approach explores the possibility of using machine learning for SDG classifications in scale.

Park Inclusive Design Index as a Systematic Evaluation Framework to Improve Inclusive Urban Park Uses: The Case of Hangzhou Urban Parks

This study aims to optimize the evaluation system of inclusive design in urban parks, emphasizing the systemic nature of sensory, cognitive, and motor capacity support and exploring its role in park design practice. Based on the capability demand model, this study constructed indicators through literature collation and focus group discussion and assigned weights through hierarchical analysis to finally construct the Park Inclusive Design Index (PIDI). Then, the PIDI was utilized to assess the inclusive design performance of 48 urban parks in Hangzhou, China. The results of this study show that the overall inclusive design level of parks is relatively low (the average PIDI < 70), especially in the provision of cognitive support (cognitive-related indicator < 4). Meanwhile, comprehensive and specialized parks performed better in inclusive design compared to community parks and leisure parks. The level of inclusive design is moderatory correlated with the park renovation time and the park area, and strongly correlated with geographic location (scenic spot parks perform better; the parks in the old city perform worse). Ten indicators in the assessment scored below 2, which reveals the current status, shortcomings, and general problems with inclusive facilities in Hangzhou’s urban parks. This study integrated the needs and ability differences of people into the indicators, providing an assessment framework with broad applicability. Inclusive performance is a long-term process, and the implementation of the evaluation framework will provide a reference guide for the design, construction, operation, and maintenance of urban parks across China and even around the world.

European Artificial Intelligence Act: an AI security approach

Purpose The purpose of this paper is to highlight the key technical challenges that derive from the recently proposed European Artificial Intelligence Act and specifically, to investigate the applicability of the requirements that the AI Act mandates to high-risk AI systems from the perspective of AI security. Design/methodology/approach This paper presents the main points of the proposed AI Act, with emphasis on the compliance requirements of high-risk systems. It matches known AI security threats with the relevant technical requirements, it demonstrates the impact that these security threats can have to the AI Act technical requirements and evaluates the applicability of these requirements based on the effectiveness of the existing security protection measures. Finally, the paper highlights the necessity for an integrated framework for AI system evaluation. Findings The findings of the EU AI Act technical assessment highlight the gap between the proposed requirements and the available AI security countermeasures as well as the necessity for an AI security evaluation framework. Originality/value AI Act, high-risk AI systems, security threats, security countermeasures.

CRS-Que : A User-Centric Evaluation Framework for Conversational Recommender Systems

An increasing number of recommendation systems try to enhance the overall user experience by incorporating conversational interaction. However, evaluating conversational recommender systems (CRSs) from the user’s perspective remains elusive. The GUI-based system evaluation criteria may be inadequate for their conversational counterparts. This paper presents our proposed unifying framework, CRS-Que , to evaluate the user experience of CRSs. This new evaluation framework is developed based on ResQue , a popular user-centric evaluation framework for recommender systems. Additionally, it includes user experience metrics of conversation (e.g., understanding, response quality, humanness) under two dimensions of ResQue (i.e., Perceived Qualities and User Beliefs). Following the psychometric modeling method, we validate our framework by evaluating two conversational recommender systems in different scenarios: music exploration and mobile phone purchase . The results of the two studies support the validity and reliability of the constructs in our framework and reveal how conversation constructs and recommendation constructs interact and influence the overall user experience of the CRS. We believe this framework could help researchers conduct standardized user-centric research for conversational recommender systems and provide practitioners with insights into designing and evaluating a CRS from users’ perspectives.

A novel evaluation framework for recommender systems in big data environments

A novel evaluation framework for recommender systems in big data environments

A multi-attribute decision-making framework for enterprise competitive intelligence system evaluation with 2-tuple linguistic neutrosophic information

Big data is changing our lives and the way we understand the world, as well as the operational patterns of business and social organizations. Fully understanding the value of data and knowing how to use big data to provide a basis for business decision-making has gradually become the most basic thinking that business organizations should possess in the era of big data. Under the thinking mode of data-driven decision-making, many information science researchers have discussed the model, architecture, operation mechanism and other aspects of big data competitive intelligence system. At the same time, more and more enterprises, such as IBM, Amazon, Google, Microsoft, Wal Mart, etc., have begun to attach importance to the development and construction of big data competitive intelligence software systems, and have achieved certain results. The enterprise competitive intelligence system evaluation in the context of big data is regarded as multi-attribute decision-making (MADM). In this paper, the Hamy mean (HM) and the power average (PA) are connected with 2-tuple linguistic neutrosophic sets (2TLNSs) to propose the 2-tuple linguistic neutrosophic numbers power HM (2TLNPHM) operator. Then, use the 2TLNPHM operator to handle MADM with 2TLNS. Finally, taking the enterprise competitive intelligence system evaluation in the context of big data as an example, the proposed method is explained. The main contributions of this study are summarized: the establishment of the 2TLNPHM operator; (2) The 2TLNPHM operator was developed to handle MADM with 2TLNS; (3) Through the empirical application of the enterprise competitive intelligence system evaluation, the proposed method is validated; (4) Some comparative studies have shown the rationality of the 2TLNPHM operator.

ESICCC as a systematic computational framework for evaluation, selection, and integration of cell-cell communication inference methods.

Cell-cell communication (CCC) is critical for determining cell fates and functions in multicellular organisms. With the advent of single-cell RNA-sequencing (scRNA-seq) and spatial transcriptomics (ST), an increasing number of CCC inference methods have been developed. Nevertheless, a thorough comparison of their performances is yet to be conducted. To fill this gap, we developed a systematic benchmark framework called ESICCC to evaluate 18 ligand-receptor (LR) inference methods and five ligand/receptor-target inference methods using a total of 116 data sets, including 15 ST data sets, 15 sets of cell line perturbation data, two sets of cell type-specific expression/proteomics data, and 84 sets of sampled or unsampled scRNA-seq data. We evaluated and compared the agreement, accuracy, robustness, and usability of these methods. Regarding accuracy evaluation, RNAMagnet, CellChat, and scSeqComm emerge as the three best-performing methods for intercellular ligand-receptor inference based on scRNA-seq data, whereas stMLnet and HoloNet are the best methods for predicting ligand/receptor-target regulation using ST data. To facilitate the practical applications, we provide a decision-tree-style guideline for users to easily choose best tools for their specific research concerns in CCC inference, and develop an ensemble pipeline CCCbank that enables versatile combinations of methods and databases. Moreover, our comparative results also uncover several critical influential factors for CCC inference, such as prior interaction information, ligand-receptor scoring algorithm, intracellular signaling complexity, and spatial relationship, which may be considered in the future studies to advance the development of new methodologies.

Value of open research data: A systematic evaluation framework based on multi-stakeholder survey

Stakeholders such as funders, data centers and data curators need specific and detailed evidence to support and justify the value of their existing or proposed open data policies and practices. A questionnaire-based evaluation framework was designed through systematic review and expert interviews, focusing on the scientific, economic, and societal values of open research data. Specifically, scientific value was sub-categorized into 24 statements centering around research process, quantity, quality, efficiency, and impact. Economic value included 6 statements related to cost savings and increased returns, while societal value was divided into 7 statements focused on efficiency and quality. The questionnaire survey was conducted among 219 stakeholders from 22 countries/regions. On average, participants (strongly) agreed that open research data has scientific (83.97%), economic (76.94%), and societal (86.89%) value, all 37 statements of the framework were supported and confirmed. This framework is one of the most comprehensive and systematic tools for measuring data value, which can offer support to diverse stakeholders, especially funders, data centers, and data curators in managing and promoting open data systems.

Towards Workplace Metaverse: A Human-Centered Approach for Designing and Evaluating XR Virtual Displays

Work is becoming more and more flexible nowadays. It can take place in the office, at home, or even on the go; tasks may encompass activities in the form of documents, multimedia, or 3D models in virtual space. Under such circumstances, personal computers (PC), the most widely used productivity devices for work today, cannot well address the diverse needs such as screen size, privacy, and flexibility to display diverse content formats (e.g., 2D to 3D), due to their fixed hardware specs. We believe the solution lies in Extended Reality (XR). In this article, we explored how XR glasses can be used for PC’s virtual extended displays, and conducted user interviews and usability tests to propose a systematic user experience design and evaluation framework. We discovered that the design space encompasses four dimensions general placement, display specs, operating system integration, and interaction behaviors) and summarized users’ corresponding preferences. We proposed a quality-of-experience (QoE) evaluation framework for XR virtual displays consisting of visual quality, visual fatigue and discomfort, as well as immersiveness, and identified clarity as the most significant factor that affects user satisfaction. Our design and evaluation frameworks could serve as a resource for both practitioners and scholars with an interest in the design and evaluation of virtual displays.

Assessing Public Transport Supply Quality: A Comparative Analysis of Analytical Network Process and Analytical Hierarchy Process

This paper delves into the realm of public transport system enhancement, a critical consideration for decision makers due to its profound impact on citizens' lives and government in-vestments. The primary objective is to assess the quality of public bus transport supply and identify the most effective improvements to heighten passenger satisfaction and attract new users. To accomplish this goal, two prominent multi-criteria decision-making approaches, namely the Analytical Network Process (ANP) and the Analytical Hierarchy Process (AHP), were employed, leveraging a dynamic questionnaire survey. The ANP method, recognized for its robustness, takes into account the interrelationships and feedback among various criteria levels, offering a systematic evaluation framework. In contrast, the AHP method overlooks these factors. The adoption of both methods was crucial in obtaining a comprehensive under-standing of experts' perceptions regarding public transport service quality. To illustrate the practical implementation of these approaches, an empirical study was conducted using a re-al-life case. This study serves as a testament to the efficacy of these decision-making methods and underscores their value in the decision-making process. Ultimately, this paper under-scores the significance of prioritizing public transport system improvements as a means to en-rich citizens' lives and bolster government investments.

A Framework of Dependence Modeling and Evaluation System for Compound Flood Events

AbstractThe coincidence and superposition of flood processes from different rivers and regions tend to form compound flood events, determined by spatial relationship between diverse flood processes that cannot be accurately depicted and evaluated by existing dependence analysis methods. A framework, integrating multi‐dimensional vine copula model and dependence evaluation system, was developed with a testing‐oriented application to explore underlying dependence between two kinds of extreme runoff series (peak discharge and flood volume) extracted from the identified compound flood events in the upper reaches of the Yangtze River. Multi‐dimensional regular vine (R‐vine) copula models were established to depict the complex and diverse dependence, and corresponding vine structure was specified by the vine structure array that can reflect the sequence of tributaries flowing into the main stream and the spatial locations of different hydrometric stations. Dependence magnitude and association status were calculated and compared according to the optimal R‐vine copula models and information theory. Comparison with existing methods demonstrated that dependence evaluation system could reflect nonlinear and local dependence characteristics and eliminate the effect of extreme runoff series from other hydrometric station on the dependence. The association status between different extreme runoff series of the upper Yangtze River and its tributaries were diverse in view of the impact of tributary flood inflow. The proposed framework can be regarded as an effective way for dependence modeling and evaluation of compound floods, thus providing a scientific reference for the risk analysis of water resources systems.

Research on the design of costume performance professional evaluation system based on information management

In the continuous improvement of the level of computer technology, the education field proposed to use information technology to build a professional evaluation system, its purpose is to replace the traditional manual management mode, improve the efficiency of practical work, shorten the practical work cycle. According to the application of the educational information evaluation system in recent years, it can not only provide technical support for information statistics and assessment work, but also ensure that the practical operation is simple and fast and the performance is outstanding. Therefore, on the basis of understanding the current situation of costume performance education in modern colleges and universities, and according to the technical theory of developing educational informatization evaluation system, this paper deeply discusses the structural framework of costume performance evaluation system with informatization management as the core, and conducts systematic evaluation with practical cases. The final results show that the overall system design meets the needs of educational management of costume performance specialty, which is of great significance to the innovation and development of college education.

Circular economy initiatives are no guarantee for increased plastic circularity: A framework for the systematic comparison of initiatives

Plastic plays a prominent role within circular economy, with many stakeholders promoting initiatives to increase plastic circularity during the manufacturing, use and end-of-life phases. Despite well-meant intentions, many initiatives are characterized by a lack of compliance with basic circular economy principles, implementation barriers, and limited effects on the intended long-term plastic circularity. This study provides a systematic evaluation framework for comparison of plastic initiatives, based on 17 criteria addressing key aspects of plastic circularity. A three-level likelihood ranking approach is applied to analyse the impact of 54 initiatives targeting plastic circularity in a European context. It was found that relatively few of these initiatives were readily implementable without considerable investments, e.g. in new waste management and recycling technologies, and changes in plastic production and product design. The results clearly suggest that current suggestions for circular economy initiatives targeting plastic may have limited effect and not lead to the intended impacts without the support of new regulations and change in plastic demand and consumption. The study stresses the importance of synergies and cooperation between stakeholders across the value chain to reach plastic circularity. The framework offers a consistent basis for decision-makers to identify critical barriers and enablers in relation to plastic circularity characteristics, but the approach may also be applied to other topic areas.