Practical Evaluation Framework Research Articles

Optimization modeling and verification from problem specifications using a multi-agent multi-stage LLM framework

This paper explores the use of Large Language Models (LLMs) in modeling real-world optimization problems. We concretely define the task of translating natural language descriptions into optimization models (NL2OPT) and provide criteria for classifying optimization problems for the NL2OPT task. Our novel multi-agent modeling framework leverages relations identifier agents and a multi-agent verification mechanism, eliminating the need for solver execution. Additionally, we introduce a straightforward and practical evaluation framework, offering a more effective assessment method compared to traditional execution-based evaluations. We have created a unique dataset tailored for optimization modeling, featuring Problem Specifications as a structured representation of optimization problems. Through comprehensive experiments, our study compares our modeling framework with existing LLM reasoning strategies, highlighting their relative effectiveness in optimization modeling tasks. We also perform ablation studies to explore the effect of different components of our modeling framework. Experimental results demonstrate that our multi-agent framework outperforms many common LLM prompting strategies.

Unboxing the blackbox of human-centric and inclusive smart city initiatives: a theory based evaluation framework

ABSTRACT The concept of smart cities has evolved from a technology and data-driven approach to a more human-centred and inclusive model. This paper explores the shift in smart city development towards a focus on human-centric and inclusive solutions while suggesting a novel evaluation framework for these smart city initiatives. Through a comprehensive literature review, the study identifies key challenges in evaluating smart city initiatives including the lack of standardized evaluation frameworks, limited focus on social and environmental outcomes, and the need for citizen participation. To address these challenges, the paper proposes a theoretical framework grounded in a realist evaluation approach. This framework incorporates a Theory of Change (ToC) that outlines the causal pathways and contextual factors underlying smart city initiatives. The proposed framework is designed to guide the evaluation of smart city initiatives by providing specific evaluation criteria and sub-criteria that are aligned with the objectives of inclusive and sustainable urban development. By highlighting the importance of human-centric and inclusive approaches and providing a practical evaluation framework, the paper aims to contribute to the ongoing discourse on smart cities and their evaluation.

Neutrosophic CEBOM-MACONT model for sustainable management of end-of-life tires

Management of end-of-life tires (ELTs) has evolved into an important sustainability requirement that should follow circular economy principles. It is increasingly important to find environmentally-friendly and cost-effective solutions for ELT management, particularly in the context of large freight transportation companies. Selecting the most sustainable solution from the set of available ELT management strategies, such as retreading, recycling, energy recovery, and landfilling, presents a decision-making challenge for authorities. This study aims to introduce a practical evaluation framework comprised of strategy alternatives and key decision-making criteria to support transportation companies in managing ELT flows. Also, the research introduces an advanced two-stage neutrosophic decision support model to solve the addressed problem and reveal the most sustainable strategy. The model is based on the integration of the cross-entropy-based optimization model (CEBOM) method and mixed aggregation by comprehensive normalization technique (MACONT) under the type-2 neutrosophic number (T2NN) environment. Prominent features of T2NN-CEBOM are hybrid weighting sub-framework and processing controllability. Distinguished characteristics of T2NN-MACONT are average referencing, triple-normalization support, modeling of risk attitude behavior, adjustable ordering schemes, and an advanced scoring system. The real-life study of one of the largest German freight transportation companies that operates along an important European transit route offers practical insights for decision-makers when evaluating ELT management strategies. The research findings show that retreading is the most sustainable solution. The eight sensitivity analyses confirm the high robustness of the introduced decision-support model. The comparative analysis reveals the superiority of T2NN-CEBOM-MACONT for employment in practical settings.

Neutrosophic LOPCOW-ARAS model for prioritizing industry 4.0-based material handling technologies in smart and sustainable warehouse management systems

Industry 4.0 technologies embedded in the warehouse management system (WMS) are needed to improve the automation of material handling activities such as receiving, storing, picking, sorting, packaging, and delivering. This research aims to introduce a neutrosophic multi-criteria group decision-making tool that is intelligible in supporting the transition and upgrading of WMS with Industry 4.0-based solutions. This advanced two-stage model is based on the integration of the logarithmic percentage change-driven objective weighting (LOPCOW) method and the additive ratio assessment (ARAS) method under the type-2 neutrosophic number (T2NN) environment. In the first stage, T2NN-LOPCOW generates an objective importance vector of decision-making criteria. In the second stage, T2NN-ARAS based on the generalized weighted Heronian mean operator provides an advantageous order of Industry 4.0-based material handling technologies. T2NN-LOPCOW-ARAS brings the following novelties: ((i) to straightforwardly represent and explore interconnection levels between weights of criteria, ((ii) to provide wide-scoping insight into the stability of initial priority order, as well as a broad spectrum of flexible solutions, ((iii) to control the normalization procedure and minimize distortions due to the double-normalization backbone. The real-life case study of a logistics company from the Serbian grocery retail sector illustrates the practical applicability of T2NN-LOPCOW-ARAS. A practical evaluation framework is defined to comprehensively assess automated guided vehicles (AGVs), collaborative robotics, and drones. The sensitivity analyses show the high robustness of the proposed framework. The comparative investigation shows that T2NN-LOPCOW-ARAS is superior to the extant methods. The research findings show that AGVs are the most favorable Industry 4.0-based material handling solution.

Practical Framework for Evaluation and Improvement of Drinking Water Treatment Robustness in Preparation for Extreme-Weather-Related Adverse Water Quality Events.

Robustness is the ability of a drinking water treatment plant (DWTP) to achieve the desired finished water quality even during adverse raw water quality events. Increasing the robustness of a DWTP is beneficial for regular operations and especially for extreme weather adaptation. This paper proposes three robustness frameworks: (a) a general framework outlining the main steps and methodology for systematic assessment and improvement of the robustness of a DWTP, (b) a parameter-specific framework applying the general framework to a water quality parameter (WQP), and (c) a plant-specific framework applying the parameter-specific framework to a DWTP. A parameter-specific framework for turbidity is presented using the turbidity robustness index (TRI) for evaluation and applied to a full-scale DWTP in Ontario, Canada. This evaluation was conducted with historical plant data, as well as bench-scale experimental data simulating extremely high-turbidity scenarios. The framework application is capable of identifying (i) less robust processes which are likely to be vulnerable during climate extremes, (ii) operational responses to increasing short-term robustness, and (iii) a critical WQP threshold beyond which capital improvements are necessary. The proposed framework provides insights into the current state of robustness of a DWTP and serves as a tool for climate adaptation planning.

Expert-centric design inspection of branded apps: a close look at marketing and interface design features

PurposeAlthough brands have developed mobile applications (apps) to offer consumers new experiences, low app usage numbers indicate the need to develop a systematic, practical evaluation framework for branded app design that specifies concrete design features.Design/methodology/approachAn expert review provides an overview of the design of current branded apps. On the basis of an extensive literature review, this article classifies state-of-the-art design features for branded apps according to a proposed evaluation framework that includes human–computer interaction (HCI)–related and marketing-related evaluation criteria. In an application of these evaluation criteria, the authors evaluate 73 branded apps issued by 11 top fast-moving consumer goods (FMCG) brands.FindingsThe expert review identifies strengths and weaknesses that are common to the design of current branded apps. These findings inform the set of design recommendations that this article offers, which includes 14 features common to all types of apps and 9 features specific to particular types of apps.Practical implicationsThis research offers practical implications for app designers, who need to address design dimensions contained in the proposed framework including the HCI-related (mobile, social and user experience design features) and marketing-related (branding and customer relationship management design features) to create effective branded apps.Originality/valueDesign elements identified in prior literature remain abstract and do not prescribe a systematic or pragmatic approach to using them in practice. This study takes a multidisciplinary perspective (HCI, marketing and design science) to establish a practical evaluation framework for branded app designs.

Evaluation framework for an efficient commuting carpool program

Hundreds of cities have already pledged to get to “net zero” by 2050. MaaS (Mobility as a Service) is a one of promising measures but commuting carpool program seems to be effective for concentrated commuting to factories in suburban areas where public transportation is insufficient. The commuting carpool is easy to understand as a service, but there are considered many parameters for designing it. Especially companies in Japan have a big role to implement its service because collaborations with public and private sectors are not sufficient viewpoint of carpooling infrastructures and financial supports. In this study, we propose an evaluation framework for companies with two phases: Basic design to make a decision for implementation through the feasibility studies, and Detailed design and Improvement to consider detailed implementation.We focus on practical evaluation framework for Basic design and conduct a case study on commuting to a newly constructed factory. We apply it to extract parameters which are necessary for the feasibility study. As for the effect of the introduction of the program, its service is able to reduce the number of vehicles by 30-66% and CO2 emissions by 373-800 tons per year.

Integrating spatial multi-criteria evaluation into the potential analysis of culture-led urban development – A case study of Tainan

Post-1980s, early-developed historical cities attempted to reverse the dilemma of urban modern development and the preservation of historical textures. Culture-led urban development then emerged as a strategy to highlight the advantages of historical cities. However, study that propose practical evaluation framework that integrate culture, historical texture, and relevant resources is less. Furthermore, cultural resources, their characteristics, and their development potential are rarely considered under spatial scales. Therefore, we aim to conduct a spatial potential evaluation framework for culture-led urban development (CLUD), shedding light on the characteristics of the historical and cultural environment and their relevance to social and economic factors. Recent studies suggest that culture-led urban development should focus more on the balance between local cultural context and development planning content, to achieve long-term sustainable development. Consequently, we applied the concept of adaptive development, which emphasizes locally appropriate assess, particularly at a regional level. In this research, a spatial multi-criteria evaluation (SMCE) model was developed for the decision-making process, which can contain multiple aspects of the urban environment and related resources. We apply the established framework to Tainan City, due to its rich cultural resources, high cultural participation and intensive support from the government. The results were calculated in different spatial units in response to various purposes. The findings are divided into several zones performing various objectives that are appropriate for specific places. The outcome demonstrates the possibility of development cooperation within the study region. In the future, the framework and the processes can be applied to evaluate other regions to enable more comprehensive understanding before policy inputs.

E-Voting System Evaluation Based on The Council of Europe Recommendations: Helios Voting

Despite the claimed benefits of e-voting initiatives, wider adoption of e-voting mechanisms and implementation processes is slower than expected. Several technical, social, and cultural challenges hinder generability and applicability of e-voting. Amongst them, the evaluation and harmonization of e-voting systems, given different legal and statutory frameworks, is still an important challenge to overcome. Yet, only a few works have addressed this topic in the field. This article aims to contribute to further understanding this unexplored topic by applying a practical evaluation framework to Helios Voting, one of the most widely used e-voting tools to date. Our framework, strongly based on the technical and security requirements issued by the Council of Europe in 2017, is a valuable source of information for election officials, researchers and voters to understand the strengths and weaknesses of Helios Voting and, as a result, to improve decision-making processes regarding the type and size of elections that can be securely handled by Helios Voting. The ultimate goal of our paper is to conceptually and practically support the gradual, secure and protocolized expansion of e-voting.

Class imbalance in gradient boosting classification algorithms: Application to experimental stroke data.

Imbalance between positive and negative outcomes, a so-called class imbalance, is a problem generally found in medical data. Imbalanced data hinder the performance of conventional classification methods which aim to improve the overall accuracy of the model without accounting for uneven distribution of the classes. To rectify this, the data can be resampled by oversampling the positive (minority) class until the classes are approximately equally represented. After that, a prediction model such as gradient boosting algorithm can be fitted with greater confidence. This classification method allows for non-linear relationships and deep interactive effects while focusing on difficult areas by iterative shifting towards problematic observations. In this study, we demonstrate application of these methods to medical data and develop a practical framework for evaluation of features contributing into the probability of stroke.

Towards a framework for measuring local government return on investment in arts and cultural development

ABSTRACTLocal governments make a significant annual investment in arts and cultural development programmes. However, the practicalities of measuring the return on this investment have been largely overlooked by researchers, and investigations into return on investment have frequently focused on the economic return. Here, it is argued that intrinsic social and cultural returns on investment are equally important. A further issue is that measurement frameworks have often taken an outputs-based approach, seeking to evaluate the performance of one-off events rather than measuring the outcomes or impact of the total investment in arts and cultural development over the life of a policy or strategy. This paper therefore takes a broader, outcomes-based approach and offers a practical evaluation framework. An implementation-ready method is presented which will allow local government policy-makers to provide evidence of medium- and long-term outcomes and impacts of their arts and cultural development programmes and weigh this against their investment.

EVALUATING THE CROSS-NATIONAL TRANSFERABILITY OF POLICIES: A CONCEPTUAL FRAMEWORK

When seeking to harness entrepreneurship and enterprise culture, governments often seek to transfer policy measures successful in another country to their own. Until now however, governments have often lacked a practical evaluation framework for selecting policy measures and then appraising the feasibility and transferability of such measures. The aim of this paper is to fill that gap. Reviewing the literature on cross-national policy transfer, this paper provides a pragmatic evaluation framework for selecting policy measures and appraising their feasibility and transferability from one country to another. This details how successful policy transfer and cross-national policy learning must be informed by prospective policy analysis and testing the features of the specific policy initiative against the specifics of the national context and circumstances, and then establishes the criteria and processes through which potential policy adopters can identify promising policies used elsewhere to tackle similar problems in their own country and assess their 'goodness of fit' prior to transfer to national realities.

Developing a practical evaluation framework for identifying critical factors to achieve supply chain agility

Supply chain agility is a key determinant of competitiveness in today’s dynamic and turbulent business environment. This paper develops a practical evaluation framework that serves to identify critical factors for achieving supply chain agility. First, we construct a reference framework of the factors that contribute to achieving agility in supply chain based on a systematic analysis of the related literature. Then, we develop a hybrid evaluation method that integrates fuzzy logic, DEMATEL (decision making trial and evaluation laboratory), and ANP (analytic network process). The proposed framework is implemented in an automotive company that is seeking to improve agility of its supply chain. It provides a systematic approach to explore and analyze influential relationships between agile-enabling factors and assigns a weight to each factor representing its relative importance. It also captures ambiguities, uncertainties, and vagueness inherent in evaluating the factors and their relationships and provides an efficient mechanism for group evaluation.

Methodology for Evaluating the Influence of Schools’ Sustainability Education Programs on Their Communities

The role of schools as catalysts for change within their communities has been widely emphasized in the context of sustainability education. Yet the literature does not provide sufficient information regarding effective methods for its evaluation. The following article offers a practical evaluation framework for assessing the influence of schools' sustainability education programs on the parents and siblings of school children, as well as on broader circles in the community. The framework is comprised of a set of indicators, methods for data collection, and measurement of influence

Evaluating quality of conceptual modelling scripts based on user perceptions

This paper presents the development of a user evaluations based quality model for conceptual modelling scripts applying Seddon’s variant of the well-known model of DeLone and McLean [W.H. DeLone, E.R. McLean, Information systems success: the quest for the dependent variable, Information Systems Journal 3(1) (1992) 60–95] for evaluating the success of information systems. Given the growing awareness among researchers and practitioners about the importance of high-quality conceptual modelling scripts, it is surprising that there is no practical evaluation framework that considers the quality of conceptual modelling scripts from the user perspective. A first research goal is therefore to determine what the appropriate dimensions are for evaluating the success or quality of conceptual modelling scripts from the user point of view. A second goal is to investigate the relationships between these quality dimensions. The paper also presents an empirical test of the proposed model of quality dimensions and their relationships. Results are presented of two experiments with 187 and 124 business students respectively, that were designed to test a set of hypotheses generated from the proposed model. The results largely support the proposed model and have implications for both theory and practice of quality evaluation of conceptual modelling scripts.