Holistic Evaluation Framework Research Articles

Policy mixes for just transitions: A holistic evaluation framework

In this paper, we develop a holistic policy evaluation framework that aims to harness a fuller potential of just transitions. Although appeals for broader understanding of just transitions are becoming louder, applicable frameworks supporting consideration of justice in the planning and evaluation of transition policies are still lacking. The evaluation framework developed integrates the multidimensional understanding of justice more consistently into the consideration of policy intervention points for sustainability transitions. We test and apply the framework to the discussions concerning just food system transitions in Finland. The results show that justice issues deserve attention across all the policy intervention points, but their relative importance differ. A better understanding of the relative importance of justice concerns is critical when designing policy mixes for more active and emancipatory just transitions governance. We close the paper by explicating further avenues for the application and development of the framework.

Comparative analysis of solar module configuration and tracking systems for enhanced energy generation in South Sakucia Union, Bhola, Bangladesh: A software based analysis

Bangladesh is blessed with an extensive range of solar energy generation possibilities; however, the primary impediment to attaining its full potential in the solar energy industry is the inadequate budget in the energy sector. As a result, determining the most economical and efficient solar module configuration for each specific scenario has become a critical necessity. This study offers a comprehensive techno-economic analysis and environmental impact assessment of four distinct solar modules: monofacial, bifacial, dual-axis solar tracker, and seasonal tilt solar module, in an open area of South Sakucia Union, Bhola district, in the southwest part of Bangladesh. By integrating energy-generation capabilities, financial metrics, and environmental benefits, this research provides a holistic evaluation framework to ensure optimal economic performance and minimal adverse environmental effects for sustainable solar solutions in Bangladesh. Utilizing PV*SOL, PVsyst, and System Advisor Model (SAM) software, this study assesses energy-generation capabilities and economic viability. Despite the dual-axis solar tracker exhibiting the highest average energy generation (149,070.3 kWh/year), its higher initial cost renders it less financially viable compared to other configurations. Financial metrics reveal that the seasonal tilt configuration is the most cost-efficient, with the lowest Levelized Cost of Electricity (LCOE) at $0.0452/kWh and the highest Net Present Value (NPV) of $52,887.70. Additionally, it has the shortest Discounted Payback Period (DPBP) at 12.69 years, a favorable Internal Rate of Return (IRR) of 9.460 %, and a Profitability Index (PI) of 1.459, indicating robust returns on investment. These findings emphasize the importance of considering both energy-generation capabilities and financial metrics when evaluating solar module configurations in the southern part of Bangladesh, serving as a valuable reference for policymakers. Moreover, meticulous environmental impact assessments assist in choosing configurations with minimal adverse effects on the environment.

Towards an evaluation framework for inclusive technological innovation in social and health care services

With the COVID-19 pandemic came an influx of digital technologies introduced into the care home sector. Many care homes were not ready for digital changes, and subsequently, the inclusivity of these innovations was variable. We develop a holistic evaluation framework to assess the inclusivity of service innovations, including the well-being of care home residents and employees. We adopt a mixed methods approach using the ‘non-adoption, abandonment, and challenges to scale-up, spread, and sustainability − complexity assessment toolkit’ (NASSS-CAT) to evaluate the introduction of tablet computers into care homes across Scotland and assess the inclusivity of the innovation. The paper offers a theoretical synthesis by indicating how the seven dimensions of the NASSS-CAT can evaluate the six stages of the ladder of inclusive innovation. Results specify how the care home sector could be more inclusive of its residents and staff by co-creating innovation, developing staff training, and supporting the well-being/activities coordinators.

Generative Artificial Intelligence Image Tools among Future Designers: A Usability, User Experience, and Emotional Analysis

Generative Artificial Intelligence (GenAI) image tools hold the promise of revolutionizing a designer’s creative process. The increasing supply of this type of tool leads us to consider whether they suit future design professionals. This study aims to unveil if three GenAI image tools—Midjourney 5.2, DreamStudio beta, and Adobe Firefly 2—meet future designers’ expectations. Do these tools have good Usability, show sufficient User Experience (UX), induce positive emotions, and provide satisfactory results? A literature review was performed, and a quantitative empirical study based on a multidimensional analysis was executed to answer the research questions. Sixty users used the GenAI image tools and then responded to a holistic evaluation framework. The results showed that while the GenAI image tools received favorable ratings for Usability, they fell short in achieving high scores, indicating room for improvement. None of the platforms received a positive evaluation in all UX scales, highlighting areas for enhancement. The benchmark comparison revealed that all platforms, except for Adobe Firefly’s Efficiency scale, require enhancements in pragmatic and hedonic qualities. Despite inducing neutral to above-average positive emotions and minimal negative emotions, the overall satisfaction was moderate, with Midjourney aligning more closely with user expectations. This study emphasizes the need for significant improvements in Usability, positive emotional resonance, and result satisfaction, even more so in UX, so that GenAI image tools can meet future designers’ expectations.

Strategic assessment of sustainable production in the international automobile sector using efficiency analysis and decision-making models

In the context of escalating global environmental concerns, the automotive industry faces pressing imperatives to align its growth with sustainable practices. This urgency, coupled with the need for a holistic evaluation framework, sets the stage for this study. Motivated by the gaps in quantitatively assessing sustainability, the study aims to discern and quantify the sustainability efforts of leading automotive manufacturers. The objective is to establish a robust, quantifiable link between operational efficiency and sustainability criteria that informs both business strategy and environmental stewardship. To achieve this, an innovative methodology combining the Malmquist Data Envelopment Analysis (Malmquist DEA) framework and the Ordinal Priority Approach (OPA) is employed. The approach analyzes efficiency changes, technological progress, and the industry's adoption of sustainable practices over a five-year period, concluding in the pre-pandemic year of 2020. The findings reveal a notable emphasis on productivity and market exposure, identifying them as pivotal to sustainable manufacturing. The OPA provides a weighted analysis of sustainability criteria based on expert evaluations, emphasizing the integration of environmental initiatives with core business strategies. The study's contribution is twofold: it introduces a novel, quantifiable approach to evaluating sustainability in the automotive industry, and it offers strategic insights for companies seeking to elevate their sustainability profile effectively. However, the reliance on pre-pandemic data and expert opinions marks the limitations of the study, indicating the necessity for further research. Future studies should aim to expand the temporal scope to include the latest industry shifts and to diversify the pool of sustainability criteria, capturing a broader spectrum of environmental and social dimensions. This continued research will enhance the industry's understanding of sustainable manufacturing in the evolving global context.

An integrated One Health framework for holistic evaluation of risks from antifungal agents in a large-scale multi-city study

A new framework for retrospective mass spectral data mining for antifungal agents (AFs) and Wastewater-Based Epidemiology (WBE) was developed as part of One Health framework to tackle risks from AFs. A large scale, multi-city study was undertaken in South-West England. Key drivers of AFs in the catchment were identified with communal wastewater discharges being the main driver for human AFs (fluconazole, ketoconazole) and agricultural runoff being the main driver for pesticide AFs (prochloraz, prothioconazole and tebuconazole). Average WBE-estimated human used fluconazole and ketoconazole PNDIs (population normalised daily intake) exceeded 300 mg day−1 1000 inh−1 and 2000 mg day−1 1000 inh−1. This is much higher than PNDPs (population normalised daily prescriptions <40 mg day−1 1000 inh−1 and <80 mg day−1 1000 inh−1 for fluconazole and ketoconazole respectively). This was expected due to both prescription and over-the-counter usage, and both oral and topical applications. Pesticide AF, prothioconazole had PNDIs <40,000 mg day−1 1000 inh −1, which gave intake: 0.43, 0.26, 0.07 mg kg−1 in City A, B, and C, likely due to accounting for external/non-human sources. This is higher than the acceptable daily intake (ADI) of 0.01 mg kg−1bw day−1, which warrants further study. Intake per kg of body weight estimated using tebuconazole was 0.86, 1.39, 0.12, 0.13, and 2.7 mg kg−1 in City A-E respectively and is likely due to external/non-human sources. Intake calculated using its metabolite was 0.02 and 0.01 mg kg−1 in City B and C respectively, which aligned with ADI (0.03 mg kg−1bw day−1). The environmental risk assessment of AFs indicated low/medium risk from fluconazole, prochloraz, and tebuconazole, medium risk from epoxiconazole, prothioconazole's metabolite, and tebuconazole, and high risk for prothioconazole in river water. High risk was estimated from fluconazole, epoxiconazole, prothioconazole and its metabolite, tebuconazole, ketoconazole in wastewater samples, which is important during raw sewage discharge events via sewer overflows.

A holistic techno-economic evaluation framework for sizing renewable power plant in a hydro-based hybrid generation system

The configuration of the hybrid generation systems (HGSs) is critical for maximizing the synergy between different power sources. However, accurately simulating the operation of HGSs over their lifespan is challenging due to the need to balance multiple objectives covering different timeframes. For example, long-term water management and short-term peak-shaving can conflict. To tackle this problem, we propose a comprehensive evaluation framework that optimizes the size of a renewable power plant integrated with an existing hydropower station. This framework combines a long-term and short-term operation model with a techno-economic analysis. Unlike traditional models, our short-term operation model prioritizes power supply quality and quantity while maintaining satisfactory peak-shaving performance for the HGS. We generated numerous daily generation plans using this model to derive the multidimensional response functions, which describe the relationship between long-term hydropower decision, renewable power output and short-term performance indicators. These response functions were then incorporated into a refined long-term operation model to simulate the HGS's extended operation. Finally, we employed a holistic evaluation framework comprising various techno-economic indicators to determine the cost-effective size of the renewable power plant. We applied this framework to the Igong Zangpo River Basin in China, a tributary of the Yarlung Zangbo River. The results revealed the following: (1) Increasing the size of the photovoltaic (PV) component resulted in a decline in technical performance, including energy use efficiency and peak-shaving capabilities; (2) Given sufficient transmission capacity, a PV size threshold of 2400 MW was identified, while the optimal PV size with the highest net present value was 1950 MW, twice the capacity of the hydropower installation (1030 MW); and (3) The optimal PV size varied from 0 to 2400 MW based on electricity price and initial investment factors. Thus, our proposed techno-economic framework effectively guides the capacity configuration of the hydro-based HGSs.

Towards trustworthy multi‐modal motion prediction: Holistic evaluation and interpretability of outputs

AbstractPredicting the motion of other road agents enables autonomous vehicles to perform safe and efficient path planning. This task is very complex, as the behaviour of road agents depends on many factors and the number of possible future trajectories can be considerable (multi‐modal). Most prior approaches proposed to address multi‐modal motion prediction are based on complex machine learning systems that have limited interpretability. Moreover, the metrics used in current benchmarks do not evaluate all aspects of the problem, such as the diversity and admissibility of the output. The authors aim to advance towards the design of trustworthy motion prediction systems, based on some of the requirements for the design of Trustworthy Artificial Intelligence. The focus is on evaluation criteria, robustness, and interpretability of outputs. First, the evaluation metrics are comprehensively analysed, the main gaps of current benchmarks are identified, and a new holistic evaluation framework is proposed. Then, a method for the assessment of spatial and temporal robustness is introduced by simulating noise in the perception system. To enhance the interpretability of the outputs and generate more balanced results in the proposed evaluation framework, an intent prediction layer that can be attached to multi‐modal motion prediction models is proposed. The effectiveness of this approach is assessed through a survey that explores different elements in the visualisation of the multi‐modal trajectories and intentions. The proposed approach and findings make a significant contribution to the development of trustworthy motion prediction systems for autonomous vehicles, advancing the field towards greater safety and reliability.

A systematic literature review of cybersecurity scales assessing information security awareness

Information Security Awareness (ISA) is a significant concept that got considerable attention recently and can assist in minimizing the risks associated with information security breaches. Several measurement scales have been developed in this regard, as measuring users’ ISA is paramount. Although ISA specific scales are very important, yet what methodological rigor they use in terms of initial conceptualization of ISA, data collection and analysis during the development, and scale validation of such scales are some unknown aspects. Therefore, we provide a comprehensive review of the existing ISA specific scales to address all the above concerns. A popular method, PRISMA, is utilized, and a total of 24 articles that match with criteria of this research are included for the final in-depth analysis. Also, a holistic evaluation framework is developed containing three phases and 19 criteria. Findings revealed that most studies treat ISA as a multi-dimensional construct, and ISA researchers rarely conduct both pilot testing and pre-text evaluation while validating and refining the initial scales. Additionally, several articles did not report some of the essential elements used for checking the rigor of factor analysis, and evidence for validities of the identified scales is inadequate. Consequently, existing ISA specific scales must be improved both in terms of the methodological thoroughness of the scale development procedure and their validities. Moreover, not only justifying why the development of a new scale is necessary, but also improving the quality of the existing scales by doing multiple iterations is significant in the future. Likewise, the inclusion of all the dimensions of ISA, while generating the initial items pool is an important aspect to be considered. A thorough discussion, recommendations for future research, conclusions, and study limitations are provided.

Are we discriminating towards guests with disabilities? Accessibility analyses of public restaurants facilities in the Republic of Slovenia

Previous studies have not considered the importance of managers? perspectives for assuring accessibility, and no international benchmarking standards for public restaurant accessibility assessment exist. This study aims to: (1) Propose a holistic evaluation framework for the overall (physical, functional, and communicational) assessment of restaurant accessibility; (2) Identify managers? perceptions of accessibility; and (3) Determine differences in restaurant accessibility according to managers? and restaurants? characteristics. This study was conducted in two stages: literature and legislative analyses and exploratory research using focus groups (the national disability organisations representatives) and field research. The European (EU) and the national legislation were thoroughly analysed and compared to the Americans with disabilities act (ADA). A self-administered questionnaire was distributed to 180 managers of small and medium-sized (SME) restaurant enterprises in Slovenia. Bivariate and cluster analyses (CA) were used. Two clusters of restaurants (less and better accessible) were identified. Results indicate that physical barriers, though strictly regulated at the EU level, still present a significant shortcoming in providing accessible services. Functional and communicational barriers prove less problematic. Results also reveal that more accessible restaurants are bigger, have more employees, invest more in IT, and their managers have greater understanding and skills in this area. The evaluation tool enables a holistic approach to accessibility analysis by following the guest?s logical movement path. Besides stricter control, policymakers, interior designers, and restaurant service providers should collaborate closely with disability organisations. Restaurant service providers should be trained on disability issues.

A Review of E-Government Assessment Frameworks: E-Readiness, Adoption, Citizen Engagement and Quality

The rapid development of digitalization has impelled countries to adopt electronic government (e-government) to improve quality and efficiency, reduce costs and establish transparency in their transactions with the citizens, thereby, strengthening the government-citizens relationship. As e-government has become pivotal for development, it is essential to understand various frameworks available in the literature for evaluating a country’s e-government at different phases, from e-readiness to the realization of end benefits. Hence, this paper briefly reviews the e-government literature focusing on e-readiness, adoption, citizen engagement and performance and quality assessment frameworks. Further, a thematic analysis of the selected studies is carried out using NVivo and QDA Miner to map the main themes and subthemes and the associations among the selected studies. This paper contributes to the e-government literature with clear academic insights into different variables and dimensions in association with the evaluation of e-government and by setting a future research agenda for developing a holistic evaluation framework.

Decent work in a seascape of livelihoods: Regional evaluation of the shrimp and groundfish fishery of the Guianas-Brazil shelf

With growing evidence of labor violations and exploitative working conditions in fisheries, ensuring decent work is imperative to protect fishers and fishworkers in the global seafood sector. This study provides the first evaluation of decent work in a shared, transboundary fishery – the shrimp and groundfish fishery of the Guianas-Brazil Shelf. Decent work in fisheries has gained increasing attention and research, yet gaps exist in our understanding of the elements of decent work, how we evaluate it, and how to enable decent work. To date, there has been limited analysis of decent work in a range of geographies and diverse fisheries contexts, including small-scale fisheries and transboundary fisheries. This study will address this gap by evaluating decent work, utilizing a new fishery-specific, holistic evaluation framework drawing from existing frameworks including the ILO Work in Fishing Convention (C188), the Monterey Framework for Social Responsibility, and the FAO Voluntary Guidelines for Securing Sustainable Small-Scale Fisheries. This evaluation details country-level challenges that put fishers and fishworkers at risk in their occupation, including illegal fishing, vessel safety, and worker representation. This paper concludes with recommendations, to be advanced with a transboundary, regional approach, to ensure decent work and strengthen existing progress, including 1) addressing widespread illegal activities, 2) adopting fisheries-specific standards like C188, 3) implementing and enforcing policies at the country and regional level, and 4) ensuring worker representation and participation leveraging cooperatives and collectives.

A Novel Holistic Framework for Evaluation of Landslide Susceptibility in the Hengduan Mountains, the Qinghai-Tibet Plateau

A Novel Holistic Framework for Evaluation of Landslide Susceptibility in the Hengduan Mountains, the Qinghai-Tibet Plateau

Stratification by Tumor Grade Groups in a Holistic Evaluation of Machine Learning for Brain Tumor Segmentation.

Accurate and consistent segmentation plays an important role in the diagnosis, treatment planning, and monitoring of both High Grade Glioma (HGG), including Glioblastoma Multiforme (GBM), and Low Grade Glioma (LGG). Accuracy of segmentation can be affected by the imaging presentation of glioma, which greatly varies between the two tumor grade groups. In recent years, researchers have used Machine Learning (ML) to segment tumor rapidly and consistently, as compared to manual segmentation. However, existing ML validation relies heavily on computing summary statistics and rarely tests the generalizability of an algorithm on clinically heterogeneous data. In this work, our goal is to investigate how to holistically evaluate the performance of ML algorithms on a brain tumor segmentation task. We address the need for rigorous evaluation of ML algorithms and present four axes of model evaluation—diagnostic performance, model confidence, robustness, and data quality. We perform a comprehensive evaluation of a glioma segmentation ML algorithm by stratifying data by specific tumor grade groups (GBM and LGG) and evaluate these algorithms on each of the four axes. The main takeaways of our work are—(1) ML algorithms need to be evaluated on out-of-distribution data to assess generalizability, reflective of tumor heterogeneity. (2) Segmentation metrics alone are limited to evaluate the errors made by ML algorithms and their describe their consequences. (3) Adoption of tools in other domains such as robustness (adversarial attacks) and model uncertainty (prediction intervals) lead to a more comprehensive performance evaluation. Such a holistic evaluation framework could shed light on an algorithm's clinical utility and help it evolve into a more clinically valuable tool.

Holistic framework for evaluating and improving information security culture

PurposeThe objective of this research was to propose and validate a holistic framework for information security culture evaluation, built around a novel approach, which includes technological, organizational and social issues. The framework's validity and reliability were determined with the help of experts in the information security field and by using multivariate statistical methods.Design/methodology/approachThe conceptual framework was constructed upon a detailed literature review and validated using a range of methods: first, measuring instrument was developed, and then content and construct validity of measuring instrument was confirmed via experts' opinion and by closed map sorting method. Convergent validity was confirmed by factor analysis, while the reliability of the measuring instrument was tested using Cronbach's alpha coefficient to measure internal consistency.FindingsThe proposed framework was validated based upon the results of empirical research and the usage of multivariate analysis. The resulting framework ultimately consists of 46 items (manifest variables), describing eight factors (first level latent variables), grouped into three categories (second level latent variables). These three categories were built around technological, organizational and social issues.Originality/valueThis paper contributes to the body of knowledge in information security culture by developing and validating holistic framework for information security culture evaluation, which does not observe information security culture in only one aspect but takes into account its organizational, sociological and technical component.

Enhancing urban flood resilience: A holistic framework incorporating historic worst flood to Yangtze River Delta, China

Rapid urbanization and climate change have increased the risk of urban flooding, causing massive infrastructure and human losses. The concept of resilience proposes new solutions to manage flood disaster. An urban flood resilience evaluation framework considering the flood disaster cycle of actual historic flood event and objective physical-socio-economic status is necessary for future flood mitigation. This paper proposes a holistic evaluation framework for evaluating urban flood resilience with VIKOR and Grey Relational Analysis (GRA) method. The proposed framework consists of indicators of resistance, coping, recovery and adaptation capacity of resilience for three stages of the flood disaster cycle, namely pre, during and post-flood. The framework has been applied to Yangtze River Delta (YRD) consisting 27 cities in China. Following a rigorous analysis, the cities are ranked and mapped, among which Nanjing stands out to be the first, whereas the entire region presents a moderate level of urban flood resilience varying from city to city. The detailed comparison with sensitivity analysis of resilience at regional, provincial and city level suggests a better resilience in pre-flood stage than post-flood stage. Finally, practical recommendations to regional and local level are provided for further flood mitigation and resilience improvement. The proposed framework is generalizable and useful to develop flood related standards, establish benchmarks, perform evaluation at regional, provincial and city levels across China and other parts of the world.

How accurately do experts perceive the effectiveness of urban freight transportation solutions in medium-sized cities

This study demonstrates and implements a holistic evaluation framework to investigate the convergence of experts' opinion towards implementing urban freight transportation solutions (policies and techniques) with recorded solution's impacts, at medium-sized cities. The framework is formulated based on a multi-criteria evaluation approach by considering sustainability disciplines (economy, environment, mobility, and society) and enablers (solution maturity, social acceptance, and user uptake). Enablers represent the readiness of the city to adopt a solution and promote its successful operation towards sustainability. Results show that the two data sources produce different results about the performance level of the same solutions, but in some cases comparable level of improvement. When considering also the enablers in the assessment, the results of the two data sources tend to diverge less. The findings of this paper contribute in a better understanding of the performance results of urban freight transportation solutions, based on experts' perceptions on a multidisciplinary set of evaluation parameters.

E-justice evaluation factors: The case of Smart Court of China

As information and communication technologies (ICT) continue to impact and shape modern society, and e-justice has gained momentum in recent years. China’s Supreme People’s Court (SPC) set up a trial management information system that connects all courts in China. Designed to be online, transparent, and intelligent, China’s Smart Court development began in 2016. In practice, SPC promoted modernization of the trial systems to improve the flow of information between courts throughout China. However, significant investments in ICT e-justice services have caused some to question whether these investments have achieved the expected ends. Thus, how to evaluate e-justice services becomes an urgent theoretical and policy issue in the process of e-justice construction in China. E-justice value is not clearly defined in theory in China, nor is easy to measure in practice. Because of the sensitive and complex nature of such evaluation, little research has been conducted in this regard. The objective of this paper is to fill this gap. Relevant literature is reviewed before the article moves on to describe various approaches withe regard to e-government and e-justice evaluation, as well as the characteristics of China’s Smart Court. Evaluation factors and constructs are found based on the Chinese circumstances. This study contributes to the development of a holistic evaluation framework for e-justice system. It also adds the Chinese case to the existing literature. Evaluation factors found in this article can also serve as a foundation for future development and study of e-justice services.

CISEF: A composite index of social, environmental and financial performance

We propose a holistic evaluation framework that banks could adopt when screening corporate entities. The framework is based on the development of a composite indicator of social, environmental and financial performance. This integrated view of evaluation is conceptually aligned with ISO standards and empirical proposals of academics and market practitioners in defining inclusive performance. We complement those proposals with a methodological framework that permits incorporation of a plethora of viewpoints in the evaluation process, reflecting the expectations of the various stakeholders in the environment of a bank. We further enhance the evaluation process with analytics of a more detailed hierarchical view of performance, fine-tuning of the evaluation process, and provide implications and suggestions to the senior executive team of a bank’s clients.

CISEF: a Composite Index of Social, Environmental and Financial Performance

We propose a holistic evaluation framework that banks could adopt when screening corporate entities. The framework is based on the development of a composite indicator of social, environmental and financial performance. This integrated view of evaluation is conceptually aligned with ISO standards and empirical proposals of academics and market practitioners in defining inclusive performance. We complement those proposals with a methodological framework that permits incorporation of a plethora of viewpoints in the evaluation process, reflecting the expectations of the various stakeholders in the environment of a bank. We further enhance the evaluation process with analytics of a more detailed hierarchical view of performance, fine-tuning of the evaluation process, and provide implications and suggestions to the senior executive team of a bank’s clients.