Evaluation Framework Research Articles

Common values for common goods: a circular regeneration model for active citizenship

Regenerating cultural heritage implies adopting a systemic and multi-scalar approach capable of considering the characteristics and flows of relations between the different components of the system. The regeneration project can be a circular strategy to reconnect people and places, generating and regenerating material and immaterial values of cultural heritage as a ‘common good’. The research proposes a circular and participatory regeneration model, validated through an evaluative framework. It supports decision-making and assesses the multidimensional impacts of reuse, redevelopment, maintenance and management practices of the commons, at all stages of the recovery project, and with respect to emerging instances and criteria adopted in existing policies.

Evaluating Ecosystem Services Incorporation in River Policy: A Multidimensional Framework Applied to China's River Chief System

ABSTRACTThis study develops a comprehensive evaluation framework to assess the incorporation of river ecosystem services (ESs) into policy content, focusing on China's River Chief System (RCS) policy. The framework introduces a quantitative tool that bridges the gap between policy formulation evaluation and policy effect evaluation by integrating a hierarchical indicator system, reference values, and a method for quantifying ES incorporation. Key findings from 32 province‐level RCS policy documents revealed selective incorporation of certain ES indicators, significant mismatches between policy focus and the typical structure of river ESs, and notable regional disparities. Provisioning services (i.e., water supply) and regulating services (i.e., pollution mitigation) are prioritized, while critical regulating services (e.g., flood mitigation and habitat maintenance) and cultural services (e.g., education and science) are underrepresented. Stakeholder and relationship indicators are limited, primarily focusing on managers and spatial relationships, while the value of river ESs is largely ignored. Policy optimization should emphasize expanding the coverage of ESs, incorporating broader stakeholder involvement, and integrating financial incentives. Drawing on international river policies, policymakers could create a more flexible and adaptive river governance structure. While the study offers a quantitative framework for policy evaluation, it acknowledges limitations, including reliance on textual data and potential biases in the reference values. Future research should include stakeholder perspectives and dynamically adjust reference values as new data become available.

Health Catch-UP!: a realist evaluation of an innovative multi-disease screening and vaccination tool in UK primary care for at-risk migrant patients

BackgroundMigrants to the UK face disproportionate risk of infections, non-communicable diseases, and under-immunisation compounded by healthcare access barriers. Current UK migrant screening strategies are unstandardised with poor implementation and low uptake. Health Catch-UP! is a collaboratively produced digital clinical decision support system that applies current guidelines (UKHSA and NICE) to provide primary care professionals with individualised multi-disease screening (7 infectious diseases/blood-borne viruses, 3 chronic parasitic infections, 3 non-communicable disease or risk factors) and catch-up vaccination prompts for migrant patients.MethodsWe carried out a mixed-methods process evaluation of Health Catch-UP! in two urban primary healthcare practices to integrate Health Catch-UP! into the electronic health record system of primary care, using the Medical Research Council framework for complex intervention evaluation. We collected quantitative data (demographics, patients screened, disease detection and catch-up vaccination rates) and qualitative participant interviews to explore acceptability and feasibility.ResultsNinety-nine migrants were assessed by Health Catch-UP! across two sites (S1, S2). 96.0% (n = 97) had complete demographics coding with Asia 31.3% (n = 31) and Africa 25.2% (n = 25), the most common continents of birth (S1 n = 92 [48.9% female (n = 44); mean age 60.6 years (SD 14.26)]; and S2 n = 7 [85.7% male (n = 6); mean age 39.4 years (SD16.97)]. 61.6% (n = 61) of participants were eligible for screening for at least one condition and uptake of screening was high 86.9% (n = 53). Twelve new conditions were identified (12.1% of study population) including hepatitis C (n = 1), hypercholesteraemia (n = 6), pre-diabetes (n = 4), and diabetes (n = 1). Health Catch-UP! identified that 100% (n = 99) of patients had no immunisations recorded; however, subsequent catch-up vaccination uptake was poor (2.0%, n = 1). Qualitative data supported acceptability and feasibility of Health Catch-UP! from staff and patient perspectives, and recommended Health Catch-UP! integration into routine care (e.g. NHS health checks) with an implementation package including staff and patient support materials, standardised care pathways (screening and catch-up vaccination, laboratory, and management), and financial incentivisation.ConclusionsClinical Decision Support Systems like Health Catch-UP! can improve disease detection and implementation of screening guidance for migrant patients but require robust testing, resourcing, and an effective implementation package to support both patients and staff.

A Comparative Study on ERNIE Bot 4.0 Turbo and ChatGPT 4o’s Performance in Evaluating First-Year Undergraduate Persuasive Essays

This study compares the performance of two prominent AI language models, ERNIE Bot 4.0 Turbo and ChatGPT 4o, in evaluating first-year undergraduate persuasive essays within the social sciences domain. Drawing from the Louvain Corpus of Native English Essays, a comprehensive collection of academic writings by British and American university students, this study aims to examine the models’ capabilities in assessing the grammatical correctness, vocabulary usage, coherence, content depth, and writing style of the essays. This study adopts a structured evaluation framework based on IELTS writing criteria to assess the models’ performance. A 40 persuasive essays from the Louvain Corpus were evaluated by both AI models and compared with human raters’ evaluations to ensure validity. The findings reveal distinct differences in the assessment styles of the two models. ChatGPT 4o exhibits a more critical approach, pinpointing areas for improvement, such as lack of argument development, coherence issues, and grammatical errors. Conversely, ERNIE Bot 4.0 Turbo offers a more balanced assessment, acknowledging essays’ strengths and suggesting improvement areas. Notably, ERNIE Bot’s evaluation highlights potential biases in AI-based assessment systems, particularly in its unequal emphasis on viewpoints. This comparative examination offers valuable perspectives on the advantages and constraints of AI models in assessing scholarly compositions, underscoring the significance of amalgamating varied AI functionalities to establish more all-inclusive and efficient feedback systems for learners. By understanding these differences, researchers and educators can better utilize AI-assisted essay evaluation systems to enhance student learning experiences.

Critical Success Factors and Acceptance of the Casemix System Implementation Within the Total Hospital Information System: Exploratory Factor Analysis of a Pilot Study.

The health care landscape is evolving rapidly due to rising costs, an aging population, and the increasing prevalence of diseases. To address these challenges, the Ministry of Health of Malaysia implemented transformation strategies such as the Casemix system and hospital information system to enhance health care quality, resource allocation, and cost-effectiveness. However, successful implementation relies not just on the technology itself but on the acceptance and engagement of the users involved. This study aims to develop and refine items of a quantitative instrument measuring the critical success factors influencing acceptance of Casemix system implementation within the Ministry of Health's Total Hospital Information System (THIS). A cross-sectional pilot study collected data from medical doctors at a hospital equipped with the THIS in the federal territory of Putrajaya, Malaysia. This pilot study's minimum sample size was 125, achieved through proportionate stratified random sampling. Data were collected using a web-based questionnaire adapted from the human, organization, and technology-fit evaluation framework and the technology acceptance model. The pilot data were analyzed using exploratory factor analysis (EFA), and the Cronbach α assessed internal reliability. Both analyses were conducted in SPSS (version 25.0; IBM Corp). This study obtained 106 valid responses, equivalent to an 84.8% (106/125) response rate. The Kaiser-Meyer-Olkin measure of sampling adequacy was 0.859, and the Bartlett test of sphericity yielded statistically significant results (P<.001). Principal component analysis identified 9 components explaining 84.07% of the total variance, surpassing the minimum requirement of 60%. In total, 9 unique slopes indicated the identification of 9 components through EFA. While no new components emerged from the other 7 constructs, only the organizational factors construct was divided into 2 components, later named organizational structure and organizational environment. In total, 98% (41/42) of the items had factor loadings of >0.6, leading to the removal of 1 item for the final instrument for the field study. EFA ultimately identified 8 main constructs influencing Casemix implementation within the THIS: system quality, information quality, service quality, organizational characteristics, perceived ease of use, perceived usefulness, intention to use, and acceptance. Internal reliability measured using the Cronbach α ranged from 0.914 to 0.969, demonstrating high reliability. This study provides insights into the complexities of EFA and the distinct dimensions underlying the constructs that influence Casemix system acceptance in the THIS. While the findings align with extensive technology acceptance literature, the results accentuate the necessity for further research to develop a consensus regarding the most critical factors for successful Casemix adoption. The developed instrument is a substantial step toward better understanding the multidimensional challenges of health care system transformations in Malaysia, postulating an underpinning for future fieldwork and broader application across other hospitals.

5.X.1. Co-designing for policy impact: Belgium’s monitoring &amp; evaluation framework for cancer care &amp; control

Abstract Cancer is one of the main public health challenges, and the second leading cause of mortality in the EU. It is Belgium’s second leading cause of adult death after circulatory diseases and has been an increasing burden of disease in recent years. Recognizing the urgency of tackling the entire disease pathway, the European Commission (EC) has launched Europe’s Beating Cancer Plan (EBCP), a policy-driven strategy, and the Mission on Cancer, a research-focused undertaking. In Belgium, Sciensano’s Cancer Centre has been charged with the coordination and implementation of these programs, pioneering the setup of a dedicated structure to coordinate them. While Belgian public, academic, private and non-for-profit entities have been highly involved in these EU initiatives since their launch, there is currently no mechanism to monitor the impact of EBCP’s uptake and implementation in Belgium. Moreover, in contrast to the investment and momentum at the European level, the last Belgian Cancer Plan dates from 2008, with its most recent evaluation being published in 2013, highlighting an important gap. To address this gap, the Cancer Centre was asked by the Federal Cabinet to develop a new monitoring and evaluation (M&amp;E) framework for cancer care and control in Belgium. With a view to avoiding traditional top-down approaches, it was decided that the framework would be designed co-creatively, by engaging patients and experts in the field. The aim was to enable the iterative and flexible monitoring of Belgian policies related to the EBCP’s implementation, while generating buy-in from stakeholders. To achieve this goal, a modified Delphi process was used, with complementary qualitative data collection in the form of focus group discussions with experts across the cancer continuum. A Steering and Patient Committee were put in place to provide stewardship over the process, and based on their feedback, additional patient workshops were run to ensure M&amp;E priorities for patients were taken into account. The framework will cover the entire spectrum of cancer prevention and control, as well as transversal themes such as inequalities and patient-centeredness. It will also anticipate future evolutions and undergo adjustments when needed. This workshop will use a Pecha Kucha approach, with panelists sharing insights from the Belgian case study. The session will guide the audience through a story of the strengths and pitfalls of co-creating an M&amp;E framework in the field of public health monitoring. Topics covered will include challenges of monitoring cancer care and control policies, approaches taken in Belgium to overcome these challenges, and the importance of involving diverse stakeholders in the process. A moderated discussion will allow participants to share their perspectives and ask questions. Attendees will walk away with a real-world example on how to challenge top-down paradigms of population-level M&amp;E frameworks and identify transferable elements for adaptation in their own settings. Key messages • Europe’s Beating Cancer Plan (EBCP) provides opportunities to tackle the second leading cause of disease in the EU but will require innovative approaches to monitor uptake and impact. • Development of Belgium policy-actionable monitoring &amp; evaluation framework shows how co-creation can challenge top-down paradigms in public health monitoring and assess the uptake and impact of EBCP. Speakers/Panelists Gabrielle Schittecatte Sciensano, Brussels, Belgium Jinane Ghattas Sciensano, Brussels, Belgium Robbe Saesen Sciensano, Maaseik, Belgium

Development of the health literacy responsiveness (HLR) evaluation framework for HLR interventions

Abstract Background Health literacy responsiveness (HLR) interventions include actions at different levels of an organization to provoke change and improve environments ensuring equitable access to and use of information and services (1). However, generic instruments measuring the effect of HLR at intervention level are inconsistent, making evidence on HLR intervention effects difficult to compare (2). The objective of this study was to develop an evaluation framework based on core outcome domains of HLR to facilitate higher consistency in future evaluations of HLR interventions. Methods HLR outcomes were identified through a narrative literature review and categorized into domains through a content mapping process. Expert consensus on core outcome domains was reached through a Delphi process. Experts within the field of HLR were invited to participate and answer three questionnaires designed to rank and clarify potential core outcome domains. Quantitative and qualitative analysis was performed to select and complete the final core outcome domains in the HLR evaluation framework (HLR-EF). Results The literature search resulted in identification of nine reviews. A total of 383 outcomes were identified from these reviews and included in the content mapping process, which resulted in a total of 26 core outcome domains. Experts (n = 15) participated in the three-round Delphi process, which resulted in consensus on 15 core outcome domains in HLR-EF allocated under five headings. Conclusions The study has successfully managed to identify a set of core outcome domains applicable in future evaluations of HLR interventions based on systematic methodologies and expert consensus. Context-sensitive manuals for the use of the HLR-EF is under development and the instrument is ready to be applied in future evaluation of HLR interventions, potentially making future measuring, reporting and comparison of results significantly easier. Key messages • Instruments facilitating evaluation of HLR interventions are non-generic and inconsistent. • HLR-EF has been developed using a narrative literature review, content mapping, and a Delphi process, and defines 15 core outcome domains for evaluation of HLR interventions.

Evaluating healthcare inequalities using ChatGPT4 – The case of potentially inappropriate medication

Abstract Issue/problem Regular and standardized monitoring of healthcare is necessary to inform decision makers and clinicians about healthcare quality. Equity is a fundamental dimension in such evaluations. We aimed to investigate if artificial intelligence (AI) large language models (LLM), such as ChatGPT4, may facilitate routine monitoring of healthcare inequalities using an established framework for healthcare quality and equity evaluation, analysis of individual heterogeneity and discriminatory accuracy (AIHDA). Description of the problem We have previously demonstrated that compared to traditional methods, AIHDA improves the evaluation of healthcare inequity. We asked ourselves if ChatGPT4 could facilitate the standardized application of the AIHDA approach in routine monitoring of healthcare. Results Using strict guidelines, we instructed a GPT model, based on ChatGPT 4, to utilize the AIHDA framework for evaluating healthcare inequalities. To demonstrate the analysis, we used the quality indicator potentially inappropriate medication among elderly and analyzed simulated data of individuals &amp;gt;75-year-old belonging to 36 socioeconomic strata and residing in 21 Swedish regions. The GPT performed an accurate analysis, correctly interpretated the results and formulated an informative short rapport with illustrative tables and figures. Lessons AI appears to facilitate routine monitoring of health care inequity using the AIHDA approach. AI also allows a constructive interaction with the user but, so far, it needs be closely supervised. Key messages • ChatGPT 4 can improve the standardized evaluation of healthcare inequity. • Although the technology is under development and must be used with caution.

Addressing the complexities of evaluating comprehensive sexuality education: an Italian case study

Abstract Background Existing evaluation methods for assessing sexuality education programs may not be suitable for Comprehensive Sexuality Education (CSE) interventions, which are globally indicated as the most effective approach for promoting sexual health and preventing STIs. This holds particular significance in countries, such as Italy, where CSE is not integrated into school curricula. In a pilot project funded by the Italian Ministry of Health since 2019 (EduForIST), we aimed to develop a composite evaluation framework to test the CSE intervention we implemented in lower and upper secondary schools over the past 4 years. Methods Based on the latest available evidence, evaluating a CSE intervention requires addressing various dimensions. These include assessing the quality of programs and their implementation, as well as measuring outcomes and impact. Through iterative consultations with experts and providers, methods, criteria, and metrics were chosen and tested for each dimension. Results Program quality was evaluated by an expert advisory board. Instruments for implementation quality assessment included: satisfaction survey among students; reflective journals and SWOT analysis drawn up by educators delivering intervention, exploring their perceptions, practices, and attitudes; surveys among teachers and parents, measuring knowledge, attitude, and practice on CSE, and observed changes among adolescents. A four-arm, Solomon, cluster design study was designed to evaluate outcomes in terms of changes in students’ knowledge. Clinical and behavioral indicators in the short, medium and long term were selected as measures of the impact of the intervention. Conclusions In the field of CSE implementation research, a multi-component approach to evaluation is needed to provide a solid basis for informing policymakers. The involvement of providers, recipients and contextual adults is essential in evaluating CSE interventions. However, assessing long-term impact requires long time frame. Key messages • Assessing CSE interventions requires a multi-component evaluation approach, considering the impact on the personal and emotional development of individuals in order to sustain behavior changes. • The proposed evaluation framework may inform scientific community, professionals, and decision-makers on effectiveness of interventions in settings where CSE is not yet included in school curricula.

Implementing mental health quality practices for vulnerable populations in five EU countries

Abstract Issue Mental ill-health has been a leading cause of disability in Europe for many years, with the COVID-19 pandemic increasing incidences of both poor mental health and mental disorders. Vulnerable groups such as children, migrants, individuals with existing mental health issues, and healthcare workers are particularly affected but often lack tailored mental health support. Description of the problem The MENTALITY project aims to alleviate mental health distress worsened by the COVID-19 pandemic focusing on vulnerable groups. It also involves developing capacity-building tools and disseminating research results to increase awareness and knowledge sharing. Five EU countries are currently piloting four distinct good practices to address mental health issues among vulnerable populations. Specifically, Poland is aiming to support migrants (including Ukrainian and Spanish-speaking populations), Latvia is focusing on the healthcare workforce, and Romania and Slovakia are concentrating on users and community services. Meanwhile, Greece and Slovakia are implementing initiatives for children. The implementation phase began in January 2024 with a duration of 6 months. Results By consulting with relevant stakeholders, a Situation Analysis and Needs Assessment framework has been conducted, allowing for informed decisions regarding the elements to be pilot. Partners have developed country specific piloting plans and external evaluators are leading a common framework for monitoring and evaluation, ensuring regular assessment of the project’s progress. Lessons The innovative MENTALITY project aspires to enhance community-based mental health systems and promote improved mental health services across EU Member States during crises by adopting a preventative approach. Preliminary results show that the initiatives within the project were much needed and have already begun to demonstrate promising outcomes in addressing mental health challenges at both individual and community levels. Key messages • The MENTALITY project aims to address mental health distress of vulnerable populations in times of crisis. • The project seeks to advance equality in health and social rights for all individuals living in the EU.

Navigating complexity: an evaluation framework for assessing co-creation in public health

Abstract Background Co-creation aims to democratize research and bridge the gap between research and practice. Key characteristics like empowerment and meaningful participation shape sustainable outcomes. However, existing frameworks like Arnstein’s Ladder of Citizen Participation broadly overlook individual method contributions. This study aims to address this gap by creating and validating a framework for intrinsic evaluation, guiding stakeholders and researchers in genuine co-creation of public health interventions. Methods We systematically screened and extracted frameworks aligned with five key co-creation characteristics to identify and evaluate methods. Through iterative testing, we identified key features for distinguishing methods and integrated them into a novel framework. Co-creation experts validated and refined this framework using the closed card sort method, qualitative assessment, and group discussions. Results The Co-Creation Rainbow framework includes key features from 20 existing frameworks, with an Individual-to-Collective Continuum, five sections (Informing, Understanding, Stimulating, Collaborating, and Collective Decision-making), and three overarching themes (Engage, Participate, and Empower). Analysis showed successful categorization of 436 methods and revealed a significant disparity in co-creation practices between researchers and practitioners. Conclusions The Co-Creation Rainbow framework provides a structured approach to evaluating and reporting co-creation processes. Acknowledging co-creation’s diversity, the framework aids in tailoring approaches for meaningful participation. By offering a unified language for diverse co-creation methods, the framework also facilitates transparent evaluation, enhancing the reliability and reproducibility of co-created public health interventions. Key messages • Employing a universal language across diverse co-creation methods is pivotal in fostering transparent evaluation, thereby bolstering the reliability of co-created public health interventions. • Effective collaboration of key stakeholders necessitates the use of suitable methods. By integrating diverse perspectives, public health interventions gain relevance, efficacy, and sustainability.

Assessing progress towards viral hepatitis B and C elimination in prisons in the European region

Abstract Background In 2016 the WHO launched the global health sector strategy for the elimination of viral hepatitis, alongside a dedicated monitoring&amp;evaluation (M&amp;E) framework to assess progress at national, regional and global levels. Despite being a priority setting for viral hepatitis elimination in Europe, prisons are rarely integrated into national monitoring efforts. Besides, prisons are particular contexts with a high viral hepatitis prevalence, where it is possible to utilize adequate micro-elimination strategies. Objectives We aimed to assess available metrics of viral hepatitis burden and coverage of prevention and control interventions in prison settings in Europe using routine sources of data. Methods We adapted the WHO M&amp;E framework for viral hepatitis elimination to the prison context. The resulting M&amp;E framework has 49 indicators of which 10 are defined as core. Data were sourced from available European prison surveillance systems at European level, namely: European Monitoring Centre for Drugs and Drug Addiction (EMCDDA) and WHO Health in Prisons European Database (HIPED). Results In total 38 countries in the European region provided data to either data sources for 2020. For WHO-HIPED, among core indicators, 3 out of 10 could be calculated for HCV, and covered prevalence, HCV treatment availability and treatment outcome. Concerning HBV, 3 core indicators out of 10 could be calculated and covered prevalence, screening availability and treatment coverage in prison. With EMCDDA surveillance system we could calculate 4 HCV core indicators and 6 HBV core indicators. Lessons Availability of health data from prison settings is scarce, hampering the current capacity to monitor the progress in the viral hepatitis elimination within prisons contexts strategies and its relative contribution to global health goals achievement. A coordinated and standardised M&amp;E framework tailored to this specific setting is needed to foster data collection and systematic monitoring. Key messages • Micro-elimination strategies in prison context contribute to the global health goals achievement. • A coordinated and standardized M&amp;V framework, adapted to the prison context, is needed to foster data collection and monitoring of the viral hepatitis elimination strategy.

Leveraging Social Media and Deep Learning for Sentiment Analysis for Smart Governance: A Case Study of Public Reactions to Educational Reforms in Saudi Arabia

The Saudi government’s educational reforms aim to align the system with market needs and promote economic opportunities. However, a lack of credible data makes assessing public sentiment towards these reforms challenging. This research develops a sentiment analysis application to analyze public emotional reactions to educational reforms in Saudi Arabia using AraBERT, an Arabic language model. We constructed a unique Arabic dataset of 216,858 tweets related to the reforms, with 2000 manually labeled for public sentiment. To establish a robust evaluation framework, we employed random forests, support vector machines, and logistic regression as baseline models alongside AraBERT. We also compared the fine-tuned AraBERT Sentiment Classification model with CAMeLBERT, MARBERT, and LLM (GPT) models. The fine-tuned AraBERT model had an F1 score of 0.89, which was above the baseline models by 5% and demonstrated a 4% improvement compared to other pre-trained transformer models applied to this task. This highlights the advantage of transformer models specifically trained for the target language and domain (Arabic). Arabic-specific sentiment analysis models outperform multilingual models for this task. Overall, this study demonstrates the effectiveness of AraBERT in analyzing Arabic sentiment on social media. This approach has the potential to inform educational reform evaluation in Saudi Arabia and potentially other Arabic-speaking regions.

Evaluation of a public health workplace staff psychosocial risk assessment and wellbeing initiative

Abstract Adverse psychosocial work environments can have a detrimental effect on staff wellbeing causing chronic stress leading to burnout. Despite legislation mandating regular risk assessments, only one third of workplaces assess psychosocial risks. A 2020 European report identified one of the main barriers being a reluctance to talk openly about psychosocial issues. This paper describes an evaluation of a staff psychosocial risk assessment and wellbeing initiative in a Public Health Department in Ireland. The initiative commenced in October 2022 during a period of major health service organisational change. Informed by positive organisational psychology conceptual frameworks, guided by the European Psychosocial risk assessment framework (PRIMA-EF) and utilising the HSE Work Positive (CI) survey tool it was possible to constructively identify and categorise psychosocial risks across the Public Health team. In the evaluation, a mixed methods approach was used based on Donabedian’s structures, processes and outcomes overarching framework and Nielsen’s Model of Process Evaluation Framework. The first three iterative quality improvement cycles of the intervention were reviewed. The evaluation demonstrated that all of Nielson’s three indicators for a successful organisational level staff wellbeing intervention had been met: there was evidence of improvement in both working conditions and staff wellbeing; iterative quality improvement cycles including appreciative inquiry staff collaborative engagement ensured staff views and key learning was captured; new organisational structures within the Department enabled the sustainability of this process. The evaluation indicates that the staff psychosocial risk assessment process is beneficial for staff wellbeing and that the methods used are appropriate. The resources used in this process continue to be readily available within the health service organisation. The evaluation supports continuation of this process in its present form. Key messages • This evaluation supports staff psychosocial risk assessment and mitigation processes and provides a framework for healthcare managers to implement such processes and comply with legislation. • This evaluation indicates that effective and sustainable collaborative staff engagement can be achieved using a positive focus which builds on team strengths and harnesses collective wisdom.

Evaluating Genetic Testing: A Systematic Review of Assessment Frameworks

Abstract Introduction The assessment of the risks and benefits of genetic/genomic tests has long been addressed using various frameworks. They are mostly ACCE-based, focus on technical aspects, but often overlook economic and organizational considerations. The few HTA-based approaches, though more comprehensive, lack validation and implementation. This review’s purpose is to identify all evaluation frameworks for genetic/genomic tests and synthesize their key aspects. Methods PubMed, Scopus, Web of Science, and Google Scholar were searched. Inclusion criteria were documents describing evaluation frameworks for genetic/genomic tests, that were original, specifically created, and covering at least three assessment domains. This study was supported by the EC and MUR under PNRR - M4C2-I1.3 Project PE_00000019 ‘HEAL ITALIA’. Results Overall, 12546 unique records were screened, of which 67 documents were assessed for eligibility. A total of 29 studies were included, reporting 24 different frameworks. These frameworks were published between 2000 and 2019, mostly from USA (50%), Canada (13%) and UK (13%). There was substantial interest in the economic facets of the technology (92%), albeit without extensive detail, and high attention was given to its technical accuracy (70-90%). The clinical value was also consistently mentioned (70-90%), similarly to legal, ethical, and social considerations (70-80%). However, there was minimal emphasis on non-health outcomes (20-50%), and insufficient attention to organizational, educational, and implementation barriers (20-50%). Discussion A pressing need exists for a universally accepted evaluation framework for genetic/genomic tests. Applying a general HTA methodology, potentially based on the EUnetHTA core model, that can integrate solid theoretical and methodological principles, and result in a validated, comprehensive, and widely shared tool for genetic test evaluation, is a viable option to foster the implementation of these technologies. Key messages • Genetic/genomic test evaluations focus on technical accuracy and clinical value, often missing economic and organizational aspects. • A universal, validated, HTA-based evaluation framework for genetic/genomic tests is needed to enhance their implementation in clinical practice.

Indicators of the clinical utility of genetic or genomic testing: a scoping review

Abstract Background Genetic or genomic tests are the cornerstones of personalized prevention. Previous reviews primarily focused on theoretical frameworks rather than actual assessments and assessed dimensions rather than specific indicators within these dimensions. Our scoping review aimed to review the dimensions and the specific indicators measured in the published assessment reports of genetic or genomic tests. Methods We selected formal assessments of genetic and genomic test used for prevention through PubMed, Web of Science, Scopus, websites of 20 different organizations, Google, and Google Scholar. Also, through a separate analysis we identified ten comprehensive dimensions of clinical utility through an analysis of 30 theoretical frameworks for genetic and genomic tests. From the included assessments, we extracted all the indicators, clustering them based on the comprehensive dimensions. Results From 3054 unique references and 12000 grey literature search results, we collected 57 assessments, all from high income-countries (in Europe (53%) and North America (47%)). The assessment methods were HTA (42%), EGAPP (25%), ACCE (21%) and other (12%). 951 disease-specific indicators were extracted from the assessments. Analytic validity (60%), clinical validity (79%) clinical efficacy (79%), economic impact (58%) were the most common dimensions (had at least one indicator). However, only 12 assessments (21%) include indicators that compared health outcomes between tested and untested groups. Equity, acceptability, legitimacy, and personal value were evaluated in less than 15% of the documents. Conclusions Our study illustrate that, although dimensions such as equity and acceptability, are significantly emphasized in traditional evaluation frameworks, these are often not considered in the assessments. Additionally, our study has underscored a significant dearth of primary evidence concerning the clinical efficacy of these technologies. Key messages • Our study highlights the prevalence of evaluations lacking indicators for equity, acceptability, and legitimacy dimensions and a lack of direct evidence of clinical efficacy. • The catalog of indicators was openly provided as a resource for researchers and institutions working in evidence-based evaluations of personalized preventive healthcare.

Validated translation of telehealth and health app usability questionnaires to 8 European languages

Abstract International projects such as the Joint Action on strengthening eHealth including telemedicine and remote monitoring for health care systems for cancer prevention and care (eCAN) are implemented in countries with different languages, requiring validation of translations of standardized tools. Evaluation frameworks for telemedicine services necessarily include the patient’s perspective, i.a. on the service usability. Due to the lack of validated questionnaires in multiple European languages we sought to develop translation for the languages of the eCAN pilot countries. Based on literature review, we selected mHealth App Usability Questionnaire (MAUQ) and Telehealth Usability Questionnaire (TUQ) and developed a conceptual definition for each item. Next, we performed forward and back translation from English into the target languages, expert review, cognitive interview (CI) with 5-10 cancer or chronic disease patients per language/per questionnaire. Qualitative CI results were scored by 2 independent researchers for consistence of each item with conceptual definitions for final review by native speaker expert. We chose 8 languages (ES, GR, HU, IT, LT, NL, PT, SI) for validation. CI involved 48 respondents (29 females, 60%; 18 aged&amp;gt;60, 38%;23 higher education, 48%) for TUQ and 48 (22 females, 46%; 16 aged&amp;gt;60, 33%; 23 higher education, 48%) for MAUQ. Overall 872 (87%) of TUQ and 728 (84%) of MAUQ item answers were classified as consistent. In more than one language we identified concepts difficult to understand: consistent navigation when moving between screens, distinction between easy to use/learn, social settings. The validation process allowed us to identify opportunities to improve translation and highlighted concepts generally more difficult to understand. We provided validated translation of MAUQ and TUQ in 8 European languages for use in eCAN pilots and future telemedicine evaluation studies. Key messages • eCAN successfully validated translations of telemedicine usability questionnaires for 8 languages. • Multilingual translation validation can provide insights into concepts difficult across languages.

Bridging ESG and SDGs Clean Air and Water Through CSR Committee: A New Pathway for Sustainability in the Utility Sector

This study aims to bridge Environmental, Social, and Governance (ESG) principles with the Sustainable Development Goals (SDGs), particularly related to clean air and water, through the role of the Corporate Social Responsibility (CSR) committee in the utility sector. As global awareness of climate change rises, utility companies worldwide are adopting ESG strategies to reduce carbon emissions, improve energy efficiency, and manage water resources. This research proposes a new evaluation framework to assess the contribution of utility companies to achieving SDGs focused on air and water quality. Using data from annual reports and Refinitiv LSEG, the study examines the relationship between ESG scores, asset size, debt-to-equity ratio, and the moderating role of CSR committees in achieving SDG goals. Regression analysis results indicate that ESG scores have a significant positive influence on company contributions to achieving SDGs, especially in terms of clean air and water quality. Additionally, the CSR committee acts as an effective moderator in optimizing asset use and managing company debt to meet sustainability goals. This study provides both theoretical and practical contributions to support evidence-based decision-making for more sustainable policies in the utility sector.

The effect of blended tube feeding compared to conventional formula on clinical outcomes in adults: A systematic review.

To explore the effect of blended tube feeding compared to conventional formulas on nutritional status, quality of life, anthropometry, diarrhoea and tube blockages in adults receiving tube feeding. The protocol was registered (PROSPERO CRD42022372443). Five databases (MEDLINE, EMBASE, CINAHL, Scopus, CENTRAL) were searched from commencement of database to 14th June 2023 to identify studies comparing blended tube feeding to conventional formulas in adults receiving tube feeding. Certainty assessment was conducted using the Cochrane Risk of Bias Tools and the Grading of Recommendations, Assessment, Development and Evaluations framework, and a narrative synthesis of results is provided. From 4227 studies screened, eight were included (total n = 763 patients, 9-215 patients). Three studies were hospital-based and five were home-based with duration from 8 days-8 months. Blended tube feeding and conventional formulas were nutritionally equivalent in only three studies; energy and protein concentration of formulas ranged from 1.7-7.1 kJ/mL and 21-68.5 g/L for the blended tube feeding groups, and 4.2-6.7 kJ/mL and 39-100 g/L for the conventional groups. No studies assessed nutrition status or quality of life using validated measures. Blended tube feeding was associated with a clinically relevant reduction in diarrhoea with a low level of certainty. For remaining outcomes, there were inconclusive findings and an overall very low certainty of evidence for each. The effect of blended tube feeding compared to conventional formulas on all outcomes remains uncertain. Blended tube feeding may reduce the incidence of diarrhoea. Future research using nutritionally equivalent comparisons and validated outcome measures is required.

A preliminary exploration of quantitative models for evaluating binocular visual perception impairments among patients diagnosed with stroke.

The objective of this study is to formulate and implement graded biological models pertaining to binocular visual perception function with the use of computer algorithms. We aim to quantitatively assess the location, severity, and degree of impairment in binocular visual perception among patients who have suffered stroke, thereby providing valuable insights into the repercussions of cerebral tissue damage on the visual system. To overcome the shortcomings of previous instruments used to assess binocular function in terms of stereoscopic effects and the challenges posed by physiological and psychological interference during examinations, this study optimized its approach by integrating polarized stereovision technology with computer graphic modeling techniques. This study employed computer models to assess binocular visual perception function in stroke patients. Computer models refer to psychophysical testing methods used to measure binocular visual perception function, including various assessment tasks such as recognizing inverted letters and assessing stereopsis during high-speed movements. The cross-into-circle test was used as a means to quantify perceptual eye position. Subsequently, the collected data was analyzed to assess the magnitude of impairment in binocular visual perception. The results of the study revealed a spectrum of binocular visual perception impairment among patients diagnosed with stroke, demonstrating discernible variations in the recognition of inverted letters and stereopsis across different movement speeds. Importantly, perceptual eye position measurements offered valuable insights into ocular misalignment. The computational models effectively quantified both the spatial distribution and severity of these identified impairments. Damage to brain tissue resulting from a stroke can give rise to notable impairments in binocular visual perception function. Graded biological models, formulated through computer algorithms, provide a systematic framework for the comprehensive evaluation and quantification of these impairments. The comprehension of the nature and extent of visual impairments observed in patients with stroke establishes a basis for the development of personalized visual perception learning methodologies. Based on such tailored approaches, we aim to facilitate the recovery of impaired visual function, thereby contributing to the broader objective of neural system rehabilitation.