National Evaluation System Research Articles

A comparative analysis of the evaluation system for comprehensive reform of agricultural water prices in China

Water is a fundamental natural resource that affects human and social development, and also plays an important role in modern agricultural production and food security. In China, agricultural water consumption accounts for the majority of the total water consumption. For a long time, China has not fully implemented water pricing and payment for agricultural water, and the lower agricultural water cost have led to the waste of water resource and the rapid conversion of agricultural water to other industries. In order to avoid agricultural water waste and achieve the goal of agricultural sustainable development, the Chinese government has launched a comprehensive reform of agricultural water prices nationwide since 2016. According to the plan, nowadays, the reform will faces overall evaluation and acceptance in 2024 and 2025. However, there is a lack of a relatively unified and flexible nationwide evaluation index system and acceptance measures. In this study, a comparative research of the evaluation system for comprehensive reform of agricultural water prices was developed. Based on existing evaluation and acceptance measures issued by four typical provincial regions, this study compared and analyzed the compliance and similarities and differences of these acceptance measures with national reform policies and requirements. From the aspects of evaluation procedure and index system, a systematic comparative analysis was conducted on the integrity and differences in the four regions. At the same time, suggestions were given for policy-making of national evaluation process and standard. The comparative analysis results showed that the evaluation indices in the four provincial regions have met all the key points of national reform policies and have a complete system, but there were significant differences in the structure of the evaluation index system in different provincial regions. It is recommended to establish a simple, relatively unified and practical national evaluation and acceptance system as soon as possible. In addition, evaluation criteria for individual specific indices should be determined by region based on the differences in economic capacity and agricultural water conservancy facilities conditions.

Lessons on the use of real-world data in medical device research: findings from the National Evaluation System for Health Technology Test-Cases.

Aim: Although the US FDA encourages manufacturers of medical devices to submit real-world evidence (RWE) to support regulatory decisions, the ability of real-world data (RWD) to generate evidence suitable for decision making remains unclear. The 2017 Medical Device User Fee Amendments (MDUFA IV), authorized the National Evaluation System for health Technology Coordinating Center (NESTcc) to conduct pilot projects, or 'Test-Cases', to assess whether current RWD captures the information needed to answer research questions proposed by industry stakeholders. We synthesized key lessons about the challenges conducting research with RWD and the strategies used by research teams to enhance their ability to generate evidence from RWD based on 18 Test-Cases conducted between 2020 and 2022. Materials & methods: We reviewed study protocols and reports from each Test-Case team and conducted 49 semi-structured interviews with representatives of participating organizations. Interview transcripts were coded and thematically analyzed. Results: Challenges that stakeholders encountered in working with RWD included the lack of unique device identifiers, capturing key data elements and their appropriate meaning in structured data, limited reliability of diagnosis and procedure codes in structured data, extracting information from unstructured electronic health record (EHR) data, limited capture of long-term study end points, missing data and data sharing. Successful strategies included using manufacturer and supply chain data, leveraging clinical registries and registry reporting processes to collect and aggregate data, querying standardized EHR data, implementing natural language processing algorithms and using multidisciplinary research teams. Conclusion: The Test-Cases identified numerous challenges working with RWD but also opportunities to address these challenges and improve researchers' ability to use RWD to generate evidence on medical devices.

CDC-Funded HIV Testing Services Outcomes in Ending the HIV Epidemic in the U.S. (EHE) and Non-EHE Jurisdictions, 2021.

Ending the HIV Epidemic in the U.S. (EHE) aims to end the HIV epidemic by focusing on 57 jurisdictions most impacted by HIV. Using 2021 data from the National HIV Prevention Program Monitoring and Evaluation system, we calculated distributions of CDC-funded HIV tests and HIV testing services outcomes in EHE and non-EHE jurisdictions. We conducted chi-square tests and robust Poisson regression to compare differences in outcomes. Of the 1,753,873 tests conducted, a greater proportion were conducted in EHE (65.7%) versus non-EHE (34.3%) jurisdictions (p<0.001). Greater number of persons newly diagnosed were identified in EHE (n=5,861) versus non-EHE (n=2,329) jurisdictions; newly diagnosed positivity was higher in EHE (0.5%) versus non-EHE (0.4%; PR=1.31, 95% CI: 1.25-1.38) jurisdictions. Among persons newly diagnosed, there were no differences in proportion of linkage to care within 30 days in EHE jurisdictions (77.9%) versus non-EHE jurisdictions (77.1%; PR=1.01, 95% CI: 0.95-1.07). Proportion of persons newly diagnosed interviewed for partner services was lower in EHE (69.0%) versus non-EHE (84.8%; PR=0.81, 95% CI: 0.76-0.87) jurisdictions. CDC-funded HIV testing services in EHE jurisdictions conducted more HIV testing and diagnosed more persons. Jurisdictions can further expand HIV testing and related services to continue striving toward EHE goals.

Negotiating between the accountability and the innovation mandates: evidence from Italian schools

ABSTRACT In Italy, the National Evaluation System (SNV), encompassing both internal and external accountability components, is designed not only to hold school actors accountable but also to stimulate innovation and change in pedagogical practices. Nevertheless, existing literature presents inconclusive and contradictory findings regarding the effects and the relationship between school performance-based accountability (PBA) and educational innovation. Utilizing sensemaking and filtering theories, this paper analyses the negotiation of PBA and innovation mandates in Italian schools, emphasizing how school actors interpret, filter, and adapt the policy expectations to their organizational and educational practices. Special attention is given to the influence of local school contexts and performative pressures on this process. The primary data collection method involves interviews with school leadership teams and teachers analysed through an ideal type case analysis. Findings reveal diverse school responses to the double mandates of PBA and innovation. Varied logics of school enactment within the contemporary policy context are identified, influenced by teacher attitudes towards PBA and schools’ socio-economic conditions. Innovation is evident in both advantaged and disadvantaged school contexts, albeit with distinct applications and interpretations. Overall, the adaptation of PBA prerogatives in schools to foster innovation and change is neither uniform nor direct, although external pressures impact educational practices across contexts. Beyond socio-economic factors and administrative and marketing accountability pressures, additional intervening elements include school leadership and staff cohesion as well as school infrastructure and material resources.

Decolonising national evaluation systems

Decolonising national evaluation systems

CDC-Funded HIV Testing and Undiagnosed HIV Infection in Ending the HIV Epidemic in the U.S. Jurisdictions.

Approximately 13% of persons with HIV (PWH) are unaware of their status. To help end HIV, it is important to understand the relationship between the rates of HIV testing and undiagnosed HIV infection. Ending the HIV in the U.S. (EHE) jurisdictions. Using 2021 data from the National HIV Surveillance System and the National HIV Prevention Program Monitoring and Evaluation system, we calculated estimated rates of undiagnosed HIV infections per 100,000 population and rates of CDC-funded HIV tests per 1,000 population. We assessed the association between the two rates using Spearman's rank correlation. We also calculated a rank difference between the two rates for each EHE jurisdiction to help identify jurisdictions with greater unfilled needs for HIV testing. Overall, CDC-funded HIV tests per 1,000 population were positively associated with estimated rate of undiagnosed HIV infection per 100,000 population (rho=0.55, p<0.001). EHE jurisdictions with the greatest magnitude of negative rank differences (i.e., higher undiagnosed HIV infection per 100,000 population and lower CDC-funded HIV tests per 1,000 population) were Prince George's County, MD; Mecklenburg County, NC; Hudson County, NJ; Bronx County, NY, and Hamilton County, OH. In general, CDC-funded HIV testing is being conducted in jurisdictions with the greatest needs. However, we also found large discrepancies between CDC-funded HIV testing and undiagnosed HIV infection rates for some jurisdictions. These jurisdictions may want to identify barriers to their HIV testing services and expand their programs to ensure that all PWH in their jurisdictions are diagnosed.

Genetic Parameters for Limousine Interbeef Genetic Evaluation of Calving Traits.

The aim of this study was to estimate across-country genetic correlations for calving traits (birth weight, calving ease) in the Limousine breed. Correlations were estimated for eight populations (Czech Republic, joint population of Denmark, Finland, and Sweden, France, Great Britain, Ireland, Slovenia, Switzerland, and Estonia). An animal model on raw performance accounting for across-country interactions (AMACI) was used. (Co)variance components were estimated for pairwise combinations of countries. Fixed and random effects were defined by each country according to its national genetic evaluation system. The average across-country genetic correlation for the direct genetic effect was 0.85 for birth weight (0.69-0.96) and 0.75 for calving ease (0.62-0.94). The average correlation for the maternal genetic effect was 0.57 for birth weight and 0.61 for calving ease. After the estimation of genetic parameters, the weighted bending procedure was used to compute the full Interbeef genetic correlation matrix. After bending, direct genetic correlations ranged from 0.62 to 0.84 (with an average of 0.73) for birth weight and from 0.58 to 0.82 (with an average of 0.68) for calving ease.

Testing a Cloud-Based Model for Active Surveillance of Medical Devices with Analyses of Coronary Stent Safety Using the Data Extraction and Longitudinal Trend Analysis (DELTA) System.

To demonstrate the use of the Data Extraction and Longitudinal Trend Analysis (DELTA) system in the National Evaluation System for health Technology's (NEST) medical device surveillance cloud environment by analyzing coronary stent safety using real world clinical data and comparing results to clinical trial findings. Electronic health record (EHR) data from two health systems, the Social Security Death Master File, and device databases were ingested into the NEST cloud, and safety analyses of two stents were performed using DELTA. This is an observational study of patients receiving zotarolimus drug-eluting coronary stents (ZES) or everolimus eluting coronary stents (EES) between July 1, 2015 and December 31, 2017. After exclusions, 3334 patients receiving EES and 1002 receiving ZES were available for study. Analysis using inverse probability weighting showed no significant difference in one-year mortality or major adverse cardiac events (MACE) for EES compared to ZES [Mortality Odds Ratio 0.94 (95% CI 0.81-1.175); p = 0.780] [MACE Odds Ratio 1.04 (95% CI 0.92-1.16; p = 0.551]). Analysis using propensity matching showed no significant difference in EES one-year mortality (547 of 992 alive and available after censoring) compared to ZES (546 of 992) [Log-Rank statistic 0.3348 (p = 0.563)]. Automated cloud-based medical device safety surveillance using EHR data is feasible and was efficiently performed using DELTA. No statistically significant differences in 1-year safety outcomes between ZES and EES were identified using two statistical approaches, consistent with randomized trial findings.

Розробка і впровадження нових організаційних структур в публічному секторі

Розробка і впровадження нових організаційних структур в публічному секторі

THE PERSPECTIVES OF THE PUBLIC POLICIES AND PROGRAMS EVALUATION SYSTEM IN ROMANIA IN THE CONTEXT OF IMPLEMENTING THE NATIONAL RECOVERY AND RESILIENCE PLAN

The implementation of the Recovery and Resilience Facility boosts the EU's power to transform Member States, especially in the field of public management. The conditioning of the Member States to link proposed investments in their plans in order to overcome the effects of the crisis generated by the COVID-19 pandemic, but also to boost the implementation of the Green Pact on structural reforms transforms this temporary instrument in the main driver for deepening European integration. In the case of Romania, one of the important results pursued as part of the reforms related to the public administration sector, correlated with those related to the reform of public finances, refers to the improvement of the planning system of public policies and programs in accordance with budgetary scheduling. In this context, it is expected that following the implementation of these structural reforms, the national system for public policies and programs evaluation will strengthen significantly. The article aims to discuss certain aspects related to the dynamics of the use of public policies and programs evaluations in various stakes specific to the public management sector. Thus, in the first part of the article, the dynamics of the institutionalization of the national system for public policies and programs evaluation is described and analysed. In the second part, the strengthening premises of the national evaluation system for public intervention are analysed as a result of the implementation of a package of structural reforms which are specific to strategic planning and public management.

Innovative curriculum design and evaluation for achieving diversity, equity and inclusion in the Indian higher education system

PurposeIndia has been withstanding increasing pressure of enrolment in the higher education system, resulting in the creation of new universities in consonance with the recommendations of the Knowledge Commission (2007). Barring a few institutions of paramount excellence, the mushrooming universities fail to conform to equitability of quality and standards, that is teaching-learning-dissemination and research, except for accommodating higher gross enrolment ratio. It has resulted in an asymmetric and sporadic development of human resources, leaving a large basket of learners out of the pursuit for aspiring higher academic, research and professional enrichment. The country needs to develop an innovative common minimum curriculum and evaluation framework, keeping in view the trinity of diversity, equity and inclusion (DEI) across the Indian higher education system to deliver human resources with equitable knowledge, skill and intellectual acumen.Design/methodology/approachThe paper has been developed using secondary information.FindingsThe manuscript has developed an innovative teaching-learning framework that would ensure every Indian HEI to follow a common minimum curriculum and partial common national evaluation system so that the learners across the country would enjoy the essence of equivalence.Originality/valueThis research has designed a comprehensive model to integrate the spirit of the “DEI” value proposition in developing curriculum and gearing common evaluation. This would enable the country to reinforce the spirit of social equity and the capacity to utilise resources with equitability and perpetuity.

Steering the Direction of Research through Organizational Identity Formation

Public research organizations respond to external pressures from national research evaluation systems, performance-based funding systems and university rankings by translating them into internal goals, rules and regulations and by developing organizational identities, profiles and missions. Organizational responses have primarily been studied at the central organizational level, and research on the steering of research has primarily focused on the impacts of performance-based funding systems. However, research evaluation exercises may also have a formative impact, especially below the central organizational level. This paper uses a case study of a research unit of a biomedical research school in the Netherlands to explore the organizational response to a relatively critical external assessment report. It shows that the participation in the Dutch research evaluation cycle legitimated the formation of a new organizational identity for the research unit, which functions as a frame that suggests to staff members a new interpretation of the type of research that is at the core of what the research unit does. We identify three additional steering mechanisms that support the enactment of the organizational identity: steering by resource allocation, by suggesting and by re-organizing. We, furthermore, explore the epistemic effects – the direction and conduct of research – of the organizational response, through interview data in combination with a bibliometric analysis.

The Impact of Digital Divide on Household Participation in Risky Financial Investments: Evidence From China

The digital divide has now become a worldwide problem and has the potential to lead to greater inequality. This paper empirically analyses the impact of the “digital access divide”, “digital use divide” and “digital inequality divide” on household participation in risky financial investments using micro data from China. The results show that all three digital divides have a positive and significant impact on the probability of households participating in risky financial investments; in addition, the digital divide between urban and rural areas and between households is also significant. Finally, the authors propose strategies for bridging the digital divide based on China’s national context, such as building a national cultivation and evaluation system of digital literacy, reducing the family’s parenting burden, improving the investment environment for residents, developing the power resources of the younger elderly, and promoting intergenerational digital feedbacks.

Academic identity at the intersection of global scientific communities and national science policies: societal impact in the UK and Netherlands

ABSTRACT This article investigates attitudes to societal impact of research as an entry point into understanding academic identities. Conceptually, we position academic identity at the intersection of global scientific fields and national science policies. We argue that the degree of alignment or misalignment between the two can create coherent academic identities, or on the contrary, tensions in academics’ identity. Empirically, we use the disciplines of philosophy and anthropology as proxies for scientific fields in the social sciences and humanities (SSH). The study is based on sixteen semi-structured interviews with mid-career philosophers and anthropologists in the United Kingdom and the Netherlands, and an analysis of how societal impact is positioned in the two national evaluation systems. We conclude that ‘coercive’ national impact policies (like the one in the UK) are less likely to be aligned with global disciplinary norms in the SSH and therefore create tensions in academic identity; these can undermine academics’ agency and be counterproductive in terms of reaching policy objectives. By contrast, ‘enabling’ national impact policies (like the one in the Netherlands) are conducive to more coherent academic identities that are better aligned with disciplinary notions of societal impact. By discussing academic identities in a comparative context, the study highlights the struggles of reconciling disciplinary and national notions of societal impact. To realise the potential societal impact of academic research, we recommend that impact is integrated into a wider ecosystem of interactions where policy-driven notions are aligned with disciplinary norms and values.

The evolution of research evaluation in China

Reforming the research evaluation system is essentially a struggle between the academic aristocracy, which professes expert approaches to evaluation, and the scientific bureaucracy, which seeks to ensure the dictate of numerical indicators. A representative example is the evolution of this confrontation in China, where a long-term commitment to international scientometric databases made it possible to achieve world leadership in terms of the number of publications. However, these achievements were accompanied by negative effects caused by deterioration in the personnel policy of universities, the reduction of academic freedoms, changes in the publication behavior of researchers, and violations of scientific ethics. This led to a reform of the national evaluation system, namely, the strengthening of the role of expertise, the priority dissemination of scientific information at the national level, and the restriction of open access publications. This considers the history of the formation of and changes in the research evaluation system in China, its positive and negative effects, as well as new scientific practices, their likely consequences, and the possibility to apply them in other countries.

Teacher Evaluation Documents and Procedures in High-Performing APR–Countries: Hong Kong and Singapore

The paper highlights the issue of teacher’s professional competence assessment and standardization in teacher professional activity as essential for improving teaching quality. The authors refer to the best-world experience in teacher evaluation and accountability in two top-performing educational systems in the world – those of Hong Kong and Singapore. This paper is an in-depth case study that aims to describe and compare the competency-based tools used there for teacher performance evaluation. The following aspects of the documents are paid attention to in the paper: the purpose of the standards/document, the target groups and, the structure. In the course of the research, some theoretical methods, including a comparative analysis, the method of evaluation and systematization of the used materials on the selected theme were used. The analysis of the professional standards for teachers in Hong Kong and Singapore has shown that, along with the differences in certain aspects, there are some features which are likely to make them effective in practice. It is assumed that using these documents contribute to holding these countries’ senior positions in performing teacher assessment systems and in the sphere of world education as well. The paper concludes by suggesting its contribution to the discussion of the problem of the development and implementation of a proper unified national teacher evaluation and accountability system based on competency-oriented teacher standards in Russia. It is also expected that the findings could be useful in the development of the Unified Federal Evaluation Materials for teacher assessment and accountability in the country.

A “three-legged model”: (De)constructing school autonomy, accountability, and innovation in the Italian National Evaluation System

The widespread adoption of school autonomy with accountability reforms in education has generated debate regarding the relationship between autonomy, innovation and accountability. While at the policy design level, these three elements are highly related, several authors highlight the contradictions among them. By analyzing key documents and interviews, this paper aims to identify the program ontology behind the current Italian National Evaluation System (SNV), with a focus on the way in which autonomy, accountability and innovation have been conceptualized and linked together. The paper also aims to explore whether pitfalls and/or tensions exist that might hamper the achievement of the SNV goals. The findings highlight the peculiarities of the Italian autonomy with accountability system, which has resulted from the involvement of different stakeholders in the design and implementation of the reforms. The findings also reveal contradictions regarding some of its premises. Various rationales (improvement, efficiency, equity and transparency) emerge that seem to have acted as drivers of the reforms, however, the influence of globalizing discourses on international competition and the benefits of datafication also appears significant. A number of contextual aspects are finally considered which hamper the expected change mechanisms, highlighting the discontinuous ground in which such policy dispositifs operate.

Integrating real-world data to assess cardiac ablation device outcomes in a multicenter study using the OMOP common data model for regulatory decisions: implementation and evaluation.

The objective of this study is to describe application of the Observational Medical Outcomes Partnership (OMOP) common data model (CDM) to support medical device real-world evaluation in a National Evaluation System for health Technology Coordinating Center (NESTcc) Test-Case involving 2 healthcare systems, Mercy Health and Mayo Clinic. CDM implementation was coordinated across 2 healthcare systems with multiple hospitals to aggregate both medical device data from supply chain databases and patient outcomes and covariates from electronic health record data. Several data quality assurance (QA) analyses were implemented on the OMOP CDM to validate the data extraction, transformation, and load (ETL) process. OMOP CDM-based data of relevant patient encounters were successfully established to support studies for FDA regulatory submissions. QA analyses verified that the data transformation was robust between data sources and OMOP CDM. Our efforts provided useful insights in real-world data integration using OMOP CDM for medical device evaluation coordinated across multiple healthcare systems.

Exploring unique device identifier implementation and use for real-world evidence: a mixed-methods study with NESTcc health system network collaborators

ObjectivesTo examine the current state of unique device identifier (UDI) implementation, including barriers and facilitators, among eight health systems participating in a research network committed to real-world evidence (RWE) generation...

자치경찰 성과평가 가이드라인 구축에 관한 연구

With the introduction of the local police system, it is divided into national police affairs and local police affairs, raising the need to revise the previously implemented national police performance evaluation system to suit the circumstances of each region. Accordingly, the municipal police committees stipulate the evaluation of autonomous police affairs as their jurisdiction, but there are still no specific guidelines for evaluating the performance of autonomous police. Therefore, this study aims to establish guidelines for the performance evaluation of autonomous police by the municipal police committees. Accordingly, three out of six performance indicators have significantly improved the current indicators, two quantitative performance indicators have been maintained, and new traffic indicators have been developed so that the municipal police committees can actually play a role in evaluating the performance of the autonomous police. In addition, it proposes a performance evaluation system to motivate evaluators to pay attention and passion by proposing various incentives such as special pledges for excellent performance, on-site inspections of overseas autonomous police, individual commendations, group commendations, and prize money. As a result, the qualitative performance indicators proposed in this study are expected to increase local residents' trust in local police by autonomously preparing plans for local police affairs, continuously executing them regardless of personnel changes or accidents, and raising appropriate policing performance.