National Evaluation System Research Articles

A stakeholder view of the development of national evaluation systems in Africa

vidence for policy-informed decision-making, budgeting and programming. National evaluation systems (NESs) are being set up across Africa, together with the processes and other monitoring and evaluation (ME) infrastructure for efficient and effective functioning.Objectives: This article seeks to document comparative developments in the growth of systems in Anglophone African countries, and provide an understanding of these systems for capacity-development interventions in these countries. It also aims to contribute to the public debate on the development of national ME systems, institutionalisation of evaluation, and use of ME evidence in the larger African context.Methods: This article uses four key dimensions as the conceptual framework of a national monitoring and evaluation system, including monitoring and evaluation systems in the executive; the functioning of parliamentary ME systems; professionalisation of evaluation and existence of an enabling environment. A questionnaire was used to collect information based on the key dimensions from government and non-governmental personnel. The Mo Ibrahim index of 2018 was used to collect information on enabling environment.Results: Findings indicate that all systems have stakeholders with different roles and contexts and are designed according to the state architecture, prevailing resources and capacities.Conclusions: This article concludes that the findings can be used as different entry points for developing and strengthening ME capacities in countries studied.

State of monitoring and evaluation in Anglophone Africa: Centre for Learning on Evaluation and Results in Anglophone Africa’s reflections

This article is an overview of what the Centre for Learning on Evaluation and Results in Anglophone Africa (CLEAR-AA) is currently learning in its work implementing monitoring and evaluation (M&E) capacity strengthening programmes in our partner countries. This article is based on the reflections drawn from the authors’ experiences and the work of CLEARAA in strengthening M&E systems across the continent. It serves as a contribution to larger ongoing strategic conversations about how to promote evidence-informed decision-making for better development outcomes. The article begins with a discussion on systems broadly and M&E systems in particular, with a specific focus on some of the historical roots of the current ways in which M&E is defined and implemented in African systems of governance. We continue to discuss the various elements that come into play in establishing and institutionalising M&E systems, in particular the ‘M&E Market’ and the demand for evidence, where we also challenge the notion of the unidirectional demand and supply chain of evaluation. The institutional architecture within which M&E systems operate is next discussed, and how the formal (and informal) laws, policies, boundaries and rules continue to provide some degree of leverage in support of these systems. The article finally addresses two key elements of developing and sustaining M&E systems: the role of leadership and developing an evaluative culture. The authors explain why these elements, which often receive less attention than the technical elements in building and strengthening M&E systems, carry such weight in sustaining national evaluation systems.

What Does PISA Tell Us About Performance of Education Systems?

The performance of education systems is an area where intense discussions take place around the world. Since international student achievement researches, such as the Program for International Student Assessment (PISA), provide information about readiness of students to modern society and participation to workforce differences in achievements between countries create great pressure for policy makers. Due to this pressure, many countries implement serious revisions in their education systems without adequately focusing on the national background of differences. Lack of national monitoring studies also increases the probability of inaccurate identifying the background of problems. In this study, the factors that function on the background of achievement differences between countries, especially in PISA studies, are tried to be determined and the most important factors that are effective in student achievement, which are independent of the cultural and social context differences of the countries, are tried to be specified. It is seen that the three major factors are teacher quality, delaying students' school tracking and allocating resources to schools, considering disadvantaged schools as priority. In addition, it is recommended to establish a national monitoring and evaluation system to evaluate international student achievement research results in healthier way and to develop more realistic policies.

The New Research Assessment Reform in China and Its Implementation

A radical reform of research assessment was recently launched in China. It seeks to replace a focus on Web of Science-based indicators with a balanced combination of qualitative and quantitative research evaluation, and to strengthen the local relevance of research in China. It trusts the institutions to implement the policy within a few months but does not provide the necessary national platforms for coordination, influence and collaboration on developing shared tools and information resources and for agreement on definitions, criteria and protocols for the procedures. Based on international experiences, this article provides constructive ideas for the implementation of the new policy. Policy highlightsIn response to the three main messages of the new policy, we suggest these possible solutions for the implementation:Farewell to “SCI worship”: With the move away from Web of Science as a standard, an integrated research information system and a national journal evaluation system is needed.From metrics to peer review: The function and weight of peer-review evaluation needs to be differentiated between the levels of the research system: individuals, units, institutions, and national agencies.New priority to local relevance: The optimal balance between globalization and local relevance must be allowed to differ by type and field of research.

From evidence to values-based decision making in African parliaments

Monitoring and Evaluation discourse in Africa has evolved to focus on building systems at a national level. While this systemic approach has many advantages, its implementation often runs up against the uncomfortable reality that governments have complex incentives to use evidence, and this evidence can equally contribute to decision making that is neither development-focused nor democratic if values are not part of the conversation. Much of the literature on public-sector reform focuses on evidence-based policy making. While relevant, it does not reflect on values, and this article will argue that acknowledging the central role values play in interpreting evidence is critical to effective national evaluation system building. To make this argument, this article will present and discuss vignettes from the parliaments of Malawi, Tanzania, Uganda, Zambia and Zimbabwe that illustrate the pivotal role values have played in interpreting and acting on evidence in a parliamentary context. Finally, it makes a case for the discourse about evidence-based policy making to consider values-based policy making as an appropriate lens for parliaments to acknowledge and engage with the complex landscape of the politics of evidence.

UNBK: TRANSFORMASI SISTEM EVALUASI STANDAR NASIONAL DI ERA REVOLUSI INDUSTRI 4.0

Education is the process of transferring knowledge. In the process, it is necessary to design a quality curriculum. In the curriculum, one of the essential things is evaluation. Educators can make evaluations to students in order to obtain information on student development. In the national education system, the state pays great attention to education. The state's attention in education is to hold a national exam to equalize the quality of education in Indonesia. The national standard evaluation system is still changing in terms of terms and technicalities. Recently, computer-based evaluation systems are the latest step and innovation. Especially in the current era of the industrial revolution 4.0, digitization systems are in all aspects of life, including in the world of education. Since 2014, the national standard education evaluation system began using the online examination system. Although in 2014, not all of the schools in Indonesia were able to conduct and access computer-based examinations with the internet network. The program requires sufficient time for socialization and technical preparation. The spirit of revolution 4.0 is marked by building a digitalized national standard evaluation system. Even in response to the development of the Industrial Revolution era 4.0, Indonesia has made innovative breakthroughs in the world of education, namely the Computer Based National Examination (UNBK) system

Research for improvement of the national evaluation program for emergency medical center in Korea

In Korea, the national evaluation program for emergency medical centers (EMCs) was introduced in 2003. EMCs in Korea have three levels of emergency centers including regional emergency centers, local emergency centers, and local emergency rooms. The evaluation system assesses the performance of EMCs at all three levels. The role of both regional emergency centers and local emergency centers administer the final treatment for severe emergency patients, although the definition of severe emergency patients is unclear. These factors may aggravate crowding at the emergency department. The national evaluation system aims to reduce the length of stay in the emergency department. However, reduction of the length of stay at emergency department may cause a conflict with the administration of final treatment for critically ill emergency patients owing to a lack of in-hospital beds. Crowding of the emergency department is not a problem of EMCs but that of the health system. In 2019, the Korean Society of Emergency Medicine performed a study to improve the national evaluation program for EMC and the performance of EMC. Here, we have summarized the results.

National evaluation program for emergency medical center

Since 2003, the national evaluation program for emergency medical centers (EMCs) has been managed by the Central Emergency Medical center which is controlled by the Ministry of Health and Welfare. Although the evaluation system for EMC has contributed to the development of the structure of emergency centers and the expansion of emergency resources (emergency medical person and equipment), it has some limitations in terms of quality control for both EMCs and emergency medical systems. One of the purposes of the evaluation program is to assess the performance of EMC in different levels. However, both regional and local emergency centers have same role that is offering of final treatment to severe emergency patients. There is no role for local emergency rooms in the emergency care of patients. In addition, the national evaluation program does not have outcome indicators that assess the performance of the EMC in emergency care. The improvement of the national evaluation system for EMC is required for the appropriate assessment of the performance of EMCs in the future.

An overview of the provincial evaluation system of the Western Cape Government of South Africa as a response to the evaluation of the National Evaluation System

Background: The Western Cape Government (WCG) has recently implemented its seventh (7th) Provincial Evaluation Plan that is aligned to the National Evaluation Policy Framework adopted by South Africa in November 2011. The National Evaluation System (NES) and its evolution at the provincial level has been aligned to its Province-wide Monitoring and Evaluation System (PWMES). The Department of the Premier in the WCG centrally drives this system. The institutionalisation of a Provincial Evaluation System (PES) serves as a model for any sub-national government to practice government evaluations at provincial, regional or local levels. To date, thirty-three (33) evaluations have been completed and a further eleven (11) are at various stages of implementation. Objectives: Since the initial rollout of the PES in the WCG, the system has developed, strengthened and matured. This comes from a need for better access to, and understanding of, evaluations. Method: The paper has drawn on the recent evaluation of the NES which selected the Western Cape as a case study; and uses the findings to put improvements in place in the areas of policy, methodology, organisation, capacity, participation with other actors, and use. Results: The main value in sharing this work, is the building of an enabling environment to roll out evaluations on a regional level, that are in line with the national evaluation approach, whilst at the same time facilitating integration with provincial strategic priorities. Conclusion: This paper reflects on the way in which the WCG of South Africa has undertaken the implementation and maturation of the NES at a sub-national level.

Zinc Biofortified Rice Varieties: Challenges, Possibilities, and Progress in India.

Zinc malnutrition is a major issue in developing countries where polished rice is a staple food. With the existing significant genetic variability for high zinc in polished rice, the development of biofortified rice varieties was targeted in India with support from HarvestPlus, Department of Biotechnology, and Indian Council of Agricultural Research of Government of India. Indian Institute of Rice Research (IIRR) facilitates rice varietal release through All India Coordinated Rice Improvement Project (AICRIP) and also supports rice biofortification program in India. Various germplasm sets of several national institutions were characterized at IIRR for their zinc content in brown rice using energy-dispersive X-ray fluorescence spectroscopy indicating the range of zinc to be 7.3 to 52.7 mg/kg. Evaluation of different mapping populations involving wild germplasm, landraces, and varieties for their zinc content showed the feasibility of favorable recombination of high zinc content and yield. Ninety-nine genotypes from germplasm and 344 lines from mapping populations showed zinc content of ≥28 mg/kg in polished rice meeting the target zinc content set by HarvestPlus. Through AICRIP biofortification trial constituted since 2013, 170 test entries were nominated by various national institutions until 2017, and four biofortified rice varieties were released. Only the test entry with target zinc content, yield, and quality parameters is promoted to the next year; thus, each test entry is evaluated for 3 years across 17 to 27 locations for their performance. Multilocation studies of two mapping populations and AICRIP biofortification trials indicated the zinc content to be highly influenced by environment. The bioavailability of a released biofortified rice variety, viz., DRR Dhan 45 was found to twice that of control IR64. The technology efficacy of the four released varieties developed through conventional breeding ranged from 48 to 75% with zinc intake of 38 to be 47% and 46 to 57% of the RDA for male and female, respectively. The observations from the characterization of germplasm and mapping populations for zinc content and development of national evaluation system for the release of biofortified rice varieties have been discussed in the context of the five criteria set by biofortification program.

Новые вызовы для российской внешней помощи в контексте достижения целей устойчивого развития

The article discusses current trends in the Russian practice of international development assistance (IDA). Despite international isolation, the Russian government continued to increase its allocations for official development assistance (ODA), which, from 2015, amount to about USD1 billion annually. The author identifies key problems in the field of IDA related to monitoring and evaluation, the achievement of the Sustainable Development Goals (SDGs) and interaction with the business sector. The author describes the actions of the government in its quest to solve these problems. Special emphasis is placed on a comparative analysis of the SDG indicators with indicators of national development goals until 2024. This makes it possible to link Russian goals with international priorities. The theoretical basis of this study is the ‘systemic change’ and ‘scaling up approach’, which are often used in research on development economics. A “large-scale approach” allows us to measure the scale of Russia's participation in construction and installation work in terms of the number of allocated resources, the number of stakeholders involved, geographical coverage, etc. The “systemic change” approach explains how Russia's development cooperation activities are aligned with national goals and the SDGs, as well as the transformation in the structure and dynamics of the system, which leads to an impact on the material conditions or behavior of stakeholders. In conclusion, the author reflects on the prospects for the creation of a national monitoring and evaluation system in the area of ​​IDA, as well as on the possibilities of contributing to the achievement of the SDGs by 2030. One of the directions could be increased participation in the formation of new international development institutions, including the creation of the New Development Bank (NDB) and the Asian Infrastructure Investment Bank (AIIB) along with the expansion of bilateral programs in the field of international development assistance.

Protection Status and Development Strategies of Traditional Villages in Northwestern Jiangxi Province

5 traditional villages in northwestern Jiangxi province are taken as the research objects in this paper, it carries on the field investigation and data arrangement on the content of the national traditional village evaluation and identification system, including the site selection and preservation of village pattern; the protection and utilization of traditional buildings; the non-material cultural heritage and its physical space environment. The results show that these traditional villages protection is faced with the problems of lack of integrity protection concept, improper construction management mode and conflict among relevant subjects. Finally, in view of the current situation, the protection and development strategies of traditional villages are put forward from three respects, Human-oriented thought, Development model and Industrial structure.

University social responsibility: perceptions and advances

Purpose The purpose of this paper is to analyze the perception of professors, students, administrative staff and academic directors about social responsibility in four universities of Rio Grande do Norte (RN) – Brazil. Design/methodology/approach The research was carried out in four universities in RN, Brazil. A survey was conducted with 1,318 students, 118 members of the administrative team, 104 professors and 29 academic directors, to whom a Likert-type questionnaire was applied, with 65 items allocated in seven dimensions as follows: organizational governance, human rights, work practices, environment, legal practices, consumer issues and community involvement and development. Exploratory factorial analysis and Cronbach’s alpha were the statistics used to analyze the data. Findings The research showed difficulties in assessing social responsibility aspects because of the lack of discussion and little socialization and debate of the data. Concerning everyday issues such as student service, environment issues, working conditions and local development is an exception because they have explicit opinions and often negative perceptions. Research limitations/implications The data used were self-reported they were subjected to bias, such as self-attribution and selective perception by community members. The lack of a theoretical model of social responsibility to study a non-profit organization. This study offers a theoretical contribution to analyzing six variables in non-profit organizations. Practical implications The data collection instrument also contribute to Brazilian universities can meet the evaluation criteria of higher education institutions in Brazil by Law 10.861/04, which established the national higher education evaluation system. Social implications This contribution allowed the authors to understand, which of them find the most consensus among stakeholders and which are the most contradictory, as well as to promote improvements in the implementation of social responsibility policies. The proposal of the instrument results in concrete actions for the adoption of the university, to accelerate the implementation of ISO 26000 in its statutes. Originality/value The study on social responsibility at universities of RN, Brazil, under dimensions of organizational governance, human rights, work practices, environment, legal practices, consumer issues, and community involvement and development is innovative and supports other universities to think their processes and better contribute in training new generations of citizens and professionals.

Building interdisciplinary research teams according to the requirements of the national research evaluation system

Aim/purpose – The paper explores the issue of building interdisciplinary research teams from the point of view of the evaluation requirements for Polish researchers and research units. The main aim of the paper is to determine the possibility of creating interdiscipli-nary research teams involving management and economics researchers.Design/methodology/approach – The author searched 30,404 journals. An assumption was made that team creation should reflect the evaluation requirements of several meas-urement factors of similarity between disciplines, these were duly developed. An analy-sis of the possibility of developing effective interdisciplinary teams to maximise the possible number of places for publications and points scored was performed and dis-cussed. The analysis was performed by considering all of the scored journals useful for the development of young researchers and well-regarded journals publishing original research prepared by experienced scientists.Findings – The analysis indicates that the relevant journals are not evenly spread among the various scientific disciplines examined. Considering the possibility of finding many shared journals for publication while achieving favourable interdisciplinary research outcomes and scoring a high number of evaluation points, researchers in the fields of economics as well as finance and management and quality sciences should mainly cooperate with researchers in the field of socio-economic geography and sociological sciences.Research implications/limitations – The analysis was based on the Polish national research evaluation system, which may limit the generalisation of the results. Originality/value/contribution – The results presented in the paper may be useful for researchers, research team managers and authorities who run research units and create effective research teams.

Research on the Development of Machine Tool Industry in China

This study focuses on the development of the high-end manufacturing equipment industry in China, represented by machine tools. The machine tool referred to herein includes not only the machine tools in traditional definition, such as lathes, milling machines, planers, drilling machines, boring machines, grinders, and gear making machines, but also additive manufacturing equipment and additive-subtractive hybrid manufacturing equipment. This study analyzes the development trends of the machine tool industry in China and the globe, and then studies the problems existing in China’s machine tool industry. Compared with world’s advanced level, China’s machine tool industry still has weaknesses in (1) basic research and key technologies for ultra-precision machine tools, (2) the machining accuracy and efficiency of large-scale machine tools, (3) the reliability and accuracy retention of the whole machine performance, (4) the research and development of common technologies of the advanced additive manufacturing and hybrid manufacturing equipment, and (5) the level of intelligence of high-end machine tools. Furthermore, we propose the key direction of China’s machine tool industry during the 14th Five-Year Plan and the following five to ten years, and put forward some policy measures for the development of machine tool industry in China, including coordinating national science and technology plans with national policy supporting and evaluation systems, and establishing and strengthening the common technology collaborative innovation system for the high-end manufacturing equipment represented by machine tools.

Development and implementation of a monitoring and evaluation system of a catering in educational organizations of the Russian Federation as a management problem

The article considers the model of the system of monitoring and evaluation of nutrition organization in educational organizations, which can be the basis of the National Monitoring and Evaluation System, which is an element of the Catering Concept in educational organizations of the Russian Federation.

Analysis of Math Test Score Gap Between White and Non-White Students in Brazilian Public Schools: SAEB 2015

This study aims to analyze and decompose the gap between White and non-White students’ test scores observed in the Brazilian National Evaluation System of Basic Education (SAEB)—2015. To do so, proficiency equations were estimated for each student race group (White, Brown, Black, and Indigenous) using the recentered influence function method, which generalizes the Oaxaca–Blinder decomposition to any point in the grade distribution. The proficiency of White students is higher than the other groups of students, and this difference increases throughout the distribution. Black students exhibit the lowest average test score, behind the Indigenous and then the Brown students. At the lower and middle levels of the math test score distribution, the racial gap is mainly due to the characteristics effect, which represents the part of the proficiency differential that is explained by the differences in observable characteristics of students. However, the gap at the upper level of grade distribution is mainly due the structural effect, which results from non-observable issues, such as discrimination and stereotype threats.

Claims-based cardiovascular outcome identification for clinical research: Results from 7 large randomized cardiovascular clinical trials

Medicare insurance claims may provide an efficient means to ascertain follow-up of older participants in clinical research. We sought to determine the accuracy and completeness of claims- versus site-based follow-up with clinical event committee (+CEC) adjudication of cardiovascular outcomes. We performed a retrospective study using linked Medicare and Duke Database of Clinical Trials data. Medicare claims were linked to clinical data from 7 randomized cardiovascular clinical trials. Of 52,476 trial participants, linking resulted in 5,839 (of 10,497 linkage-eligible) Medicare-linked trial participants with fee-for-service A and B coverage. Death, myocardial infarction (MI), stroke, and revascularization incidences were compared using Medicare inpatient claims only, site-reported events (+CEC) only, or a combination of the 2. Randomized treatment effects were compared as a function of whether claims-based, site-based (+CEC), or a combined system was used for event detection. Among the 5,839 study participants, the annual event rates were similar between claims- and site-based (+CEC) follow-up: death (overall rate 5.2% vs 5.2%; adjusted κ 0.99), MI (2.2% vs 2.3%; adjusted κ 0.96), stroke (0.7% vs 0.7%; adjusted κ 0.99), and any revascularization (7.4% vs 7.9%; adjusted κ 0.95). Of events detected by claims yet not reported by CEC, a minority were reported by sites but negatively adjudicated by CEC (39% of MIs and 18% of strokes). Differences in individual case concordance led to higher event rates when claims- and site-based (+CEC) systems were combined. Randomized treatment effects were similar among the 3 approaches for each outcome of interest. Claims- versus site-based (+CEC) follow-up identified similar overall cardiovascular event rates despite meaningful differences in the events detected. Randomized treatment effects were similar using the 2 methods, suggesting claims data could be used to support clinical research leveraging routinely collected data. This approach may lead to more effective evidence generation, synthesis, and appraisal of medical products and inform the strategic approaches toward the National Evaluation System for Health Technology.

Evaluation2 – Evaluating the national evaluation system in South Africa: What has been achieved in the first 5 years?

Background: South Africa has pioneered national evaluation systems (NESs) along with Canada, Mexico, Colombia, Chile, Uganda and Benin. South Africa’s National Evaluation Policy Framework (NEPF) was approved by Cabinet in November 2011. An evaluation of the NES started in September 2016.Objectives: The purpose of the evaluation was to assess whether the NES had had an impact on the programmes and policies evaluated, the departments involved and other key stakeholders; and to determine how the system needs to be strengthened.Method: The evaluation used a theory-based approach, including international benchmarking, five national and four provincial case studies, 112 key informant interviews, a survey with 86 responses and a cost-benefit analysis of a sample of evaluations.Results: Since 2011, 67 national evaluations have been completed or are underway within the NES, covering over $10 billion of government expenditure. Seven of South Africa’s nine provinces have provincial evaluation plans and 68 of 155 national and provincial departments have departmental evaluation plans. Hence, the system has spread widely but there are issues of quality and the time it takes to do evaluations. It was difficult to assess use but from the case studies it did appear that instrumental and process use were widespread. There appears to be a high return on evaluations of between R7 and R10 per rand invested.Conclusion: The NES evaluation recommendations on strengthening the system ranged from legislation to strengthen the mandate, greater resources for the NES, strengthening capacity development, communication and the tracking of use.

Evaluation systems and the pace of change – The example of Swedish higher education1

The aim of this article is to illuminate and discuss evaluation and evaluation systems in relation to the pace of change. It is argued that evaluation promotes and accelerates change. The article will thus contribute to a critical scrutiny of evaluation as a societal phenomenon and as a widespread practice in education. To accomplish this aim, the inherent purpose of evaluation and evaluation systems is brought forward. National evaluation systems for Swedish higher education are used as an empirical example. An analysis using Rosa’s three aspects of social acceleration (technical acceleration, acceleration of social change, and acceleration of the pace of life) is offered to demonstrate how the evaluation systems are related to, sustain, and promote an increase of the pace of change (acceleration) in educational practice in higher education.