Education Assessment System Research Articles

Construction and Application of OBE-based Multiple Formative Assessment System in the “Micro-lecture + PAD Class”

To construct a scientific and effective curriculum assessment system of higher education is an effective measure to improve classroom teaching quality, a powerful guarantee to enhance students’ classroom participation and enthusiasm, and an important way to achieve fair learning evaluation. Based on a brief introduction of OBE and a comprehensive review of the current research situation of formative assessment, this paper analyzes the existing problems in the curriculum evaluation of higher education, constructs an OBE-based multiple formative evaluation system, and tries to apply it in the “micro-lecture + PAD class”. Finally, the author carries out a controlled experiment and makes a quantitative analysis of the relevant data obtained in the experiment. The results of the study show that the OBE-based multiple formative assessment system plays a positive role in promoting students’ academic performance, improving their autonomous learning ability, and enhancing their self-confidence.

Implikasi Kebijakan Kampus Merdeka Belajar terhadap Manajemen Kurikulum dan Sistem Penilaian Pendidikan Menengah Serta Pendidikan Tinggi

This policy of the Minister of Education and Culture deserves to be called a change that is quite extreme, this then raises some doubts in the minds of academics. Among the big questions that arise from the policy of independent Learning, Free Campus ”is 1. What is the definition of Government policy? 2. How is the concept of Free Learning? 3. What are the implications of the independent learning policy on curriculum management and assessment systems in secondary schools? 4. What are the implications of the policy of independent learning for curriculum management and assessment systems in higher education? This visionary policy of the Minister of Education and Culture deserves appreciation, however, several problems arise from the implementation of the policy of "Free Learning, Independent Campus" that must be resolved immediately. This paper attempts to review the form of the Freedom Learning policy and implementation challenges.

The K-Means Algorithm for Generating Sets of Items in Educational Assessment

In a national-scale educational assessment system, such as the National Examination, the need for several sets of questions that have the same level of difficulty is very required to avoid cheating by students. Therefore, the objective, which is to make a set of questions with the same level of difficulty automatically, is done in this research. It used a machine learning approach, namely K-Means. To achieve this goal, several following procedures need to be implemented. Firstly, we need to create banks of questions to be assigned to students. Then, we build training data by determining the value of each question based on Bloom's Taxonomy, item characters/types, and other parameters. Then, with utilizing K-Means, several cluster centers are obtained to represent the uniformity of the questions in the cluster members. By using several heuristics criteria defined previously, several sets or packages of questions that have the same characteristics and difficulty levels are obtained. From the experiments conducted, the analysis with descriptive (i.e., mean, standard deviation, and data visualization) and inference (i.e., ANOVA) statistics of results are presented showing that questions of each sets have the same characteristics to ensure the fairness of examinations. Moreover, by using this system, the contents of the questions in the generated set do not need to be the same, the package of questions can be generated automatically quickly, and the level of the difficulties can be measured and guaranteed.

Country's Higher Education Assessment System

The higher education system is an important part of a country, based on this, a comprehensive model that allows analyzing the higher education system of any country was developed. Each aspect is evaluated by a small model, and different methods are used to analyze them according to the different influencing factors and their interralationships, The final scores were summed up by the weights calculated by the hierarchical this analysis method and percentaged to obtain the total score.And then we analyze how the model reflects the health and sustainability of the education system. Using this model, we obtained scores for the United States, Japan, Australia and China. With this model, we can not only know the health of the whole higher education system, but also the development and sustainability of the system in all aspects, so that we can plan accordingly.

Semantic-Based Self-Critical Training for Question Generation

Question generation is a conditioned language generation task that consists in generating a context-aware question given a context and the targeted answer. Train language modelling with a mere likelihood maximization has been widely used while suffering from exposure bias and the discordance between the training and the test metrics. In the way of addressing this issue, The presented work portrays a fully Transformer-based reinforcement learning generator-evaluation architecture for neural question generation. To edge the flexibility of the generation, a semantic-based reward score was externally infused during the training to drive the training of the language model. The global architecture is laid out in a generator-evaluator fashion optimized directly to n-gram and semantic-based metrics. Evaluation metrics for language modelling only based on n-gram overlapping do not consider semantic relations between reference and candidate sequences. To improve the evaluation step, a two-fold evaluation was carried out. On the one side, an n-gram overlapping evaluation using the BLEU score. On the other side, a semantic-based assessment using BERTScore and NUBIA. The results were corroborated by a binary human evaluation of the semantic relatedness of the generated question and the ground truth. The results obtained showed that use a semantic-based REINFORCE algorithm for the question generation syntactically reshapes the generated questions while preserving their underlying semantic meaning. Many downstream applications can be drawn from a successful question generation including the enlargement of question answering datasets, the improvement of conversational systems, the enhancement of autonomous educational assessment systems, and so forth.

O Sistema Nacional de Avaliação da Educação Superior:

A avaliação do sistema educacional entrou na agenda do governo brasileiro com mais intensidade na década de 1990, como meio de subsidiar as reformas do sistema de ensino. Em 2004 foi criado o Sistema Nacional de Avaliação da Educação Superior (SINAES) que reconfigurou a concepção e o modelo de avaliação institucional das instituições de ensino superior. Em 2008, com a criação da Rede Federal de Educação Profissional, Científica e Tecnológica e dos institutos federais (IF), suas unidades passaram a ofertar esse nível de ensino e, para fins de regulação, tiveram que se adequar ao SINAES. Esse artigo tem o objetivo de examinar os pressupostos teóricos que subsidiam a política de avaliação da educação superior e averiguar de que modo os institutos federais inserem-se nessa política e como vêm se articulando para atender as diretrizes do SINAES. Para o estudo, foram selecionadas pesquisas em nível de pós-graduação em bancos oficiais de teses e dissertações, entre as quais foram identificados seis trabalhos com potencial para subsidiar esse debate. Os resultados revelam que o SINAES não contempla a configuração das instituições de educação profissional, não permite a avaliação institucional de sua totalidade e ainda pode ocultar a realidade dos IF e inibir seu desenvolvimento institucional.

Implementasi Sistem Informasi Pendidikan Muhammadiyah (Sidikmu) Dalam Layanan Administrasi Sistem Pe Nilaian Al Islam, Kemuhammadiyahan Dan Bahasa Arab (Ismuba) Pada Era Industri 4.0 Di Sekolah Dasar Muhammadiyah 1 Krian, Sidoarjo

The development of the world of technology cannot be avoided, educational institutions as centers of education and learning must follow these technological developments so that they can exist more. In the industrial era 4.0 which was marked by advances in information and communication technology, accelerating and providing various kinds of convenience in accessing information and data on various aspects of life. This acceleration and convenience has become a necessity and a lifestyle. Therefore, schools as educational institutions that have various kinds of data must be willing to make changes, adjust to these demands, so that the education and learning process can take place properly, quickly, and accurately. Schools must take advantage of information and communication technology. Carry out the education and learning process online. Consequently, it must have facilities and infrastructure, in the form of information and communication technology. However, the reality in the field is that not all schools have and apply this technology, due to various limitations, financial and workforce quality. The problem then is, is this true, information and communication technology is a major need, without this technology learning cannot take place optimally, the continuity of the institution and its existence is threatened. From here the researcher was moved to research on the implementation of information systems online. Aims to find out and understand the effectiveness of online information system implementation in Era 4.0 as a database and educational information service center at the Krian 1 Muhammadiyah Primary School.
 This type of qualitative research uses a phenomenological approach (field research). This study used data collection techniques in the form of observation, interviews, and documents. The collected data were analyzed through 3 stages, namely data collection, data reduction, data presentation, and conclusion drawing. To check the validity of the data, triangulation was used.
 The results obtained from this study are the implementation of the Muhammadiyah Education Information System (Sidikmu) in the administrative services of the Al Islam, Kemuhammadiyahan, and Arabic education assessment system in the Industrial era 4.0 at SD Muhammadiyah 1 Krian Sioarjo implemented in integral automation. An information system that not only functions as a database but also as a center for educational information services for student guardians. So that educational institutions can move according to the era which is completely online and instant. Sidikmu is effective and efficient in improving the quality and productivity of the ISMUBA education assessment system administration services at SD Muhamadiyah 1 Krian, Sidoarjo. The effectiveness and efficiency of Sidikmu's work automatically integrate all existing data in schools, making it easier for the administrative process for assessments, especially on ISMUBA subjects. The main obstacle is that not all Ismuba teachers master the technology, so training is needed, the provision of facilities, and guaranteed awards for Creative and Innovative Ismuba teachers

Multiple imputation in big identifiable data for educational research: An example from the Brazilian education assessment system

Abstract Almost all quantitative studies in educational assessment, evaluation and educational research are based on incomplete data sets, which have been a problem for years without a single solution. The use of big identifiable data poses new challenges in dealing with missing values. In the first part of this paper, we present the state-of-art of the topic in the Brazilian education scientific literature, and how researchers have dealt with missing data since the turn of the century. Next, we use open access software to analyze real-world data, the 2017 Prova Brasil , for several federation units to document how the naïve assumption of missing completely at random may substantially affect statistical conclusions, researcher interpretations, and subsequent implications for policy and practice. We conclude with straightforward suggestions for any education researcher on applying R routines to conduct the hypotheses test of missing completely at random and, if the null hypothesis is rejected, then how to implement the multiple imputation, which appears to be one of the most appropriate methods for handling missing data.

SAEB E PNLD: DISSONÂNCIAS E IMPLICAÇÕES DAS AVALIAÇÕES DE LARGA ESCALA NO CONTEXTO EDUCACIONAL BRASILEIRO

Este trabalho apresenta os resultados de uma investigação que teve como objetivo analisar o diálogo referencial entre o Sistema de Avaliação da Educação Básica (SAEB) e o Programa Nacional do Livro Didático (PNLD). A produção das informações se deu por meio da análise de documentos do SAEB, na busca de referências sobre o PNLD e de documentos do PNLD buscando referências ao SAEB. Os resultados indicam que o diálogo entre as duas políticas é praticamente nulo e que provoca um efeito drástico da avaliação educacional brasileira, o que leva ao questionamento dos resultados que o SAEB produz acerca da Educação Brasileira. As análises evidenciam a existência de um ciclo de fracasso como produto da falta de diálogo entre as referidas políticas.

Psda: A tool for extracting knowledge from symbolic data with an application in Brazilian educational data

Symbolic polygonal data analysis is a new type of framework to extract valuable knowledge from a new structure of data using regular polygon built from data in class, big data, and complex data. This paper introduces a toolbox for symbolic polygonal data, named psda, that contains the main descriptive measures for this type of variable, e.g., mean, variance, correlation, and a polygonal linear regression model (plr). It is applied at the Brazilian Basic Education Assessment System (SAEB), giving a new perspective to the managers of the counties to realize the public policy in the Brazilian educational system. The hypothesis test showed that the polygonal linear regression model presented the best performance compared to some symbolic interval regression models in the SAEB application.

O papel dos organismos internacionais na reforma gerencial da educação brasileira: os sistemas de avaliação educacional em larga escala como instrumentos da pedagogia política do capital.

For three decades, capital has been plunged into an organic crisis that has conditioned the maintenance of its accumulation bases, which led to a process of bourgeois recomposition in order to restructure work and production, as well as reconfigure the relationship between the State and civil society. This process affects all spheres of social life, including human development policies, which assume a strategic character, both for training new workers and for the ethical and moral conformity of civil society. In this context, it is clear that international organizations, notably UNESCO, the World Bank Group and the OECD have undergone significant transformations, assuming a new role in the planning and formulation of the educational policies of their Member States, in order to create a consensus on around a political pedagogy of capital, for which large-scale educational assessment systems are a strategic element.

Functional Literacy of High School Students-Working With Text: Opportunities for Applying Educational Skills in Self-Cognition Classes

It is considered the reading literacy of high school students in the article, which can be developedthrough self-cognition classes. Currently, reading literacy is more considered in language and literaturecourses. The skills of working with text and electronic text are important for the modern student. It isevaluated in the international educational achievement assessment systems PISA, PIRLS. Since the currentgeneration of students is called the digital generation, which is dominated by the visual code ofperception, in order to form verbal and logical thinking, pedagogues began to pay more attention to thedevelopment of reading skills and understanding of the text by students.The purpose of the article is to analyze the phenomenon of functional reading literacy and actualizationthe problem of using techniques for developing functional reading literacy to identify values inworking with educational texts in self-cognition classes, offer recommendations. Reading literacy skillsare aimed at finding the meaning and application of the information obtained from the text, its understanding,and the development of students ‘ cognitive abilities. The article presents tasks to identify thevalue-semantic content of the text in self-cognition classes. Through some of the tasks, students canlearn to compose texts and convey their meaning to their classmates, their understanding of stories, andtheir vision of values. These tasks can be used during storytelling, discussion of stories, discussion of the“Teacher’s Gift”, or during creative work.Key words: reader’s literacy, values of self-cognition, value-semantic training, self-cognition classes.

OBJECTIVE STRUCTURED CLINICAL EXAMINATION AS ADEQUATE METHODS FOR ASSESSING THE PROFESSIONAL COMPETENCE FOR INTERSHIP DOCTOR

Abstract. Providing quality training for future medical specialists is an urgent problem of our time. The state program for reforming the medical sphere requires educational institutions to train specialists who possess not only a high level of theoretical knowledge, but also a qualified level of practical skills, communicative properties. The article analyzes in detail the experience of implementing an objective structured exam (OSСЕ) to assess the knowledge and skills of interns on the stage of final certification in the specialty "Infectious Diseases", which was held at the Department of Infectious Diseases and Epidemiology of Ivano-Frankivsk National Medical University in 2019 and 2020. In percentage terms, the exam performance was as follows: “excellent” - 11.1%, “good” - 88.8%. Criteria for the effectiveness and objectivity of the system for assessing practical and communicative skills in the OSСЕ system are determined. The role of OSСЕ in the assessment system in medical education is described. The technique of the exam at the Department of Infectious Diseases and Epidemiology is described in detail. The expediency of introducing OSСЕ for conducting state certification of specialists has been proved. To evaluate the results, a descriptive-analytical method was used.
 Conclusions are drawn and directions for further improving the methodology for conducting and preparing for OSСЕ and improving the methodology for its assessment are identified. According to the results of OSKI conducted by interns in the specialty "Infectious Diseases" at the Department of Infectious Diseases and Epidemiology, in order to optimize the examination at the preparation stage, it was decided: to form examination booklets with a sufficient database, start preparing for the exam already in the first year of training of interns paying more attention training communication skills, examination algorithms, treatment, emergency care; to evaluate the intermediate certification of interns after each thematic section using a structured exam technique; to improve the station assessment system for a more thorough differentiation of the result.
 It is worth noting that during the exam at one of the stations, most future specialists had low results of passing the skills, therefore, at a meeting of the department, it was decided to carefully review the tasks of this station and make appropriate adjustments.
 Тhe use of structured response algorithms by the examiner helps to increase the reliability of the assessment, and therefore, the impartiality and objectivity of the exam and it is the objectivity of the assessment of the exam that depends on its structure (number of stations, questions).
 So compared with the traditional version of clinical trials, OSKI has many positive advantages. OSKI is a reliable and impartial exam, the main purpose of which is to standardize the tasks for the examinees, which increases the commitment of the examinees because it completely eliminates the fact of the "unfortunate task". During the OSKI, not only the level of theoretical knowledge was evaluated, but also the competence of future specialists in the application of the knowledge and practice was tested. Without applying theoretical knowledge to analytical skills by interns, it is impossible to actually complete a practical task, interpret the results of the examination, establish a diagnosis, etc. A positive point is that due to the presence on the OSKI of question and answer algorithms, teachers adhere to established rules and assessment standards, can fairly evaluate the intern during a practical skill and minimize the subjective influence of the examiner on the evaluation process of the intern.

Índice de Desenvolvimento da Educação Básica (IDEB), observado e projetado, das escolas estaduais do norte do Brasil e do estado do Amapá entre 2007 e 2017

O Índice de Desenvolvimento da Educação Básica (IDEB) representa um indicador educacional medido pelo Instituto Nacional de Estudos e Pesquisas Educacionais (INEP) para avaliar a qualidade das escolas brasileiras. Sua composição é analisada através do fluxo escolar e do desempenho de aprendizagem em Língua Portuguesa e Matemática compilados pelo Sistema de Avaliação da Educação Básica (SAEB). Nessa perspectiva, o objetivo da presente pesquisa foi comparar o IDEB observado e projetado, das escolas estaduais do Norte do Brasil e do Estado do Amapá entre 2007 e 2017. Para tanto, realizou-se pesquisa de cunho bibliográfico. Verificou-se que o Estado do Amapá ainda apresenta um dé ficit de rendimento de aprendizagem em relação à média dos outros estados da região Norte do Brasil. Diante do exposto, o artigo sugere que o IDEB cumpra sua função social descrita na meta 7 do Plano Nacional de Educação (PNE) que trata o indicador como norteador de políticas públicas a serem implementadas no cotidiano escolar.

MP17: Evaluation of a national competency-based assessment system in emergency medicine: A CanDREAM study

Introduction: In 2018, Canadian postgraduate specialist Emergency Medicine (EM) programs began implementing a competency-based medical education (CBME) assessment system. To support improvement of this assessment program, we sought to evaluate its short-term educational outcomes nationally and within individual programs. Methods: Program-level data from the 2018 resident cohort were amalgamated and analyzed. The number of Entrustable Professional Activity (EPA) assessments (overall and for each EPA) and the timing of resident promotion through program stages was compared between programs and to the guidelines provided by the national EM specialty committee. Total EPA observations from each program were correlated with the number of EM and pediatric EM rotations. Results: Data from 15 of 17 (88.2%) EM programs containing 9,842 EPA observations from 68 of the 77 (88.3%) Canadian EM specialist residents in the 2018 cohort were analyzed. The average number of EPAs observed per resident in each program varied from 92.5 to 229.6 and correlated strongly with the number of blocks spent on EM and pediatric EM (r = 0.83, p < 0.001). Relative to the guidelines outlined by the specialty committee, residents were promoted later than expected and with fewer EPA observations than suggested. Conclusion: We present a new approach to the amalgamation of national and program-level assessment data. There was demonstrable variation in both EPA-based assessment numbers and promotion timelines between programs and with national guidelines. This evaluation data will inform the revision of local programs and national guidelines and serve as a starting point for further reaching outcome evaluation. This process could be replicated by other national assessment programs.

SISTEMA DE AVALIAÇÃO DO ENSINO MUNICIPAL: PESQUISA AVALIATIVA À LUZ DA METODOLOGIA QUADRIPOLAR

Este artigo é um recorte da dissertação de Mestrado da autora principal desta pesquisa, estudo ao qual lhe foi dado nova abordagem, centrando-se na pesquisa avaliativa à luz da metodologia quadripolar proposta por Bruyne et al. (1977). Versa-se sobre o Sistema de Avaliação do Ensino Municipal em Jijoca de Jericoacoara, Ceará, e seus efeitos na aprendizagem dos alunos do 2º ano dos anos iniciais. A pesquisa, prioritariamente qualitativa, apropriou-se de recursos metodológicos quantitativos e qualitativos, dentre eles a metodologia quadripolar. A análise de dados quantitativos foi realizada por meio do programa estatístico IBM SPSS Statistics 20 e os qualitativos, por meio do programa Atlas.ti 7. Como principais resultados, evidenciou-se que essa sistemática contribui para que haja equidade entre as escolas e a orientação da prática docente. Todavia, os educadores dessa cidade reconhecem que aspectos como a metodologia de divulgação dos resultados e o nível das questões das avaliações ministradas precisam ser revistos para que ela seja mais significativa quanto às suas contribuições para o processo de ensino-aprendizagem das crianças.

Governance structure and standard setting in educational assessment

ABSTRACTSeldom have comparative studies of educational assessment systems been undertaken, especially regarding their standard setting procedures. This study examines the effects of governance structures on the power relations in standard setting in the dominant school-leaving or university-entrance examinations. We present acritical analysis of the published research and policy documents, including sense-checking with senior assessment practitioners from 22 jurisdictions. The nature of standard setting systems in three cases of Ireland, the USA and India are described in detail to showcase the differences between the following three models of governance systems: nationalised, commercial market and quasi-market. The contribution of this article, then, is to describe the three models of governance systems, to classify the 22 jurisdictions using the three models, and to generate propositions inductively. Thus, the article provides aconceptual basis for extension of this work to other cases to advance the literature cumulatively by theory-building.

Performance Audit: Applying the Principle of Effectiveness in a Case Study

This article, based on a performance audit focused on the principle of effectiveness, aims at analyzing the impact resulting from the adoption of private textbooks for the primary school network of a municipality located in South of Brazil. For this, an assessment methodology known as difference-in-differences has been applied to data from the Brazilian Basic Education Assessment System, revealing the impact, on the municipality schools grades, for the use of textbook material other than those provided free of charge by the Federal government.

Repositório institucional da Universidade Federal da Bahia: apontamentos para a construção preliminar de um instrumento de avaliação

This article seeks to identify in the literature approaches that contribute to the construction of the assessment instrument for the Institutional Repository of the Federal University of Bahia, in order to serve the repository and contribute to the institutional assessment of the National Higher Education Assessment System (SINAES). The methodology used was the strategic search for the term “evaluation of repositories” and observation of the Repository. The results show the need for discussion, given the lack of standardized documents that support the repositories evaluation. We verified some initiatives, such as the Federal Universities of Santa Catarina and Ceará. We believe that debates on the theme can be intensified and that individual initiatives tend to build criteria and evaluation standards for repositories, so that they can be used as a resource for the evaluation of other repositories as well as institutional evaluation as a whole.

Using Electronic Assessment System in Higher Education: Challenges and Solutions for Lectures, Students and Institutions

Assessment is an important component of education especially in the context of the learning and teaching environment. In higher education, assessment is the heart of formal education. Nowadays, technology plays an important role in the assessment process. Therefore, using technology development has an impact on improving the assessment system. Focus on the using of electronic assessment (e-assessment) systems is increasingly developing at the higher education level. Technological developments have led to a shift in concepts and ways of assessing moving from face-to-face assessment to digital-based assessment, namely an end-to-end assessment process using ICT. E-assessment system is provides many benefits for the effectiveness and efficiency of learning in higher education for lecturers, students and institutions. Many e-assessment models are developed to meet various objectives, especially in the purpose of educational learning. However, there are many challenges in its process and application. This paper aims to provide an example of e-assessment system application, describe challenges in the use of e-assessment systems in higher education based on literature, journals and research results and provide recommendations on how to deal with all of these challenges so that the assessment objectives can be achieved. The method that used is to figure out and describe qualitatively. Overall conclusion is the e-assessment system has become one of the requirements in higher education with many models. There are still many questions for lecturers, students and institutions that pose challenges in the application of e-assessments related to validity and reliability, accuracy, fairness, security, infrastructure and human resources. Based on this fact then the recommendation of these challenges in the successful implementation of e-assessment in higher education are paying attention to the characteristics of e-assessment, designing assessments according the needs, and creating a complete and accessible assessment system for all.