Formative Assessment System Research Articles

Evaluation of Automatic Item Generation Utilities in Formative Assessment Application for Korean High School Students

The recent interests in research in the assessment field have been rapidly shifting from decision-maker-centered assessments to learner-centered assessments (i.e., diagnostic and/or formative assessments). In particular, it is a very important research topic in this field to analyze how these learner-centered assessments are developed more practical and valid by combining information (or intelligent) and communication technologies (ICT) and psychometric advances, and how these systems contribute to learning. Automatic Item Generation (AIG), which is in the spotlight recently, is a representative example of building a next-generation assessment theory and practice by integrating ICT into psychometrics.AIG has very broad and promising features in mass production, intelligent item calibration and management, especially learner-centered assessment, and so on. However, these claims have not yet been fully validated in real education settings, and it is in dire need of evaluating the utilities of AIG applications in practical educational settings. Based on these needs, the purposes of this research are, firstly, to introduce the benefits and challenges of AIG to properly assess AIG’s utilities. And, secondly, this study provides empirical evaluation results of AIG’s utilities within a formative assessment system for Korean high school students and teachers.The results of this study have important implications in that it is the first empirical study to evaluate the usefulness of AIG in the formation evaluation application of the actual educational environment, Korean high school setting. The results of the theoretical and empirical evaluation of the AIG will also be useful to researchers and practitioners who wish to evaluate the benefits and issues of future AIG-based educational services or theory development.

Making Formative Assessment Feasible: A Collaborative Approach Supporting Grant Funded Projects

Formative assessment is a means of unlocking the ‘black box’ of an intervention to explore and understand cause and effect, and is, therefore, vital for improvements that lead to success. However, most funders focus on summative evaluation, and projects with limited resources struggle to meet that requirement, leaving formative investigations ideal, but infeasible. Yet, formative assessment is essential when conducting an intervention that must create results with a small budget. This paper discusses the Formative Assessment Systems for Advanced Technological Education (FAS4ATE) project, which used a high level, theory based participatory approach to address these issues in small grant funded projects. FAS4ATE brought the leaders and evaluators from a variety of small projects together with experts in formative assessment and evaluation in two webinars and a workshop. Logic models were the core activity and teaching tool used to identify formative assessment needs throughout the life cycle of individual projects. The logic models were then analysed across projects to provide specifications for the development of tools and resources to support formative assessment. The paper describes the project's process, key strategy, products and recommendations. It will be particularly valuable for those who are interested in evaluation capacity building, both in terms of individual learning and creating evaluation systems.

Video podcast and illustrated text feedback in a web‐based formative assessment environment

AbstractIt is well known that computer‐based formative assessment and timely feedback enhance effective student learning but there is still a debate about what type of feedback should be given, being the text‐based the most used feedback in practice. Although the use of video content as a learning resource has recently increased in both educational and non‐educational contexts, there is very little research on its effectiveness as assessment feedback and the existing results are contradictory. For that reason, we have combined in this work the use of a web‐based formative assessment system (Siette) where we have integrated a specific type of video podcast (“modular teaching mini‐videos”: MTMs) and equivalent illustrated text. We have carried out two experiments with a twofold purpose, first to compare the effect of video feedback versus correct response feedback alone and secondly to compare the effect of video feedback versus equivalent illustrated text feedback. In the context of our research, a Statistic Course at university level, our results show, as expected, that there is a statistically significant positive effect in favor of video feedback over correct response feedback alone. Surprisingly, in contrast to our expectations, based on our context of acquisition of procedural knowledge, there is a statistically non‐significant effect of video feedback versus equivalent illustrated text feedback. From the students’ satisfaction survey, there is not statistically significant differences in the overall score of the activity based on video podcast feedback or equivalent illustrated text, which is an indication that we have really obtained equivalent feedback materials.

Formative assessment, work regulation, organization, engagement, tracking and attendance in Spanish Universities

The Assessment Systems Scale, a 34-item questionnaire, has been proven a robust, reliable and valid instrument to evaluate assessment procedures at the university level. However, the sample used to validate it was rather small, and proper validation of instruments requires testing on multiple independent samples. The current studies evaluated the reliability and the validity of a reduced version consisting in only 23 items, as well as its interactions. Study 1 determined the validity and reliability of the reduced version of the Assessment System Scale in the context of a large sample of Spanish university students. 3.428 students and 52 teachers from 17 different universities across Spain agreed to participate. Results showed a 3-factor structure and high internal consistency (α = .835). In Study 2 descriptive and inferential statistics showed that formative assessment significantly linked students’ work regulation and organization, engagement, tracking and attendance. Students who participate in formative assessment systems understand how necessary is class attendance, because it favors greater engagement and, as a consequence, better regulation and monitoring of the content throughout the teaching-learning process. A significant relationship between peer review and more effective feedback to improve the understanding of the tasks found. Moreover, the definition of clear evaluation criteria correlated positively with the registration of individual work or group. A level of self-regulation work differently depending on the degree studied was found. We conclude that it is necessary to implement formative assessment systems using tools and systematic procedures that ensure truly an alternative to more traditional methodological approaches

Design, Development and Evaluation of Academic Oral English Curriculum Reform

The postgraduate students’ oral English proficiency has long been considered to be unsatisfactory in tailoring ever-increasing global engagement and international academic cooperation. Numerous Chinese universities are currently undergoing English curriculum reform for enhancing postgraduate students’ pragmatic communicative capacity especially in involving in global professional interaction and articulating at international academic settings. To assess the effect of the curriculum innovation, a 670-postgraduate-student questionnaire was surveyed at Southeast University to evaluate its pilot reform encompassing teaching, learning and assessment, namely, “Collaborative Teaching Mode and Interactive Learning Model”, “Major-related Teaching Content” and “Formative Assessment System Integrated in Summative Assessment System”. The data showcases that the teaching content merged by Simulated International Conference on major-relevant themes has been popularly acknowledged by students. The Sino-foreign collaborative teaching mode and multiple interactive learning model have proven to boost students’ enthusiasm and confidence in improving oral English proficiency. The formative assessment system can significantly propel teamwork spirit and arouse students’ earnest to practice oral English. The research provides viable modes for oral English curriculum reform in Chinese tertiary educational institutions.

Challenges With Initial Implementation of an Intelligent Formative Assessment System

With a grant from the National Board of Medical Examiners' Stemmler Fund, the University of Illinois at Chicago (UIC) and University of Central Florida (UCF) have introduced an intelligent formative assessment system (www.Osmosis.org) into the preclinical phase of their MD curricula. This web‐browser based and mobile application presents students with individualized regimens of study questions that aim to leverage the testing effect, spaced repetition and the mixed‐practice effect to aid long term retention. In the first semester of use, 90 of 120 UCF students tried to use the system answering an average of 136 items per individual though subsequent use of the system has continually decreased. We performed two initial focus groups with first year medical students to learn about the strengths and weaknesses of the IFAS and what factors have affected students' decisions about what study aids to adopt. Groups of 4–5 students gave informed consent and the discussions were audio taped for transcription and thematic analysis. A preliminary coding of the audio files suggests that student decisions about what study aids to use in medical school are strongly affected by advice from upperclassmen in the early part of the first year. Near‐peer advice appears to have high face validity and is readily accepted. The data also suggested that concerns for lack of time and demand for efficiency weigh most heavily on student decisions about what tools to use. The data suggest that high quality orientation is needed when introducing new technology, with widespread consultation in the learning community, including faculty and upperclassmen. New applications must be highly intuitive and easy to use for successful adoption.Support or Funding InformationNational Board of Medical Examiners' Stemmler Fund

Investigation of Formative Assessment of Learning (INFORMAL): The Performance Indicator Tool (PIT)

This pilot study evaluated the student user experience of a performance indicator tool (PIT) for formative assessment of a cohort of fourth year medical undergraduates undertaking a 12 week rotation. Scoping and elaboration for the standalone web-based tool was based on the existing paper-based formative assessment system for which training and secure access was made available through the Virtual Campus at King’s College London. Thirty-eight participants were randomly assigned to four groups. A baseline survey on formative assessment was carried out in Term 1 by all groups. In Term 2, two groups received the PIT and two the paper-based system. The PIT and paper-based groups were further split into those who gained formative assessment at the end, and those at the middle and end of rotation. Evaluation was by online student satisfaction questionnaires and focus groups. Term 1 baseline survey (n = 13) revealed a lack of one-to-one formal feedback. In evaluation by satisfaction questionnaire (n = 8) and focus groups (n = 30), students valued personalised feedback, and mid-rotation feedback allowed students to reflect prior to the end of rotation. However, some students felt that their tutor did not know them as well as others, thus adversely affecting their feedback. A functional web-based PIT was developed including tutor–student discussion spaces. Students found feedback very valuable. Those involved found the PIT easy to access and use, preferring formative assessment two thirds into the rotation. The PIT has wider applicability within the College for all healthcare faculties and may help improve student satisfaction.

The teaching reform of the movement system diseases in the center of the organ system

For better implementation of outstanding physician education program , Chongqing Medical University carried out the centering on the organ curriculum reform. In motor system disease teaching, we integrated motor system curriculums, reduced the classroom theory teaching time, strengthened the discussion and practice teaching, proceeded teacher training and teaching demonstration before class and adopted the model of problem-based learning (PBL) teaching, carried out clinical skills teaching, established formative assessment system to comprehensively evaluate students' learning effects and accept motor system disease teaching certification and other measures, which improved the students' knowledge system, enhanced the students' ability of analysis and solve problems. Key words: Organ systems; Motor systems; Outstanding physician education; Problem-based learning (PBL)

Academic Performance on First-Year Medical School Exams: How Well Does It Predict Later Performance on Knowledge-Based and Clinical Assessments?

ABSTRACTNumber of appearances in the bottom quartile of 1st-year medical school exams were used to represent the extent to which students were having academic difficulties. Medical educators have long expressed a desire to have indicators of medical student performance that have strong predictive validity. Predictors traditionally used fell into 4 general categories: demographic (e.g., gender), other background factors (e.g., college major), performance/aptitude (e.g., medical college admission test scores), and noncognitive factors (e.g., curiosity). These factors, however, have an inconsistent record of predicting student performance. In comparison to traditional predictive factors, we sought to determine the extent to which academic performance in the 1st-year of medical school, as measured by examination performance in the bottom quartile of the class in 7 required courses, predicted later performance on a variety of assessments, both knowledge based (e.g., United States Medical Licensing Examination Step 1 and Step IICK) and clinical skills based (e.g., clerkship grades and objective structured clinical exam performance). Of all predictors measured, number of appearances in the bottom quartile in Year 1 was the most strongly related to performance in knowledge-based assessments, as well as clinically related outcomes, and, for each outcome, bottom-quartile performance accounted for additional variance beyond that of the traditional predictors. Low academic performance in the 1st year of medical school is a meaningful risk factor with both predictive validity and predictive utility for low performance later in medical school. The question remains as to how we can incorporate this indicator into a system of formative assessment that effectively addresses the challenges of medical students once they have been identified.

THE NEED FOR A MORE EFFICIENT USER NOTIFICATION SYSTEM IN USING SOCIAL NETWORKS AS UBIQUITOUS LEARNING PLATFORMS

While carrying out formative assessment activities over social network services (SNS), it has been noted that personalized notifications have a high chance of “the important post getting lost” in the notification feed. In order to highlight this problem, this paper compares within a posttest only quasi-experiment, a total of 104 first year undergraduate students, all of which are prospective ICT teachers, in two groups. A formative assessment system in the ubiquitous learning context is delivered over an SNS in both groups. In the first group, the SNS has been used for the entire assessment task. In the second group, the questions have been delivered and responses were received over mobile phone “SMS” messages, while the SNS was used solely for providing feedback. The cases were compared in terms of voluntary participation rates and academic success. Both response rates and academic success have been significantly higher in the SMS group. When asked their reasons for not responding to questions; the SNS-only group frequently reported “not noticing the questions being sent”. This may indicate a flaw in message design for using social networks as LMS's. Sensible use of push-messages is advised.

Application of Big Data on Formative Assessment System for Foreign Language Learning.

Application of Big Data on Formative Assessment System for Foreign Language Learning.

Focusing on the Formative: Building an Assessment System Aimed at Student Growth and Development

This Perspective addresses the need for an integrated system of formative and summative assessment in undergraduate medical education with a focus on the formative. While acknowledging the importance of summative assessment, which asks whether trainees have met criteria for progression, the authors propose that a formatively focused assessment system can best accomplish a central task of competency-based medical education: transmitting feedback to learners in a format and a manner that will help them to improve, develop, and grow. Formative assessment should not be seen as a set of singular events but, rather, as a process that is organized and integrated over time, much like the cycle of quality improvement in medicine. To justify this position, the authors discuss its conceptual underpinnings and rationale, including the need to prepare learners for the formatively focused assessment system of graduate medical education. Next, the authors identify assessment strategies that could be employed, as well as the characteristics of an institutional culture and the learner-teacher relationship necessary for a learner-centered, improvement-focused assessment system to succeed. Finally, an infrastructure for such a system is proposed. This consists of a foundation of well-articulated and disseminated milestones for achievement and four pillars: faculty development, learner development, longitudinal academic advising and coaching, and documentation of developing competence. The authors conclude by suggesting that the guidelines proposed are analogous to the principles of continuity and coordination of care, so much valued in the world of medicine yet often overlooked in the world of education.

Exploratory application of formative assessment in internship education for clinical anesthesia

The formative assessment system has been applied to the internship education for the clinical anesthesia with the aim to improve students' initiative and to evaluate their outcomes more compre-hensively. The students' performance in the shift exchange, case discussion, raising question, solving question at the time points of after the preclinical train, one month and 3 months into the anesthesia internship, and after the completion of internship, and their capability in preoperative patient assessment, condition report, clinical practice, review writing have been evaluated to determine the educational quality and to instruct the improvement of educational approach. Assess process takes into account both the individuality and the general character of the students and feedbacks the evaluation result to improve the practice teaching The implementation of the evaluation can promote students' autonomous learning and comprehensively evaluate students' practice process. Key words: Formative assessment; Internship teaching; Anesthesia

Developing and pilot testing of a tool for "clinicosocial case study" assessment of community medicine residents.

Practical and clinical skills teaching should constitute a core part of the postgraduate curriculum of Community Medicine. The clinicosocial case study is a method to enhance learners' skills but there is no generally accepted organized system of formative assessment and structured feedback to guide students. A new tool based on the principles of mini-Clinical Evaluation Exercise (mini CEX) was developed and pilot tested as a 'clinicosocial case study' assessment of community medicine residents with feedback as a core component. Ten core domains of clinicosocial skills were identified after reviewing the relevant literature and input from local experts in community medicine and medical education. We pilot tested the tool with eight faculty members to assess five residents during clinicosocial case presentations on a variety of topics. Kappa statistic and Bland Altman plots were used to assess agreement between faculty members' average assessment scores. Cronbach's alpha was used to test the internal consistency with faculty members as domains. All 95% confidence limits using the Bland-Altman method were within the predetermined limit of 2 points. The overall Kappa between two faculty members was fair ranging from 0.2 to 0.3. Qualitative feedback revealed that both faculty and residents were enthusiastic about the process but faculty suggested further standardization, while residents suggested streamlining of the process. This new assessment tool is available for objective and unbiased assessment of residents through 'clinicosocial case study,' which enriches learning through comprehensive feedback. Further validation in different settings is needed.

Examining Schools’ Distributed Instructional Leadership Capacity: Comparison of Elementary and Secondary Schools

The purpose of this study is to elucidate the fundamental differences between elementary and secondary schools in distributed instructional leadership practices by comparing results determined by the Comprehensive Assessment of Leadership for Learning (CALL) survey. The data set used in this study was derived from the 2011-1013 CALL survey: a 360-degree, on-line, formative assessment and feedback system for K-12 school leadership. A total number of 4311 voluntary teachers, school administrators and other school staff participated in this study. The analysis included three integral steps. First, descriptive statistics were carried out to explain how distributed instructional leadership was rated on the CALL survey. Second, an independent-samples t-test was employed to compare the mean scores of elementary and secondary schools on the CALL survey. Third, ratios of elementary and secondary school staff in the top scoring 20 percentile and bottom scoring 20 percentile were compared to each other. The results suggested that all five distributed-instructional leadership domains which were addressed in the CALL had very close mean scores, indicating that none of the broad leadership areas was being ignored and elementary and secondary schools differ only in terms of the leadership practices related to monitoring teaching and learning.

Construction of formative quality evaluation system for continuing medical education project

This article first standardizes the operating procedures of continuing medical education (CME) programs, to guide the design, reporting and holding until the completion of the whole feedback process, and then tries to constitute the formative quality assessment system according to the preset assessments, the implementing assessments and summative assessments. The result of the input of each process is effectively and visually checked, the effect is cleared, and the problem is found out. Assessment information is feedbacked to the project manager, new solutions for deficiencies and problems are found in the late work so as to achieve the optimal management of CME, and provide reference for the continuous improvement of teaching quality of CME programs. Key words: Continuing medical education project; Standardized procedures; Formative assessment system; Quality

Evaluación Formativa en Educación Superior: Impacto en Estudiantes con Ansiedad a las Matemáticas

Our objective was to implement and evaluate a formative assessment system in a mandatory course of the degree in Psychology. With this system, students received feedback from the tests they performed. We found a positive correlation between feedback classes’ attendance and students’ grades. The correlation between math anxiety and course's performance was not significant, suggesting that feedback could have helped to reduce the negative impact of math anxiety on performance.

Primary Grades Teachers’ Instructional Decisions During Online Mathematics Professional Development Activities

This study examines primary grades teachers’ instructional decisions in their mathematics classroom during their participation in a year-long professional development program on formative assessment. Teachers participated in 40 h of face-to-face workshops followed by 40 h of classroom-embedded activities that were facilitated in an asynchronous online format. Inductive analysis of teachers’ online discussion forum posts and their frequency of using an internet-based formative assessment system indicated significant variance in teachers’ use of the formative assessment tool. Some teachers used the assessment system regularly and reported modifying instruction based on the data collected through formative assessment. However, some teachers demonstrated difficulty determining how the assessments aligned to specific mathematics standards, and how the assessment should inform their use of curriculum. Implications for future research include the need to triangulate data from instructional plans, surveys, and classroom observations, while implications for practice include the need to more explicitly support teachers’ instructional planning process using specific curricular resources.

����������� ������� � ���������� ��������������� ����������� �����������

This article describes one of the most important issues in pedagogy at the present time - the question of assessment of students. What should be the
 evaluation in terms of educational paradigm shift, what role do the mark and the assessment play and what is the difference between them. The article
 also touched upon the historical question of the appearance of "the mark" and "lesson point". It describes the process of formative assessment, gives the
 author´s definition and identifies the basic tools of formative assessment. Portfolio and student record book are considered as the main elements of the
 system of formative assessment. The article also mentions the main directions of assessing the achievement of the expected results of development of
 the basic educational program, including assessment of metasubject results using standardized metasubject works.

The effect of regulation feedback in a computer-based formative assessment on information problem solving

This study examines the effect of regulation feedback in a computer-based formative assessment in the context of searching for information online. Fifty 13-year-old students completed two randomly selected assessment tasks, receiving automated regulation feedback between them. Student performance was (self-)graded by students and by experts. Expert, as well as student (self)grades showed a significant increase between Task 1 and Task 2. However, further analysis of the expert grades showed significant improvement in performance for girls only. Furthermore, the formative assessment system traced the number of searches and the number of websites consulted per student to complete the two assignments. On average, the results showed that students consulted significantly more websites for Task 2, compared to Task 1. The average number of searches did not differ significantly between Tasks 1 and 2. On the other hand, significant differences were found for those students who, during the evaluation of their performance on Task 1, explicitly stated that they would increase their searches.