High-stakes Summative Assessment Research Articles

‘You are the hero’: teaching great literature in a London Jesuit school

This article argues that Ignatian pedagogy in the Jesuit tradition allows great literature to become transformative. It takes as a case study an extracurricular course – ‘Becoming a Hero: From Homer to Harry Potter’ – taught to a socially diverse group of key stage three ‘gifted and talented’ children in a London Jesuit comprehensive school. While secondary teaching is normally geared towards high-stakes summative assessment, this course aims instead at personal growth. Young people discover themselves as the heroes in a perennial plotline which appears throughout both classic texts and young adult fiction. The teacher (himself a Jesuit) had previously taught the course to undergraduates at Harvard University. Adapting it for younger children, he makes the case for ‘gifted and talented’ provision and narrates his use of assessment for prior learning and creative teaching techniques in order to ensure holistic – spiritual, intellectual, and emotional – learning beyond the curriculum.

Medical specialty certification exams studied according to the Ottawa Quality Criteria: a systematic review

BackgroundMedical specialty certification exams are high-stakes summative assessments used to determine which doctors have the necessary skills, knowledge, and attitudes to treat patients independently. Such exams are crucial for patient safety, candidates’ career progression and accountability to the public, yet vary significantly among medical specialties and countries. It is therefore of paramount importance that the quality of specialty certification exams is studied in the scientific literature.MethodsIn this systematic literature review we used the PICOS framework and searched for papers concerning medical specialty certification exams published in English between 2000 and 2020 in seven databases using a diverse set of search term variations. Papers were screened by two researchers independently and scored regarding their methodological quality and relevance to this review. Finally, they were categorized by country, medical specialty and the following seven Ottawa Criteria of good assessment: validity, reliability, equivalence, feasibility, acceptability, catalytic and educational effect.ResultsAfter removal of duplicates, 2852 papers were screened for inclusion, of which 66 met all relevant criteria. Over 43 different exams and more than 28 different specialties from 18 jurisdictions were studied. Around 77% of all eligible papers were based in English-speaking countries, with 55% of publications centered on just the UK and USA. General Practice was the most frequently studied specialty among certification exams with the UK General Practice exam having been particularly broadly analyzed. Papers received an average of 4.2/6 points on the quality score. Eligible studies analyzed 2.1/7 Ottawa Criteria on average, with the most frequently studied criteria being reliability, validity, and acceptability.ConclusionsThe present systematic review shows a growing number of studies analyzing medical specialty certification exams over time, encompassing a wider range of medical specialties, countries, and Ottawa Criteria. Due to their reliance on multiple assessment methods and data-points, aspects of programmatic assessment suggest a promising way forward in the development of medical specialty certification exams which fulfill all seven Ottawa Criteria. Further research is needed to confirm these results, particularly analyses of examinations held outside the Anglosphere as well as studies analyzing entire certification exams or comparing multiple examination methods.

Practice to Perform: Formative Mini-Skills Checkoffs Prior to High-Stakes Summative Assessment for Practice-Ready Students.

In previous years, students expressed their concerns regarding the high-stress nature of the summative skills checkoff, and pass rates were not as high as expected. To evaluate the impact of mini-skills checkoff assessments and the use of standardized patients (SPs) on summative skills pass rates and nursing students' perceptions of this teaching approach. A mixed-methods approach was used with a convenience sample of 46 sophomore nursing students. There was a significant difference in the summative skills checkoff pass rates after including mini-skills checkoffs and SPs ( P < .01). Four dominant themes emerged: emotional impact, preparation and learning, confidence and competence, and readiness. Evidence from this pilot study suggests the inclusion of continuous, low-stakes assessments over time, with learner feedback bridges the gap between clinical practice, simulation, and classroom learning.

Student-Created Physics Problems as an Independent and Equitable Assessment Tool

Traditional high-stakes summative assessments—timed, in-class exams accounting for a large percentage of the term’s overall grade—have often received criticism from the educational community. Such assessments tend to prize a particular “narrow bundle of skills,” and have been shown in some contexts to produce disparate outcomes between different demographic groups. Alternative low-stakes assessments (assignments with significantly less impact on the overall grade) have shown potential to improve student engagement and close demographic gaps. At the same time, principles from physics education research suggest to us that students may benefit from the challenge of writing their own physics problems. Conjoining these ideas, we document an assignment (“The Problem Project”) in which students were asked to create and solve a problem of their own design, as an alternative form of assessment, and provide exploratory research into its equity impacts.

Video-based formative and summative assessment of surgical tasks using deep learning

To ensure satisfactory clinical outcomes, surgical skill assessment must be objective, time-efficient, and preferentially automated—none of which is currently achievable. Video-based assessment (VBA) is being deployed in intraoperative and simulation settings to evaluate technical skill execution. However, VBA is manual, time-intensive, and prone to subjective interpretation and poor inter-rater reliability. Herein, we propose a deep learning (DL) model that can automatically and objectively provide a high-stakes summative assessment of surgical skill execution based on video feeds and low-stakes formative assessment to guide surgical skill acquisition. Formative assessment is generated using heatmaps of visual features that correlate with surgical performance. Hence, the DL model paves the way for the quantitative and reproducible evaluation of surgical tasks from videos with the potential for broad dissemination in surgical training, certification, and credentialing.

Are high school students accurate in predicting their AP exam scores?: Examining inaccuracy and overconfidence of students’ predictions

ABSTRACT We examined whether students were accurate in predicting their test performance in both low-stakes and high-stakes testing contexts. The sample comprised U.S. high school students enrolled in an advanced placement (AP) statistics course during the 2017–2018 academic year (N = 209; Mage = 16.6 years). We found that even two months before taking the AP exam, a high stakes summative assessment, students were moderately accurate in predicting their actual scores (κweighted = .62). When the same variables were entered into models predicting inaccuracy and overconfidence bias, results did not provide evidence that age, gender, parental education, number of mathematics classes previously taken, or course engagement accounted for variation in accuracy. Overconfidence bias differed between students enrolled at different schools. Results indicated that students’ predictions of performance were positively associated with performance in both low- and high-stakes testing contexts. The findings shed light on ways to leverage students’ self-assessment for learning.

Low Inter-Rater Reliability of a High Stakes Performance Assessment of Teacher Candidates

The Performance Assessment for California Teachers (PACT) is a high stakes summative assessment that was designed to measure pre-service teacher readiness. We examined the inter-rater reliability (IRR) of trained PACT evaluators who rated 19 candidates. As measured by Cohen’s weighted kappa, the overall IRR estimate was 0.17 (poor strength of agreement). IRR estimates ranged from −0.29 (worse than expected by chance) to 0.54 (moderate strength of agreement); all were below the standard of 0.70 for consensus agreement. Follow-up interviews of 10 evaluators revealed possible reasons we observed low IRR, such as departures from established PACT scoring protocol, and lack of, or inconsistent, use of a scoring aid document. Evaluators reported difficulties scoring the materials that candidates submitted, particularly the use of Academic Language. Cognitive Task Analysis (CTA) is suggested as a method to improve IRR in the PACT and other teacher performance assessments such as the edTPA.

Validity Evidence for Procedure-specific Competence Assessment Tools in General Surgery: A Scoping Review.

This study aims to provide a focused and detailed assessment of the validity evidence supporting procedure-specific operative assessment tools in general surgery. Competency-based assessment tools should be supported by robust validity evidence to be used reliably for evaluation of operative skills. The contemporary framework of validity relies on five sources of evidence: content, response process, internal structure, relation to other variables, and consequences. A systematic search of 8 databases was conducted for studies containing procedure-specific operative assessment tools in general surgery. The validity evidence supporting each tool was assessed and scored in alignment with the contemporary framework of validity. Methodological rigour of studies was assessed with the Medical Education Research Study Quality Instrument. The educational utility of each tool was assessed with the Accreditation Council for Graduate Medical Education framework. There were 28 studies meeting inclusion criteria and 23 unique tools were assessed. Scores for validity evidence varied widely between tools, ranging from 3 - 14 (maximum 15). Medical Education Research Study Quality Instrument scores assessing the quality of study methodology were also variable (8.5-15.5, maximum 16.5). Direct reporting of educational utility criteria was limited. This study has identified a small group of procedure-specific operative assessment tools in general surgery. Many of these tools have limited validity evidence and have not been studied sufficiently to be used reliably in high-stakes summative assessments. As general surgery transitions to competency-based training, a more robust library of operative assessment tools will be required to support resident education and evaluation.

Development, production and evaluation of 2-dimensional transfer tattoos to simulate skin conditions in health professions education

BackgroundMoulages can greatly extend the possibilities of simulation in teaching and assessment. Since moulages that fit an educator’s exact needs are often unavailable commercially, this paper explains how 2-dimensional transfer tattoos can be independently developed, produced, and evaluated.MethodsFrom representative photographs of the specific skin condition an analogue copy of the pathological finding is drawn. Once validated by the medical expert, it can be digitized by scanning and processed using graphics software. The final digital image file is printed onto transfer paper. Once applied and fixed onto the intended wearer, usually a simulated patient, its authenticity can be confirmed, and further transfer tattoos can be produced.ResultsUsing this moulage technique we produced 10 different 2-dimensional transfer tattoos to date, including hematoma, Janeway lesions and splinter nails. These moulages are used in clinical skills training, formative and high-stakes summative assessment in undergraduate medical and nursing programs.ConclusionsBy sharing our development process for 2-dimensional transfer tattoos, health profession educators can produce their own that best fit their local educational needs. Due to their high authenticity and standardization, 2-dimensional transfer tattoos are ideal for use in high-stakes assessment.

06 A systematic review of validity evidence for the fundamentals of laparoscopic surgery (FLS) exam in gynecology

FLS (Fundamentals in Laparoscopic Surgery) is a test and a curriculum to test and teach laparoscopic surgery skills. Our aim for this systematic review was to examine validity evidence supporting or refuting FLS specifically as a high-stakes summative assessment tool in gynecology. We systematically searched for studies on FLS as an assessment tool in gynecology on January 30, 2019. We organized validity evidence for cognitive and manual skills portions based on validity elements of current validity framework. From 1,971 citations identified 9 studies were included involving 319 participants. For the cognitive portion of the test, results were mixed in five studies in relationship with other variables category. For the manual portion of the test, most of the studies focused on relationships with other variables validity element with mixed findings. Concerning findings in the manual skills portion were the lack of transferability of skills to the operating room, limited mixed evidence for improvement in operating room performance, and worse performance by OBGYNs compared to other specialties. We were not able to find evidence supporting content, response process or consequences validity elements in either the cognitive or the manual skills portions of the test. Validity evidence was mixed in relation with other variables category, but other validity evidence was lacking. Given the high-stakes use of this assessment, we suggest requiring further evidence and/or re-evaluating utilization of this test.

Use and Potential Misuse of Milestones.

By 2001, the Accreditation Council for Graduate Medical Education (ACGME) 6 core competencies were established, which introduced graduate medical education's (GME's) transition to competency-based medical education (CBME).1 Competencies are teachable attributes that residents and fellows must develop in order to carry out professional roles. Milestones describe the developmental pathway for achieving each competency in more than 150 GME specialtes and subspecialties. Yet confusion remains about their optimal use as well as misuse, which may produce adverse consequences for all stakeholders (eg, residents, faculty, programs, sponsoring institutions [SIs]).Milestones are narrative statements that describe skills, knowledge, and behaviors for performance in each of the 6 core competency domains and are arranged sequentially from attributes of novices to attributes that are aspirational.2 Milestone-based assessments enable faculty to provide GME learners specific formative feedback and improvement goals through the defined progression of levels toward expertise.3 Aggregated Milestone data from within and across programs can facilitate program, SI, and review committee (RC) learning and improvement.Milestones themselves are not assessments. They are constructs used by the GME community to build assessments. As constructs, Milestone assessments are open to interpretation and may be affected by context and other factors such as the skill and biases of assessors. These features can be mitigated during formative feedback sessions, but they make Milestone-based assessments inappropriate for high-stakes summative assessments of learning or competence. Thus, Milestone assessments should never be shared with certifying, credentialing or licensing boards or agencies, or potential employers, or provided as evidence of competence for legal purposes.4 When used for these unintended purposes, faculty may be reluctant to give honest assessments and clinical competency committees (CCCs) may be tempted to inflate Milestone judgements. Milestone security can facilitate the integrity of the assessment processes.

Editorial

England's state education system is increasingly controlled from the centre while being highly fragmented. Elements within the system, and forces acting on it, lean towards privatisation. Public, preparatory and private schools endure. Grammar schools persist, vitiating fair educational opportunities in many localities and preventing the sustained development in practice of a better conception of education for all. At every age and stage, high-stakes summative assessment enforces a narrowed curriculum, straitjackets pedagogical approaches and drains the joy from teaching and learning. Labelling young people by so-called 'ability', and segregating cohorts into 'ability' sets, conspire to limit what is possible for individuals and groups. Assumptions about 'ability' or a specious 'ability-range' are yet to be eradicated. Nor have we yet shaped pedagogy commensurate with the conception of children as the people best trusted to know how they learn, as intrinsically driven to learn, and able when conditions are conducive always to go on learning. Nevertheless, the comprehensive ideal is alive, thanks to the dedication of practitioners and educationalists, the commitment of parents and carers, and the work of trade unions and activist groups campaigning for it on the ground. As a journal, FORUM will continue to be a platform from which to argue for comprehensive education. Selection, division, elitism and all that goes to make up a competitive mindset in education will be challenged in our pages. FORUM will spread news of the success of comprehensive education in this country and beyond. It will explore the implications and meet the many challenges of the comprehensive ideal, and so take it forward.

Promoting and Supporting a Doctor of Nursing Practice Program of Scholarship.

Mentoring in the context of the faculty-student relationship can be one in which each benefits. Creative approaches are warranted to help discriminate the boundaries between the student and the teacher as a mentor, which are often ill-defined and blurred given the high-stakes summative assessments of performance. Six specific, measurable, achievable, relevant and time-based (SMART) goals were established for a mentor and mentee relationship spanning a 2-year period based on a vision for building a program of scholarship. Weekly mentoring sessions were used to monitor progress toward goal attainment. While completing the Doctor of Nursing Practice program, the mentee expanded professional networks, obtained funded fellowships, expanded knowledge and skills related to the substantive area of scholarship, and disseminated that work through professional presentations and publications coauthored with the mentor. Cognizant of the boundaries between course-related products and professional presentations and publications, the goal-driven mentor-mentee relationship is transformational to both. [J Nurs Educ. 2020;59(9):526-530.].

Investigating the diversity of scientific methods in high-stakes chemistry examinations in England

ABSTRACTThe traditional description of “the scientific method” as a stepwise, linear process of hypothesis testing through experimentation is a myth. Although the teaching and learning of the scientific method have been a curriculum and assessment goal, the notion of the ‘scientific method’ itself has been identified as being problematic. Many researchers have recognised there is no single scientific method. However, there does not seem to be any useful guidelines for how best to deal with the nature of scientific methods in school science, including in high-stakes summative assessment. The article presents the use of a framework to illustrate the diversity of scientific methods that goes beyond the traditional limitations of a scientific method, to provide a more comprehensive and inclusive account, including non-manipulative parameter measurements. The framework not only clarifies the definition of scientific methods but also is adapted as an analytical framework to trace how scientific methods are framed in high-stakes chemistry examination papers from three examination boards in England. Such analyses can potentially point to what is emphasised in chemistry lessons, given how instrumental high-stakes testing is for driving teaching and learning. Results from an empirical investigation of examination questions are presented, highlighting an imbalance in the representation of methods in chemistry tests.

Beyond Reflection: Using ePortfolios for Formative Assessment to Improve Student Engagement in Non-Majors Introductory Science

Success rates in non-majors introductory biology courses tend to be lower than those in courses for majors. The use of student learning ePortfolios correlates with increased retention and improved learning. Additionally, when ePortfolios are used for periodic formative assessment, they have been shown to stimulate student response to feedback and improve the quality of student work—both indicators of increased student engagement. In our study, the implementation of low-stake assignments in ePortfolio fostered a dialogue between the instructor and student outside of class time and provided opportunities for formative assessment of individual student and overall class learning prior to larger, high-stakes summative assessments. Furthermore, ePortfolio allowed the instructor to give students feedback on their work, creating opportunities for confirmation of learning or extended learning outside of class. Through intentional and embedded use of ePortfolios, we have created a learning environment that fosters more interaction with course material outside of class, better assignment turn-in rates, and improved exam scores, for increased success rates in the course. These results describe a promising intervention that can improve success rates in introductory biology courses.

Teaching for deeper political learning: a design experiment

Is in-depth political learning possible in college-preparatory courses known for curricular breadth at an accelerated pace plus a high-stakes exam? A multidisciplinary research team conducted design-based implementation research (DBIR) for seven years across three school systems for the purpose of achieving deeper learning in an ‘advanced’ high school government and politics course. This article reports findings from the final year of research and development. The design centres on cyclical learning, projects (simulated political processes) and strategic use of texts as resources for learning the curriculum. Quantitative analysis shows comparable achievement to students in traditional classrooms on the high-stakes summative assessment despite the dramatic departure from test-prep instruction. Qualitative analyses focus on two of the design elements: learning from simulations and learning from text. The first allowed students to ‘experience’ political beliefs, institutions, and conflict, although simulated; the second required students to learn subject matter not only from simulations and teacher lectures but from texts, too. The discussion shows how these two in combination with cyclical learning required skilful attention to content selection. We conclude that when deeper learning is the goal, content selection cannot be elided or presumed, particularly with respect to the articulation of depth and breadth in curriculum and instruction.

Use of Student Response Systems for Summative Assessments

Student Response Systems (Clickers) have been adopted by a number of instructors to increase interactions, student engagement and/or formative assessment and feedback especially in large group sessions. Clickers are well known as tools for active learning strategy particularly in formative assessments. However, they are rarely used in assessments that count. With the advent of clickers with a display screen, attempts are being made to use clickers in summative exams. We examined the feasibility of their use in high stakes summative assessment by piloting such an assessment in a simulated setting. Utilizing the lessons learned in the pilot study, clickers were used in formative and summative assessments in various iterations of a computer course taught by one of the authors (CC). At the end of the course, perceptions of students on the use of clickers for high-stake examinations were obtained using an online survey. The instructor was interviewed to identify factors that facilitated clicker use and the challenges faced. In general, students were accepting of the use of this technology in high stakes exams and found it engaging and satisfying, primarily because of instant feedback. The instructor found the process less time consuming and efficient, and more secure compared to scan sheets. Clickers are best used for examinations of short duration, with multiple-choice questions or questions with minimal text or mathematical entry.

Assessment and the aims of the curriculum: An explorer’s journey

This article considers lessons learnt through involvement in several assessment projects. Early experience, in university work and in school examinations, led to an opportunity to help establish a novel system of assessment for an innovative school curriculum. Different lessons were then learnt from work on a national survey of school students’ learning of science, and different lessons again while leading a group to advise the UK government on a new scheme for national testing of all students. Many welcomed the group’s advice but politicians rejected it; however, the recoil from this defeat led to very rewarding work on formative assessment. The article ends with reflection on the conflict between the summative and the formative and ways to resolve that conflict, along with the full benefit of formative approaches that investment can secure to help teachers share responsibility for high-stakes summative assessments.

Using summative and formative assessments to evaluate EFL teachers’ teaching performance

Using classroom observations (formative) and student course experience survey results (summative) to evaluate English lecturers’ teaching performances is not new in practice, but surprisingly only a few studies have investigated this issue in a higher education context. This study was conducted in an English department of a large university in Vietnam. The data include: (1) semi-structured interviews with all the full-time lecturers, (2) two department heads and (3) course experience surveys from English as a foreign language (EFL) students (N = 2886). Three lessons can be learned: (1) formative assessments do not seem to have an effect on promoting better teaching practices when their feedback is not helpful in improving high-stakes summative assessment results, (2) without sharing a common definition of good teaching practices among assessors, summative assessments appear to make the feedback from formative assessments less meaningful and applicable, and (3) as a result, the combination of formative and summative assessments tends to make EFL lecturers’ self-assessment practices less effective.

Reducing the need for guesswork in multiple-choice tests

The humble multiple-choice test is very widely used within education at all levels, but its susceptibility to guesswork makes it a suboptimal assessment tool. The reliability of a multiple-choice test is partly governed by the number of items it contains; however, longer tests are more time consuming to take, and for some subject areas, it can be very hard to create new test items that are sufficiently distinct from previously used items. A number of more sophisticated multiple-choice test formats have been proposed dating back at least 60 years, many of which offer significantly improved test reliability. This paper offers a new way of comparing these alternative test formats, by modelling each one in terms of the range of possible test taker responses it enables. Looking at the test formats in this way leads to the realisation that the need for guesswork is reduced when test takers are given more freedom to express their beliefs. Indeed, guesswork is eliminated entirely when test takers are able to partially order the answer options within each test item. The paper aims to strengthen the argument for using more sophisticated multiple-choice test formats, especially for high-stakes summative assessment.