Low-stakes Assessments Research Articles

Practice to Perform: Formative Mini-Skills Checkoffs Prior to High-Stakes Summative Assessment for Practice-Ready Students.

In previous years, students expressed their concerns regarding the high-stress nature of the summative skills checkoff, and pass rates were not as high as expected. To evaluate the impact of mini-skills checkoff assessments and the use of standardized patients (SPs) on summative skills pass rates and nursing students' perceptions of this teaching approach. A mixed-methods approach was used with a convenience sample of 46 sophomore nursing students. There was a significant difference in the summative skills checkoff pass rates after including mini-skills checkoffs and SPs ( P < .01). Four dominant themes emerged: emotional impact, preparation and learning, confidence and competence, and readiness. Evidence from this pilot study suggests the inclusion of continuous, low-stakes assessments over time, with learner feedback bridges the gap between clinical practice, simulation, and classroom learning.

Incorporating test-taking engagement into the item selection algorithm in low-stakes computerized adaptive tests

In low-stakes assessment settings, students’ performance is not only influenced by students’ ability level but also their test-taking engagement. In computerized adaptive tests (CATs), disengaged responses (e.g., rapid guesses) that fail to reflect students’ true ability levels may lead to the selection of less informative items and thereby contaminate item selection and ability estimation procedures. To date, researchers have developed various approaches to detect and remove disengaged responses after test administration is completed to alleviate the negative impact of low test-taking engagement on test scores. This study proposes an alternative item selection method based on Maximum Fisher Information (MFI) that considers test-taking engagement as a secondary latent trait to select the most optimal items based on both ability and engagement. The results of post-hoc simulation studies indicated that the proposed method could optimize item selection and improve the accuracy of final ability estimates, especially for low-ability students. Overall, the proposed method showed great promise for tailoring CATs based on test-taking engagement. Practitioners are encouraged to consider incorporating engagement into the item selection algorithm to enhance the validity of inferences made from low-stakes CATs.

Students’ Perceptions of Low Stakes Positioning Tests at the Start of Higher STEM Education: A Mixed Methods Approach

In Flanders, open admission into higher education has led to heterogeneity in academic preparedness of incoming STEM students. Higher education institutions offer low stakes positioning tests to these students in order to help them assess their level of starting competences. Due to the unique nature of these tests, little can be inferred about students’ perception of these tests. In this study, we used mixed methods to investigate these perceptions. Semi-structured interviews (n=20) and thematic analysis revealed that students do not always understand the stakes, objective and structure of positioning tests as intended by the organizers. A questionnaire (n=189) revealed that students considered chemistry problems better reflecting certain study programs than mathematics problems and that they wrongly think chemistry questions are more predictive than mathematics questions. Our findings provide insight in STEM students’ multifaceted perceptions of and interaction with low stakes assessment.

"It Changed How I Think"-Impact of Programmatic Assessment Upon Practicing Physicians: A Qualitative Study.

Programmatic assessment is a systematic approach used to document and assess learner performance. It offers learners frequent formative feedback from a variety of contexts and uses both high- and low-stakes assessments to determine student progress. Existing research has explored learner and faculty perceptions of programmatic assessment, reporting favorable impact on faculty understanding of the importance of assessment stakes and feedback to learners while students report the ability to establish and navigate towards goals and reflect on their performance. The Cleveland Clinic Lerner College of Medicine (CCLCM) of Case Western Reserve University adopted programmatic assessment methods at its inception. With more than 18years' experience with programmatic assessment and a portfolio-based assessment system, CCLCM is well-positioned to explore its graduates' perceptions of their programmatic assessment experiences during and after medical school. In 2020, the investigators interviewed 26 of the 339 physician graduates. Participants were purposefully sampled to represent multiple class cohorts (2009-2019), clinical specialties, and practice locations. The investigators analyzed interview transcripts using thematic analysis informed by the frameworks of self-determination theory and professional identity formation. The authors identified themes and support each with participant quotes from the interviews. Based on findings, the investigators compiled a series of recommendations for other institutions who have already or plan to incorporate elements of programmatic assessment into their curricula. The authors concluded by discussing future directions for research and additional avenues of inquiry.

Stakes of Assessments in Residency: Influence on Previous and Current Self-Regulated Learning and Co-Regulated Learning in Early Career Specialists.

Assessments drive learning but the influence of the stakes of the assessments on self-regulated (SRL) during and after residency are unknown. As early career specialists (ECS) must continue learning independently, the answer to this is important as it may inform future assessments with the potential to promote life-long learning after graduation. We utilized constructivist grounded theory to explore the perspectives of eighteen ECS on the influence of stakes of assessments within residency on their SRL during training and in current practice. We conducted semi-structured interviews. We initially set out to examine the influence of the stakes of assessments on SRL during residency and after graduation. However, it was apparent that learners increasingly engaged with others in co-regulated learning (CRL) as the perceived stakes of the assessments increased. The individual learner's SRL was embedded in CRL in preparation for the various assessments in residency. For low-stakes assessments, the learner engaged in less CRL, taking less cues from others. As stakes increased, the learner engaged in more CRL with peers with similar intellectual level and supervisors to prepare for these assessments. SRL and CRL influenced by assessments in residency had a knock-on effect in clinical practice as ECS in: 1) developing clinical reasoning, 2) improving doctor-patient communication and negotiation skills, and 3) self-reflections and seeking feedback to deal with expectations of self or others. Our study supported that the stakes of assessments within residency reinforced SRL and CRL during residency with a continued effect on learning as ECS.

A Pharmacy Drug Knowledge Assessment Pilot: Who Will Fly Farthest and What Downs the Plane?

To evaluate the effectiveness of a sequenced drug knowledge pilot in third professional year students in a capstone course. A three-phase drug knowledge pilot was conducted in spring 2022. Students completed a total of thirteen assessments, including nine low-stakes quizzes, three formative tests, and a final summative comprehensive exam. Results from the previous year's cohort (historical control) who only completed a summative comprehensive exam were compared to the pilot (test group) results to assess effectiveness. The faculty spent over 300 h developing content for the test group. The pilot group had a mean score of 80.9% on the final competency exam, which was one percent lower than the control group who had a less rigorous intervention. A sub-analysis was conducted that removed the students who failed (<73%) the final competency exam, and no significant difference in the exam score was found. One practice drug exam was found to be moderately correlated and significant (r = 0.62) with the final knowledge exam performance in the control. The number of attempts on the low-stakes assessments had a low correlation with the final exam score in the test group compared to the control (r = 0.24). The results of this study suggest a need to further investigate the best practices for knowledge-based drug characteristic assessments.

Implementation and effects of low stakes quiz feedback on exam performance in a health professional program

Introduction: The inclusion of interactive learning approaches in graduate education including mobile polling, game-based learning, physiology apps, eLearning activities and tablet use to supplement course content is an established means to improve student engagement. Strategies that continue to support learning asynchronously with the purpose of driving deeper critical thinking and reasoning has the potential to further advance course outcomes. The purpose of this investigation was to assess whether the incorporation of detailed asynchronous feedback to low stakes quizzes as an additional learning tool translates into improved exam performance in the Foundational Sciences. Methods: ExamSoft assessment data was collected across 2 semesters from 101 Doctor of Physical Therapy students enrolled in a Foundational Sciences course series encompassing cardiopulmonary, pediatrics and neuro blocked content. Exam performance was compared between 2 cohorts (non-feedback, n=53 and feedback group, n=48) following the implementation of detailed question feedback on low stakes semester quizzes with the second cohort. Exam performance on specific subject areas was also reviewed for select exam questions that directly aligned with quiz questions. Results: Independent samples t-test analyses indicated that learners in the feedback group demonstrated consistently improved exam performance for midterm (p &lt; 0.001 for summer midterm and p &lt; 0.01 for fall midterm) and final exams (p &lt; 0.01 for both summer and fall final exams) across the 2 semesters when compared to the non-feedback group. Similar results were observed when specific exam questions were reviewed across physiology content areas. Vestibular physiology exam performance was significantly (p = 0.05) improved in the feedback group, while basal ganglia physiology showed a trend but was not significant. Pharmacology exam performance was significantly improved for the fall final exam (p &lt; 0.001), but not for the fall midterm. Conclusions: Instructor tailored quiz question feedback, which requires student review to view the correct answer choice is an additional learning opportunity that results in improved course outcomes. This approach to providing feedback to low stakes assessments focuses the in-class time on new material because the quiz feedback can be reviewed asynchronously as needed by the learner. Using the learning management system quiz feedback feature provides a valuable strategy to further support learning outside of the classroom and provides an opportunity for students to synthesize and apply content at a deeper level. This is the full abstract presented at the American Physiology Summit 2023 meeting and is only available in HTML format. There are no additional versions or additional content available for this abstract. Physiology was not involved in the peer review process.

Student-Created Physics Problems as an Independent and Equitable Assessment Tool

Traditional high-stakes summative assessments—timed, in-class exams accounting for a large percentage of the term’s overall grade—have often received criticism from the educational community. Such assessments tend to prize a particular “narrow bundle of skills,” and have been shown in some contexts to produce disparate outcomes between different demographic groups. Alternative low-stakes assessments (assignments with significantly less impact on the overall grade) have shown potential to improve student engagement and close demographic gaps. At the same time, principles from physics education research suggest to us that students may benefit from the challenge of writing their own physics problems. Conjoining these ideas, we document an assignment (“The Problem Project”) in which students were asked to create and solve a problem of their own design, as an alternative form of assessment, and provide exploratory research into its equity impacts.

What Moves the Needle on DFW Rates and Student Success in General Chemistry? A Quarter-Century Perspective

Second-term general chemistry is a key gateway course at our institution for a variety of STEM (Science, Technology, Engineering, and Mathematics) majors. Herein, we examine data on DFW% (i.e., % of D and F grades and withdrawals) in our general chemistry courses over roughly a quarter-century period, focusing on sections taught by the authors. The data are discussed in light of changes in curriculum, pedagogy, and assessments, which included introduction of recitation sections, classroom response systems (clickers), and online homework, restructuring of course content into a modular format, followed by a redesign of the course modules along an anchoring concepts scheme, introduction of the flipped classroom, and introduction of regular low-stakes quizzing. We find that the lowest DFW%, roughly 5%, occurs in classrooms combining an active learning pedagogy, core concepts curriculum, and low-stakes assessments with instructor intervention. Trends in the DFW populations are examined for White, non-Hispanic students, first-generation students, and by binary gender. We find that women are increasingly underrepresented in the DFW population, which may reflect the recent incorporation of lower stakes assessments. To the degree that our lowest DFW rates are consistent with that found by other researchers using different but related approaches, these may represent a baseline target for DFW rates in general chemistry II. Finally, while a trend toward lower DFW rates is found for both general chemistry I and II, the follow-on course, organic chemistry I, shows an inverse trend, which emphasizes the need for larger scale reform.

Teaching and Assessing Systems Thinking in First-Year Chemistry

Powerful arguments have been made recently to advocate for the introduction of systems thinking in chemistry education to equip graduates to address sustainability challenges. This contribution describes systems thinking activities and an assessment tool that enables meaningful learning in first-year organic chemistry as students engage with the molecular-level foundation and real-world implications of the system under consideration. The activity incorporates concept maps and systems-oriented concept mapping extensions (SOCMEs) as visualization tools to scaffold the development of systems thinking skills. A rubric based on the SOLO taxonomy was designed and used to assess evidence of systems thinking skills demonstrated on SOCMEs. The activity was implemented in a small and a large group of 18 and 219 participants respectively, where findings from both groups are presented. The grading of six SOCMEs by three independent raters proved to be inconsistent at higher thinking SOLO levels, where rater experience was flagged as an area for future improvement to ensure raters are proficient in assessing systems thinking. However, meaningful progress was made to assess the quality of learning. The rubric was an effective tool for formative and low-stakes assessment to drive the teaching and learning of systems thinking in chemistry.

Simulation vs workplace-based assessment in resuscitation: a cross-specialty descriptive analysis and comparison.

Simulation-based assessment can complement workplace-based assessment of rare or difficult to assess Entrustable Professional Activities (EPAs). We aimed to compare the use of simulation-based assessment for resuscitation-focused EPAs in three postgraduate medical training programs and describe faculty perceptions of simulation-based assessment. EPA assessment scores and setting (simulation or workplace) were extracted from 2017-2020 for internal medicine, emergency medicine, and surgical foundations residents at the transition to discipline and foundations of discipline stages. A questionnaire was distributed to clinical competency committee members. Eleven percent of EPA assessments were simulation-based. The proportion of simulation-based assessment did not differ between programs but differed between transition (38%) and foundations (4%) stages within surgical foundations only. Entrustment scores differed between settings in emergency medicine at the transition level only (simulation: 4.82 ± 0.60 workplace: 3.74 ± 0.93). 70% of committee members (n=20) completed the questionnaire. Of those that use simulation-based assessment, 45% interpret them differently than workplace-based assessments. 73% and 100% trust simulation for high-stakes and low-stakes assessment, respectively. The proportion of simulation-based assessment for resuscitation focused EPAs did not differ between three postgraduate medical training programs. Interpretation of simulation-based assessment data between committee members was inconsistent. All respondents trust simulation-based assessment for low-stakes, and the majority for high-stakes assessment. These findings have practical implications for the integration simulation into programs of assessment.

Rapid Guessing in Low-Stakes Assessments: Finding the Optimal Response Time Threshold with Random Search and Genetic Algorithm

Rapid guessing is an aberrant response behavior that commonly occurs in low-stakes assessments with little to no formal consequences for students. Recently, the availability of response time (RT) information in computer-based assessments has motivated researchers to develop various methods to detect rapidly guessed responses systematically. These methods often require researchers to identify an RT threshold subjectively for each item that could distinguish rapid guessing behavior from solution behavior. In this study, we propose a data-driven approach based on random search and genetic algorithm to search for the optimal RT threshold within a predefined search space. We used response data from a low-stakes math assessment administered to over 5000 students in 658 schools across the United States. As we demonstrated how to use our data-driven approach, we also compared its performance with those of the existing threshold-setting methods. The results show that the proposed method could produce viable RT thresholds for detecting rapid guessing in low-stakes assessments. Moreover, compared with the other threshold-setting methods, the proposed method yielded more liberal RT thresholds, flagging a larger number of responses. Implications for practice and directions for future research were discussed.

"The Most Crushing Thing": Understanding Resident Assessment Burden in a Competency-Based Curriculum.

Competency-based medical education (CBME) was expected to increase the workload of assessment for graduate training programs to support the development of competence. Learning conditions were anticipated to improve through the provision of tailored learning experiences and more frequent, low-stakes assessments. Canada has adopted an approach to CBME called Competence by Design (CBD). However, in the process of implementation, learner anxiety and assessment burden have increased unexpectedly. To mitigate this unintended consequence, we need a stronger understanding of how resident assessment burdens emerge and function. This study investigates contextual factors leading to assessment burden on residents within the framework of CBD. Residents were interviewed about their experiences of assessment using constructivist grounded theory. Participants (n=21) were a purposive sample from operative and perioperative training programs, recruited from 6 Canadian medical schools between 2019 and 2020. Self-determination theory was used as a sensitizing concept to categorize findings on types of assessment burden. Nine assessment burdens were identified and organized by threats to psychological needs for autonomy, relatedness, and competence. Burdens included: missed opportunities for self-regulated learning, lack of situational control, comparative assessment, lack of trust, constraints on time and resources, disconnects between teachers and learners, lack of clarity, unrealistic expectations, and limitations of assessment forms for providing meaningful feedback. This study contributes a contextual understanding of how assessment burdens emerged as unmet psychological needs for autonomy, relatedness, and competence, with unintended consequences for learner well-being and intrinsic motivation.

ISTART: Adaptive Comprehension Strategy Training and Stealth Literacy Assessment

The Interactive Strategy Training for Active Reading and Thinking (iSTART) game-based intelligent tutoring system (ITS) was developed with a foundation of comprehension theory and principles of learning science to improve students’ comprehension of complex scientific texts. iSTART has been shown to improve reading comprehension for learners from middle school through adulthood, particularly lower knowledge readers, through strategy instruction and game-based practice. This paper describes iSTART, the theoretical foundations that have guided iSTART development, and evidence for the feasibility of game-based practice to improve learning outcomes. This paper also introduces a novel method of assessing students’ reading comprehension through game-based literacy assessments that have been incorporated in iSTART. The development of these stealth assessments was guided by recent work emphasizing the need for rapid, dynamic, and low stakes assessments that evaluate students’ reading skills in the context of brief, dynamic games. Stealth assessments can generate estimates of multiple aspects of students’ reading comprehension quickly and within a motivating environment. The work described in this paper is a promising method to assess students’ literacy in an unobtrusive and authentic way that may lead to improved learning outcomes for students.

Implementation of Entrustable Professional Activities assessments in a Canadian obstetrics and gynecology residency program: a mixed methods study.

BackgroundSince the implementation of competency-based medical education (CBME) across residency training programs in Canada, there has been limited research understanding how entrustable professional activity (EPA) assessments are used by faculty supervisors and residents.ObjectiveThis study examines how EPA assessments are used in an Obstetrics and Gynecology residency program and the impact of implementation on both groups.MethodsA mixed methods study design was used. Part one involved the aggregation of descriptive data of EPA assessment completion for postgraduate year 1 and 2 residents from July 2019 to May 2020. Part two involved a thematic analysis of semi-structured interviews of residents and faculty.ResultsThere was significant uptake of EPA assessments across community and teaching hospitals with widespread contribution of assessment data from faculty. However, both residents and faculty reported that the intended design of EPA assessments as low-stakes assessments to provide formative feedback is not how EPA assessments are experienced. Residents and faculty noted the increased level of administrative burden and related perceived stress amongst the resident group.ConclusionsThe implementation of EPA assessments is feasible across a variety of sites. However, previous measurement challenges remain. Neither residents nor faculty perceive the value of EPAs to improve feedback, despite their intended nature.

The education passport: connecting programmatic assessment across learning and practice.

Competency-based medical education (CBME) shifts us from static assessment of learning to developmental assessment for learning. However, implementation challenges associated with CBME remain a major hurdle, especially after training and into practice. The full benefit of developmental assessment for learning over time requires collaboration, cooperation, and trust among learners, regulators, and the public that transcends each individual phase. The authors introduce the concept of an “Education Passport” that provides evidence of readiness to travel across the boundaries between undergraduate medical education, graduate medical education, and the expanse of practice.The Education Passport uses programmatic assessment, a process of collecting numerous low stakes assessments from multiple sources over time, judging these data using criterion-referencing, and enhancing this with coaching and competency committees to understand, process, and accelerate growth without end. Information in the Passport is housed on a cloud-based server controlled by the student/physician over the course of training and practice. These data are mapped to various educational frameworks such Entrustable Professional Activities or milestones for ease of longitudinal performance tracking. At each stage of education and practice the student/physician grants Passport access to all entities that can provide data on performance. Database managers use learning analytics to connect and display information over time that are then used by the student/physician, their assigned or chosen coaches, and review committees to maintain or improve performance. Global information is also collected and analyzed to improve the entire system of learning and care.Developing a true continuum that embraces performance and growth will be a long-term adaptive challenge across many organizations and jurisdictions and will require coordination from regulatory and national agencies. An Education Passport could also serve as an organizing tool and will require research and high-value communication strategies to maximize public trust in the work.

Exploring the pre-instruction gender gap in physics.

There is a substantial body of work in physics education looking at gender disparities in physics. Recent work has linked gender disparities in college physics course performance to disparities in high school physics preparation, but to our knowledge, the origin of the disparity in high school physics preparation is still underexplored. In a select sample, we found that women on average had lower force and motion conceptual evaluation (FMCE) pre-scores (the FMCE is a short conceptual assessment of Newton's laws), and FMCE pre-score entirely mediated the effects of high school preparation and social-psychological factors on exam performance. The gender gap in FMCE pre-scores could not be explained by differences in the number of physics courses taken in high school. Instead, we find that the gender gap in the FMCE is partially explained by female students' higher levels of general test anxiety. We hypothesize that the format of the FMCE, a timed assessment, triggers stereotype threat in female students despite being a low-stakes assessment. Therefore, instructors and researchers should take care in interpreting the results of such concept inventory scores and should re-think the way they assess understanding of physics concepts. Results of this work aligned with previous findings on gender disparity in timed exams call upon investigating gender equitable assessment formats for evaluating physics knowledge to replace timed assessments, either high or low stakes.

Not-reached Items: An Issue of Time and of test-taking Disengagement? the Case of PISA 2015 Reading Data

ABSTRACT Many low-stakes assessments, such as international large-scale surveys, are administered during time-limited testing sessions and some test-takers are not able to endorse the last items of the test, resulting in not-reached (NR) items. However, because the test has no consequence for the respondents, these NR items can also stem from quitting the test. This article, by means of mixture modeling, investigates heterogeneity in the onset of NR items in reading in PISA 2015. Test-taking behavior, assessed by the response times on the first items of the test, and the risk of NR item onset are modeled simultaneously in a 3-class model that distinguishes rapid, slow and typical respondents. Results suggest that NR items can come from a lack of time or from disengaged behaviors and that the relationship between the number of NR items and ability estimate can be affected by these non-effortful NR responses.

Investigating the Effect of Differential Rapid Guessing on Population Invariance in Equating.

Score equating is an essential tool in improving the fairness of test score interpretations when employing multiple test forms. To ensure that the equating functions used to connect scores from one form to another are valid, they must be invariant across different populations of examinees. Given that equating is used in many low-stakes testing programs, examinees' test-taking effort should be considered carefully when evaluating population invariance in equating, particularly as the occurrence of rapid guessing (RG) has been found to differ across subgroups. To this end, the current study investigated whether differential RG rates between subgroups can lead to incorrect inferences concerning population invariance in test equating. A simulation was built to generate data for two examinee subgroups (one more motivated than the other) administered two alternative forms of multiple-choice items. The rate of RG and ability characteristics of rapid guessers were manipulated. Results showed that as RG responses increased, false positive and false negative inferences of equating invariance were respectively observed at the lower and upper ends of the observed score scale. This result was supported by an empirical analysis of an international assessment. These findings suggest that RG should be investigated and documented prior to test equating, especially in low-stakes assessment contexts. A failure to do so may lead to incorrect inferences concerning fairness in equating.

Shaping the right conditions in programmatic assessment: how quality of narrative information affects the quality of high-stakes decision-making

BackgroundProgrammatic assessment is increasingly being implemented within competency-based health professions education. In this approach a multitude of low-stakes assessment activities are aggregated into a holistic high-stakes decision on the student’s performance. High-stakes decisions need to be of high quality. Part of this quality is whether an examiner perceives saturation of information when making a holistic decision. The purpose of this study was to explore the influence of narrative information in perceiving saturation of information during the interpretative process of high-stakes decision-making.MethodsIn this mixed-method intervention study the quality of the recorded narrative information was manipulated within multiple portfolios (i.e., feedback and reflection) to investigate its influence on 1) the perception of saturation of information and 2) the examiner’s interpretative approach in making a high-stakes decision. Data were collected through surveys, screen recordings of the portfolio assessments, and semi-structured interviews. Descriptive statistics and template analysis were applied to analyze the data.ResultsThe examiners perceived less frequently saturation of information in the portfolios with low quality of narrative feedback. Additionally, they mentioned consistency of information as a factor that influenced their perception of saturation of information. Even though in general they had their idiosyncratic approach to assessing a portfolio, variations were present caused by certain triggers, such as noticeable deviations in the student’s performance and quality of narrative feedback.ConclusionThe perception of saturation of information seemed to be influenced by the quality of the narrative feedback and, to a lesser extent, by the quality of reflection. These results emphasize the importance of high-quality narrative feedback in making robust decisions within portfolios that are expected to be more difficult to assess. Furthermore, within these “difficult” portfolios, examiners adapted their interpretative process reacting on the intervention and other triggers by means of an iterative and responsive approach.