Educational Assessment Program Research Articles

The Association of Psychotherapy With Burnout, Depression, and Measures of Well-Being in Residents and Fellows: A Pilot Study.

Resident physicians experience high rates of burnout and depression but rarely prioritize their well-being or seek mental health care. The Accreditation Council for Graduate Medical Education mandated that training programs prioritize resident wellness and emotional and mental health to ensure readily available and accessible mental health care. To help meet that requirement and circumvent barriers to accessing care, the University of California San Diego Healer Education Assessment & Referral (HEAR) Program offers residents and fellows short-term therapy for coping with challenges that threaten their well-being. This report describes the results of a pilot study designed to evaluate the feasibility and effectiveness of the HEAR Program's resident therapy program. The cohort included residents and fellows who completed at least 1 postbaseline assessment from January to May 2022. Measures of fulfillment, burnout, self-compassion, quality of life, depression, and suicidal ideation were assessed and compared before and up to 12 weeks after enrollment. Of the 39 residents who consented to participation, 30 completed at least 1 postbaseline assessment. Most outcomes improved after therapy, with significant increases in fulfillment (mean [SE] coefficient, 0.24 [0.08]; z score, 2.86; P = .004), self-compassion (mean [SE] coefficient, 0.37 [0.07]; z score, 5.72; P < .001), and quality of life ( P < .001) and significant reductions in burnout (Stanford burnout scale: mean [SE] coefficient, -0.27 [0.07]; z score, -4.01; P < .001; single-item burnout scale: mean [SE] coefficient, -0.34 [0.08]; z score, -4.37; P < .001) and depression severity (mean [SE] coefficient, -1.08 [0.25]; z score, -4.36; P < .001). This pilot study noted improvements in fulfillment, compassion, quality of life, and function, as well as reductions in burnout and depression severity, among resident physicians. Future studies in larger cohorts are needed to validate these findings and inform further optimization of this program.

EFFORTS TO IMPROVE TEACHERS' ABILITY TO DEVELOP ASSESSMENT PROGRAMS THROUGH IN-HOUSE TRAINING ACTIVITIES

School development aims to achieve 8 National Education Standards (SNP) Government Regulation (PP) RI No. 32 of 2013 concerning changes to PP No. 19 of 2005 concerning National Education Standards. The achievement of 8 SNPs, specifically for Assessment Standards, is carried out by conducting In House Training (IHT) activities. The purpose of this study is to improve the achievement of 8 (eight) National Education Standards, Provide excellent and optimal services to students and parents and the community. The methods used for initial data collection before IHT and after are: Document mining using the teacher data checklist method that collects education assessment programs before IHT, Interviews, Tests, Postes, Document mining using the teacher data checklist method that collects education assessment programs after IHT. The result was a significant difference from the pretest result of 46.67 and the posttest result of 72.29. Based on the existing results, it can be concluded that the increasing ability of teachers in compiling educational assessment programs, preliminary data before IHT was obtained 6 people did not compile educational assessment programs, 42 people compiled education assessment programs but were incomplete and no one compiled a complete education assessment program. After the implementation of IHT, data were obtained: there were no teachers who did not compile an educational assessment program, 2 people compiled an incomplete but then improved and 46 people compiled a complete education assessment program.

Bayesian Model Assessment for Jointly Modeling Multidimensional Response Data with Application to Computerized Testing.

Computerized assessment provides rich multidimensional data including trial-by-trial accuracy and response time (RT) measures. A key question in modeling this type of data is how to incorporate RT data, for example, in aid of ability estimation in item response theory (IRT) models. To address this, we propose a joint model consisting of a two-parameter IRT model for the dichotomous item response data, a log-normal model for the continuous RT data, and a normal model for corresponding paper-and-pencil scores. Then, we reformulate and reparameterize the model to capture the relationship between the model parameters, to facilitate the prior specification, and to make the Bayesian computation more efficient. Further, we propose several new model assessment criteria based on the decomposition of deviance information criterion (DIC) the logarithm of the pseudo-marginal likelihood (LPML). The proposed criteria can quantify the improvement in the fit of one part of the multidimensional data given the other parts. Finally, we have conducted several simulation studies to examine the empirical performance of the proposed model assessment criteria and have illustrated the application of these criteria using a real dataset from a computerized educational assessment program.

Moral Injury.

Moral Injury.

Digital Module 18: Automated Scoring https://ncme.elevate.commpartners.com

AbstractIn this digital ITEMS module, Dr. Sue Lottridge, Amy Burkhardt, and Dr. Michelle Boyer provide an overview of automated scoring. Automated scoring is the use of computer algorithms to score unconstrained open‐ended test items by mimicking human scoring. The use of automated scoring is increasing in educational assessment programs because it allows scores to be returned faster at lower cost. In the module, they discuss automated scoring from a number of perspectives. First, they discuss benefits and weaknesses of automated scoring, and what psychometricians should know about automated scoring. Next, they describe the overall process of automated scoring, moving from data collection to engine training to operational scoring. Then, they describe how automated scoring systems work, including the basic functions around score prediction as well as other flagging methods. Finally, they conclude with a discussion of the specific validity demands around automated scoring and how they align with the larger validity demands around test scores. Two data activities are provided. The first is an interactive activity that allows the user to train and evaluate a simple automated scoring engine. The second is a worked example that examines the impact of rater error on test scores. The digital module contains a link to an interactive web application as well as its R‐Shiny code, diagnostic quiz questions, activities, curated resources, and a glossary.

Evaluation of a National Competency-Based Assessment System in Emergency Medicine: A CanDREAM Study

In 2018, Canadian postgraduate emergency medicine (EM) programs began implementing a competency-based medical education (CBME) assessment program. Studies evaluating these programs have focused on broad outcomes using data from national bodies and lack data to support program-specific improvement. We evaluated the implementation of a CBME assessment program within and across programs to identify successes and opportunities for improvement at the local and national levels. Program-level data from the 2018 resident cohort were amalgamated and analyzed. The number of entrustable professional activity (EPA) assessments (overall and for each EPA) and the timing of resident promotion through program stages were compared between programs and to the guidelines provided by the national EM specialty committee. Total EPA observations from each program were correlated with the number of EM and pediatric EM rotations. Data from 15 of 17 (88%) programs containing 9842 EPA observations from 68 of 77 (88%) EM residents in the 2018 cohort were analyzed. Average numbers of EPAs observed per resident in each program varied from 92.5 to 229.6, correlating with the number of blocks spent on EM and pediatric EM (r = 0.83, P < .001). Relative to the specialty committee's guidelines, residents were promoted later than expected (eg, one-third of residents had a 2-month delay to promotion from the first to second stage) and with fewer EPA observations than suggested. There was demonstrable variation in EPA-based assessment numbers and promotion timelines between programs and with national guidelines.

Sustainability and Outcomes of a Suicide Prevention Program for Nurses.

We now know that nurses are at greater risk for suicide than others in the general population. It is known that job stressors are prevalent in nurses who die by suicide. Yet, little is known about targeted suicide prevention for nurses. The first nurse-centric Healer Education Assessment and Referral (HEAR) suicide prevention program was piloted for 6months in 2016. The HEAR program was effective in identifying at-risk nurses. The purpose of this paper is to report the 3-year sustainability and outcomes of this nurse suicide prevention program. Descriptive statistics are provided of program outcomes over the course of 3years. Over the 3years, 527 nurses have taken advantage of the screening portion of the program. Of these, 254 (48%) were Tier 1 high risk, and 270 (51.2%) were Tier 2 moderate risk. A startling 48 (9%) had expressed thoughts of taking their own life, 51 (9.7%) had a previous suicide attempt, whereas only 79 (15%) were receiving counseling or therapy. One hundred seventy-six nurses received support from therapists electronically, over the phone, or in person; 98 nurses accepted referral for treatment. The number of group emotional debriefs rose from eight in 2016 to 15 in 2017 to 38 in fiscal year 2019. Many of the debriefs are now requested (vs. offered), demonstrating the development of a culture open to reaching out for mental health treatment. The initial success of this pilot program has been sustained. A nurse suicide prevention program of education, assessment, and referral is feasible, well-received, proactively identifies nurses with reported suicidality and facilitates referral for care. The HEAR program has provided service to physicians and residents for 10years and now supports effectiveness in nurses. The HEAR program is portable and ready for replication at other institutions.

A Graphical Transition Table for Communicating Status and Growth.

This paper introduces a simple and intuitive graphical display for transition table based accountability models that can be used to communicate information about students’ status and growth simultaneously. This graphical transition table includes the use of shading to convey year to year transitions and different sized letters for performance categories to depict yearly status. Examples based on Michigan’s transition table used on their Michigan Educational Assessment Program (MEAP) assessments are provided to illustrate the utility of the graphical transition table in practical contexts. Additional potential applications of the graphical transition table are also suggested.

Introduction to the Special Issue on Research and Development on Large-Scale Educational Assessment Programs

Introduction to the Special Issue on Research and Development on Large-Scale Educational Assessment Programs

Update on the UC San Diego Healer Education Assessment and Referral (HEAR) Program

Burnout, depression and suicide are rampant amongst health care professionals. Current evidence shows the problem is worsening. In the aftermath of physician suicides, the Physician Wellbeing Committee created the UC San Diego Healer Education Assessment and Referral (HEAR) Program in 2009 in collaboration with the American Foundation for Suicide Prevention (AFSP). This article chronicles the HEAR program from inception through June 2017. Initially created to address medical students, residents and faculty physician duress, HEAR has now expanded to embrace pharmacists, nurses and clinical staff within UC San Diego Health. HEAR operates through two mechanisms: 1) a program of ongoing education and outreach, and 2) encouragement of all personnel to annually engage in the online, anonymous, interactive screening program, created by the AFSP (known as SDSQ at UC San Diego Health). Since inception in May 2009 through June 2017, 1,537 UC San Diego health care personnel have been screened, 320 individuals have dialogued with a counselor either in person, by phone or electronically, and more than 300 have been referred confidentially for evaluation and treatment by a mental health professional, usually a community psychiatrist. While tracking death by suicide remains challenging, we have reason to believe that the prevalence of suicide has diminished during this time. The UC San Diego HEAR Program is one cost-effective model for addressing this current crisis in U.S. health care. This AFSP model has been adopted by many other schools of medicine and is now ready for use with clinicians of all disciplines.And whoever saves a life, it is considered as if he saved an entire world.— Mishnah Sanhedrin 4:5; Babylonian Talmud Tractate Sanhedrin 37a

Relationship of Teacher Training and School Characteristics to Middle School State Assessment Results

Purpose:To examine the relationship between the South Carolina middle school physical education assessment results and the school characteristics. In addition, the relationship between teacher training attendance and student achievement were determined.Method:Student performance on four physical education indicators in 63 middle schools (and 116 teachers) were reported to the South Carolina Physical Education Assessment Program. Multiple regression analyses were conducted to determine the relationships between school characteristics as predictors of the performance indicator. ANOVAs were conducted to determine the relationship to teacher training and the performance indicators.Results:Statewide averages of student performance indicated that slightly over 50% of middle school students were rated as competent in all physical education indicators except health-related fitness (31.2%). The variability was high among all indicators. The correlations between the poverty index and the physical education indicators were significant and low. Teachers who attended data collection training sessions scored higher on all performance indicators, particularly health-related fitness knowledge. Teachers who attended professional development had significantly higher scores on motor skills, health-related fitness knowledge, and the overall weighted scores and approached significance on the health-related fitness performance.Discussion/Conclusion:This study suggests that teachers and the programs they deliver have a greater impact on student learning than do school characteristics. Teacher training and professional development is warranted. Most compelling is that the results of this study provide a strong argument against the practice of using student scores from other academic content areas to evaluate teacher effectiveness in physical education.

The Field of Psychological Assessment: Where it Stands and Where it’s Going – A Personal Analysis of Foci, Gaps, and Implications for EJPA

The Field of Psychological Assessment: Where it Stands and Where it’s Going – A Personal Analysis of Foci, Gaps, and Implications for <i>EJPA</i>

Effects of an Educational Gymnastics Course on the Motor Skills and Health-Related Fitness Components of PETE Students

Many physical education teacher education (PETE) programs seek to develop teacher candidates’ content knowledge through various physical activity courses. However, limited empirical evidence exists linking college physical activity courses to the development of skill or fitness. The purpose of the study was to examine the effects of an educational gymnastics course on PETE students’ motor skill proficiency and health-related fitness components. Participants (N = 22) were PETE students enrolled in an educational gymnastics course. Data were collected from two primary sources: (1) four individual skills tests and the South Carolina Physical Education Assessment Program (SCPEAP) Elementary School Educational Gymnastics assessments and (2) the FitnessGram test battery. Findings suggest that an educational gymnastics course can improve PETE students’ content knowledge/motor skill proficiency. Additionally, a relationship may exist between certain fitness indicators and motor skill level in educational gymnastics. This study may guide PETE programs in making decisions regarding the inclusion of educational gymnastics as a content course in their programs, as such a course may help ensure that teacher candidates are physically educated individuals who are able to demonstrate competent movement performance. Subscribe to TPE

Proximity of public schools to major highways and industrial facilities, and students’ school performance and health hazards

Children with consistent exposure to air pollution have increased asthma, chronic respiratory problems, and neurobehavioral dysfunction. However, many schools are located in close proximity to highways and industrial facilities which are key sources of air pollution to children. The goal of this study is to explore the association between the proximity from schools to highways and industrial facilities, and children’s school performance and health hazards. We measured the distances from 3,660 Michigan public schools to highways and industrial facilities, and linked these to the Michigan Educational Assessment Program test performance rate and the National Air Toxics Assessment’s respiratory and neurological hazards. We found that schools located closer to highways and industrial facilities had higher risks of respiratory and neurological diseases than those located farther away. We also found that schools located closer to major highways had a higher percentage of students failing to meet the state standards than the latter after controlling for the location of schools, student expenditure, school size, student–teacher ratio, and free lunch enrollment. In addition, a larger percentage of black, Hispanic, or economically disadvantaged children attended schools nearest to pollution emissions than white students.

La evaluación de la escritura a la finalización de la escolaridad primaria según las pruebas TERCE y ONE /

This paper aims to analyze the form that takes the writing assessment in two campaigns that explore students learning at the end of primary education, which corresponds to the sixth year. We are referring to the Third Regional Comparative Study (UNESCO, 2015), organized by the Latin American Laboratory for Assessment of the Quality of Education (LLECE-UNESCO) and the National Campaign for Evaluation (ONE) conducted in 2013 by the National Directorate for Information and Assessment of the Quality of Education (DiNIECE, 2014) in Argentina. Additionally, it takes up theoretical contributions derived from research on the processes involved in writing and presents the main results obtained by the subjects examined in the two aforementioned studies, considering some central aspects of the curricular designs for this level in the reference country. This is relevant as the TERCE starts from the curricular analysis of the participating countries. It should be noted that the investigation about writing has been belatedly incorporated to the educational quality assessment programs, mainly due to the difficulty of analyzing textual productions in large-scale approaches. However, it is a core skill within the conceptual framework of life skills (Atorresi, 2005).

Predicting Proficiency on Statewide Assessments: A Comparison of Curriculum-Based Measures in Middle School

ABSTRACTA key tool in multitiered systems of support is the use of curriculum-based measures to predict which students are at risk for academic failure. However, there are few studies that examine which measures are most suitable for students in middle school. The authors examine the reliability of predicting outcomes on state assessments for 3 commonly used curriculum-based measures at the middle school level. Data were collected from a middle school in the Midwest that regularly administers 3 different curriculum-based measures. Reading Curriculum-Based Measure (R-CBM), Maze, and Multiple-Choice Reading Comprehension (MCRC) were given to students in Grades 7 (n = 238) and 8 (n = 256). Logistic regression was used to examine each measure in predicting outcomes on the Michigan Education Assessment Program Reading assessment. Results indicated that MCRC more accurately predicted outcomes than R-CBM or maze (Grade 7 eβ = 1.75, Grade 8 eβ = 1.68). Limitations and recommendations for future research are discussed.

On the Teaching Management Mode of Sino-foreign Educational Cooperation Program in Higher Vocational Colleges

In order to further standardize the management of Sino-foreign cooperative education, to improve the quality, a guiding principle has been established in the Regulation of the People's Republic of China on Chinese-foreign Cooperative Education, which is wider to the outside world, running and managing the programs according to the law, and promoting the development of co-education (Sino-foreign cooperative education assessment program). The Ministry of education mainly mentioned the quality guarantee system in the process of Sino-foreign cooperative education institution, teaching quality guarantee system is the key process to the feasible evaluation. This paper, based on its author’s own experience of managing the Sino-foreign cooperative education program, gives her thought on how to improve the management of sino-foreign co-education. With China's accession to the world trade organization, opening to the outside world has entered a new stage. How to cope with the internationalization trend of China's higher education is an unavoidable and must-considered question (Cai, 2002). As a supplement of China's higher education form, Sino-foreign cooperative education has obtained fast development in our country. To do well in the construction of the international cooperation program, there are a lot of work to do, among which the most important thing is to do well in teaching management. For Sino foreign cooperative education program teaching management has its own characters, we must build appropriate teaching management system and management model.

Can compassionate healthcare be measured? The Schwartz Center Compassionate Care Scale™

ObjectiveAssess psychometric characteristics of an instrument to measure patient ratings of treating physicians’ compassionate care in a recent hospitalization. MethodsWe used Cronbach's alpha to examine scale reliability, exploratory and confirmatory factor analysis to examine scale structure of two sets of items on compassionate care. We used Mokken analysis to determine if items in each set belonged to a unidimensional scale. ResultsResults indicated that both sets of items had strong reliability when used to rate individual physicians (Cronbach's α=.97 and .95). A one factor model was a good fit to both sets of items. Mokken analysis supported unidimensional scales. Both sets of items correlated with an overall measure of patient satisfaction with physicians and even more strongly with an item measuring emotional support. ConclusionsA patient-rated scale reliably measured hospital physicians’ compassion and correlated significantly with an item measuring overall patient satisfaction and a specific measure of satisfaction with emotional support. Measurement of compassionate healthcare should be included in research, educational assessment, and quality improvement programs. Practice implicationsClinicians should participate in efforts to enhance their ability to demonstrate compassionate care including eliciting and sharing information and acting collaboratively to ameliorate patients’ socioemotional concerns and needs.

Assessment of Reading Proficiency in Urban Education: The Case of Charter Schools in Michigan, United States

The purpose of this qualitative study was to examine the 7th grade reading performance among a random sample of urban public school academies (charter schools) in Wayne County, Michigan, compared with a random sample of traditional urban public schools within the same geographic area. This study was conducted using the fall 2012 Michigan Educational Assessment Program (MEAP) reading test scores, as reported by the Michigan Department of Education. A total of 11 public charter schools were selected and represented 15% of the total public charters in Wayne County that serviced middle school aged children. The 11 randomly selected Traditional Public Schools used for the comparison represented 10% of these traditional public schools that were located within a 5-mile radius of the public charter schools. Contingency tables were constructed (type of school vs. reading proficiency) and the data was analyzed using the nonparametric chisquare test for independence. According to the data, the traditional public schools had a higher percentage of grade 7 students who were proficient readers when compared with the public charter schools (30.7% vs. 18.6%, respectively). The difference, however, was not statistically significant and the null hypothesis was not rejected. These results suggested that the public charter schools were not outperforming traditional public schools in grade 7 reading proficiency.

On the potential balance among compulsory education outcomes through econometric and multiobjective programming analysis

This paper sheds new light on the relationship between inputs and outputs in the framework of the educational production function. In particular, it is geared at gaining a better understanding of which factors may be affected in order to achieve an optimal educational output level. With this objective in mind, we analyze teacher-based assessments (actual marks) in three different subjects using a multiobjective schema. For much of the analysis we use data from a recent (2010) Survey – ESOC10, linked with the results from an educational assessment program conducted among 11 and 15 year-old students and with the administrative records on teacher-based scores. Following the statistical and econometric analysis of these data, they are used to build a multiobjective mixed integer model. A reference point approach is used to determine the profile of, potentially, the most “successful and balanced” students in terms of educational outcomes. This kind of methodology in multiobjective programming allows generating “very balanced” solutions in terms of the objective values (subjects). Finally, a sensitivity analysis is used to determine policies that can be carried out in order to improve the performance levels of primary and secondary education students. Particularly, policy makers should be more concerned with the need to promote some cultural habits – such as reading –, from both the students’ and parents’ side. Additionally, policy efforts should be focused on making the vocational pathways available to Spanish youth more appealing, with the aim of taking advantage of the particular skills of students not succeeding in the academic track.