Faculty Evaluation Research Articles

Redesigning a Course Evaluation Instrument: Experience, Practical Guidance, and Lessons Learned

Course evaluation instruments (CEIs) are widespread, influential components of faculty professional development and evaluation processes. Given their importance to continuous instructional improvement and their weight in promotion, tenure, and other evaluation processes, it is critical that business schools and higher education institutions more broadly ensure a well-designed instrument. While the literature covering course evaluation instruments is extensive, coverage of the CEI revision process is virtually nonexistent. This article seeks to fill that gap, providing a case study of a successful CEI change process at an AACSB accredited business school. We offer practical recommendations for a revision process, including a sample timeline, revised instrument, and lessons learned from experience.

749 Medical Student Milestones for Neurosurgery Sub-Interns: A Pilot Study of a Novel Competency-Based Evaluation Tool to Quantitatively Differentiate Residency Applicants

INTRODUCTION: Resident selection is guided by few objective, quantitative datapoints. Utility of a standardized letter of recommendation (SLOR) has been limited by “grade inflation”. The Milestones program in neurosurgery residency standardizes “competence” definitions, recognizes domains of competence beyond patient care and medical knowledge, and highlights deficient areas. A similar approach for neurosurgery sub-interns could provide objective performance data to aid applicant evaluation. METHODS: Medical student milestones were adapted from the residency milestones or newly created according to group consensus of relevant competencies. Competency levels were graded ordinally 1 (level of 3rd-year medical student) to 4 (2nd-year resident). Faculty and self-evaluations were completed for 35 sub-interns across eight programs. Evaluators were surveyed on its usefulness. Average and cumulative milestone scores (CMS) within and across programs were evaluated using descriptive statistics. Kendall’s W tested inter-rater agreement. CMS were compared against percentile assignments on LOR using analysis of variance with post-hoc testing. RESULTS: The average faculty rating overall was 3.20, similar to that expected of an intern. Students scored highest in coachability and feedback (3.49; 3.67) and lowest in bedside procedural aptitude (2.90; 2.85) in faculty and self-evaluations. Median CMS was 26.5 (IQR 21.75-29.75; range 14-32). CMS in four programs evaluating at least five students ranged at least 11 points. A program with three faculty raters demonstrated scoring agreement across five students (p = 0.024). CMS differed significantly between SLOR percentile assignments, despite 25% of students assigned to top 5%ile while CMS-driven assignment of percentiles significantly differentiated quartiles of students. Faculty and students strongly endorsed the milestones form. CONCLUSIONS: The medical student milestones form was well-received and considered useful in objectively differentiating students within and across programs.

Students Evaluation of Faculty: Linguistic and Axiological Perspective.

The article considers a speech genre borrowed from American and Western European educational tradition, which is an axiologically oriented text of a student’s comment on the teacher. The article continues the study of evaluative speech genres of educational discourse, considering the problem of speech genre borrowing from an axiological perspective. The aim of this study is a linguistic and axiological analysis of students’ public evaluative utterances commenting on their attitude to teaching. Speech borrowing is one of the least studied problems in linguistics, which determines the relevance of the present study along with the importance of the sphere of education itself for the culture of society as a whole. Based on the communicative and semiotic concept of speech genre, the author identifies and analyzes the value aspects of student evaluation of teaching quality, identifies general and national specific features of English- and Russian-language evaluation texts, identifies and describes factors of conflictogenic evaluative statements of this genre, distinguishing features of functioning of pragmatic category of categorization in Russian- and English-language evaluative texts. The article reveals the absence of clear speech-genre criteria in Russian students’ evaluative comments unlike in English-language practices, which can be explained by the fact of speech genre borrowing and the lack of relevant speech-genre competence in the Russian-language educational discourse. The article considers the linguistic and ecological aspect of this speech genre from moral and ethical and legal points of view. The article reveals the dependence of the linguistic form of conceptualization of negative evaluation on communicative and ethical competence, the formation of which depends on the conceptual certainty of the speech genre. The author concludes that the utilitarian and pragmatic borrowing of communicative patterns from a foreign culture can be a threat to the spiritual and moral state of society.

Integrated Care Education for General Psychiatry Residents in the US: a Review of the Literature.

The purpose of this review was to synthesize published literature describing integrated care education available to general psychiatry residents in the United States (US) in order to better understand curricular models and summarize curriculum barriers and facilitators. The authors searched electronic databases for articles describing integrated care education for general psychiatry residents. Minimum inclusion criteria were focus on an ambulatory integrated care curriculum, description of the study population and training program, publication in English, and program location in the US. Data extracted included trainee, faculty, or collaborator evaluations, educational model, level of care integration, and barriers or facilitators to implementation. The literature search identified 18 articles describing curricula at 26 residency programs for inclusion. Most programs offered clinical and didactic curricula to advanced trainees across a variety of care integration levels. Common barriers included fiscal vulnerability and difficulties identifying team members or clarifying team member roles. Common facilitators included institutional and interdepartmental support, dedicated space, and faculty supervision. No statistical analysis was able to be performed due to study heterogeneity. This review found a relatively small number of articles written about integrated care education for psychiatry residents. Resident evaluation suggests this training is valuable regardless of curriculum structure, training years, or level of care integration. Dedicated funding, staff, and space were crucial for successful curricula. This review highlights a need for more rigorous research characterizing and evaluating integrated care education.

Bibliometrics and Qualitative Assessment: a Pragmatist Approach

Abstract In this essay we explore whether and how we should use bibliometrics in hiring, promoting, and granting in the academy. We suggest a Deweyan-Hickmanian pragmatist approach to reflecting on the technology of bibliometrics as a resource for inherently qualitative judgements in these deliberations. We begin with a literature review of current work evaluating the role and use of bibliometrics in the academy, from advocating for them to questioning their construct validity and assessing their limitations and/or dangerous consequences. In the next section, we take a step back and provide a Deweyan philosophical analysis of the validity and practical use of qualitative assessment. In our conclusion we reflect on what pragmatism can teach about recent algorithmic tools used for faculty evaluation.

Development and psychometric testing of the Diagnostic Competency During Simulation-based (DCDS) learning tool.

Despite diagnostic errors impacting an estimated 12 million people yearly in the United States, educational strategies that foster diagnostic performance among nurse practitioner (NP) students remain elusive. One possible solution is to focus explicitly on competencies fundamental for diagnostic excellence. Currently, no educational tools were found that comprehensively address individual diagnostic reasoning competencies during simulated-based learning experiences. Our research team developed and explored psychometric properties of the "Diagnostic Competency During Simulation-based (DCDS) Learning Tool." Items and domains were developed based on existing frameworks. Content validity was determined by a convenience sample of eight experts. Inter-rater reliability was determined by four faculty rating eight simulation scenarios. Final individual competency domain scale content validity index (CVI) scores ranged between 0.9175 and 1.0; total scale CVI score was 0.98. The intra-class correlation coefficient (ICC) for the tool was 0.548 (p < 0.0001, 95 % confidence interval CI [0.482-0.612]). Results suggest that the DCDS Learning Tool is relevant to diagnostic reasoning competencies and may be implemented with moderate reliability across varied simulation scenarios and performance levels. The DCDS tool expands the landscape of diagnostic reasoning assessment by providing NP educators with granular, actionable, competency-specific assessment measures to foster improvement.

INTRODUCTION TO SECTION II - VOLUME 27, ISSUE 1

In addition to the special issue papers, this first issue of 2023 also includes ten articles from our regular submission process. Topic include access, inclusion, synchronous online learning, student satisfaction, student evaluation of online faculty, student perspectives on engagement, faculty adoption of online teaching and more.

Natural Language Processing of Learners' Evaluations of Attendings to Identify Professionalism Lapses.

Unprofessional faculty behaviors negatively impact the well-being of trainees yet are infrequently reported through established reporting systems. Manual review of narrative faculty evaluations provides an additional avenue for identifying unprofessional behavior but is time- and resource-intensive, and therefore of limited value for identifying and remediating faculty with professionalism concerns. Natural language processing (NLP) techniques may provide a mechanism for streamlining manual review processes to identify faculty professionalism lapses. In this retrospective cohort study of 15,432 narrative evaluations of medical faculty by medical trainees, we identified professionalism lapses using automated analysis of the text of faculty evaluations. We used multiple NLP approaches to develop and validate several classification models, which were evaluated primarily based on the positive predictive value (PPV) and secondarily by their calibration. A NLP-model using sentiment analysis (quantifying subjectivity of the text) in combination with key words (using the ensemble technique) had the best performance overall with a PPV of 49% (CI 38%-59%). These findings highlight how NLP can be used to screen narrative evaluations of faculty to identify unprofessional faculty behaviors. Incorporation of NLP into faculty review workflows enables a more focused manual review of comments, providing a supplemental mechanism to identify faculty professionalism lapses.

Feudalism, meritocracy and sexual harassment

Scientific literature has clarified that the contexts of more intense power relations favor sexual harassment and impunity. This article presents the results of a research that has analyzed a legislation aimed at combating sexual harassment that preceded six months earlier the selection and evaluation of faculty. The communicative research methodology followed the criteria of social impact and creation, which are currently required by the European Union research program. In addition to a documentary review, sixteen interviews were conducted, twelve with teachers who led the research that gave rise to these reforms and three with victims who managed to become survivors. The result shows an affirmation shared by all interviewees: the passing of legislation against sexual harassment weakened feudal relationships within universities and made members of the Spanish parliament aware of the need to overcome these relationships through meritocratic evaluation. This result can contribute to a twofold process: that feminisms support democratic and social reforms in universities and that those who elaborate scientific policies know that guaranteeing gender equality and overcoming sexual harassment is the way to promote scientific productivity.

A longitudinal examination: association between part-time faculty and student academic achievement at public two-year colleges

ABSTRACT A considerable number of studies have investigated the relationship between part-time faculty employment and student academic achievement, such as credential completion. However, most studies used cross-sectional data, making it difficult to properly assess the changes in this relationship. In this study, we drew longitudinal data from IPEDS, and linear and quadratic growth models to understand the dynamic relationship between part-time faculty and graduation rates at public two-year colleges. Rigorous analysis results based on growth models and imputed datasets suggested part-time faculty employment is negatively associated with student graduation rates at public community colleges. This research finding might have important implications for resource allocation relevant to faculty employment, institutional excellence, and academic achievement at community colleges.

Review: Radical Roots: Public History and a Tradition of Social Justice Activism, edited by Denise D. Meringolo

Review: <i>Radical Roots: Public History and a Tradition of Social Justice Activism</i>, edited by Denise D. Meringolo

Interprofessional telehealth simulations for pharmacy and nursing students: Development and evaluation of an online experience

IntroductionThe authors evaluated student achievement of interprofessional education (IPE) core competencies during two formats (one hybrid and one completely online) of an IPE simulation designed for pharmacy and nursing students. MethodsThis IPE simulation was designed to teach students to use distance technologies to collaborate on patient care. In 2019, pharmacy (n = 83) and nursing (n = 38) students attended the hybrid (in-person and online) IPE simulation (SIM 2019) with the use of a telepresence robot. In 2020, pharmacy (n = 78) and nursing (n = 48) students attended the simulations completely online (SIM 2020), without the use of a robot. Both sessions aimed to achieve IPE core competencies through interprofessional student collaboration sessions using telehealth distance technologies. Students completed quantitative and qualitative evaluation surveys for both simulations. During SIM 2020, faculty and students used an observation tool to directly assess student team collaboration skills. ResultsStatistically significant improvements in self-assessment of IPE core competency scores were found in both formats of the simulation sessions. There were no statistical differences in faculty ratings with student ratings of team skills using the direct observation of team collaboration. Qualitative results indicated that students found interprofessional collaboration to be the most important lesson learned from the activity. ConclusionsBoth formats for the simulation achieved core competency learning objectives. IPE is an essential experience for health care education and is achievable online.

Examining the effects of early patient care and biomedical science integration on predoctoral dental student competence and confidence.

Our study investigates early experiential learning as a method of curricular integration by allowing students to begin their clinical experience in the first year of the programme, as well as distributing biomedical classes throughout the predoctoral dental school curriculum. This study utilises a quasi-experimental design with two different groups, Standard Curriculum Group and Integrated Curriculum Group, n=87. Data were collected from 2017 to 2021. We found that, on average, it took 608 h less for the participants in an integrated curriculum group to reach clinical competence in comparison to peers who did not experience the same methods of integration in their programme. These data were collected through daily faculty evaluations of students' progression as well as participants' own self-assessment. Our results indicate that participants in the Integrated Curriculum Group also experienced a positive effect on their confidence in their ability to apply the biomedical sciences to patient care. Our findings demonstrate that predoctoral dental programmes may be able to bring about positive outcomes for students' clinical confidence and competence by providing patient care opportunities early in the programme and sequencing the biomedical sciences throughout the curriculum. As such, it appears that early experiential learning may be a viable option for curricular integration that can have a positive effect on both students' confidence in their clinical abilities and their progression to clinical competence.

Discipline-Specific Writing Support in Graduate Nursing.

Scholarly writing is an essential skill for a doctorally prepared nurse. Despite the importance of writing, there are limited data on how best to support students in the process of writing skill acquisition. This article describes the development, growth, and effects of a writing support program for graduate nursing students. A writing support program was initiated with a focus on providing discipline-specific support. Activities included oneon-one coaching, workshops, and collaboration with faculty to offer writing content within or alongside courses. Student and faculty evaluations show the benefits of the writing support program to enhance student confidence and skill in writing. Development of a writing support program has provided a valuable resource to graduate students and faculty in the school of nursing. Writing must be intentionally included in doctoral programs and supported with an array of practices including discipline-specific feedback. [J Nurs Educ. 2023;62(4):253-256.].

Oral examination of fourth-year medical students on surgery rotations allows faculty to assess Core Entrustable Professional Activities for entering Residency (CEPAR): Proof-of-concept and analysis of student submissions

BackgroundAs a community-based medical school which recruited faculty preceptors new to teaching, we sought to create objective assessments for fourth-year surgery experiences via administration of an oral exam. Students provided three authentic cases, which faculty used as a springboard to ascertain student proficiency in five entrustable professional activities: 1-oral presentation, 2-recognition of urgency/instability, 3-calling consults, 4-transitions of care, 5-informed consent. We present proof-of-concept and analysis of student case submissions. MethodsTwenty-seven student submissions (79 cases in total) were evaluated for case complexity, level-appropriateness, and an estimation of the ability to conduct a quality exam based on the information provided (subjective measures). Objective metrics included word count, instruction adherence, inclusion of figures/captions. A resident-in-training rated cases via the same metrics. In-examination data was separately culled. ResultsThe average word count was 281.70 (SD 140.23; range 40–743). Figures were included in 26.1% of cases. Faculty raters scored 29.0% as low-complexity, 37.7% medium-complexity, and 33.3% high-complexity. Raters felt 62.3% of cases provided enough information to conduct a quality exam. The majority of cases submitted (65.2%) were level-appropriate or higher. The resident rater scored cases more favorably than surgeons (Cohen's kappa of −0.5), suggesting low inter-rater agreement between those of differing experience levels. ConclusionStudent's case submissions lessened faculty burden and provided assessors with adequate information to deliver a quality exam to assess proficiency in clinical skills essential for residency. Cases demonstrated sufficient complexity and level-appropriateness. The request to correlate case rating with exam performance is under review by our institution's assessment office. Near-peer tutoring by resident alumni is a program under development.

Critique of Approaches for Evaluating Teaching and Proposal for a New Institutional Policy.

A review of current literature provides ample evidence that Student Evaluation of Teaching, as currently practiced, can be a biased and flawed evaluation of educator performance and student learning. Due to this, many institutions have developed their own internal multimodal approach for faculty evaluation and professional advancement review that removes the underlying bias associated with survey results. Ross University School of Veterinary Medicine created a taskforce to evaluate the current teaching assessment practices and provide an alternative method. Based on an extensive literature review, an institutional policy was created using a multimodal approach, combining student evaluations of teaching scores with peer review of teaching, review of teaching materials, mentorship, self-ratings, and self-reflection. The implementation plan for this policy is described to allow for a teaching evaluation process built on a philosophy of mentorship, guidance, and self-reflection; as well as the use of external professional exam metrics to review the institutional academic quality at large.

Lessons learned through the implementation of a rural longitudinal integrated clerkship at the University of Missouri-Columbia.

Created in 1995, The University of Missouri School of Medicine's Rural Track Pipeline Program was designed to address physician shortages in rural Missouri through medical student participation in a series of clinical and non-clinical programs over the course of their medical training to influence graduates to choose rural practice. To increase the likelihood of students choosing rural practice, a 46-week longitudinal integrated clerkship (LIC) was implemented at one of nine existing rural training sites. Over the course of the academic year, quantitative and qualitative data was collected to evaluate effectiveness of the curriculum and for quality improvement purposes. Data collection is in progress and includes student evaluation of the clerkship, faculty evaluations of students, student evaluations of faculty, student aggregate clerkship performance, and qualitative data from student and faculty debrief sessions. Based on data collected, changes are being made to the curriculum for the following academic year to enhance the student experience. The LIC will also be offered at an additional rural training site beginning in June of 2022, and then expanded to a third site in June of 2023. As each LIC is unique, our hope is our experience and lessons learned will help others in developing a LIC or improving an existing LIC.

Negotiating dance academia and parenting: a conversation between colleagues and friends

ABSTRACT This article aims to reveal the challenges of working mothers in dance academia in respect to balancing of teaching, service, administration, and research demands and parenting of young teens. Through a conversation, the authors aim to find solidarity in our similar situations in different geographic locations by exchanging ideas regrading practicalities and demands of dance academia in relationship to parentings. United by similar experiences, we aim to share our everyday accounts of working in dance academia, the physical challenges of ageing, efforts to stay relevant and question the evaluation and promotion process in order to create more empathic working conditions for parents in academia. Spurred by similar challenges of seeking work/family balance and specific demands of dance academia, the authors seek to analyze and discuss the issues surrounding motherhood and academia and advocate for more emphatic working conditions and promotion processes. This collaborative auto-ethnographic research aims to disturb the rigid structures of academia in relation to parenthood, professional development, and quality of life, which serves as a method to excavate cultural experience of parents, specifically mothers, working in dance academia and question dynamics of faculty evaluation, promotion, and professional progress.

National medical commission and faculty evaluation – Aura of confusion

National medical commission and faculty evaluation – Aura of confusion

Enabling possibility: Reform of faculty appointments and evaluation

Enabling possibility: Reform of faculty appointments and evaluation