Abstract
Abstract: The evaluation of knowledge is an important step in the teachers’ work, because it is fundamental that the evaluation results be valid and reliable. Several faculty development programs have been undertaken to improve teachers’ skills on item writing. Few studies, however, have explored the benefits of providing feedback to multiple-choice item writers. Progress Testing is a longitudinal assessment of students’ knowledge gain. Their results are useful for generating feedback to students, teachers and medical school managers. The aim of this report is to present a successful experience of faculty development on high-quality item writing for Progress Testing through feedback about the fate of the items written by faculty members. Annually, faculties from medical schools participating in a consortium for the application of the Progress Test are asked to formulate new items to compose the test version. However, the fate of the items is not known by their authors, that is, they do not know if the items are incorporated into the test, if the items have good quality, what is the students’ performance in each of the items, and what is the psychometric function of the items. In 2017, one of the schools participating in the consortium offered its item authors a feedback regarding the flaws in item writing, modification of items by the review committee and students’ performance, as well as psychometric function in each item. The percentage of flawed item (items without enough quality to be selected for the test) was more than 30%. There was an inverse relation between the number of flawed item and the number of items included in the final exam, when analyzing the different content areas of the test (that is, Internal Medicine, Surgery, Pediatrics, Obstetrics and Gynecology, Public Health, and Basic Sciences). In the following year, we observed a decrease in the number of flawed item (less than 10%) and an increase in the number of eligible items to be selected for the Progress Testing. Therefore, giving feedback to item writers seems to be a good strategy for developing the faculty’s ability on writing good quality items.
Highlights
Progress testing has been recognized as an important feedback tool for students, faculties, program directors, and schools.[1]
Previous literature has focused on item development procedures and item construction guidelines,[10,11,12] and several studies have strongly demonstrated the efficacy of faculty development programs for developing better items and more reliable exams.13-15Furthermore, even though a large number of studies have suggested the need for faculty development programs with longitudinal monitoring of evaluative processes,[16,17,18] few papers have provided feedback for item writers.[19,20]
Many studies have evaluated the effect of faculty development programs on item writing, but most of these comprise lower-level Kirkpatrick model,[15] and there is evidence demonstrating that item writing flaws can harm the outcome of high-stake examinations.[26,27,28]
Summary
Progress testing has been recognized as an important feedback tool for students, faculties, program directors, and schools.[1]. Few studies have examined how feedback for Progress Test-item writers is generated, or even if it is offered. Previous literature has focused on item development procedures and item construction guidelines,[10,11,12] and several studies have strongly demonstrated the efficacy of faculty development programs for developing better items and more reliable exams.13-15Furthermore, even though a large number of studies have suggested the need for faculty development programs with longitudinal monitoring of evaluative processes,[16,17,18] few papers have provided feedback for item writers.[19,20] This concern is especially important because it has been clearly demonstrated that flawed items compromise the validity and reliability of a test.[21]
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
