S171 INTRODUCTION: A secure, comprehensive computerized reporting system for resident evaluation has been designed and pilot-tested. It successfully addresses issues including security, ease of use, reporter bias quantification and removal [1], prompt statistical identification of trainees in need of special attention. The training program as a whole, and individual faculty are screened for gender and ethnic bias [2]. The statistical operations are all validated by multiple internal cross checks. METHODS: A secure world wide web (WWW) site has been created using the Linux operating system and UNIX relational database and web-serving applications. Multiple layers of security include name-password identification of users in addition to dual key encryption to NSA standards of all traffic on web connections to and from the system. All data is backed up two-hourly. Easy access to the system is by recent versions of popular web browsers, including Internet Explorer (Microsoft) and Navigator (Netscape), running on PC, Mac or Unix platforms. Data entry is by mouse indentification of the resident name (with picture), followed by pointing to visual analog scales for Knowledge, Judgement and Practical Skill. Written comment is required for extreme grades, but is otherwise optional. Without written comment, the entire process requires only 5 mouse clicks, and takes but seconds to complete. The pilot-test took less than 2 minutes to submit all 23 evaluations. Faculty are reminded of the requirements of the American Board of Anesthesiologists before submitting reports. Graphic and text format report generation is automated and shows individual trainees, peer comparisons, ethnic/gender comparisons, longitudinal trend studies of performance and identification of residents whose performance differs statistically significantly from their peers. Reporter bias is removed by an original statistical analysis of all reports given by all faculty [1]. The statistics and logic were ported from a desktop spreadsheet application developed with Lotus 1-2-3 (r4 and later, Lotus Development Corp). RESULTS: (Table 1)Table 1Deliberate bias towards gender and an ethnic group were inserted in one set of "faculty" reports, and was correctly detected [2]. All statistical crosschecks for arithmetic and logic performed without fault. Security was tested by an independent professional networked data systems security consultant, who continues to fail to breach the confidentiality or functionality of the system. DISCUSSION: Spreadsheet resident data analysis methods previously validated with over 3000 data points form the logical core of the new online system. It satisfies our institutional requirements for data collection and is easily modified for use with training programs at other institutions. Following successful validation of the system with dummy data, we now collect real contemporaneous data on a weekly basis. Faculty compliance is high, and user comment has been almost universally favorable to date. The project continues.
Read full abstract