Board 300 - Research Abstract Validating a Behaviour Assessment Tool for Simulated Neonatal Environment (Submission #765)

Abstract

Introduction/Background The role of behavioural skills in the provision of safe and effective neonatal care is well recognized.1 Behaviour Assessment Tool (BAT) has been validated for use in simulated neonatal and paediatric environment to assess resuscitation skills.2 The aim of this study was to develop a modified version of the tool and validate it to assess trainee’s performance in stabilizing an acutely unwell newborn. Methods Ethical approval was gained to study the performance of paediatric trainees. The modified Delphi method was used to develop a weighted scoring tool. After two rounds of Delphi process, consensus was achieved with Kendall W Coefficient of 0.73 demonstrating good concordance. Video recording of the performance of two trainees was used to train four assessors in use of the tool. Performance of a further sixteen trainees was recorded and was assessed by four assessors. Assessors were blinded to trainee’s identity and their years of experience. Results The scoring tool showed good interrater reliability (ICC = 0.80, CI: 0.58 – 0.92). Results of factor analysis indicated that unexplained variance could be reduced to less than 10% using only three assessors and to approximately 7% using four assessors (Figure 1). Mean of scores achieved by junior trainees was 34.03 (±5.42)compared to senior trainees 38.84 (±1.71). Results of the Independent Samples Mann–Whitney U Test demonstrated a statistically significant difference (P = 0.015) in the distribution of scores achieved by junior and senior trainees. Conclusion The study demonstrates that reliable and valid measurements of behavioural skills can be obtained from simulated neonatal environments using this scoring tool. There was a statistically significant difference in the mean scores achieved by senior and junior trainees. However, both set of trainees scored high scores. It is possibly due the fact that trainees had a spread of experience from one to seven years therefore some of the tasks on the checklist were done equally well by both groups of trainees. It is recognized that human factors such as lack of clear leadership, teamwork and communication, rather than technical failures, represent the greatest threat to complex systems like healthcare.3 There is a need to address human factors training among healthcare professionals. In order to provide robust feedback and training in this area, it is important to have tools to assess behavioural skills in a valid and reliable manner. We feel that our tool can help to assess doctors in training and support the development of a robust training programme for the doctors of the future.