Effect of training and strict guidelines on the reliability of risk adjustment systems in paediatric intensive care

Abstract

Many risk-adjustment systems have significant interobserver variability in everyday clinical practice. This can be partly corrected by strict guidelines and training. These issues have not been well studied in the paediatric setting. We assessed the reliability of two widely used paediatric scoring systems, the Paediatric Risk of Mortality (PRISM) and Paediatric Index of Mortality (PIM), before and after a special training program. Prospective observational multi-centred cohort study. Twenty-seven physicians from five paediatric intensive care units (PICUs) assessed severity of illness in 20 selected patients using PRISM and PIM scores before and after a special training program. Physicians were divided according to level of PICU experience: intensivists (>3 years experience, n=12), fellows (6-30 months experience, n=6) and residents (<6 months experience, n=9). Intraclass correlation was used to compare scoring reliability before and after training. Wide variability in PRISM and PIM scoring was observed before training (intraclass correlation for PRISM scores 0.24-0.73, intraclass correlation for PIM scores 0.16-0.33). Training and implementation of guidelines led to significant increases in interobserver agreement (intraclass correlation 0.74-0.86 for PRISM and 0.88-0.95 for PIM scores), although some variability remained. Our results show that the reliability of PRISM and PIM risk adjustment systems in daily clinical practice is much lower than expected. Training and guidelines can significantly increase interobserver agreement. These factors should be taken into account when using these systems for benchmarking, or to compare quality of care between different PICUs.