Inter-rater reliability among psychiatrists when assessing depression according to the Montgomery–Åsberg Depression Rating Scale

Abstract

Introduction Montgomery–Åsberg Depression Rating Scale (MADRS) is a validated tool for rating the depth of depression. The structured interview guide for the MADRS (SIGMA) is an interview guide that has been developed in order to increase the inter-rater reliability. Patients often meet more than one psychiatrists during their hospitalization for depression. A divergent rating of depression between psychiatrists could affect both the treatment and the outcome. This makes knowledge of the inter-rater reliability among psychiatrists important. Aim The primary aim of this study was to measure the inter-rater reliability between psychiatrists when rating depression using the MADRS. Methods Ten in-patients, who were diagnosed with depression, were filmed while being interviewed using the SIGMA. The patients were after that instructed to rate themselves using the self-rating version of the MADRS. Ten psychiatrists rated the pre-recorded interviews according to the MADRS. The inter-rater reliability was measured using intra-class correlation (ICC). Results The mean ICC for the total MADRS score was 0.952 (95% CI 0.891–0.986; p<.001) and Cronbach’s alpha 0.961. ICC values for each item ranged between 0.866 and 0.978 (p<.001). Cronbach’s alpha ranged between 0.905 and 0.984. The ICC values, when comparing the psychiatrists rating to the patients rating, ranged between 0.307 and 0.809 (p<.001). Conclusion All of the ICC values in the study, except when comparing the psychiatrists rating to the patients self-rating, were considered to be excellent. This study confirms the findings of reliability found in similar studies which involved fewer raters and not exclusively psychiatrists.