Inter- and intra-rater agreement of the rehabilitation activities profile

Abstract

The objective of the study was to determine the inter- and intra-rater agreement of the Rehabilitation Activities Profile (RAP). The RAP is an assessment method that covers the domains of communication, mobility, personal care, occupation and relationships. Each domain consists of items which are further divided in sub-items for in-depth analysis. The RAP allows quantification of the severity of disabilities, handicaps and perceived problems of a patient with regard to the items and sub-items. For this purpose ordinal 4-point Likert scales were constructed. The RAP can be used for goal setting and evaluation of rehabilitation. Because of the broad intended use of the RAP and its construction, a special design for the reliability study was needed. The study was carried out in 5 rehabilitation facilities with the participation of various professions. The items and sub-items of the RAP were divided over these professions according to their expertise. Pairs of interviewers were formed that questioned a patient. For the determination of inter- and intra-rater agreement each pair of interviewers was allowed to question a patient only once. To establish the intra-rater agreement, video recordings were made during the interviews. The median (weighted) kappa value and percentage of agreement about the severity grading of a disability or handicap for all items and sub-items exceeded 0.84 and 81%, respectively, with regard to the inter- and intra-rater agreement. For the severity grading of perceived problems these values were 0.91 and 86%. The interpretation of kappa was hindered by two paradoxes recently described in the literature. The paradox “high agreement but low kappa” manifested itself in particular. It is concluded that inter- and intra-rater agreement of the RAP can be considered to be good to very good.