An instrument for quality assurance in work capacity evaluation: development, evaluation, and inter-rater reliability

André Strahl,Jörg Gehrke,Heiner Vogel,Christian Gerlich,Annette Müller-Garnn,Georg W Alpers

doi:10.1186/s12913-019-4387-4

Abstract

BackgroundEmployees insured in pension insurance, who are incapable of working due to ill health, are entitled to a disability pension. To assess whether an individual meets the medical requirements to be considered as disabled, a work capacity evaluation is conducted. However, there are no official guidelines on how to perform an external quality assurance for this evaluation process. Furthermore, the quality of medical reports in the field of insurance medicine can vary substantially, and systematic evaluations are scarce. Reliability studies using peer review have repeatedly shown insufficient ability to distinguish between high, moderate and low quality. Considering literature recommendations, we developed an instrument to examine the quality of medical experts’ reports.MethodsThe peer review manual developed contains six quality domains (formal structure, clarity, transparency, completeness, medical-scientific principles, and efficiency) comprising 22 items. In addition, a superordinate criterion (survey confirmability) rank the overall quality and usefulness of a report. This criterion evaluates problems of inner logic and reasoning. Development of the manual was assisted by experienced physicians in a pre-test. We examined the observable variance in peer judgements and reliability as the most important outcome criteria. To evaluate inter-rater reliability, 20 anonymous experts’ reports detailing the work capacity evaluation were reviewed by 19 trained raters (peers). Percentage agreement and Kendall’s W, a reliability measure of concordance between two or more peers, were calculated. A total of 325 reviews were conducted.ResultsAgreement of peer judgements with respect to the superordinate criterion ranged from 29.2 to 87.5%. Kendall’s W for the quality domain items varied greatly, ranging from 0.09 to 0.88. With respect to the superordinate criterion, Kendall’s W was 0.39, which indicates fair agreement. The results of the percentage agreement revealed systemic peer preferences for certain deficit scale categories.ConclusionThe superordinate criterion was not sufficiently reliable. However, in comparison to other reliability studies, this criterion showed an equivalent reliability value. This report aims to encourage further efforts to improve evaluation instruments. To reduce disagreement between peer judgments, we propose the revision of the peer review instrument and the development and implementation of a standardized rater training to improve reliability.

Highlights

Employees insured in pension insurance, who are incapable of working due to ill health, are entitled to a disability pension
While it is clearly important to verify the accuracy of these evaluations, there are no official guidelines on how to perform an external quality assurance for this evaluation process
Assessing difficulties applying the manual, peers indicated in 10% of the 72 reviews to had issues applying the superordinate criterion

Summary

Introduction

Employees insured in pension insurance, who are incapable of working due to ill health, are entitled to a disability pension. To determine whether employees are eligible for a disability allowance following injury or illness, a work capacity evaluation is conducted. This generally involves a physical examination by a social-medical physician. Experts acting within the same context and with access to identical medical information may have different judgments due to systematic differences in how information is processed This observable phenomenon is attributable to variety of factors: different organisational principles for how to combine information in an appropriate way, different weights for specific pieces of information, and differences in understanding of the importance of information for the judgment. It is important to develop a tool that assesses the external quality assurance for work capacity evaluation

Objectives

Methods

Results

Discussion

Conclusion