Inter-rater Agreement in the Assessment of Solvent Exposure at a Car Assembly Plant

Abstract

This study investigated the feasibility and the reproducibility of a retrospective assessment of solvent exposure in the painting operations of a car assembly plant. Five industrial hygienists (the raters) reviewed summary documents on plant operations and solvent exposure levels, developed from plant records and interviews. The raters independently reviewed 695 department-job-year combinations compiled from 29 work histories and used semi-quantitative scores to rate solvent exposure. Inter-rater agreement was evaluated by computing the percent concordance, index and the intra-class correlation coefficient (r1). There was discordance among raters, a large proportion of which pertained to low exposure levels (1–5 ppm). The computation of the r1, is sensitive to the distribution of exposure levels. A low r1 can be observed for a rare exposure even if the percent concordance is high. Inter-rater agreement was good for cumulative exposure scores computed for frequent exposures: coatings (r1=0.67), aromatic hydrocarbons (r1=0.66), and petroleum distillates (r1=0.62).