Reliability of the Advanced REACH Tool (ART)

Abstract

The aim of this study was to assess the reliability of the Advanced REACH Tool (ART) by (i) studying interassessor agreement of the resulting exposure estimates generated by the ART mechanistic model, (ii) studying interassessor agreement per model parameters of the ART mechanistic model, (iii) investigating assessor characteristics resulting in reliable estimates, and (iv) estimating the effect of training on assessor agreement. Prior to the 1-day workshop, participants had to assess four scenarios with the ART. During two 1-day workshops, 54 participants received 3-h training in applying the mechanistic model and the technical aspects of the web tool. Afterward, the participants assessed another four scenarios. The assessments of the participants were compared with gold standard estimates compiled by the workshop instructors. Intraclass correlation coefficients (ICCs) were calculated and per model parameter and the percentage agreement and Cohen kappa statistics were estimated. The ICCs showed good agreement before and almost perfect agreement after training. However, substantial variability was observed between individual assessors' estimates for an individual scenario. After training, only 42% of the assessments lay within a factor of three of the gold standard estimate. The reliability appeared to be influenced by several factors: (i) information provided by text and video hampered the assessors gaining additional information required to make the assessments, (ii) for some parameters, the guidance documentation implemented in the tool may have been insufficient, and (iii) in some cases, the assessors were not able to implement the information explicitly provided. The ART is an expert tool and extensive training is recommended prior to use. Improvements of the guidance documentation, consensus procedures, and improving the training methods could improve the reliability of ART. Nevertheless, considerable variability can be expected between assessors using ART to estimate exposure levels for a given scenario.