The use of item response theory models to evaluate scales designed to measure knowledge of, and attitudes toward, antibiotic use and resistance in Swedish dairy producers

Abstract

Scales are frequently used to quantify latent traits that can not be measured directly (eg knowledge of a subject, attitude toward an issue, severity of a clinical condition). Historically, little attention has been paid within veterinary medical sciences to evaluating how well these scales perform. In this study methods that can be used in this evaluation were reviewed and applied to scales quantifying two unmeasurable traits: knowledge of, and attitudes toward, antibiotic use and resistance in Swedish dairy farms. Items (questionnaire questions) that did not fit well with the assumed unmeasurable trait were identified and removed. Item response theory models were applied to the remaining items to determine how well they contributed to the scale using item characteristic curves and a variety of information functions. A graphical evaluation of the fit of the models to the data was then carried out. For the knowledge scale, the evaluation identified a number of items which did not fit well with the scale. It subsequently demonstrated that the scale did a good job of discriminating among producers with below average knowledge, but had very little ability to separate average producers from those with a high level of knowledge. Similarly, the attitude scale did well at discriminating among producers with an average or poor attitude toward reducing their own antibiotic use, but had little ability to distinguish among those with above average attitudes. Item response theory models are valuable for determining how well items contribute to predicting an individual's value of the latent variable and whether or not there are “gaps” in the scale which are not measured well. Guidelines for the use of these methods in the development or refinement of scales are presented. Their post-hoc use to verify the performance of scales in preparation for subsequent analyses is also discussed.