VGC analysis: application of the ROC methodology to visual grading tasks

Abstract

To determine clinical image quality in radiography, visual grading of the reproduction of important anatomical landmarks is often used. The rating data from the observers in a visual grading study with multiple scale steps is ordinal, meaning that non-parametric rank-invariant statistical methods are required. However, many visual grading methods incorrectly use parametric statistical methods. This work describes how the methodology developed in receiver operating characteristics (ROC) analysis for characterising the difference in the observer's response to the signal and no-signal distributions can be applied to visual grading data for characterising the difference in perceived image quality between two systems. The method is termed visual grading characteristics (VGC) analysis. In a VGC study, the task of the observer is to rate her confidence about the fulfilment of image quality criteria. Using ROC software, the given ratings for the two systems are then used to determine the VGC curve, which describes the relationship between the proportions of fulfilled image criteria for the two compared systems for all possible decision thresholds. As a single measure of the difference in image quality between the two compared systems, the area under the VGC curve can be used.