What to expect from net reclassification improvement with three categories.

Abstract

The net reclassification improvement (NRI) has become a popular measure of incremental usefulness of markers added to risk prediction models. However, the expected magnitude of the three-category NRI is not well understood, leading researchers to rely on statistical significance. In this paper, we describe a slight modification to the original definition of the NRI, which weighs each reclassification by the number of categories by which a given individual is reclassified. This modification resolves some recent criticisms of the three-category NRI and at the same time has a minimal impact on its magnitude. Then we show that using this modified definition, the event and nonevent NRIs have simple interpretations as sums of changes in sensitivities and specificities calculated at the risk thresholds. We exploit this relationship to arrive at closed-form solutions for the NRI under normality within the event and nonevent subgroups. We observe that the size of the intermediate risk category and the event rate have limited impact on the magnitude of the NRI. As expected, the NRI increases with the strength of the added marker, and this relationship appears fairly proportional for markers with non-weak net effect size (above 0.25). Furthermore, we conclude that using the NRI as a metric, it is harder to improve models that already perform well.