Impact of bone suppression imaging on the detection of lung nodules in chest radiographs: analysis of multiple reading sessions

Abstract

Observer studies are frequently performed to test new modalities. Correct study design is important to generate reliable results. Two most frequently used observer study designs are the sequential and the independent reading design. We investigated the effect of different observer study designs on reader performance results and statistical power. The study included multiple assessments of chest radiographs (CXR) with bone suppression images (BSI) for the detection of lung nodules. In a fully crossed study design 8 observers assessed first radiographs without and with BSI sequentially. Secondly they scored radiographs independently having BSI available from the beginning. Five months later, the same readers scored the same cases again in an independent reading session, completing the three scorings for CXRs with BSI. Observer performance was compared using multi reader multi case (MRMC) receiver operating characteristics (ROC). To estimate reader variance, Dorfman, Berbaum, Metz (DBM) variance component estimates were calculated. No significant difference between the sequential and the independent reading sessions could be found (p=0.51; p=0.61). Both reading designs showed increased performance with BSI, with a significant increase for the sequential and the independent reading session after five months (p=0.002; p=0.007). Total observer variance between sequential and independent reading design remained the same. A strong increase of uncorrelated components was found in the independent reading sessions, masking the ability to demonstrate differences in observer performance across modalities. In conclusion, results of the sequential and the independent study design did not show a significant difference. The independent study design had less power compared to the sequential study design due to a strong increase of uncorrelated variance components.