Retrospective review of the drop in observer detection performance over time in lesion-enriched experimental studies.

Sian Taylor-Phillips,Markus C. Elze,Elizabeth A. Krupinski,Kathryn Dennick,Claudia Mello-Thoms,Alastair G. Gale,Aileen Clarke

doi:10.1007/s10278-014-9717-9

Abstract

The vigilance decrement describes a decrease in sensitivity or increase in specificity with time on task. It has been observed in a variety of repetitive visual tasks, but little is known about these patterns in radiologists. We investigated whether there is systematic variation in performance over the course of a radiology reading session. We re-analyzed data from six previous lesion-enriched radiology studies. Studies featured 8–22 participants assessing 27–100 cases (including mammograms, chest CT, chest x-ray, and bone x-ray) in a reading session. Changes in performance and speed as the reading session progressed were analyzed using mixed effects models. Time taken per case decreased 9–23 % as the reading session progressed (p < 0.005 for every study). There was a sensitivity decrease or specificity increase over the course of reading 100 chest x-rays (p = 0.005), 60 bone fracture x-rays (p = 0.03), and 100 chest CT scans (p < 0.0001). This effect was not found in the shorter mammography sessions with 27 or 50 cases. We found evidence supporting the hypothesis that behavior and performance may change over the course of reading an enriched test set. Further research is required to ascertain whether this effect is present in radiological practice.

Highlights

The pattern of radiologic performance over the course of a workday has previously been investigated [1, 2], but there is little published research on whether or not radiology performance varies over the course of a single reading session
As many radiologic screening exams are viewed in quick succession, and contain several regions of interest which are difficult to classify as signal or noise, the radiologic screening task may have both low signal salience and a high perceived event rate for the reader
We investigated whether participant behavior and performance vary with time while examining an enriched test set of radiology cases, including chest and bone x-rays, mammograms, and chest CT scans

Summary

Introduction

The pattern of radiologic performance over the course of a workday has previously been investigated [1, 2], but there is little published research on whether or not radiology performance varies over the course of a single reading session (circa 10–100 cases depending on modality/exam). A vigilance decrement is a decline in sensitivity to detect targets with time on task, and was first observed over a 30 min session in World War II radar operators [3]. This decline is steeper in highly demanding tasks such as those which have a high event rate, a high working memory load, and low signal salience [4]. As many radiologic screening exams are viewed in quick succession, and contain several regions of interest which are difficult to classify as signal or noise, the radiologic screening task may have both low signal salience and a high perceived event rate for the reader

Methods

Results

Conclusion