LROC Curve Research Articles

Evaluation of physician performance using a concurrent-read artificial intelligence system to support breast ultrasound interpretation

PurposeThe purpose of this study was to compare the diagnostic performance and the interpretation time of breast ultrasound examination between reading without and with the artificial intelligence (AI) system as a concurrent reading aid. Material and methodsA fully crossed multi-reader and multi-case (MRMC) reader study was conducted. Sixteen participating physicians were recruited and retrospectively interpreted 172 breast ultrasound cases in two reading scenarios, once without and once with the AI system (BU-CAD™, TaiHao Medical Inc.) assistance for concurrent reading. Interpretations of any given case set with and without the AI system were separated by at least 5 weeks. These reading results were compared to the reference standard and the area under the LROC curve (AUCLROC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for performance evaluations. The interpretation time was also compared between the unaided and aided scenarios. ResultsWith the help of the AI system, the readers had higher diagnostic performance with an increase in the average AUCLROC from 0.7582 to 0.8294 with statistically significant. The sensitivity, specificity, PPV, and NPV were also improved from 95.77%, 24.07%, 44.18%, and 93.50%–98.17%, 30.67%, 46.91%, and 96.10%, respectively. Of these, the improvement in specificity reached statistical significance. The average interpretation time was significantly reduced by approximately 40% when the readers were assisted by the AI system. ConclusionThe concurrent-read AI system improves the diagnostic performance in detecting and diagnosing breast lesions on breast ultrasound images. In addition, the interpretation time is effectively reduced for the interpreting physicians.

Receiver Operating Characteristic (ROC) Analysis of Image Search-and-Localize Tasks

Receiver operating characteristic (ROC) analysis for the common image search-and-localize task, in which readers search an image for lesion or lesions not knowing a priori any exists, has been studied for over four decades. However, a satisfactory solution seems elusive. We show that the ROC curve predictive of clinical outcomes where readers are penalized appropriately for not correctly localizing known lesions cannot be obtained because it is a missing data problem. Further, this ROC curve is between the case-based ROC curve where readers are not penalized and the lesion-based ROC curve where penalty applies. Moreover, the lesion-based ROC curve is the LROC curve proposed by Starr et al. We show maximum-likelihood (ML) estimation of the LROC curve, validation of this procedure with Monte Carlo simulations, and its application to reader ROC datasets. Monte Carlo simulations validated ML estimation of area under the LROC curve (AUC) and its variance. Example applications showed that ML estimate of LROC curve fits experimental datasets. The ROC curve predictive of clinical performance cannot be estimated from reader ROC data alone because it is a missing data problem, and is between the case-based ROC curve where readers are not penalized for not correctly identifying known lesions and the lesion-based ROC curve where penalty applies. The lesion-based ROC curve is the LROC curve proposed by Starr et al. and can be estimated via ML estimation.

Improving PET spatial resolution and detectability for prostate cancer imaging

Prostate cancer, one of the most common forms of cancer among men, can benefit from recent improvements in positron emission tomography (PET) technology. In particular, better spatial resolution, lower noise and higher detectability of small lesions could be greatly beneficial for early diagnosis and could provide a strong support for guiding biopsy and surgery. In this article, the impact of improved PET instrumentation with superior spatial resolution and high sensitivity are discussed, together with the latest development in PET technology: resolution recovery and time-of-flight reconstruction. Using simulated cancer lesions, inserted in clinical PET images obtained with conventional protocols, we show that visual identification of the lesions and detectability via numerical observers can already be improved using state of the art PET reconstruction methods. This was achieved using both resolution recovery and time-of-flight reconstruction, and a high resolution image with 2 mm pixel size. Channelized Hotelling numerical observers showed an increase in the area under the LROC curve from 0.52 to 0.58. In addition, a relationship between the simulated input activity and the area under the LROC curve showed that the minimum detectable activity was reduced by more than 23%.

Design Study of a Whole-Body PET Scanner With Improved Spatial and Timing Resolution

Current state-of-art whole-body PET scanners achieve a system spatial resolution of 4-5 mm with limited sensitivity. Since the reconstructed spatial resolution and image quality are limited by the count statistics, there has not been a significant push for developing higher resolution whole-body PET scanners. Our goal in this study is to investigate the impact of improved spatial resolution together with time-of-flight (TOF) capability on lesion uptake estimation and lesion detectability, two important tasks in whole-body oncologic studies. The broader goal of this project is the development of a new state-of-art TOF PET scanner operating within an MRI while pushing the technology in PET system design. We performed Monte Carlo simulations to test the effects of crystal size (4 mm and 2.6 mm wide crystals), TOF timing resolution (300ps and 600ps), and 2-level depth-of-interaction (DOI) capability. Spatial resolution was calculated by simulating point sources in air at multiple positions. Results show that smaller crystals produced improved resolution, while degradation of resolution due to parallax error could be reduced with a 2-level DOI detector. Lesion phantoms were simulated to measure the contrast recovery coefficient (CRC) and area under the LROC curve (ALROC) for 0.5 cm diameter lesions with 6:1 activity uptake relative to the background. Smaller crystals produce higher CRC, leading to increased ALROC values or a reduction in scan time. Improved timing resolution provides faster CRC convergence and once again leads to an increase in ALROC value or reduced scan time. Based on our choice of timing resolution and crystal size, improved timing resolution (300ps) with larger crystals (4 mm wide) has similar ALROC as smaller crystals (2.6 mm wide) with 600ps timing resolution. A 2-level DOI measurement provides some CRC and ALROC improvement for lesions further away from the center, leading to a more uniform performance within the imaging field-of-view (FOV). Given a choice between having either an improved spatial resolution, improved timing resolution, or DOI capability, improved spatial or timing resolution provide an overall higher ALROC relative to a 2-level DOI detector.

TU‐C‐103‐05: Image Quality and Dose Reduction Evaluation of a New CT Iterative Reconstruction Algorithm Using Model Observers

Purpose: To evaluate the impact on image quality of a new iterative reconstruction (IR) algorithm for CT systems and quantitate the amount of dose reduction by using mathematical model observers. Methods: Two phantoms with low‐contrast inserts (head mode and body mode) were used to evaluate task performance for both location known signal detection and location unknown signal detection tasks. The phantom consisted of low contrast cylindrical signals with contrasts ranging from −2 Hounsfield units (HU) to ‐ 10 HU and sizes ranging from 5mm to 7mm. The phantoms were scanned by multi row CT scanner using a range of tube currents 30&– 400mAs, and energy 120 kVp. The performance of a traditional filtered backprojection (FBP) algorithm and the new IR were evaluated using the channelized Hotelling observer. Seven channels designed to match human performance were selected to evaluate the image quality. Area under ROC curves and LROC curves were used to determine the image quality. Results: Evaluation of model observer performances indicates that a 3x dose reduction can be reached using the new IR algorithm in two modes (head and body) while maintaining image quality for location known and location unknown detection tasks. Conclusion: Overall, image quality as measured by task performance is greatly improved by using the IR compared to traditional FBP at equal mAs and kVp. Thus, this study indicates that a significant dose reduction in CT systems can be achieved by using this new IR algorithm. This new IR and the phantoms presented in this study will be used in future investigations of more complicated imaging tasks such as size and contrast estimation. GE Healthcare, Waukesha, WI

TH-C-L100F-09: Operating Characteristics Predicted by Models of Location Specific Diagnostic Tasks

Operating characteristics summarize observer performance in diagnostic tasks. Examples are the ROC, the LROC, the FROC, and the AFROC. Models of observer performance differ in their abilities to predict operating characteristics. For example, ROC models can predict the ROC curve, but cannot predict LROC, FROC or AFROC curves. Location specific models are, in historical order, the FROCFIT model, Swensson's LROC model and the search model (SM). In principle any of these models can predict all of the characteristic curves noted above. The purpose of this work was to compare the ability of these models to predict operating characteristics. CAD data consisting of mark‐rating pairs on 450 images were used to determine the various operating characteristics. For example, the highest rating on each image determined the ROC curve; the lesions rated higher than the highest noise determined the ordinate of the LROC, etc. All models agreed with the observed ROC data. Swensson's model agreed with the observed LROC curves but departed mildly from the observed AFROC curve and strongly from the observed FROC curve. The FROCFIT model departed mildly from the AFROC curve and strongly from the FROC curve. The search model provided good fits to all operating characteristics. In conclusion this work provides additional validation of the search model method of analyzing free‐response data.

Reducing Artifacts in Pelvic Bone SPECT: An Assessment of Lesion Detectability Using Numerical and Human Observers

In pelvic bone SPECT using Tc-99m labeled compounds, physical effects such as nonhomogeneous attenuation and the accumulation of activity into the bladder during the data acquisition process can often result in data inconsistencies. With filtered backprojection (FBP) reconstruction, this may result in streak artifacts. Various methods for attenuation compensation can be employed to account for some of these streaks, but if the rate of uptake of tracer in to the bladder is sufficient, the resultant streaks may be significant enough to impair lesion detection. We have investigated various reconstruction methods in an effort to reduce these artifacts. In order to assess the impact of inconsistent bladder activities on the detection of pelvic lesions, SPECT imaging was simulated using the Zubal voxelized phantom, for cases of both a static bladder activity and a changing activity distribution. Reconstructions were performed using FBP, ordered subset-expectation maximization (OSEM) and dynamic expectation maximization (dEM) and was assessed for lesion detectability using a channelized, nonprewhitening (CNPW) numerical observer model. This observer model was used to optimize reconstruction strategies for a human LROC observer study. The human LROC observer study was performed in order to assess the various reconstruction methods in terms of lesion detectability. Three human observers were used in this test. The results of this test indicate that FBP performs significantly worse than static OSEM iterative reconstruction with attenuation correction when assessed using the area under the LROC curve (maximal ALROC=0.47 for FBP versus 0.71 for OSEM). Visually, the dEM algorithm produces images with slightly reduced streak artifacts compared to OSEM, but this improvement was not reflected in significantly improved ALROC values. In fact, when assessed using human LROC methodology, detectability actually decreased slightly when using dEM (maximal ALROC=0.71 for OSEM versus 0.66 for dEM), although this reduction was not determined to be statistically significant. It is possible that the slightly reduced performance of the dEM algorithm may be due, in part, to not performing an optimization in the number of reconstruction iterations as was performed for the OSEM method

Single-slice versus multi-slice display for human-observer lesion-detection studies

Use of a single-slice (2D) display for observer studies may bias results acid reduce the studies' clinical generalizability. Human observers perform better at the task of lesion detection with 3D-processed images than they do with 2D-processed images when the images are presented using a 2D display. However, 3D-processing techniques incorporate information from out-of-plane or adjacent slices into an image and thus provide more information to the observer than does a similar 2D-processing method. Observer performance with 2D-processing methods may improve if the adjacent-slice information is provided by way of a multi-slice (3D) display. 3D processing also introduces 3D distracters which may not be present with 2D processing. The authors investigated, with a human-observer LROC study, the impact of 2D versus 3D display on FBP and OSEM reconstruction followed by 2D and 3D filtering. Three display modes were used: single-slice, multi-slice, and multi-slice with cine. The emulated clinical task was the detection and localization of small gallium lesions in thoracic SPECT scans. Results indicate that 3D display generally improves performance over the 2D display, as measured by the area under the LROC curve and the probability of correct localization. The improvement is greater for 2D-than for 3D-filtered reconstruction, leading to a reduction in the significance of the differences between them.

The Intercomparison of a Black and White and a Color Display: An Example of the Use of Receiver Operating Characteristic Curves

A black and white display was compared to a color display using the heated object spectrum with the aim of verifying which display was better for the detection of small abnormalities in a reasonably complicated background. Receiver operating characteristic curves with location (LROC) were generated for each type of display for each observer. The data were analyzed by taking the LROC curves in pairs, for each observer, fitting I binary ROC curve using a model, and using both parametric and nonparametric statistical tests for the paired differences. It was found, in this study, that the black and white display was significantly "better" than the color display. There was no significant difference in the time required to make a decision between the two displays. The technique of using paired ROC curves is suggested as being an appropriate and powerful test for intercomparing different imaging procedures and techniques in medicine.

Detection and display of defects in medical scintigraphic images

To achieve the evaluation of the lesion detectability of a program implemented by us on a minicomputer system to indicate possible defects on liver scans, we simulated a set of geometrical phantoms with different patterns of cold lesions to obtain LROC curves in different cases. Then, to check its clinical value, the program was applied to 44 liver scintigrams of patients among which 7 occurred to be pathological. These liver scintigrams had been evaluated by 6 clinicians so that a comparison between the program and the clinicians could be performedd on the basis of the corresponding LROC curves. Moreover, once an abnormality had been detected by the program, the abnormality was processed in order to perform a psychovisual evaluation of the image more easily: an adapted local 2-level display of the possible defect was superimposed on the scan to enhance it and to facilitate diagnosis.

An intercomparison of computer assisted data processing and display methods in radioisotope scintigraphy using mathematical tumours

Several computer assisted processing and display methods are evaluated using a series of 100 normal brain scintigrams, 50 of which have had single 'mathematical tumours' superimposed. Using a standard rating system, or in some cases quantitative estimation, LROC curves are generated for each method and compared.