Detection Performance Of Human Observers Research Articles

Modeling human observer detection in undersampled magnetic resonance imaging reconstruction with total variation and wavelet sparsity regularization.

Task-based assessment of image quality in undersampled magnetic resonance imaging provides a way of evaluating the impact of regularization on task performance. In this work, we evaluated the effect of total variation (TV) and wavelet regularization on human detection of signals with a varying background and validated a model observer in predicting human performance. Human observer studies used two-alternative forced choice (2-AFC) trials with a small signal known exactly task but with varying backgrounds for fluid-attenuated inversion recovery images reconstructed from undersampled multi-coil data. We used a 3.48 undersampling factor with TV and a wavelet sparsity constraints. The sparse difference-of-Gaussians (S-DOG) observer with internal noise was used to model human observer detection. The internal noise for the S-DOG was chosen to match the average percent correct (PC) in 2-AFC studies for four observers using no regularization. That S-DOG model was used to predict the PC of human observers for a range of regularization parameters. We observed a trend that the human observer detection performance remained fairly constant for a broad range of values in the regularization parameter before decreasing at large values. A similar result was found for the normalized ensemble root mean squared error. Without changing the internal noise, the model observer tracked the performance of the human observers as the regularization was increased but overestimated the PC for large amounts of regularization for TV and wavelet sparsity, as well as the combination of both parameters. For the task we studied, the S-DOG observer was able to reasonably predict human performance with both TV and wavelet sparsity regularizers over a broad range of regularization parameters. We observed a trend that task performance remained fairly constant for a range of regularization parameters before decreasing for large amounts of regularization.

Modeling human observer detection in undersampled magnetic resonance imaging (MRI).

Task-based assessment of image quality in undersampled magnetic resonance imaging (MRI) using constraints is important because of the need to quantify the effect of the artifacts on task performance. Fluid-attenuated inversion recovery (FLAIR) images are used in detection of small metastases in the brain. In this work we carry out two-alternative forced choice (2-AFC) studies with a small signal known exactly (SKE) but with varying background for reconstructed FLAIR images from undersampled multi-coil data. Using a 4x undersampling and a total variation (TV) constraint we found that the human observer detection performance remained fairly constant for a broad range of values in the regularization parameter before decreasing at large values. Using the TV constraint did not improve task performance. The non- prewhitening eye (NPWE) observer and sparse difference-of-Gaussians (S-DOG) observer with internal noise were used to model human observer detection. The parameters for the NPWE and the internal noise for the S-DOG were chosen to match the average percent correct (PC) in 2-AFC studies for three observers using no regularization. The NPWE model observer tracked the performance of the human observers as the regularization was increased but slightly over-estimated the PC for large amounts of regularization. The S-DOG model observer with internal noise tracked human performace for all levels of regularization studied. To our knowledge this is the first time that model observers have been used to track human observer detection for undersampled MRI.

On the correlation between second order texture features and human observer detection performance in digital images

Image texture, the relative spatial arrangement of intensity values in an image, encodes valuable information about the scene. As it stands, much of this potential information remains untapped. Understanding how to decipher textural details would afford another method of extracting knowledge of the physical world from images. In this work, we attempt to bridge the gap in research between quantitative texture analysis and the visual perception of textures. The impact of changes in image texture on human observer’s ability to perform signal detection and localization tasks in complex digital images is not understood. We examine this critical question by studying task-based human observer performance in detecting and localizing signals in tomographic breast images. We have also investigated how these changes impact the formation of second-order image texture. We used digital breast tomosynthesis (DBT) an FDA approved tomographic X-ray breast imaging method as the modality of choice to show our preliminary results. Our human observer studies involve localization ROC (LROC) studies for low contrast mass detection in DBT. Simulated images are used as they offer the benefit of known ground truth. Our results prove that changes in system geometry or processing leads to changes in image texture magnitudes. We show that the variations in several well-known texture features estimated in digital images correlate with human observer detection–localization performance for signals embedded in them. This insight can allow efficient and practical techniques to identify the best imaging system design and algorithms or filtering tools by examining the changes in these texture features. This concept linking texture feature estimates and task based image quality assessment can be extended to several other imaging modalities and applications as well. It can also offer feedback in system and algorithm designs with a goal to improve perceptual benefits. Broader impact can be in wide array of areas including imaging system design, image processing, data science, machine learning, computer vision, perceptual and vision science. Our results also point to the caution that must be exercised in using these texture features as image-based radiomic features or as predictive markers for risk assessment as they are sensitive to system or image processing changes.

Objective assessment of task performance: a comparison of two FFDM detectors using an anthropomorphic breast phantom.

Current digital mammography systems primarily employ one of two types of detectors: indirect conversion, typically using a cesium-iodine scintillator integrated with an amorphous silicon photodiode matrix, or direct conversion, using a photoconductive layer of amorphous selenium (a-Se) combined with thin-film transistor array. The goal of this study was to evaluate a methodology for objectively assessing image quality to compare human observer task performance in detecting microcalcification clusters and extended mass-like lesions achieved with different detector types. The proposed assessment methodology uses a novel anthropomorphic breast phantom fabricated with ink-jet printing. In addition to human observer detection performance, standard linear metrics such as modulation transfer function, noise power spectrum, and detective quantum efficiency (DQE) were also measured to assess image quality. An Analogic Anrad AXS-2430 a-Se detector used in a commercial FFDM/DBT system and a Teledyne Dalsa Xineos-2329 with CMOS pixel readout were evaluated and compared. The DQE of each detector was similar over a range of exposures. Similar task performance in detecting microcalcifications and masses was observed between the two detectors over a range of clinically applicable dose levels, with some perplexing differences in the detection of microcalcifications at the lowest dose measurement. The evaluation approach presented seems promising as a new technique for objective assessment of breast imaging technology.

Channelized Hotelling observer correlation with human observers for low-contrast detection in liver CT images.

Task-based image quality procedures in CT that substitute a human observer with a model observer usually use single-slice images with uniform backgrounds from homogeneous phantoms. However, anatomical structures and inhomogeneities in organs generate noise that can affect the detection performance of human observers. The purpose of this work was to assess the impact of background type, uniform or liver, and the viewing modality, single- or multislice, on the detection performance of human and model observers. We collected abdominal CT scans from patients and homogeneous phantom scans in which we digitally inserted low-contrast signals that mimicked a liver lesion. We ran a rating experiment with the two background conditions with three signal sizes and three human observers presenting images in two reading modalities: single- and multislice. In addition, channelized Hotelling observers (CHO) for single- and multislice detection were implemented and evaluated according to their degree of correlation with the human observer performance. For human observers, there was a small but significant improvement in performance with multislice compared to the single-slice viewing mode. Our data did not reveal a significant difference between uniform and anatomical backgrounds. Model observers demonstrated a good correlation with human observers for both viewing modalities. Human observers have very similar performances in both multi- and single-slice viewing mode. It is therefore preferable to use single-slice CHO as this model is computationally more tractable than multislice CHO. However, using images from a homogeneous phantom can result in overestimating image quality as CHO performance tends to be higher in uniform than anatomical backgrounds, while human observers have similar detection performances.

Investigation on slice direction dependent detectability of volumetric cone beam CT images.

We investigate the detection performance of transverse and longitudinal planes for various signal sizes (i.e., 1 mm to 8 mm diameter spheres) in cone beam computed tomography (CBCT) images. CBCT images are generated by computer simulation and images are reconstructed using an FDK algorithm. For each slice direction and signal size, a human observer study is conducted with a signal-known-exactly/background-known-exactly (SKE/BKE) binary detection task. The detection performance of human observers is compared with that of a channelized Hotelling observer (CHO). The detection performance of an ideal linear observer is also calculated using a CHO with Laguerre-Gauss (LG) channels. The detectability of high contrast small signals (i.e., up to 4-mm-diameter spheres) is higher in the longitudinal plane than the transverse plane. It is also shown that CHO performance correlates well with human observer performance in both transverse and longitudinal plane images.

Task-based evaluation of a 4D MAP-RBI-EM image reconstruction method for gated myocardial perfusion SPECT using a human observer study

We evaluated the performance of a new 4D image reconstruction method for improved 4D gated myocardial perfusion (MP) SPECT using a task-based human observer study. We used a realistic 4D NURBS-based Cardiac-Torso (NCAT) phantom that models cardiac beating motion. Half of the population was normal; the other half had a regional hypokinetic wall motion abnormality. Noise-free and noisy projection data with 16 gates/cardiac cycle were generated using an analytical projector that included the effects of attenuation, collimator-detector response, and scatter (ADS), and were reconstructed using the 3D FBP without and 3D OS-EM with ADS corrections followed by different cut-off frequencies of a 4D linear post-filter. A 4D iterative maximum a posteriori rescaled-block (MAP-RBI)-EM image reconstruction method with ADS corrections was also used to reconstruct the projection data using various values of the weighting factor for its prior. The trade-offs between bias and noise were represented by the normalized mean squared error (NMSE) and averaged normalized standard deviation (NSDav), respectively. They were used to select reasonable ranges of the reconstructed images for use in a human observer study. The observers were trained with the simulated cine images and were instructed to rate their confidence on the absence or presence of a motion defect on a continuous scale. We then applied receiver operating characteristic (ROC) analysis and used the area under the ROC curve (AUC) index. The results showed that significant differences in detection performance among the different NMSE-NSDav combinations were found and the optimal trade-off from optimized reconstruction parameters corresponded to a maximum AUC value. The 4D MAP-RBI-EM with ADS correction, which had the best trade-off among the tested reconstruction methods, also had the highest AUC value, resulting in significantly better human observer detection performance when detecting regional myocardial wall motion abnormality. We concluded that the NMSE-NSDav trade-off was shown to agree with observer performance for the detection task of the regional motion abnormality, and the optimized 4D MAP-RBI-EM method with ADS corrections provides significant improvement compared to 3D FBP and 3D OS-EM with ADS corrections in detecting regional myocardial wall motion abnormali in 4D gated MP SPECT.

Simulated lesion, human observer performance comparison between thin-section dedicated breast CT images versus computed thick-section simulated projection images of the breast

The objective of this study was to compare the lesion detection performance of human observers between thin-section computed tomography images of the breast, with thick-section (>40 mm) simulated projection images of the breast.Three radiologists and six physicists each executed a two alterative force choice (2AFC) study involving simulated spherical lesions placed mathematically into breast images produced on a prototype dedicated breast CT scanner. The breast image data sets from 88 patients were used to create 352 pairs of image data. Spherical lesions with diameters of 1, 2, 3, 5, and 11 mm were simulated and adaptively positioned into 3D breast CT image data sets; the native thin section (0.33 mm) images were averaged to produce images with different slice thicknesses; average section thicknesses of 0.33, 0.71, 1.5 and 2.9 mm were representative of breast CT; the average 43 mm slice thickness served to simulate simulated projection images of the breast.The percent correct of the human observer’s responses were evaluated in the 2AFC experiments. Radiologists lesion detection performance was significantly (p < 0.05) better in the case of thin-section images, compared to thick section images similar to mammography, for all but the 1 mm lesion diameter lesions. For example, the average of three radiologist’s performance for 3 mm diameter lesions was 92% correct for thin section breast CT images while it was 67% for the simulated projection images. A gradual reduction in observer performance was observed as the section thickness increased beyond about 1 mm. While a performance difference based on breast density was seen in both breast CT and the projection image results, the average radiologist performance using breast CT images in dense breasts outperformed the performance using simulated projection images in fatty breasts for all lesion diameters except 11 mm. The average radiologist performance outperformed that of the average physicist observer, however trends in performance were similar.Human observers demonstrate significantly better mass-lesion detection performance on thin-section CT images of the breast, compared to thick-section simulated projection images of the breast.

73.5L: Late‐News Paper: Observer Strategies for Assessment of Temporal Response — Effect of Technology

AbstractWe compare the detection performance of human observers in LCDs and CRTs using artificial complex backgrounds. Our results suggest that the degradation in performance at high browsing speeds observed in LCDs is not seen or much more subtle in CRTs, even at 50 frames per second.

Volumetric model and human observer comparisons of tumor detection for whole-body positron emission tomography 1

Rationale and objectives The performances of model and human observers were compared for detecting and localizing tumors in whole-body positron emission tomography imaging. Volumetric implementations of model observers were used instead of standard planar implementations to reproduce the common practice of using volumetric image displays to assess whole-body positron emission tomography images. Materials and methods Observer studies with simulated data were used to compare three different acquisition protocols for an average patient size. Multiple realizations of simulated whole-body data with multiple added 1-cm diameter spherical lesions (targets) per image volume for efficiency were used. The location and contrast ratio of the targets were chosen randomly within ranges determined by initial calibration studies. Human observer studies were performed using a volumetric image display routinely used in clinical practice, and human observer detection performance was quantified using an alternate free-response operating characteristic analysis. The human detection performances were compared with the performances of volumetric implementations of the non-prewhitening matched filter and the channelized hotelling observer and also to the target contrast measured in the reconstructed image. Results Human observer detectability was generally well described as a linear function of these three figures of merit for the detection task considered. The best correlations ( r = 0.96, ρ = 0.98) were achieved with the channelized hotelling observer and non-prewhitening matched filter model observers. Conclusion The use of volumetric model observers provides a means for quantitative comparisons of different protocols and also provides a useful tool for the optimization of key parameters in whole-body positron emission tomography imaging.

Optimization of iterative reconstructions of /sup 99m/Tc cardiac SPECT studies using numerical observers

In this paper, we investigate the use of a numerical observer to optimize ordered-subset expectation maximization (OSEM) reconstructions for the detection of coronary artery disease (CAD). The parameters optimized were the iteration number and the full-width at half-maximum of three-dimensional Gaussian postfiltering. The numerical observer employed in the optimization was the channelized Hotelling observer (CHO). The CHO had been used previously to rank tumor detection accuracy for different reconstruction strategies in Ga-67 images, showing good agreement with the rankings of human observers. The intent of this paper was to determine if this CHO could also be employed for the detection of CAD. Results indicate that when grayscale (quantized) images are used, the CHO optimization results correlate well with human observers. On the other hand, when the CHO was used with floating-point images, it provided very good detection performance even when the images were excessively filtered. This result was at odds with the human-observer performance which showed a decrease in detection accuracy with highly smoothed images. This reflects the need to better model the detection task of the human observers who usually view and rank grayscale images and by appropriately modeling the image noise that quantization introduces, we show that the CHO can better match human-observer detection performance.

視覚評価における検出能の観察距離依存性

The detection performance of human observers has been used widely in the evaluation of imaging systems. In observer performance experiments, the detectability is affected by various factors such as object size, viewing distance, illumination, selected observer, order of reading samples and so forth. For the viewing distance, there are two situations where firstly observers are allowed to choose a suitable viewing distance and secondly they are not allowed to do so and must use a fixed viewing distance. The purpose of this paper is to assess whether the detectability of observer performance experiments is dependent on the viewing distance. The contrast-detail (C-D) diagram described the relationship between the threshold contrast in visually detectable by a human observers and viewing distance (20-200cm) using four different radiographic systems. Our results indicated that detectability by detection performance increased when observers adopted a short viewing distance for small objects and long viewing distance for large objects. It is possible that the selected viewing distance changes the evaluation of screen/film systems. Therefore, it is better that the designer of the experiment does not institute the fixed viewing distance.

X線画像の信号検出と視覚特性の重要性

Observer performance experiments using ROC analysis indicated that the detectability of subtle microcalcifications in digitized mammograms was improved by unsharp-mask filtering. However, the processed mammograms still provided lower accuracy than the original mammogram. Similar studies with mild interstitial infiltrates and subtle pneumothoraces in chest radiographs showed that the use of a pixel size substantially larger than 0.1 mm may result in some loss of diagnostic accuracy, and that a 2048×2048 matrix may be a satisfactory compromise for digital chest radiography. The unsharp masking may improve diagnostic accuracy for pneumothorax, while radiologists' performance in identifying interstitial infiltrates was degraded by the image processing. The contrast-detail (C-D) diagram was measured by using simulated square patterns superimposed on radiographic noise. Experimental variations in conventional C-D curves due to observer differences, within-reader variability, and image noise outcomes were quite large, and therefore it was necessary to increase the total number of observations in order to obtain reliable results. The relationship between physical and visual image quality for the task of detecting plastic beads was studied on thirty-seven different radiographic imaging conditions using a 2AFC visual detection experiment. Results indicated that human observer detection performance most closely resembled that of a sub-optimal statistical decision process. The effects of imaging and display conditions on the detectability of low-contrast objects in DSA images were investigated by using 18AFC experiments. We found that the SNR obtained from the perceived statistical decision theory model, which includes the observer's internal noise, can accurately predict the detectability of low-contrast objects in DSA images. The effect of pixel size on the SNR and threshold detection of low-contrast radiologic pattern was investigated theoretically for digital radiographic systems. No substantial difference between 0.1 and 0.2 mm pixel sizes was found in threshold contrasts for square objects. No substantial difference between 0.1 and 0.2 mm pixel sizes was found in threshold contrasts for square objects ranging from 0.1 to 20 mm, and thus we conclude that 0.2 mm pixel would be sufficiently small for visualization of most image detail in many radiologic examinations. The effects of image processing with unsharp masking, Metz filters, matched filters and optimal statistical filter on the detection of simulated square objects superimposed on radiographic mottle were studied theoretically and experimentally. The usefulness and limitations of these filters were discussed.

A comparison of physical image quality indices and observer performance in the radiographic detection of nylon beads

Although various models have been proposed in an attempt to predict the usefulness of a radiographic image in terms of its physical characteristics, no previous work has shown whether a single physical image quality index, such as a signal-to-noise ratio, can reliably predict the performance of a human observer over a broad range of image characteristics. The authors studied the relationship between physical and visual image quality for the task of detecting nylon beads in radiographs. Thirty-seven imaging cases with different combinations of physical image characteristics were considered; these included variations in object size and magnification, X-ray beam quality, screen-film system, screen-film contact, film density and illumination, and viewing distance. For each imaging case, visual image quality was quantified in terms of observer performance in a 2AFC visual detection experiment. Physical image quality indices were calculated according to eight different models of the detection process; these indices combined data regarding object size and attenuation, screen-film system MTF, film gradient, noise Wiener spectrum, and visual system response. The results of this work indicate that, for the conditions studied, human observer detection performance most closely resembles that of a suboptimal statistical decision process.