Medical Image Perception Research Articles

Association between Proficiency and Idiosyncratic Biases in Medical Image Perception

Association between Proficiency and Idiosyncratic Biases in Medical Image Perception

Special Issue Editorial: Medical Image Perception and Observer Performance.

The editorial introduces the JMI Special Section on Medical Image Perception and Observer Performance.

Serial dependence in perception across naturalistic generative adversarial network-generated mammogram.

Human perception and decisions are biased toward previously seen stimuli. This phenomenon is known as serial dependence and has been extensively studied for the last decade. Recent evidence suggests that clinicians' judgments of mammograms might also be impacted by serial dependence. However, the stimuli used in previous psychophysical experiments on this question, consisting of artificial geometric shapes and healthy tissue backgrounds, were unrealistic. We utilized realistic and controlled generative adversarial network (GAN)-generated radiographs to mimic images that clinicians typically encounter. Mammograms from the digital database for screening mammography (DDSM) were utilized to train a GAN. This pretrained GAN was then adopted to generate a large set of authentic-looking simulated mammograms: 20 circular morph continuums, each with 147 images, for a total of 2940 images. Using these stimuli in a standard serial dependence experiment, participants viewed a random GAN-generated mammogram on each trial and subsequently matched the GAN-generated mammogram encountered using a continuous report. The characteristics of serial dependence from each continuum were analyzed. We found that serial dependence affected the perception of all naturalistic GAN-generated mammogram morph continuums. In all cases, the perceptual judgments of GAN-generated mammograms were biased toward previously encountered GAN-generated mammograms. On average, perceptual decisions had 7% categorization errors that were pulled in the direction of serial dependence. Serial dependence was found even in the perception of naturalistic GAN-generated mammograms created by a GAN. This supports the idea that serial dependence could, in principle, contribute to decision errors in medical image perception tasks.

Visual hindsight bias for abnormal mammograms in radiologists.

Hindsight bias-where people falsely believe they can accurately predict something once they know about it-is a pervasive decision-making phenomenon, including in the interpretation of radiological images. Evidence suggests it is not only a decision-making phenomenon but also a visual perception one, where prior information about an image enhances our visual perception of the contents of that image. The current experiment investigates to what extent expert radiologists perceive mammograms with visual abnormalities differently when they know what the abnormality is (a visual hindsight bias), above and beyond being biased at a decision level. experienced mammography readers were presented with a series of unilateral abnormal mammograms. After each case, they were asked to rate their confidence on a 6-point scale that ranged from confident mass to confident calcification. We used the random image structure evolution method, where the images repeated in an unpredictable order and with varied noise, to ensure any biases were visual, not cognitive. Radiologists who first saw an original image with no noise were more accurate in the max noise level condition [area under the curve ] than those who first saw the degraded images (; difference: ), suggesting that radiologists' visual perception of medical images is enhanced by prior visual experience with the abnormality. Overall, these results provide evidence that expert radiologists experience not only decision level but also visual hindsight bias, and have potential implications for negligence lawsuits.

Serial Dependence in Dermatological Judgments.

Serial Dependence is a ubiquitous visual phenomenon in which sequentially viewed images appear more similar than they actually are, thus facilitating an efficient and stable perceptual experience in human observers. Although serial dependence is adaptive and beneficial in the naturally autocorrelated visual world, a smoothing perceptual experience, it might turn maladaptive in artificial circumstances, such as medical image perception tasks, where visual stimuli are randomly sequenced. Here, we analyzed 758,139 skin cancer diagnostic records from an online app, and we quantified the semantic similarity between sequential dermatology images using a computer vision model as well as human raters. We then tested whether serial dependence in perception occurs in dermatological judgments as a function of image similarity. We found significant serial dependence in perceptual discrimination judgments of lesion malignancy. Moreover, the serial dependence was tuned to the similarity in the images, and it decayed over time. The results indicate that relatively realistic store-and-forward dermatology judgments may be biased by serial dependence. These findings help in understanding one potential source of systematic bias and errors in medical image perception tasks and hint at useful approaches that could alleviate the errors due to serial dependence.

Contributions of global and local processing on medical image perception.

The influential holistic processing hypothesis attributes expertise in medical image perception to cognitive processing of global gist information. However, it has remained unclear whether or how experts use rapid global impression of images for their subsequent diagnostic decisions based on the focal sign of cancer. We hypothesized that continuous-global and discrete-local processes jointly attribute to radiological experts' detection of mammogram, with different weights and temporal dynamics. We examined experienced versus inexperienced observers' performance at first (500ms) versus second (2500ms) mammogram image presentation in an abnormality detection task. We applied a dual-trace signal detection (DTSD) model of receiver operating characteristic (ROC) to assess the time-varying contributions of global and focal cancer signals on mammogram reading and medical expertise. The hierarchical Bayesian DTSD modeling of empirical ROCs revealed that mammogram expertise (experienced versus inexperienced observers) manifests largely in a continuous-global component for the detection of the gist of abnormality at the early phase of mammogram reading. For the second presentation of the same mammogram images, the experienced participants showed increased task performance that was largely driven by better processing of discrete-local information, whereas the global processing of abnormality remained saturated from the first exposure. Modeling of the mouse trajectory of the confidence rating responses further revealed the temporal dynamics of global and focal processing. These results suggest a joint contribution of continuous-global and discrete-local processes on medical expertise, and these processes could be analytically dissociated.

Visual adaptation to medical images: a comparison of digital mammography and tomosynthesis.

Radiologists and other image readers spend prolonged periods inspecting medical images. The visual system can rapidly adapt or adjust sensitivity to the images that an observer is currently viewing, and previous studies have demonstrated that this can lead to pronounced changes in the perception of mammogram images. We compared these adaptation effects for images from different imaging modalities to explore both general and modality-specific consequences of adaptation in medical image perception. We measured perceptual changes induced by adaptation to images acquired by digital mammography (DM) or digital breast tomosynthesis (DBT), which have both similar and distinct textural properties. Participants (nonradiologists) adapted to images from the same patient acquired from each modality or for different patients with American College of Radiology-Breast Imaging Reporting and Data System (BI-RADS) classification of dense or fatty tissue. The participants then judged the appearance of composite images formed by blending the two adapting images (i.e., DM versus DBT or dense versus fatty in each modality). Adaptation to either modality produced similar significant shifts in the perception of dense and fatty textures, reducing the salience of the adapted component in the test images. In side-by-side judgments, a modality-specific adaptation effect was not observed. However, when the images were directly fixated during adaptation and testing, so that the textural differences between the modalities were more visible, significantly different changes in the sensitivity to the noise in the images were observed. These results confirm that observers can readily adapt to the visual properties or spatial textures of medical images in ways that can bias their perception of the images, and that adaptation can also be selective for the distinctive visual features of images acquired by different modalities.

Hip fracture or not? The reversed prevalence effect among non-experts’ diagnosis

Despite numerous investigations of the prevalence effect on medical image perception, little research has been done to examine the effect of expertise, and its possible interaction with prevalence. In this study, medical practitioners were instructed to detect the presence of hip fracture in 50 X-ray images with either high prevalence (Nsignal = 40) or low prevalence (Nsignal = 10). Results showed that compared to novices (e.g., pediatricians, dentists, neurologists), the manipulation of prevalence shifted participant’s criteria in a different direction for experts who perform hip fracture diagnosis on a daily basis. That is, when prevalence rate is low (pfracture-present = 0.2), experts held more conservative criteria in answering “fracture-present,” whereas novices were more likely to believe there was fracture. Importantly, participants’ detection discriminability did not vary by the prevalence condition. In addition, all participants were more conservative with “fracture-present” responses when task difficulty increased. We suspect the apparent opposite criteria shift between experts and novices may have come from medical training that made novices to believe that a miss would result in larger cost compared to false positive, or because they failed to update their prior belief about the signal prevalence in the task, both would suggest that novices and experts may have different beliefs in placing the optimal strategy in the hip fracture diagnosis. Our work can contribute to medical education training as well as other applied clinical diagnosis that aims to mitigate the prevalence effect.

Idiosyncratic biases in the perception of medical images.

Radiologists routinely make life-altering decisions. Optimizing these decisions has been an important goal for many years and has prompted a great deal of research on the basic perceptual mechanisms that underlie radiologists' decisions. Previous studies have found that there are substantial individual differences in radiologists' diagnostic performance (e.g., sensitivity) due to experience, training, or search strategies. In addition to variations in sensitivity, however, another possibility is that radiologists might have perceptual biases-systematic misperceptions of visual stimuli. Although a great deal of research has investigated radiologist sensitivity, very little has explored the presence of perceptual biases or the individual differences in these. Here, we test whether radiologists' have perceptual biases using controlled artificial and Generative Adversarial Networks-generated realistic medical images. In Experiment 1, observers adjusted the appearance of simulated tumors to match the previously shown targets. In Experiment 2, observers were shown with a mix of real and GAN-generated CT lesion images and they rated the realness of each image. We show that every tested individual radiologist was characterized by unique and systematic perceptual biases; these perceptual biases cannot be simply explained by attentional differences, and they can be observed in different imaging modalities and task settings, suggesting that idiosyncratic biases in medical image perception may widely exist. Characterizing and understanding these biases could be important for many practical settings such as training, pairing readers, and career selection for radiologists. These results may have consequential implications for many other fields as well, where individual observers are the linchpins for life-altering perceptual decisions.

Normal blindness: when we Look But Fail To See.

Humans routinely miss important information that is 'right in front of our eyes', from overlooking typos in a paper to failing to see a cyclist in an intersection. Recent studies on these 'Looked But Failed To See' (LBFTS) errors point to a common mechanism underlying these failures, whether the missed item was an unexpected gorilla, the clearly defined target of a visual search, or that simple typo. We argue that normal blindness is the by-product of the limited-capacity prediction engine that is our visual system. The processes that evolved to allow us to move through the world with ease are virtually guaranteed to cause us to miss some significant stimuli, especially in important tasks like driving and medical image perception.

Controllable Medical Image Generation via GAN.

Medical image data is critically important for a range of disciplines, including medical image perception research, clinician training programs, and computer vision algorithms, among many other applications. Authentic medical image data, unfortunately, is relatively scarce for many of these uses. Because of this, researchers often collect their own data in nearby hospitals, which limits the generalizabilty of the data and findings. Moreover, even when larger datasets become available, they are of limited use because of the necessary data processing procedures such as de-identification, labeling, and categorizing, which requires significant time and effort. Thus, in some applications, including behavioral experiments on medical image perception, researchers have used naive artificial medical images (e.g., shapes or textures that are not realistic). These artificial medical images are easy to generate and manipulate, but the lack of authenticity inevitably raises questions about the applicability of the research to clinical practice. Recently, with the great progress in Generative Adversarial Networks (GAN), authentic images can be generated with high quality. In this paper, we propose to use GAN to generate authentic medical images for medical imaging studies. We also adopt a controllable method to manipulate the generated image attributes such that these images can satisfy any arbitrary experimenter goals, tasks, or stimulus settings. We have tested the proposed method on various medical image modalities, including mammogram, MRI, CT, and skin cancer images. The generated authentic medical images verify the success of the proposed method. The model and generated images could be employed in any medical image perception research.

Advancing Research on Medical Image Perception by Strengthening Multidisciplinary Collaboration.

Medical image interpretation is central to detecting, diagnosing, and staging cancer and many other disorders. At a time when medical imaging is being transformed by digital technologies and artificial intelligence, understanding the basic perceptual and cognitive processes underlying medical image interpretation is vital for increasing diagnosticians' accuracy and performance, improving patient outcomes, and reducing diagnostician burnout. Medical image perception remains substantially understudied. In September 2019, the National Cancer Institute convened a multidisciplinary panel of radiologists and pathologists together with researchers working in medical image perception and adjacent fields of cognition and perception for the "Cognition and Medical Image Perception Think Tank." The Think Tank's key objectives were to identify critical unsolved problems related to visual perception in pathology and radiology from the perspective of diagnosticians, discuss how these clinically relevant questions could be addressed through cognitive and perception research, identify barriers and solutions for transdisciplinary collaborations, define ways to elevate the profile of cognition and perception research within the medical image community, determine the greatest needs to advance medical image perception, and outline future goals and strategies to evaluate progress. The Think Tank emphasized diagnosticians' perspectives as the crucial starting point for medical image perception research, with diagnosticians describing their interpretation process and identifying perceptual and cognitive problems that arise. This article reports the deliberations of the Think Tank participants to address these objectives and highlight opportunities to expand research on medical image perception.

Visual Hindsight Bias for Mammogram Abnormalities in Expert Radiologists

Hindsight bias – where people believe they can accurately predict something only once they know about it – is a pervasive decision making phenomenon, including in interpretation of radiological images (e.g., Berlin, 2000). However, some evidence suggests it is not only a decision making phenomena but also a visual perception one, where prior information about an image genuinely enhances our visual perception of that image (Harley et al. 2004). For example, the random image structure evolution (RISE) method of Sadr & Sinha (2004), where objects appear in random order embedded in different amounts of noise, demonstrates that primed images are genuinely recognized more easily than those that have not been seen before, a visual hindsight bias. The current experiment tests to what extent expert radiologists show a visual hindsight bias to mammograms with visible abnormalities when they are embedded in noise (e.g., to what extent they see the images differently when they know what the abnormality is, rather than just being biased at a decision level). N=40 experienced mammograph readers were presented with a series of unilateral abnormal mammograms (half had masses, half had calcifications), for three seconds each. After the presentation of each image, observers were asked to rate their confidence on a 6-point scale ranging from confident mass to confident calcification. Critically, we used the RISE method where the images in a sequence were repeated in an unpredictable order, and with varied noise. We found that radiologists who first saw a clear, original image were more accurate in the max noise level condition (d’=0.49) than those who first saw the degraded images (d’=0.20; p=0.017), suggesting that radiologists’ visual perception of medical images is enhanced by prior knowledge of the abnormality. Overall, these results provide preliminary evidence that radiologists experience not only decision level but also visual hindsight bias.

Clinical Validation Is the Key to Adopting AI in Clinical Practice.

Clinical Validation Is the Key to Adopting AI in Clinical Practice.

Visual Perception: How Better Imaging Can Make Things Worse

Summary Visual search is critical to daily life and to socially important tasks — from cancer screening to airport security. New research shows how a technological advancement can interact with the human visual system to improve search for one type of target while making matters worse for another. Part of the problem is that we are surprisingly bad at knowing where we have looked.

Controllable Medical Image Generation via Generative Adversarial Networks.

Radiologists and pathologists frequently make highly consequential perceptual decisions. For example, visually searching for a tumor and recognizing whether it is malignant can have a life-changing impact on a patient. Unfortunately, all human perceivers-even radiologists-have perceptual biases. Because human perceivers (medical doctors) will, for the foreseeable future, be the final judges of whether a tumor is malignant, understanding and mitigating human perceptual biases is important. While there has been research on perceptual biases in medical image perception tasks, the stimuli used for these studies were highly artificial and often critiqued. Realistic stimuli have not been used because it has not been possible to generate or control them for psychophysical experiments. Here, we propose to use Generative Adversarial Networks (GAN) to create vivid and realistic medical image stimuli that can be used in psychophysical and computer vision studies of medical image perception. Our model can generate tumor-like stimuli with specified shapes and realistic textures in a controlled manner. Various experiments showed the authenticity of our GAN-generated stimuli and the controllability of our model.

What can an echocardiographer see in briefly presented stimuli? Perceptual expertise in dynamic search

BackgroundExperts in medical image perception are able to detect abnormalities rapidly from medical images. This ability is likely due to enhanced pattern recognition on a global scale. However, the bulk of research in this domain has focused on static rather than dynamic images, so it remains unclear what level of information that can be extracted from these displays. This study was designed to examine the visual capabilities of echocardiographers—practitioners who provide information regarding cardiac integrity and functionality. In three experiments, echocardiographers and naïve participants completed an abnormality detection task that comprised movies presented on a range of durations, where half were abnormal. This was followed by an abnormality categorization task.ResultsAcross all durations, the results showed that performance was high for detection, but less so for categorization, indicating that categorization was a more challenging task. Not surprisingly, echocardiographers outperformed naïve participants.ConclusionsTogether, this suggests that echocardiographers have a finely tuned capability for cardiac dysfunction, and a great deal of visual information can be extracted during a global assessment, within a brief glance. No relationship was evident between experience and performance which suggests that other factors such as individual differences need to be considered for future studies.

Special Section Guest Editorial: Medical Image Perception and Observer Performance.

.Guest editors provide an introduction to the Special Section on Medical Image Perception and Observer Performance.

Multi-path convolutional neural network in fundus segmentation of blood vessels

There is a close correlation between retinal vascular status and physical diseases such as eye lesions. Retinal fundus images are an important basis for diagnosing diseases such as diabetes, glaucoma, hypertension, coronary heart disease, etc. Because the thickness of the retinal blood vessels is different, the minimum diameter is only one or two pixels wide, so obtaining accurate measurement results becomes critical and challenging. In this paper, we propose a new method of retinal blood vessel segmentation that is based on a multi-path convolutional neural network, which can be used for computer-based clinical medical image analysis. First, a low-frequency image characterizing the overall characteristics of the retinal blood vessel image and a high-frequency image characterizing the local detailed features are respectively obtained by using a Gaussian low-pass filter and a Gaussian high-pass filter. Then a feature extraction path is constructed for the characteristics of the low- and high-frequency images, respectively. Finally, according to the response results of the low-frequency feature extraction path and the high-frequency feature extraction path, the whole blood vessel perception and local feature information fusion coding are realized, and the final blood vessel segmentation map is obtained. The performance of this method is evaluated and tested by DRIVE and CHASE_DB1. In the experimental results of the DRIVE database, the evaluation indexes accuracy (Acc), sensitivity (SE), and specificity (SP) are 0.9580, 0.8639, and 0.9665, respectively, and the evaluation indexes Acc, SE, and SP of the CHASE_DB1 database are 0.9601, 0.8778, and 0.9680, respectively. In addition, the method proposed in this paper could effectively suppress noise, ensure continuity after blood vessel segmentation, and provide a feasible new idea for intelligent visual perception of medical images.

Deep learning can be used to train naïve, nonprofessional observers to detect diagnostic visual patterns of certain cancers in mammograms: a proof-of-principle study.

.The scientific, clinical, and pedagogical significance of devising methodologies to train nonprofessional subjects to recognize diagnostic visual patterns in medical images has been broadly recognized. However, systematic approaches to doing so remain poorly established. Using mammography as an exemplar case, we use a series of experiments to demonstrate that deep learning (DL) techniques can, in principle, be used to train naïve subjects to reliably detect certain diagnostic visual patterns of cancer in medical images. In the main experiment, subjects were required to learn to detect statistical visual patterns diagnostic of cancer in mammograms using only the mammograms and feedback provided following the subjects’ response. We found not only that the subjects learned to perform the task at statistically significant levels, but also that their eye movements related to image scrutiny changed in a learning-dependent fashion. Two additional, smaller exploratory experiments suggested that allowing subjects to re-examine the mammogram in light of various items of diagnostic information may help further improve DL of the diagnostic patterns. Finally, a fourth small, exploratory experiment suggested that the image information learned was similar across subjects. Together, these results prove the principle that DL methodologies can be used to train nonprofessional subjects to reliably perform those aspects of medical image perception tasks that depend on visual pattern recognition expertise.