Background: Using images in multiple‐choice formats for comprehension testing in aphasia is common. It is generally assumed that persons being assessed perceive the content of the images represented in such tasks. However, specific visual characteristics of individual images may influence visual attention, which may influence accuracy in the selection of a correct target image corresponding to a verbal stimulus. The validity of test responses may be confounded by (1) physical stimulus features, such as size, and (2) semantic content conveyed by images, such as image familiarity. This study is supported by a grant (# DC 0015301 A1) from the National Institute on Deafness and other Communication Disorders of the National Institutes of Health. We thank Natalie Douglas for her important contributions, Yoon‐Soo Lee for creation and editing of visual stimuli, and Sonny Kim for statistical consultation. Aims: The first aim was to develop a rating instrument to assess visual stimulus properties and semantic content conveyance in multiple‐choice images, based on an extensive review of empirical literature and validated by experts in graphic design. The second aim was to study the degree of relationship between viewers' subjective ratings of images selected from published aphasia batteries, using the rating instrument, and eye movement measures recorded as independent viewers looked at the same images. The third aim was to compare the viewers' actual eye movement indices of disproportionate visual attention to an ideal value of evenly proportionate visual attention for each image set. Methods & Procedures: A rating instrument, based on an extensive review of literature and assessed and revised by graphic design and eye‐tracking experts, was developed to identify such influences within multiple‐choice images and was assessed through empirical testing of viewers' eye movement patterns as they looked at images from published aphasia tests. A total of 20 adults rated 20 image sets from five aphasia batteries. Eye movements were recorded for a separate group of 40 adults viewing the same images. Outcomes & Results: Ratings were not statistically correlated with eye movement responses. All multiple‐choice image sets prompted significantly disproportionate visual attention. Conclusions: Results highlight the importance of: (1) considering the possible influence of visual stimulus confounds on any given patient's test performance, and (2) better controlled image design for multiple‐choice test images to improve the validity of assessment. Further research is needed to improve subjective and objective means of assessment of images and guidelines for improved design of multiple‐choice image displays.