A central issue in speech perception is how listeners resolve variability. Gender diversity presents an opportunity to examine how listeners learn and represent one dimension of sociophonetic variability arising from an evolving social category, speaker gender. To assess listeners’ perceptual organization of speaker gender, recordings were obtained from 30 cisgender (15 men, 15 women) and 30 transgender (1 agender, 15 non-binary, 7 transgender men, 7 transgender women) speakers. In an auditory free classification paradigm, listeners categorized speakers reading the same or unique sentences by perceived general similarity and gender identity. Multidimensional scaling revealed gradient organization of speaker gender with masculinity/femininity and gender prototypicality as the two most salient dimensions. Cluster analyses showed fewer perceptual speaker groups and attention to fewer indexical speech features when speakers produced unique sentences. Instructions to explicitly categorize speakers by perceived gender identity substantially simplified the hierarchical structure of speakers and shifted listeners’ attention towards masculinity/femininity over gender prototypicality. Results suggest that listeners engage in fine-grained, multivalent analysis of speaker gender that cannot be adequately captured by a “male” vs “female” dichotomy. Assumptions of a gender binary in speech communication research may require a critical re-examination to accommodate multidimensional and gradient representation of speaker gender.