Fungi represent a central yet often overlooked domain of clinically relevant pathogens that have become increasingly important in human disease. With unique adaptive lifestyles that vary widely across species, human fungal pathogens show remarkable diversity in their virulence strategies. The majority of these fungal pathogens are opportunistic, primarily existing in the environment or as commensals that take advantage of immunocompromised hosts to cause disease. In addition, many fungal pathogens have evolved from non-pathogenic lifestyles. The extent of genetic diversity and heritability of virulence traits remains poorly explored in human fungal pathogens. Genetic variation caused by mutations, genomic rearrangements, gene gain or loss, changes in ploidy, and sexual reproduction have profound effects on genetic diversity. These mechanisms contribute to the remarkable diversity of fungal genomes and have large impacts on their prevalence in human disease, virulence, and resistance to antifungal therapies. Here, we focus on the genomic structure of the most common human fungal pathogens and the aspects of genetic variability that contribute to their dominance in human disease.