Phylogenetic comparative studies suggest that the direction of deviation from bilateral symmetry (sidedness) might evolve through genetic assimilation; however, the changes in sidedness inheritance remain largely unknown. We investigated the evolution of genital asymmetry in fish of the family Anablepidae, in which males' intromittent organ (the gonopodium, a modified anal fin) bends asymmetrically to the left or the right. In most species, males show a 1 : 1 ratio of left-to-right-sided gonopodia. However, we found that in three species left-sided males are significantly more abundant than right-sided ones. We mapped sidedness onto a new molecular phylogeny, finding that this left-sided bias likely evolved independently three times. Our breeding experiment in a species with an excess of left-sided males showed that sires produced more left-sided offspring independently of their own sidedness. We propose that sidedness might be inherited as a threshold trait, with different thresholds across species. This resolves the apparent paradox that, while there is evidence for the evolution of sidedness, commonly there is a lack of support for its heritability and no response to artificial selection. Focusing on the heritability of the left : right ratio of offspring, rather than on individual sidedness, is key for understanding how the direction of asymmetry becomes genetically assimilated.