BackgroundIn most animal species, males and females exhibit differences in behavior and morphology that relate to their respective roles in reproduction. DM (Doublesex/MAB-3) domain transcription factors are phylogenetically conserved regulators of sexual development. They are thought to establish sexual traits by sex-specifically modifying the activity of general developmental programs. However, there are few examples where the details of these interactions are known, particularly in the nervous system.Methodology/Principal FindingsIn this study, we show that two C. elegans DM domain genes, dmd-3 and mab-23, regulate sensory and muscle cell development in a male neural circuit required for mating. Using genetic approaches, we show that in the circuit sensory neurons, dmd-3 and mab-23 establish the correct pattern of dopaminergic (DA) and cholinergic (ACh) fate. We find that the ETS-domain transcription factor gene ast-1, a non-sex-specific, phylogenetically conserved activator of dopamine biosynthesis gene transcription, is broadly expressed in the circuit sensory neuron population. However, dmd-3 and mab-23 repress its activity in most cells, promoting ACh fate instead. A subset of neurons, preferentially exposed to a TGF-beta ligand, escape this repression because signal transduction pathway activity in these cells blocks dmd-3/mab-23 function, allowing DA fate to be established. Through optogenetic and pharmacological approaches, we show that the sensory and muscle cell characteristics controlled by dmd-3 and mab-23 are crucial for circuit function.Conclusions/SignificanceIn the C. elegans male, DM domain genes dmd-3 and mab-23 regulate expression of cell sub-type characteristics that are critical for mating success. In particular, these factors limit the number of DA neurons in the male nervous system by sex-specifically regulating a phylogenetically conserved dopamine biosynthesis gene transcription factor. Homologous interactions between vertebrate counterparts could regulate sex differences in neuron sub-type populations in the brain.