This study extends the literature on the impact of the family of origin on gender identity by theorizing about refusing gender. We define refusing gender as the intimate refusal of gender identity by family members that is perceived as intentional and deliberate by transgender and nonbinary people in the United States. In this article, we demonstrate how refusing gender is intimate, perceived as intentional, embedded within existing family instabilities, and disruptive of family relationships. This study is based on interviews from a racially diverse group of 25 transgender and nonbinary adults in Texas, half of whom report high rates of adverse childhood experiences (ACEs). Findings contribute to gender theory by revealing the importance of the family in recognizing gender identity. This research makes novel connections between existing family instability, including histories of abuse, and gender refusal. We embed gender recognition within persistent family dynamics, including long-standing family instabilities and family violence, arguing that these family dynamics persist in the lives of adult children. Overall, these findings demonstrate how cisnormativity is reproduced in family life, filling significant gaps in theorizing about transgender and nonbinary family life.