Previous research has demonstrated the positive associations with parental support with better mental health outcomes in gender-diverse young people. However, less is known about what happens within a family structure if an unsupportive parent or family member leaves the family unit and sees the child only on the condition that they present in a way that conforms to their birth gender. This paper will focus on the first interviews with nine families within the wider sample of 39 families in the LOGIC-Q (Longitudinal Outcomes of Gender Identity in Children - qualitative) study who reported they had experienced rejection by a parent or family member after the young person had disclosed their gender non-conformity. These families' narratives are examined through two themes: How rejection impacts visitation, and Moving past the loss of rejection. The families all spoke about how they had adapted to the loss of their family members. They described how any negative impact would have been worse had they stayed living together, or were in contact with, the rejecting family member. Here, it becomes important then to understand the context of why a gender-diverse young person may choose to no longer see a parent or family member.