Transgender (trans) and gender-diverse (GD) adolescents and young adults have remained largely invisible in health research. Previous research shows worse outcomes in health indicators for trans and GD people, compared to cisgender controls. Research on the impact of sexual violence focuses on mainly cisgender female adult victims. This study assessed the impact of sexual violence on the quality of life (QoL) and mental wellbeing (GHQ-12) among trans and GD adolescents and young adults, while taking into account the possible role of gender nonconformity in sexual violence and mental wellbeing. An online, anonymous survey and interviews/focus groups were conducted between October 2021 and May 2022 in Belgium. Multiple analyses of covariance (ANCOVAs) were used to assess the associations between sexual violence, mental wellbeing, and gender nonconformity, while controlling for different background variables (gender identity, sexual orientation, age, economic vulnerability, etc.). The interviews and focus groups were used to validate associations between variables that were hypothesized as important. The quantitative sample consisted of 110 respondents between 15 and 25 years old, with 30 trans respondents (27.3%) and 80 GD respondents (72.7%). A total of 73.6% reported experiences with sexual violence over the past two years (n = 81). The mean QoL score was 5.3/10, and the mean GHQ-12 score was 6.6/12. Sexual violence was not significantly associated with QoL (p = 0.157) and only marginally significantly associated with GHQ-12 (p = 0.05). Changing one's physical appearance to conform to gender norms, out of fear of getting attacked, discriminated against, or harassed was significantly associated with QoL (p = 0.009) and GHQ-12 (p = 0.041). The association between sexual violence and changing one's physical appearance to conform to gender norms was analyzed, to assess a possible mediation effect of sexual violence on mental wellbeing. No significant association was found (p = 0.261). However, the interviews suggest that sexual violence is associated with changing one's physical appearance, but this association is not limited to only trans and GD victims of sexual violence. Non-victims also adjust their appearance, out of fear of future sexual victimization. Together with the high proportion of sexual violence, as well as the lower average QoL and higher average GHQ-12 scores among trans and GD adolescents and young adults, compared to general population statistics, this highlights the need for policy makers to create more inclusive environments.