Asexual, lesbian, gay, bisexual, and trans (ALGBT) individuals face worse life conditions and violence rates than their heterosexual cisgender counterparts. Brazil is often highlighted for having one of the highest rates of hate-related homicides against ALGBTs in the world. However, to date, Brazil’s ALGBT population has not been investigated with a representative sample, and basic information such as population size or sociodemographic characteristics are mostly based in non-systematic data. We aimed to assess the proportion of asexual, lesbian, gay, bisexual, trans and non-binary adults in Brazil, their sociodemographic characteristics, and self-reported violence rates. In 2018, a sample (n = 6000) of the Brazilian adult population answered a face-to-face survey assessing sociodemographic characteristics, gender identity, sexual orientation, and self-reported psychological, physical, verbal, and sexual violence. Among Brazilian adults, 12.04% are ALGBT: 5.76% asexual, 0.93% lesbian, 1.37% gay, 2.12% bisexual, 0.68 trans, and 1.18% non-binary. Compared to heterosexual cisgender men, most ALGBT individuals have worse socioeconomic indicators and higher rates of self-reported psychological and verbal violence. All ALGBT groups and heterosexual cisgender women reported sexual violence more often than heterosexual cisgender men. It was reported between 4 up to 25 times more often by heterosexual cisgender women and trans individuals, respectively. The rates of the other ALGBT groups sit among the two. Our findings provide evidence of the important size of the ALGBT Brazilian population, as well as their socioeconomic vulnerability, and concerning violence levels experienced by the group. Policy makers may refer to the present article in order to mitigate this population’s vulnerability and to better understand its sociodemographic characteristics.