The gender-diverse community faces numerous barriers that make physical activity difficult to access, meaning they are less physically active than their cisgender counterparts, compounding the existing health and wellbeing inequalities experienced by this group. The current study conducted a systematic review of sporadic literature relating to trans, non-binary and intersex people’s experience of physical activity with the aim of understanding how individuals negotiate and experience physical activity environments. A systematic searching of 8 databases and screening highlighted 30 academic papers for review. A narrative synthesis of the experiences of trans, non-binary and intersex individuals formed a three-layer model of influence. The first layer encapsulated the systemic and systemic influences such as transphobia and trans-moral panic. The second layer included external influences such as others’ behaviour, gendered spaces and exclusionary policies. The third layer was made up of internal influences such as internalised transphobia and gender concealment. The focus on fitting gender non-conforming individuals into a binary system through gendered spaces, pressures to switch teams and provide ‘proof’ of gender conformity set a dangerous precedent of the physical activity space as an unwelcoming and unsafe place for those who do not conform to the gender binary. If there should be a hope to address the health inequalities faced by the trans, non-binary and intersex community, the focus must move away from conformity and the debate of ‘fairness’ towards true inclusion and creating an environment where gender minorities can feel safe to engage with physical activity.