Unequal power and gender norms expose adolescent girls to higher risks of HIV, early marriages, pregnancies and coerced sex. In Uganda, almost half of the girls below the age of 18 are already married or pregnant, which poses a danger to the lives of young girls. This study explores the social construction of gender norms from early childhood, and how it influences adolescents’ agency. Contrary to the mainstream theory of agency, which focuses on the ability to make informed choices, adolescents’ agency appears constrained by context-specific obstacles. This study adopted qualitative research approaches involving 132 participants. Of these, 44 were in-depth interviews and 11 were focus group discussions, parcelled out into separate groups of adolescents (12–14 years), teachers, and parents (n = 88), in Western Uganda. Data were analysed manually using open and axial codes, and conclusions were inductive. Results show that gender norms are established early in life, and have a very substantial impact on the agency of young adolescents. There were stereotypical gender norms depicting boys as sexually active and girls as restrained; girls’ movements were restricted; their sexual agency constrained; and prevention of pregnancy was perceived as a girl’s responsibility. Programs targeting behavioural change need to begin early in the lives of young children. They should target teachers and parents about the values of gender equality and strengthen the legal system to create an enabling environment to address the health and wellbeing of adolescents.