Cell identities are defined by intrinsic transcriptional networks and spatio-temporal environmental factors. Here, we explored multiple factors that contribute to the identity of adipose stem cells, including anatomic location, microvascular neighborhood, and sex. Our data suggest that adipose stem cells serve a dual role as adipocyte precursors and fibroblast-like cells that shape the adipose tissue’s extracellular matrix in an organotypic manner. We further find that adipose stem cells display sexual dimorphism regarding genes involved in estrogen signaling, homeobox transcription factor expression and the renin-angiotensin-aldosterone system. These differences could be attributed to sex hormone effects, developmental origin, or both. Finally, our data demonstrate that adipose stem cells are distinct from mural cells, and that the state of commitment to adipogenic differentiation is linked to their anatomic position in the microvascular niche. Our work supports the importance of sex and microvascular function in adipose tissue physiology.