Breast tissue composition (epithelium, non-fatty stroma, and adipose) changes qualitatively and quantitatively throughout the lifespan, and may mediate relationships between risk factors and breast cancer initiation. We sought to identify relationships between tissue composition, risk factors, tumor characteristics, and gene expression. Participants were 146 patients from the Polish Breast Cancer Study, with data on risk factor and clinicopathological characteristics. Benign breast tissue composition was evaluated using digital image analysis of histologic sections. Whole-genome microarrays were performed on the same tissue blocks. Mean epithelial, non-fatty stromal, and adipose proportions were 8.4% (SD = 4.9%), 27.7% (SD = 24.0%), and 64.0% (SD = 24.0%), respectively. Among women <50 years old, stroma proportion decreased and adipose proportion increased with age, with approximately 2% difference per year (P < 0.01). The variation in epithelial proportion with age was modest (0.1% per year). Higher epithelial proportion was associated with obesity (7.6% in nonobese vs. 10.1% in obese; P = 0.02) and with poorly differentiated tumors (7.8% in well/moderate vs. 9.9% in poor; P = 0.05). Gene expression signatures associated with epithelial and stromal proportion were identified and validated. Stroma-associated genes were in metabolism and stem cell maintenance pathways, whereas epithelial genes were enriched for cytokine and immune response pathways. Breast tissue composition was associated with age, body mass index, and tumor grade, with consequences for breast gene expression. Breast tissue morphologic factors may influence breast cancer etiology. Composition and gene expression may act as biomarkers of breast cancer risk and progression.