Obesity fosters worse clinical outcomes in both premenopausal and postmenopausal women with breast cancer. Emerging evidence suggests that an android body fat distribution in particular is deleterious for breast cancer prognosis. The extent of adipose tissue dysfunction, especially how it relates to breast cancer prognostic factors and anthropometric measurements, has not been fully investigated. Our objective was to examine if markers of adipose tissue dysfunction, such as hypertrophy and macrophage accumulation, are relevant for the pathophysiology of breast cancer and its associated prognostic factors in a well-characterised cohort of women with breast cancer who did not receive treatment before surgery. A consecutive series of 164 women with breast cancer provided breast adipose tissue sample. Multivariate generalised linear models were used to test associations of anthropometric indices and prognostic factors with markers of adipose tissue dysfunction. We found associations of breast adipocyte size and macrophage infiltration (number of CD68+ cells/100 adipocytes) with adiposity, particularly a strong association between breast adipocyte size and central obesity, independent of total adiposity, age and menopausal status (βadj = 0.87; p = 0.0001). We also identified relationships of adipocyte hypertrophy and macrophage infiltration with prognostic factors, such as cancer stage and tumour grade (p < 0.05). RNA expression of pro-inflammatory cytokines (IL6, TNF) and leptin was also increased as a function of adipocyte size and CD86+/CD11c+ macrophage number/100 adipocytes (p < 0.05). Our findings support the model of dysfunctional adipose tissue in obesity-associated breast cancer.