The adipose tissue (AT) surrounding breast cancer (BC) plays a pivotal role in cancer progression and represents an optimal source for new biomarker discovery. The aim of this work was to investigate whether specific AT factors may have prognostic value in estrogen receptor-positive (ER+) BC. Proteoglycan Versican (VCAN), Insulin-like Growth Factor 1 (IGF1), Reticulon 4B (RTN4), chemokines CCL5 (also known as RANTES) and interleukin 8 (IL-8) are expressed in AT and may play important roles in BC progression. Peritumoral AT and tumoral biopsies were obtained from patients with ER+ BC (N = 23). AT specimens were collected also from healthy women (N = 17; CTRL-AT). The analysis of gene expression by qPCR revealed significantly higher mRNA levels of VCAN, IGF1, RTN4, and CCL5 in BC-AT compared to the CTRL-AT, and no difference in IL-8 mRNA levels. VCAN positively correlated with patient Body Mass Index (BMI) in BC-AT, while not in CTRL-AT. Moreover, VCAN and IGF1 positively correlated with RTN4 and negatively with CCL5. Interestingly, VCAN correlated with tumoral Ki67, while IGF1 with tumoral OCT4 that, in turn, correlated with tumoral Ki67 and patient BMI. Thus, peritumoral AT content of VCAN, and IGF1 are related to BC proliferation and aggressiveness.