This study investigated whether the textural features of peritumoral adipose tissue (AT) on [18F]fluoro-2-deoxy-2-d-glucose (FDG) positron emission tomography/computed tomography (PET/CT) can predict the pathological response to neoadjuvant chemotherapy (NAC) and progression-free survival (PFS) in breast cancer patients. We retrospectively enrolled 147 female breast cancer patients who underwent staging FDG PET/CT and completed NAC and underwent curative surgery. We extracted 10 first-order features, 6 gray-level co-occurrence matrix (GLCM) features, and 3 neighborhood gray-level difference matrix (NGLDM) features of peritumoral AT and evaluated the predictive value of those imaging features for pathological complete response (pCR) and PFS. The results of our study demonstrated that GLCM homogeneity showed the highest predictability for pCR among the peritumoral AT imaging features in the receiver operating characteristic curve analysis. In multivariate logistic regression analysis, the mean standardized uptake value (SUV), 50th percentile SUV, 75th percentile SUV, SUV histogram entropy, GLCM entropy, and GLCM homogeneity of the peritumoral AT were independent predictors for pCR. In multivariate survival analysis, SUV histogram entropy and GLCM correlation of peritumoral AT were independent predictors of PFS. Textural features of peritumoral AT on FDG PET/CT could be potential imaging biomarkers for predicting the response to NAC and disease progression in breast cancer patients.