Purpose To determine the concordance and accuracy of imaging surrogates of immunohistochemical (IHC) markers and the molecular classification of breast cancer. Materials and Methods A total of 3050 patients from 17 public breast cancer data sets containing IHC marker receptor status (estrogen receptor/progesterone receptor/human epidermal growth factor receptor 2 [HER2]) and their molecular classification (basal-like, HER2-enriched, luminal A or B) were analyzed. Diagnostic accuracy and concordance as measured with the κ statistic were calculated between the IHC and molecular classifications. Simulations were performed to assess the relationship between accuracy of imaging-based IHC markers to predict molecular classification. A simulation was performed to examine effects of misclassification of molecular type on patient survival. Results Accuracies of intrinsic subtypes based on IHC subtype were 71.7% (luminal A), 53.7% (luminal B), 64.8% (HER2-enriched), and 81.7% (basal-like). The κ agreement was fair (κ = 0.36) for luminal A and HER2-enriched subtypes, good (κ = 0.65) for the basal-like subtype, and poor (κ = 0.09) for the luminal B subtypes. Introduction of image misclassification by simulation lowered image-true subtype accuracies and κ values. Simulation analysis showed that misclassification caused survival differences between luminal A and basal-like subtypes to decrease. Conclusion There is poor concordance between triple-receptor status and intrinsic molecular subtype in breast cancer, arguing against their use in the design of prognostic genomic-based image biomarkers. © RSNA, 2018 Online supplemental material is available for this article.