Breast cancer is the most commonly diagnosed cancer in women, with distant metastasis being the main cause of breast cancer-related deaths. Elucidating the changes in the tumor and immune ecosystems that are associated with metastatic disease is essential to improve understanding and ultimately treatment of metastasis. Here, we developed an in-depth, spatially resolved single-cell atlas of the phenotypic diversity of tumor and immune cells in primary human breast tumors and matched distant metastases, using imaging mass cytometry to analyze a total of 75 unique antibody targets. While the same tumor cell phenotypes were typically present in primary tumors and metastatic sites, suggesting a strong founder effect of the primary tumor, their proportions varied between matched samples. Notably, the metastatic site did not influence tumor phenotype composition, except for the brain. Metastatic sites exhibited a lower number of immune cells overall, but had a higher proportion of myeloid cells as well as exhausted and cytotoxic T cells. Myeloid cells showed distinct tissue-specific compositional signatures and increased presence of potentially matrix remodeling phenotypes in metastatic sites. This analysis of tumor and immune cell phenotypic composition of metastatic breast cancer highlights the heterogeneity of the disease within patients and across distant metastatic sites, indicating myeloid cells as the predominant immune modulators that could potentially be targeted at these sites.