The tumor microenvironment (TME) is connected to immunotherapy responses, but it remains unclear how cancer cells and host tissues differentially influence the immune composition within TME. Here, we performed single-cell analyses for autologous samples from liver metastasized colorectal cancer to disentangle factors shaping TME. By aligning CD45+ cells across different tissues, we classified exhausted CD8+ Tcells (Texs) and activated regulatory Tcells as M-type, whose phenotypes were associated with the malignancy, while natural killer and mucosal-associated invariant Tcells were defined as N-type, whose phenotypes were associated with the niche. Tcell receptor sharing between Texs in primary and metastatic tumors implicated the presence of common peripheral non-exhausted precursors. For myeloid cells, a subset of dendritic cells (DC3s) and SPP1+ macrophages were M-type, and the latter were predominant in liver metastasis, indicating its pro-metastasis role. Our analyses bridge immune phenotypes of primary and metastatic tumors, thereby helping to understand the tumor-specific contexture and identify the pro-metastasis components.