Keloid is a major type of skin fibrotic disease, with one prominent feature of extensive accumulation of extracellular matrix (ECM) components, and another feature of pain/itching, which is closely related to the peripheral nervous system (PNS). However, the molecular pathogenesis of these two prominent features still needs to be further explored. In the present study, we performed single-cell RNA sequencing (scRNA-seq) on clinical earlobe keloid samples and adjacent normal skin samples and constructed a keloid atlas of 31,379 cells. All cells were clustered into 13 major cell types using cell-type-specific markers. Among them, fibroblast, vascular endothelial cells, and smooth muscle cells were defined as the ECM-related populations according to their ECM-associated functions. Also, we found that Schwann cells (SCs) were the main neuron cells of PNS in the skin. Interestingly, the cell proportions of ECM-related populations, as well as SC were increased significantly in the earlobe keloid compared to the adjacent normal tissues, suggesting an important role of these cell types in the development of the earlobe keloid. Comprehensive cell-cell interaction analysis at the single-cell level revealed a strong interaction between SC and ECM-related subgroups which might be mediated by SEMA3C signaling pathways and MK/PTN gene family, which are found to be mainly involved in promoting cell proliferation and migration. Moreover, further exploration of the interactions of ECM-related populations and SC in different keloids, including earlobe keloid, back keloid, and chest keloid revealed an increasing amount of TGFβ-TGFβ receptor interactions in chest/back keloids as compared to earlobe keloid, which suggested the anatomic site-specific pathogenesis in different keloids. Altogether, these findings suggested the interactions between ECM-related populations and SC contributing to the earlobe keloid formation and helped us to better understand the pathogenesis of keloids.