The micro-texture of the aggregate plays a very important role in the anti-skid performance of the asphalt pavement. Currently, the polished stone value (PSV) index is mostly used for the characterization of the aggregate, but the micro-texture is rarely considered. In addition, there are few studies that compares micro-textures of different aggregates and their relationship with PSV. In this study, eight different aggregates were selected, including three types of sandstone, limestone, basalt, diabase, gabbro and granite. The PSV and micro-texture of the aggregates during the polishing process were tested by the british pendulum tester (BPT) and the laser scanning confocal microscope (LSCM) respectively, and the relationship between PSV and the micro-texture was regressed. Finally, scanning electron microscope (SEM) images were generated and used to explain the reason for the texture differences in the aggregate from the perspective of the micro-structure. It was found that there was no significant difference on the anti-skid performance of the aggregates before polishing. SEM images showed an unevenness on the surface of sandstone with lots of pits, which is conducive for long-term maintenance of texture. Consequently, sandstone had the best anti-skid performance after polishing, followed by magmatic rock, and limestone. Although, polishing effect reduces the micro-texture of the aggregates, but the micro-texture of the sandstone aggregate was not significantly affected before and after polishing. Overall, the texture of arkose was the largest, followed by red sandstone, quartz sandstone, granite, diabase, gabbro, basalt, limestone. There is an obvious correlation between the micro-texture and the anti-skid performance of the aggregate. The aggregates had better anti-skid performance when the micro-texture index Ra was kept above 4 μm. The multiple linear equation between micro-texture and PSV is obtained by regression analysis, which can be used to predict the PSV of the aggregate. Based on the findings from this study, sandstone is highly recommended as the first choice aggregate for anti-skid performance, followed by magmatic rock and finally limestone.