This study aims to explore the effect of rock heterogeneity caused by meso-structure differences on the macro-mechanical properties and fracture evolution process. With the help of computed tomography, the meso-structure characteristics of coarse-grained red sandstone are obtained, starting from the meso-structure level. The digital characterization of heterogeneous rocks is completed with the use of image-processing methods, and a rock equivalent crystalline model is constructed. A numerical simulation study is performed with the particle flow PFC3D 5.0 program. The results show that (1) the main form of rock heterogeneity is a meso-structure difference, and this determines the rock’s macro-mechanical properties and failure characteristics. The meso-structure difference mainly plays a role in the post-peak damage weakening stage of rock, and the slight difference leads to damage in different development modes. (2) The crack initiation starts at the cemented and crystalline grid surfaces, which is randomly distributed. The crack evolution process is selective and affected by the types and distribution of the surrounding mineral particles. The development of cracks is dominated by secondary cracks, distributed in strips, in the area of mineral particles with strong mechanical properties. In contrast, cracks are densely developed and spread around in a cluster in the area of mineral particles with weak mechanical properties.