Tracking the crack propagation in concrete at the mesoscopic level is of great importance for revealing the crack pattern and failure mechanism, due to the huge threat of cracking to structure safety. This paper develops a phase field modeling scheme with mesostructure of concrete composite to investigate its crack propagation behavior. With the classical phase field model and the staggered algorithm implementation by three-dimensional (3D) finite element method (FEM), crack propagation within concrete’s mesostructure composed of coarse aggregate, mortar matrix, interfacial transition zone (ITZ) and initial defects including micropores and microcracks is modeled. The proposed modeling scheme is tested on single-aggregate samples and complex mesostructured concrete samples subjected to uniaxial loading, showing reasonable accordance with the experimental observation and common failure modes of concrete material. The results indicate that mesostructural configurations are important factors affecting the crack pattern of concrete. The presence of coarse aggregates is prone to initiate and induce interfacial cracking in the ITZ effect, but also exerts the blocking effect on the crack growth. And the initial defects (micropores and microcracks) are also the important influential factor on crack initiation, propagation and the formation of the major crack. It is found that the higher the volume fractions of aggregates and initial defects, the cracking of concrete initiates earlier, propagates faster and penetrates more easily when subjected to the same loading condition.