Concrete is a multiphase composite material composed of mortar, aggregate, and interface transition zone (ITZ). The mesoscopic components of concrete have an important influence on its anti-penetration performance. In this study, a series of penetration experiments with large-caliber ogive-nosed projectiles penetrating concrete targets are carried out. The test results show that the damage to the concrete target consists of crater and tunnel zones and increases with increasing impact velocity. Then, a local background grid method is proposed to establish a 3D mesoscopic model of concrete, based on the arrangement characteristics of the sequence number of the finite elements. Compared with the traditional 3D mesoscopic concrete modeling method, the proposed method can effectively improve the modeling efficiency. Subsequently, numerical simulations are performed based on the 3D mesoscopic model, with the simulation and experimental results in good agreement, verifying the effectiveness of the model. Finally, the verified 3D mesoscopic model is employed to investigate the effects of shape, volume fraction, size interval, and strength of the concrete aggregates on the depth of penetration (DOP) and deflection of the projectile. The simulation results indicate that the shape of the aggregate has a negligible effect on both uniaxial compressive strength and DOP. Therefore, spherical aggregates are used to improve modeling efficiency. Increasing the volume fraction and strength of the aggregates can significantly enhance the anti-penetration performance of concrete. The influence of aggregate size interval on DOP is slight, but it has a significant impact on projectile and trajectory deflection at the same aggregate volume fraction. The uneven lateral resistance on both sides of the projectile, caused by the random distribution of aggregates, is a major factor in deflection.