In the construction of island and reef engineering, coral concrete shows a good application prospect due to its abundant raw materials. However, the porous and fragile mechanical characteristics of coral reefs limit their use as coarse aggregate in the preparation of coral concrete materials. This study utilized hard and dense granite as the coarse aggregate and regarded coral sand concrete as a two-phase composite material consisting of spherical granite coarse aggregate (GCA) and coral mortar. It investigated the enhancement effect of granite on coral concrete from a microscopic perspective. Five 3D mesoscopic models with different GCA contents and randomly distributed aggregates were established to reveal the variation patterns and failure mechanisms of coral sand concrete under impact loading with GCA. The findings demonstrate that the K&C model can effectively simulate the dynamic compression behavior of coral mortar and granite materials. Under the action of a half-sine incident wave, the dynamic compressive strength of the samples increases with the increase in GCA, demonstrating that the addition of GCA can effectively enhance the impact resistance of coral sand concrete. As the content of GCA increases, the sensitivity of the samples to the loading wave amplitude also increases accordingly.