Brittleness has a significant influence on rock failure under compression; however, the mechanism is rarely comprehensively discussed. This study numerically investigates the brittleness effect on microcracking behavior of crystalline rock using a grain-based model implemented into a two-dimensional particle flow code, with a focus on the discussion of how rock brittleness affects the failure mechanism. The simulated failure mode changes from tension to shear with decreasing rock brittleness, which is consistent with previous laboratory test results. As the brittleness gradually decreases in the model, the grain boundary (GB) tensile crack to shear crack ratio increases, and the corresponding fractures change from vertical or subvertical to an angle about 45° along the vertical direction. The propagation and coalescence of generated microcracks result in a transition of failure pattern from splitting to shear under uniaxial compression with a decreasing brittleness level in the rock. A transition from GB tensile crack to shear crack is also observed under direct tension when the brittleness index gradually decreases. The tension to shear transition mechanism is closely related to the relative strength of the mineral grain and mineral bonding. The relative strength of mineral and mineral bonding could be used as a parameter to characterize rock brittleness from a microscale viewpoint.