In order to use recycled materials in the composition of greener and cleaner concrete products, it is vital to ensure that the produced concrete have the appropriate level of strength. This paper investigates the effect of gradation (size) and percentage of recycled rubber granules on the physical parameters (slump and density), strength parameters (tensile and compressive strength values), and in particular the crack initiation and propagation characteristics (under the opening, shearing, and tearing modes). The tensile strength, compressive strength, fracture toughness (KIc, KIIc, and KIIIc), and fracture energy values at the initial crack initiation and ultimate failure propagation stages were determined by testing specimens containing 0, 1, 2, 4, and 12% of fine and coarse recycled rubber granules. Rubber particles were graded in two distinct ways so that they could serve as both filler and granules. In contrast to fine rubber granules, the addition of coarse granules had a positive effect on mode-II and mode-III fracture toughness. But, the addition of fine and coarse rubber granules reduces mode I fracture toughness. The analysis of fracture values indicates that coarse rubber granules have a positive effect on the post-peak failure energy absorption (i.e., the crack propagation stage) of all fracture modes. However, this increase in fracture energy for the fine rubber combination was only observed for mode-II fracture stress. The fracture toughness and fracture energy values of the similar concrete were empirically determined based on their corresponding tensile strength. The link between fracture toughness and fracture energy values was also discovered. According to the experimental results, the concrete containing 4% coarse rubber granules provides good mechanical and cracking resistance properties.