The performance of cement-based materials can be controlled and tailored by adjusting the characteristics of reinforced nano inclusions. Therefore, the lateral size effect of graphene on the nano/micro-structures of ultra-high performance concrete (UHPC) was explored, and then the mechanical properties were investigated to analyze the structure–property correlation of composites in this paper. The test results show that due to nucleation site effect and the formation of core–shell elements, incorporating graphene with lateral size of > 50 µm improves the polymerization degree and mean molecule chain length of C-S-H gel by 242.6% and 56.3%, respectively. Meanwhile, the porosity and average pore volume of composites is reduced by 41.4% and 43.4%. Furthermore, graphene can effectively inhibit the initiation and propagation of cracks by crack-bridging, crack-deflection, pinning and being pulled-out effect, and the wrinkling characteristic is conductive to the enhancement of pinning effect. These improvements on nano- and micro- structures result in that the compressive strength, compressive toughness and three-point bending modulus of UHPC are increased by 43.5%, 95.7% and 39.1%, respectively, when graphene with lateral size of > 50 µm and dosage of 0.5% is added. Compared to graphene with lateral size of > 50 µm, graphene with average lateral size of 10 µm has less folds and larger effective size, then reducing the distance between core–shell elements. Hence, the addition of graphene with average lateral size of 10 µm leads to 21.1% reduction for Ca(OH)2 crystal orientation index, as well as 30.0% increase for three-point bending strength. It can be, therefore, concluded that the lateral size of graphene obviously influences the nano/micro-structures of UHPC, thus leading to the significantly different reinforcing effects of graphene on mechanical behaviors of UHPC.