To investigate the effect of waste glass material particle sizes on the mechanical and durability properties of concrete, a two-phase experimental approach was conducted. First, comprehensive tests were performed to examine the effects of glass sand and glass powder of different particle sizes on concrete performance. Subsequently, based on the experimental results, an orthogonal test was designed to optimize the replacement amounts of composite particle sizes. The results indicated that an appropriate amount of glass sand can enhance the mechanical properties and durability of concrete, while excessive amounts or larger particle sizes may have adverse effects. The pozzolanic effect of glass powder also contributes to performance improvement, but the replacement rate should be limited to 20%. Under composite particle size conditions, the compressive, tensile, and shear strengths of concrete increased by 35.56%, 21.74%, and 13.79%, respectively, while durability significantly improved, with water absorption reduced by 20.73% and chloride ion permeability decreased by 63.10%. At a total replacement rate of 20%, the optimal proportions were determined to be 2.86% for 0.6 mm glass sand, 1.43% for 1.18 mm glass sand, 8.57% for 50–60 μm glass powder, and 7.14% for 60–70 μm glass powder. The incorporation of composite particle sizes improved the microstructure of concrete, thereby enhancing its mechanical and durability properties.