Using coal gangue as subgrade filler (CGSF) can address the accumulated issues of coal mine waste, but also save the constructing costs, which has the important ecological and engineering practical value. The well-graded limit of CGSF was captured based on the fractal model grading equation (FMGE). The large-scale vibration compaction test (LCT) shows that particle grading has a remarkable influence on the compaction of CGSF, with the increase of fine particle content, the dry density of sample first increases and then decreases. On this basis, the optimal range of particle grading was obtained. A series of large-scale triaxial tests (LTT) were conducted to investigate the mechanical behaviors of CGSF. The results shown that (1) the peak deviatoric stress shows a well-quadratic correlation with the fractal dimension. The optimal grading range of CGSF captured by LTT and LCT is basically the same, hence the suggestion of acquiring the optimal grading through LCT was given. (2) Increasing the compaction degree (DC) can significantly improve the strength of CGSF if DC is smaller, while when DC is greater, the strength improved by increasing DC will not be significant. (3) The grading and DC have a significant impact on the cohesive force of CGSF, while have little effect on internal friction angle. In addition, the research results provide good guidance for the constructing of a highway subgrade in practice.