Conventional asphalt pavement is the dominant mode of passenger and freight traffic in Pakistan. As a result, asphalt pavements suffer from various failures, where rutting, corrugation, and fatigue cracking are significant. Fine aggregates and mineral fillers play a pivotal role in providing structural integrity in asphalt pavements when subjected to traffic and the environment. The current study aims to examine the effects of various locally accessible fine aggregate and mineral filler materials on the interlocking properties of asphalt mixtures in relation to internal friction angle, rutting resistance, and controlling environmental pollution as an indirect benefit, thereby reducing wastes. Four distinct asphalt samples were prepared using cinders, stone dust, natural sand, and surkhi as fine aggregates and mineral fillers, as a full replacement, as per ASTM D1559, confirming the Asphalt Institute’s gradation for asphalt wearing course. Optimum binder contents (OBC) of 4.40%, 4.1%, 6.57%, and 6.63% by weight of Marshall specimen were concluded for asphalt samples containing stone dust, natural sand, cinder, and surkhi, respectively. The results revealed that surkhi, natural sand, stone dust, and cinder all showed a diminishing tendency in developing interlocking properties in asphalt mixtures at internal friction angles of 35°, 33.7°, 32°, and 28.4°, respectively. The wheel tracking test results revealed that the asphalt samples made with surkhi as fine aggregates and fillers have the highest rut resistance, whereas samples made with cinders as fine aggregates and fillers have the lowest rut resistance. The direct shear test showed that fine aggregates with a larger angle of internal friction are significantly more stable in terms of rut resistance than fine aggregates with a smaller angle of internal friction. The current research will help to prevent pavement rutting and corrugation by adding surkhi into asphalt pavements, with the reduction in brick kiln waste providing an indirect benefit.