This research sought to evaluate the effects of incorporating extruded polystyrene (XPS) into porous asphalt mixtures and to understand the role of various filler combinations, specifically coconut shell ash (CSA) and rice husk ash (RHA), on the mixture’s performance. For this purpose, asphalt mixtures were subjected to the Cantabro test under both unconditioned and conditioned modes. Mixtures were treated with XPS in varying proportions ranging from 7% to 11%. The filler combinations were further varied, with mixtures containing 7% XPS showing Cantabro loss results of 35.71% for 1SD:3CSA and 43.44% for 1SD:1RHA, and those with 11% XPS demonstrating significant reductions to 19.24% and 18.93% respectively. The findings indicate that introducing XPS notably enhances the mixture’s stability and resistance to abrasion and disintegration. CSA-based mixtures generally showcased a more pronounced positive effect on stability than those with RHA. When subjected to moisture conditioning, all mixtures exhibited an increased sensitivity, though the 1SD:1RHA mixture showed the highest resilience against moisture damage. In conclusion, integrating XPS in porous asphalt mixtures can lead to significant performance improvements. The choice of filler, its proportion, and its interaction with XPS are critical factors in determining the mixture’s overall performance, especially its resistance to moisture and abrasion.