The use of microwave enhanced material as aggregate for asphalt mixture has emerged as a crucial focus in improving the microwave heating performance of asphalt mixture. However, there is a marginal diminishing effect in microwave heating, which limited further improvement in the microwave heating property of the asphalt mixture. In order to explain the mechanism of the marginal diminishing effect, a three-phase (microwave enhanced aggregate, ordinary aggregate, asphalt sand) microscopic model of asphalt mixture was established using MATLAB and COMSOL based on the random aggregate placement and finite translation algorithm. The distribution of electromagnetic field, temperature field and the heat transfer efficiency were analyzed to investigate the mechanism of microwave enhanced aggregates influencing microwave heating efficiency. The results show that the microwave enhanced aggregate changes the heating efficiency of non-microwave enhanced phases (ordinary aggregate, asphalt sand) as follows: firstly, the change of electromagnetic fields distribution may affect the electromagnetic loss power of non-microwave enhanced phases. The other is that the heat transfer rate from microwave enhanced aggregate to the non-microwave enhanced phase will affect the heating efficiency of asphalt mixture. However, it was found that the self-generated heat of non-microwave enhanced phase accounts for over 80% of its internal energy increment. The heat transfer rate of microwave enhanced aggregates decreased as its substitute amount increased. The decrease in impedance matching of asphalt mixture with air and heat transfer efficiency are the two main reasons for the marginal diminishing effect of asphalt mixture containing microwave enhanced aggregates.