The interfacial characteristics between fiber-modified asphalt mastics and aggregates play a crucial role in the performance of asphalt mixtures. To explore these interfacial interaction characteristics, three types of fibers (basalt, glass, and polyester), five levels of fiber content (ranging from 0.1% to 0.5% by mixture mass), and two types of aggregates (limestone and basalt) were selected, and an optimal mixture design was established. Contact angle tests, binder bond strength tests, and disk-shaped compact tension tests were conducted to investigate the impact of fiber type, fiber content, and aggregate type on interface characteristics. The results indicate that the addition of fibers can enhance the interfacial strength, with basalt fibers exhibiting superior reinforcement effects. As fiber content increases, the influence of aggregate type on interfacial strength decreases. Under ambient conditions, a composite failure mode was observed, while at low temperatures, cohesive failures dominated. The incorporation of fibers led to an increase in interfacial cohesion strength. The interfacial interaction strength between fiber asphalt mastic and aggregates exhibits a favorable linear correlation with the fracture energy and peak crack mouth opening displacement of the asphalt mixture. Overall, this study elucidates the microscopic mechanisms involved in fiber asphalt mixtures, laying the foundation for further in-depth research.