The knowledge of mineral filler characteristics and their impacts on filler–asphalt interfacial behavior is not systematic and completed yet. In this paper, eight mineral fillers prepared using the same milling procedure were investigated. The filler–asphalt interfacial behavior was analyzed by differential scanning calorimetric and dynamic shear rheometer tests. Results reveal that in contrast to form factor and specific surface area, particle porosity, particle density, aspect ratio, angularity index, fractal dimension and feature roughness decrease with decreasing particle size. Limestone filler exhibits more regular shape, less significant angularity and richer surface texture than basalt filler. Of all particle characteristics studied, form factor and specific surface area are suggested to be the determining factors affecting filler–asphalt interfacial behavior. Moreover, microscopic morphology analysis provides an insight to interpret the differences explored in mastic interfacial property. The results help employ applicable particle characteristics that fabricate mastic with a stable interface system.