Nanoindentation has been applied in the field of asphalt mixtures, but, at the nano-scale, changes in the composition of the mixture and material properties can have a significant impact on the results. Therefore, it is necessary to investigate the feasibility of nanoindentation tests on different types of asphalt mixtures with different gradations and the influence of material properties and test methods on nanoindentation results. In this paper, the nanoindentation test results on three kinds of asphalt mixture (AC-13, SMA-13, and OGFC-13) with different aggregate gradations were investigated. The load-displacement curves and moduli obtained from the nanoindentation tests were analyzed. In addition, nanoindentation tests were carried out before and after polishing with different ratios of filler and asphalt (RFA) (0.8–1.6). On this basis, the morphology of asphalt specimens with different RFAs is observed by scanning electron microscopy (SEM) imaging. The results indicate that using the nanoindentation test to characterize the mechanical behavior of asphalt mixture, the confidence level of the dense-graded mixture is low, and non-dense-graded mixtures are used as much as possible. Moreover, results illustrate that the nanoindentation modulus tends to increase as the RFA increases. and the SEM chart shows that the higher the mineral powder content in the mastic, the more complex the bitumen and mineral powder interaction surface, confirming the influence of mineral powder content on the nanoindentation test results. Furthermore, the effect of polishing is almost insignificant.