The strength and shape of aggregates significantly affect the internal skeleton structure (ISS) of the asphalt mixtures, which affects the service life of asphalt pavement. ISS is a critical factor that decides the load transfer path in the asphalt mixture. To improve the performance of the asphalt mixture, three aggregates Granite, Diabase, and Limestone with different strengths and shapes were used. Two-dimensional image processing was used to acquire the ISS including contact properties, distribution and orientation of aggregates. Marshall, wheel tracking and three-point bending beam tests were carried out to evaluate the relationship between skeleton indexes with mechanical properties. Results indicated that the coarse aggregate voids filling method could obtain mineral gradations with the same volume combinations for the three aggregates. The crushing value, loss-angles value, roundness, and flakiness index have an excellent relationship with contact characteristics, showing that strength and shape significantly affect the ISS of the asphalt mixture. The results of Marshall, rutting, and bending stiffness modulus are consistent with the internal structural index, proving that low-temperature and high-temperature performance of asphalt mixture are directly affected by aggregate contact properties. There is a weak linear relationship of strength parameters with mechanical properties because Diabase is three times stronger than Granite but has a lower performance demonstrating that the strength of aggregate has a minimal effect on the performance and stability of the asphalt mixture. The results obtained in this study could be used to improve the utilization efficiency of aggregate.