The skeleton structure of asphalt mixture is the main carrier to resist load. At present, the identification and evaluation of skeleton contact chains need to be further investigated. This study quantitatively analyzes the mesostructural characteristics of asphalt mixture skeleton contact chains from the perspective of complex networks by using digital image processing (DIP) technique. A self-developed meso-structure intelligent identification software is developed, the evaluation parameters of the contact chain complex networks are proposed, and five types of asphalt mixture skeleton contact chains are compared. Additionally, the multi-scale relationship between the meso and macro parameters of the contact chains has been established, and a design flow chart of the asphalt mixture skeleton mesostructure based on rutting prediction has been proposed. The results indicate that the skeleton structure of the asphalt mixture has the characteristics of a scale-free network, and the degree distribution of each coarse aggregate is inhomogeneous, exhibiting a good power-law distribution; the length of the aggregate to aggregate contact chains conforms to the normal distribution, and the asphalt mixture skeleton clustering coefficient is close to zero; the smaller the skeleton modularity of the asphalt mixture, the greater the number of coarse aggregates within the longest contact chain and the fewer the modules, indicating that the asphalt mixture can form a more perfect skeleton structure. The measures to improve the skeleton performance of asphalt mixture include focusing on the passing rates of the 2.36 and 4.75 mm aggregates, obtaining a maximum amount of coarse aggregate in the contact chains, and achieving a smaller number of modules and modularity in the network. This study presents a fast and effective technical evaluation tool and detection method for asphalt mixture indoor design and pavement construction quality detection and also provides guidance for the fine design and performance improvement of asphalt mixtures.