To evaluate the impact of coarse aggregate morphology on the mechanical performance of the asphalt mixture, the coarse aggregates with diverse wearing times were prepared and the corresponding asphalt mixture samples were compacted. The morphology characteristics of coarse aggregate under diverse wearing times were evaluated by an image measurement system. The dynamic modulus test and repeated loading creep tests were carried out to evaluate the viscoelastic properties and permanent deformation resistance of the asphalt mixture. The results indicated that the angularity index and three-dimensional sphericity could well characterize the wearing morphology. With the increase of wearing time, the angularity index of coarse aggregate gradually decreased, and the three-dimensional sphericity gradually increased. The balanced complex modulus Ge* and rheological parameter RG could effectively characterize the viscoelastic properties of asphalt mixture. As the angularity index increased, Ge* increased linearly, and RG decreased linearly. However, the Ge* and RG showed the opposite trend with the increase of three-dimensional sphericity. The flow number FN and the combination parameter εp/FN were the indexes to evaluate resistance to permanent deformation of the asphalt mixture. As the angularity index increased, the value of εp/FN decreased linearly, and FN increased linearly. With the increase of three-dimensional sphericity, εp/FN increases linearly and FN decreases linearly. The research results can provide a reference for the selection of coarse aggregate considering the morphological characteristics, and then improve the performance of asphalt mixture.