A microstructure-based viscoplastic continuum model is developed for the permanent deformation of asphalt concrete (AC). The model accounts for several phenomena that influence the permanent deformation of AC at high temperatures. These phenomena include strain rate dependency, confining pressure dependency, dilation, aggregate friction, anisotropy, and damage. The material anisotropy was included in the model by replacing the stress invariants in the yield function with invariants of both the stress and a microstructure tensor, which describes the aggregate orientation distribution. The components of the microstructure tensor were determined using image analysis techniques (IAT) conducted on digital images taken from two-dimensional cut sections of AC. Furthermore, damage was included in the model based on the effective stress theory to account for crack and air void growth that significantly reduces the load-carrying capacity of the material. Experimental data from triaxial compressive strength tests conducted at five strain rates and three confining pressures were used to develop a methodology to determine the material parameters that characterize AC permanent deformation. The model predictions were in a good agreement with the experimental measurements.