This study proposes a mechanical model for the shear modulus of asphalt mixtures based on granular mechanics theory. An aggregate interface shear test is conducted to validate the proposed model with the asphalt mixture particle shear analysis system and a pressure film. Results show that the pressure film can measure accurately the contact area, stress distribution, and total stress of the contact interface. It is demonstrated that more than 65% of the contact area bears only 37.5% of the total load, and the contact interface exhibits an uneven stress distribution. The binder type has a significant influence on the shear force at the contact interface. As the penetration of the asphalt decreases, the softening-point temperature increases; as the rutting factor increases, the tangential elastic stiffness also increases. The proposed mechanical model can distinguish the influences of the binder and gradation on the shear modulus of asphalt mixtures.