The surface free energy (SFE) of aggregate and asphalt binder can be utilized to quantify the adhesion between the aggregate and binder. The SFE value varies with the changing surface characteristics. This study investigated the effect of surface texture on the aggregate’s SFE when applying the contact angle measurement. A surface treatment protocol for granite, diabase, basalt, and limestone was designed. A dicing saw, grinder and abrasive paper were utilized and were combined into three groups to treat the aggregates’ surface — Groups A, B, and C. The group in which aggregate was not treated was labeled Group D. The aggregate image measurement system (AIMS) was employed to quantify the surface characteristic. After surface treating, four levels of surface texture index were obtained. The surface texture indexes of the treated aggregates followed an increasing rank: A < B < C < D. A higher surface texture index indicated higher surface roughness. Subsequently, the sessile drop method was utilized to measure the contact angle and obtain the SFE of the aggregates that been treated through Groups A, B, and C. The gas adsorption method was applied to determine the SFE of aggregates in Group D. With the increasing surface texture index, the contact angle also increased and resulted in an increasing SFE value. An exponential-type model was established to describe the relationship between surface texture index and SFE. The model was first employed to fit the SFE data of aggregates in the A, B, C, and D groups, which showed good satisfaction of model fitting. The model was also applied to fit the SFE data of aggregates in the A, B, and C groups and to predict the SFE values based on the surface texture index obtained in Group D, which indicated good predictability. The fitting parameters showed a variable coefficient within 20% in the two fitting methods, which confirmed the model was reliable.