Graphene is an auspicious nanomodified material for asphalt pavements. This study aims to evaluate the effect of graphene composites (GCs) on the adhesion characteristics of asphalt based on surface-free energy (SFE) theory. The surface-free energy of graphene-modified asphalt was measured by the sessile drop method. The work of adhesion and stripping between asphalts and two kinds of aggregates were calculated through using their surface-free energy components. The macroadhesion property between asphalts and aggregates was obtained by the improved boiling method. Immersion Marshall test and freeze–thaw splitting test were conducted to verify the effectiveness of surface-free energy indexes. The results demonstrated that GC significantly increased total surface-free energy and dispersion components of asphalt but reduced the polarity components. The adhesion work between asphalt and both aggregates increased after modification, whereas the stripping work decreased after modification. GC can substantially improve the adhesion and resistance to moisture damage at the asphalt–aggregate interface, and the effect on asphalt-limestone is more obvious. There are strong correlations between macro evaluation indicators and surface-free energy calculated indexes, where the macro indicators are more strongly correlated with stripping work. Limestone is generally better at resisting moisture damage than granite. GC remarkably improved the water stability of the mixture, which further indicates that energy indexes are a good method to quantize the moisture damage resistance of hot mix asphalt (HMA). The high dosage of graphene-modified asphalt is potentially applicable to porous asphalt concrete (PAC) pavement with extra-large voids.