The adsorption of polymers affects the cost and oil recovery in oil reservoir exploitation and the flocculation effect in the treatment of oil sand tailings. The adhesion and adsorption of a hydrophobically modified polyacrylamide (HMPAM), i.e., P(AM-NaAA-C16DMAAC), on silica and asphaltene were investigated using surface force measurements, thermodynamic analysis and quartz crystal microbalance with dissipation (QCM-D) measurement. Our study indicates that HMPAM polymer has strong interaction with both silica and asphaltene. The adhesion force of HMPAM on silica was stronger than that on asphaltene surface. Consistently, the adsorption of HMPAM was also greater on silica surface, with a more rigid layer formed on the surface. For HMPAM/silica system, the attractive interaction and the strong adhesion are mainly driven by the hydrogen bonding and electrostatic interaction. For HMPAM/asphaltene system, it is mainly due to hydrophobic interaction between the long hydrocarbon chains of HMPAM and asphaltene. Furthermore, continuous adsorption of HMPAM was detected and multiple layers formed on both silica and asphaltene surfaces, which can be attributed to the hydrophobic chains of HMPAM polymers. This work has illustrated the interaction mechanism of HMPAM polymer on hydrophilic silica and hydrophobic asphaltene surfaces, which provide insight into the industrial applications of hydrophobically modified polymer.