Cold expansion (CE) serves as a practical surface enhancement to improve the fatigue life of hole structures by improving surface integrity in both macro-scale and micro-scale. Due to the inaccessibility and high cost of experimental measurements, the physical relation between surface integrity and fatigue life are always implicit, serving as the major challenge for accurate life prediction. To address this issue, a novel method is proposed by introducing physical information to traditional data-driven method, where surface integrity enriched by multi-scale simulation is mapped to fatigue life via machine learning (ML) mechanism. As integrated to four typical ML algorithms, the proposed physics-enhanced data-driven method exhibit outstanding capability for accuracy improvement, decreasing the scatter band by amplitude between 27.3 % and 71.4 %. The proposed method offers a promising option for fatigue life prediction on surface treated structures with limited physical information.