This paper introduces a method for predicting the fatigue life of single crystals, which fuses image data and physical data by exploiting machine learning techniques. The proposed approach establishes a comprehensive study by progressively contributing a new database to the field, enabling the inference of service conditions, including temperature and σmax, as well as inherent material properties from fracture images, such as crystal orientation. We evaluate the effectiveness of the proposed method via its application to the fatigue life prediction of DD6 alloy, which achieves promising performance across various crystal orientations. The extensive experimental analysis and evaluation suggest that our proposed method has high predictive accuracy and computational efficiency of our proposed method. These results show that our approach holds promise for enhancing our understanding of material fatigue and improving predictive precision in failure analysis.