Understanding the fatigue behavior and accurately predicting the fatigue life of laser powder bed fusion (L-PBF) parts remain a pressing challenge due to complex failure mechanisms, time-consuming tests, and limited fatigue data. This study proposes a physics-informed data-driven framework, a multimodal transfer learning (MMTL) framework, to understand process-defect-fatigue relationships in L-PBF by integrating various modalities of fatigue performance, including process parameters, XCT-inspected defects, and fatigue test conditions. It aims to leverage a pre-trained model with abundant process and defect data in the source task to predict fatigue life nondestructively with limited fatigue test data in the target task. MMTL employs a hierarchical graph convolutional network (HGCN) to classify defects in the source task by representing process parameters and defect features in graphs, thereby enhancing its interpretability. The feature embedding learned from HGCN is then transferred to fatigue life modeling in neural network layers, enabling fatigue life prediction for L-PBF parts with limited data. MMTL validation through a numerical simulation and real-case study demonstrates its effectiveness, achieving an F1-score of 0.9593 in defect classification and a mean absolute percentage log error of 0.0425 in fatigue life prediction. MMTL can be extended to other applications with multiple modalities and limited data.