Fish-scale-like melt pool structures and internal defects are unique characteristic in additively manufactured (AM) metals. These features play a critical role in the damage and fracture processes under different service conditions, governing the material's performance. However, the relationship between these damage features and loading conditions, as well as the spatial interactions between melt pool structures and internal defects remains incompletely understood. By in situ time-lapse synchrotron X-ray tomography and diffraction, we identify the initiation and growth events of life-limiting damage under tensile, low cycle fatigue (LCF), and high cycle fatigue (HCF) loading. A novel transition from meso-structure insensitive, defect-dominated short fatigue crack propagation to a meso-structure sensitive mechanism occurs as the plastic zone expands of a growing crack. Under tension and LCF, the damage accumulation gradually increases and micro-voids nucleate at the melt pool boundaries (MPBs) after which the crack path follows the MPBs. In contrast, under HCF, surface defects initiate fatigue cracking and the MPBs have a very limited effect on the crack propagation path. Finally, physics-informed machine learning method is introduced to develop a novel methodology for predicting fatigue life by including three-dimensional features of defects in AM parts.