Laser powder bed fusion (LPBF) is one of the high-precision additive manufacturing techniques for producing complex 3D components. It is well known that defects appear in additive-manufactured parts, and they deeply affect the fatigue properties; even heat treatment is performed after printing. In order to meet the safe-life design requirements of additive-manufactured aircraft structures, the effects of build direction and heat treatment on defects and fatigue properties need to be quantified. Hence, Ti6Al4V alloy samples with different build directions were designed and printed by LPBF. X-ray computed tomography was used to quantitatively analyze the defect size, the sphericity, and the defect orientation. And their effects on fatigue properties were studied. An extended effective defect size and a defect-based fatigue anisotropy evaluation process are proposed to qualify the effects of the defect size, sphericity, and defect orientation. It is shown that the build direction can affect the porosity distribution and maximum defect size, while the annealing treatment can cause the coalescence of small defects and higher porosity. The defect orientation exhibited a fluctuating trend of 0°–90°–0°–90°–0° as the volume increased. The elongated lack of fusion defects related to the build direction was the main crack source and could lead to fatigue anisotropy of LPBF Ti6Al4V.