Selective laser melting (SLM) exhibits more and more utilization potentialities in today’s aviation industry. In order to ensure safe operation of SLM produced aero parts, it is important to have the fatigue performance well-characterized, and fatigue life prediction methods developed by considering the initial feature of this material. In this study, high cycle fatigue tests under stress ratios of −1, 0.1 and 0.5 at room temperature for the SLM Ti6Al4V alloy were carried out, and the fracture morphologies of fatigue specimens were observed. The results show that the subsurface pore is the main source of crack initiation for fatigue failure. By quantitative and statistical analysis of the pore characteristics in the crack source zone of all fractured specimens, it is revealed that the anisotropy and dispersion of fatigue performance are closely related to the difference of the shape, size, location and density of the pores in the crack source zone of specimens. The relationship between pore size and location, such as the ratio of radius to the distance to the specimen surface, plays an important role in estimating the fatigue life of the specimen. Aiming at the nature of defect-induced fatigue damage, the fatigue life of the SLM Ti6Al4V alloy was reasonably predicted using the fracture mechanics method based on the small-crack theory and the long crack growth data.