Laser Powder Bed Fusion (LPBF) process, is becoming more and more widespread in industry. The possibilities of microstructural control offered by this process are an opportunity to study the contribution of the different length scales of microstructure to the fatigue behavior. This paper is devoted to the understanding of this fatigue behavior resulting from the interactions between the process induced defects and the different polycrystal length scales. Two distinct defect − microstructure competition regimes have been identified. The first concerns microstructures containing large Lack of Fusion (LoF) defects. These LoFs drastically reduce the fatigue life, while microstructure has no influence on the fatigue strength. The second regime concerns microstructures containing small defects. A limited effect of the polycrystalline microstructure was revealed. Furthermore, this paper demonstrates that the ratio between damage initiation defect size and grain size, used in literature to describe the defect and microstructure sensitivity of fatigue strength, is not applicable over a wide range of defects and microstructures, such as obtained by additive manufacturing processes. Finally, the comparison between the fatigue behavior of different microstructure and defect features shows that producing a finer microstructure improves fatigue strength despite the presence of a significant defect population.