This study aims to provide evidence on the harm-to-benefit ratio of Fecal Immunochemical Test (FIT)-based colorectal cancer (CRC) screening by previous fecal Hemoglobine (f-Hb) concentrations, as reflected in Number Needed to Screen (NNS) and Number Needed to Scope (NNSc). Participants in up to four FIT screening rounds of the Dutch CRC screening program were included. The main outcomes of this study were the NNS and NNSc to detect one CRC and/or AN in screening round two, three or four, conditional on previous f-Hb concentrations. Outcomes were compared between participants using chi-squared tests, and logistic regression. In total, 2,428,883 study participants completed at least two consecutive FITs, 1,308,684 completed three FITs, and 150,958 completed four FITs. There were 31,400, 16,060, and 2,007 AN detected by round, respectively. The NNS for individuals with vs without a history of detectable f-Hb differed significantly irrespective of screening round. Individuals without detectable f-Hb in previous negative FITs had almost nine times the NNS to detect one AN compared to those with detectable f-Hb (OR 8.71, 95%CI 8.51-8.92). A similar directional pattern was observed for NNSc, although the differences were smaller (OR 2.7, 95%CI 2.7-2.8). The harm-to-benefit ratio of FIT-based screening is substantially greater in individuals without vs. with prior detectable f-Hb. Less intensive screening should be considered for this lower-risk group.