This study aimed to provide evidence on the harm-to-benefit ratio of fecal immunochemical test (FIT)-based colorectal cancer (CRC) screening by previous fecal hemoglobin (f-Hb) concentrations, as reflected in the number needed to screen (NNS) and number needed to scope (NNSc). Participants in up to 4 FIT screening rounds of the Dutch CRC screening program were included. The main outcomes of this study were the NNS and NNSc to detect 1 CRC and/or advanced neoplasia (AN) in screening rounds 2, 3, or 4, conditional on previous f-Hb concentrations. Outcomes were compared between participants using chi-square tests and logistic regression. In total, 2,428,883 study participants completed at least 2 consecutive FITs, 1,308,684 completed 3 FITs, and 150,958 completed 4 FITs. There were 31,400, 16,060, and 2007 ANs detected by round, respectively. The NNS for individuals with vs without a history of detectable f-Hb differed significantly irrespective of screening round. Individuals without detectable f-Hb in previous negative FITs had almost 9 times the NNS to detect 1 AN compared with those with detectable f-Hb (odds ratio, 8.71; 95% confidence interval, 8.51-8.92). A similar directional pattern was observed for NNSc, although the differences were smaller (odds ratio, 2.7; 95% confidence interval, 2.7-2.8). The harm-to-benefit ratio of FIT-based screening is substantially greater in individuals without vs with prior detectable f-Hb. Less intensive screening should be considered for this lower-risk group.