We hypothesized that common breast cancer risk alleles are associated with incidences of breast cancer and other cancers in the general population, and identify low risk women among those invited for screening mammography. About 35 441 individuals from the Danish general population were followed in Danish health registries for up to 21 years after blood sampling. After genotyping 72 breast cancer risk loci, each with 0–2 alleles, the sum for each individual was calculated. We used the simple allele sum instead of the conventional polygenic risk score, as it is likely more sensitive in detecting associations with risks of other endpoints than breast cancer. Breast cancer incidence in the 19 010 women was increased across allele sum quintiles (log-rank trend test; P = 1×10 − 12), but not incidence of other cancers (P = 0.41). Age- and study-adjusted hazard ratio for the fifth versus the first allele sum quintile was 1.82 (95% confidence interval; 1.53–2.18). Corresponding hazard ratios per allele were 1.04 (1.03–1.05) and 1.05 (1.02–1.08) for breast cancer incidence and mortality, similar across risk factors. In 50-year-old women, the starting age for screening mammography in Denmark, the average 5-year breast cancer risk was 1.5%, overall and 1.1%, 1.4%, 1.6%, 1.7%, 2.1%, for the first through fifth quintile, respectively. Based on age, nulliparity, familial history, and allele sum, 25% of women aged 50–69 years, and 94% of women aged 40–49 years, had absolute 5-year breast cancer risks ≤ 1.5%. Using polygenic risk score led to similar results. Common breast cancer risk alleles are associated with incidence and mortality of breast cancer in the general population, but not with other cancers. After including breast cancer allele sum in risk assessment, 25% of women currently being offered screening mammography had an absolute 5-year risk below the cutoff of average risk for a 50-year-old woman.