High selenium (Se) and low cadmium (Cd) and arsenic (As) contents in rice grains were good for human health. The genetic basis and relationship of Se, Cd and As concentrations in rice grains are still largely unknown. In the present study, large variations were observed in Se, Cd and As concentrations in brown and milled rice in normal and Se treatment conditions in 307 rice accessions from 3K Rice Genomes Project. Se fertilizer treatment greatly increased Se concentrations but had no obvious changes in concentrations of Cd and As both in brown and milled rice. Total of 237 QTL were identified for Se, Cd and As concentrations in brown and milled rice in normal and Se treatment conditions as well as ratio of concentrations under Se treatment to normal conditions. Only 19 QTL (13.4%) were mapped for concentrations of Se and Cd, Se and As, and Se, Cd and As in the same or adjacent regions, indicating that most Se concentration QTL are independent of Cd and As concentration QTL. Forty-three favorable alleles were identified for 40 candidate genes by gene-based association study and haplotype analysis in 14 important QTL regions. Se-enriched rice variety will be developed by pyramiding favorable alleles at different Se QTL and excluding undesirable alleles at Cd and As QTL, or combining favorable alleles at Se QTL with the alleles at Se-sensitive QTL by marker-assisted selection.