Selenium (Se) accumulation in rice (Oryza sativa L.) has become a major global concern. Se offers multiple health benefits in humans; however, its inadequate or excessive intake can be harmful. Therefore, determining the factors driving Se abundance and bioavailability in paddy soils is essential to ensure the safety of human Se intake. This study investigated the accumulation, bioavailability, and distribution of Se in 820 paddy soil and rice grain samples from Luxi County, China to assess how soil properties (soil organic matter [SOM], cation exchange capacity [CEC], and pH), geographical factors (parent materials, elevation, and mean annual precipitation [MAP] and temperature [MAT]), and essential micronutrients (copper [Cu], zinc [Zn], and manganese [Mn]) govern Se accumulation and bioavailability in paddy soils. Results showed that the average soil Se content was 0.36 mg kg−1, which was higher than that in China (0.29 mg kg−1). Alternatively, the average rice grain Se content was 0.032 mg kg−1, which was lower than the minimum allowable content in Se-rich rice grains (0.04 mg kg−1). Five studied parent materials all had a significant effect on soil Se content but had little effect on Se bioavailability (p < 0.05). CEC, elevation, and SOM, as well as the soil contents of Cu, Zn, and Mn were positively correlated with soil Se content, but pH, MAP, and MAT were negatively correlated. Correspondingly, Se bioavailability was negatively correlated with SOM and soil Zn content, but positively correlated with MAP and grain contents of Cu, Zn, and Mn. Furthermore, partial least squares path analysis revealed the interactive impacts of the influencing factors on Se accumulation and bioavailability in soils. On this basis, prediction models were established to predict Se accumulation and bioavailability in paddy soils, thereby providing theoretical support for developing efficient control measures to meet Se challenges in agriculture.