Determination of the concentration of nutrients in the plant is key information for evaluating crop nutrient removal, nutrient use efficiency, fertilizer recommendations guidelines, and in turn for improving food security and reducing environmental footprints of crop production. Diffuse infrared (IR) reflectance spectroscopy is a powerful, rapid, cheap, and less pollutant analytical tool that could be substituted for traditional laboratory methods for the determination of the concentration of nutrients in plants. However, its accuracy for predicting the concentration of nutrients in rice plants is poorly known. This study aimed i) to determine macro- and micronutrients concentration that can be accurately predicted by near-infrared (NIR, 7498–4000 cm−1), mid-infrared (MIR, 4000–600 cm−1), or their combination (NIR-MIR, 7498–600 cm−1) spectra, ii) to identify the most suitable spectral range with the best prediction potential for the simultaneous analysis of nutrients concentrations in rice plants (straw and paddy) and iii) to assess the influence of agro-ecological zone and production system on nutrients concentrations in straw and paddy (unhulled grains) samples. Second-derivative spectra were fitted against plant laboratory reference data using partial least-squares regression (PLSR) to estimate six macronutrients (N, P, K, Ca, Mg, and S) and seven micronutrients (Na, Fe, Mn, B, Cu, Mo, and Zn) concentration in paddy and rice straw samples collected at harvest from 1628 farmers’ fields in 20 sub-Saharan African (SSA) countries. The modeling prediction potential was assessed by coefficient of determination (R2), root mean square error (RMSE), mean absolute error (MAE), and ratio of performance to interquartile distance (RPIQ). Good prediction models (0.75 < R2 ≤ 0.95) were obtained for 7 nutrients concentrations consisting of N, P, K, Ca, Mg, Mn, and Cu. Satisfactory predictions (0.62 ≤ R2 ≤ 0.75) were obtained for S, Fe, and B. NIR, MIR, and combined NIR-MIR diffuse reflectance spectroscopy demonstrated the best prediction potential for 3, 1, and 6 of these 10 well-predicted nutrients concentrations, respectively. All nutrients concentrations both in straw and paddy were moderate to highly variable (CV = 15–111%). Agro-ecological zone and production system had a significant impact on most nutrients concentrations both in rice straw and paddy. N, P, and K concentrations in rice in irrigated lowland (IL) fields were higher than in rainfed lowland (RL) and upland (RU). From the studied fields, 2%, 16%, and 16% of straw samples were deficient in N, P, and K, respectively. K deficiency occurred in all three production systems, whereas P deficiency mainly occurred in the rainfed upland systems. Overall, the combined NIR-MIR diffuse reflectance spectroscopy has a good potential to be applied as an alternative method in the determination of macronutrient concentrations in rice plants. Further investigation on the relationships between soil attributes, rice productivity, and nutrients concentration data determined in this study is needed for providing fundamental information for site-specific soil management and fertilizer recommendations in SSA.