The use of controlled-release urea (CRU) has been recommended over that of conventional urea to improve rice grain yield and nitrogen use efficiency. However, the underlying agronomical and physiological mechanisms need to be better understood. In this study, field trials over four site-years, and a big container experiment were carried out to explore CRU effects on rice yield and NUE, with the main aims to identify the key yield components contributing to the superior rice yield with CRU use, and to evaluate differences in dry matter, nitrogen (N) accumulation, translocation and yield formation with different N fertilizer practices. Four N treatments were investigated: control with 0 kg N ha−1 (CK), farmers' fertilizer practice (FFP) with 150 kg N ha−1 as urea basal application, modified fertilizer practice (MFP) with 150 kg N ha−1 as split urea application (40% at transplanting, 30% at tillering and 30% at the panicle stages), and CRU treatment with 150 kg N ha−1 as CRU basal application. Results showed that the CRU increased rice yields by 10.8 and 5.6% over FFP and MFP, respectively. The N recovery efficiency and N agronomic efficiency for CRU were significantly higher than that obtained from MFP and FFP treatments. The analysis of yield components revealed that the higher grain yields using CRU were accounted for mainly by increased panicle and spikelet numbers per m2, which resulted from higher N uptake. In addition, results from the container experiment with comparable experimental design to field trials illustrated that both post-anthesis dry matter production and translocation were critical for high grain yields using CRU, while the former seemed more important. Relative to MFP and FFP, CRU maintained higher flag leaf SPAD and photosynthetic rate, as well as higher root oxidation activity (ROA) and N uptake during grain filling. Furthermore, CRU increased the activities of key enzymes involved in N assimilation in flag leaves, including GS, GOGAT, and NR. CRU effects on such underground and aboveground processes were proposed to contribute to high rice yield.