ContextWith the increasing global food and oil security problems, improvement in rapeseed (Brassica napus L.) yield become urgent. Moreover, previous studies rarely used the semi-dwarf and compact plant type of rapeseed to improve yield potential. ObjectiveHerein, we aimed to investigate the effects of different nitrogen application rates and plant density on root growth and yield-related traits using three genotypes with different plant types to clarify various regulatory mechanisms using semi-dwarf and compact plant type mutants to achieve high yield. MethodsSplit-split-plot experiments with three nitrogen application rates (N1: N2: N3, 120: 240: 360 kg ha−1) and three plant densities (D1: D2: D3, 15 × 104: 45 × 104: 75 × 104 plants·ha−1) were conducted during the three growing seasons (2019–2022) using semi-dwarf mutant HS5sca, and HS5 (wild type) as well as their F1 hybrid HS5+/sca. ResultsWith decreasing nitrogen application rate (N3 to N2), the expression level of IAA signal response genes BnaA07. GH3 and BnaA03. IAA13 was decreased in root, while increased in shoot at the early-flowering stage. Moreover, IAA content was decreased in roots and aboveground parts, whereas soluble sugar content of root bleeding sap was increased and volume of root bleeding sap was decreased in three genotypes at early-flowering stage. Additionally, nitrogen accumulation per plant and yield per plant was decreased, while population nitrogen accumulation and population yield were increased with decreasing nitrogen application rate in three studied genotypes. On the other side, with increasing plant density, the expression of BnaA07. GH3 and BnaA03. IAA13 genes were first increased and then decreased in root tissues. In addition, root surface area, the total volume of root bleeding sap, and nitrogen accumulation were decreased, which decreased dry matter and yield per plant. However, total root area, population nitrogen accumulation, and population yield in the three genotypes were increased, and yield was the highest at D2. The population yield of HS5sca, HS5+/sca, and HS5 achieved the largest increase of 26.0%, 10.3%, and 9.5% under N2D2 in 2019, 6.9%, 8.4%, and 4.7% under N3D2 in 2020, respectively, as compared to N3D1. Moreover, the morphological indices of HS5+/sca were greater than other genotypes, with the highest population yield. ConclusionsAppropriate nitrogen application rate with plant density optimized the root architecture and aboveground type of HS5+/sca that improved auxin level through affecting the downstream auxin-related genes, as well as improved dry matter accumulation and distribution, which increased yield potential. Compared with the two parents, F1 genotype HS5+/sca was suitable for nitrogen fertilizer (240 kg ha−1) and plant density (45–75 × 104 plants ha−1). Implications+ /sca heterozygous genotypes might promote root growth under low nitrogen application rate and high plant density. A strong root system is an important guarantee of high yield; thus, this heterozygous site could use to introduce excellent hybrids in rapeseed production to further improve root traits and optimize cultivation factors to increase seed yield.