Seed number is one of the key traits related to plant evolution/domestication and crop improvement/breeding. In rapeseed germplasm, the seed number per pod (SNPP) shows a very wide variation from several to nearly 30; however, the underlying causations/mechanisms for this variation are poorly known. In the current study, the genetic and cytological bases for the natural variation of SNPP in rapeseed was firstly and systematically investigated using the representative four high-SNPP and five low-SNPP lines. The results of self- or cross-pollination experiment between the high- and low-SNPP lines showed that the natural variation of SNPP was mainly controlled by maternal effect (mean = 0.79), followed by paternal effect (mean = 0.21). Analysis of the data using diploid seed embryo–cytoplasmic–maternal model further showed that the maternal genotype, embryo, and cytoplasm effects, respectively, explained 47.6, 35.2, and 7.5% of the genetic variance. In addition, the analysis of combining ability showed that for the SNPP of hybrid F1 was mainly determined by the general combining ability of parents (63.0%), followed by special combining ability of parental combination (37.0%). More importantly, the cytological observation showed that the SNPP difference between the high- and low-SNPP lines was attributable to the accumulative differences in its components. Of which, the number of ovules, the proportion of fertile ovules, the proportion of fertile ovules to be fertilized, and the proportion of fertilized ovules to develop into seeds accounted for 30.7, 18.2, 7.1, and 43.9%, respectively. The accordant results of both genetic and cytological analyses provide solid evidences and systematic insights to further understand the mechanisms underlying the natural variation of SNPP, which will facilitate the development of high-yield cultivars in rapeseed.