As one of the main yield components, grain number per panicle (GNP) played critical role in the rice yield improvement. The identification of natural advantageous variations under different situations will promote the sustainable genetic improvement in rice yield. This study was designed to identify natural genotypes in a rice mini-core collection, to examine the genotypic effects across the indica and japonica genetic background in different environments, and excavating the superior genotypes that had drove the modern genetic improvement. The association mapping of GNP was carried out using a mini-core collection including 154 indica and 119 japonica accessions in seven different environments. Genotypic effects of each genotype for each QTL were calculated and genotype frequency distortion between the commercial rice cultivars and landraces was screened by χ2-test. In total, 74 QTLs containing stable and sensitive QTLs in various environments were detected. Within them, 20 positive and 24 negative genotypes in indica, and 24 positive and 16 negative genotypes in japonica were identified. When checking the accumulation of positive genotypes identified in indica across cultivars in each of the two subspecies, it indicated that increased number of positive genotypes identified in indica results in the substantially increased GNP in both indica and japonica across all of the environments, while this trend was not obvious for the positive genotypes identified in japonica especially in short day environments. Moreover, the positive and negative genotype frequency distortion between the landraces and commercial rice cultivars indicated that both positive selection of positive genotypes and negative selection of negative genotypes had driven the genetic improvement on GNP. Our findings suggested that the accumulation of positive genotypes and purifying negative genotypes played equivalently important roles in the improvement of rice yield, but the efficient use for some QTLs or genotypes depends on the comprehensive evaluation of their effect under diverse genetic backgrounds and environments.