Recently developed ‘super’ rice cultivars with greater yield potentials often suffer from the problem of poor grain filling, especially in inferior spikelets. Here, we studied the activities of enzymes related to starch metabolism in rice stems and grains, and the microstructures related to carbohydrate accumulation and transportation to investigate the effects of different water regimes on grain filling. Two ‘super’ rice cultivars were grown under two irrigation regimes of well-watered (WW) and alternate wetting and moderate soil drying (AWMD). Compared with the WW treatment, the activities of ADP glucose pyrophosphorylase (AGPase), starch synthase (StSase) and starch branching enzyme (SBE), and the accumulation of non-structural carbohydrates (NSCs) in the stems before heading were significantly improved, and more starch granules were stored in the stems in the AWMD treatment. After heading, the activities of α-amylase, β-amylase, sucrose phosphate synthase (SPS) and sucrose synthase in the synthetic direction (SSs) were increased in the stems to promote the remobilization of NSCs for grain filling under AWMD. During grain filling, the enzymatic activities of sucrose synthase in the cleavage direction (SSc), AGPase, StSase and SBE in the inferior spikelets were increased, which promoted grain filling, especially for the inferior spikelets under AWMD. However, there were no significant differences in vascular microstructures. The grain yield and grain weight could be improved by 13.1 and 7.5%, respectively, by optimizing of the irrigation regime. We concluded that the low activities of key enzymes in carbon metabolism is the key limitation for the poor grain filling, as opposed to the vascular microstructures, and AWMD can increase the amount of NSC accumulation in the stems before heading, improve the utilization rate of NSCs after heading, and increase the grain filling, especially in the inferior spikelets, by altering the activities of key enzymes in carbon metabolism.