To explore the internal driving force of the growth of rice yield per unit area in China, a model based on varying-coefficient production function is proposed in this study, which comes from the idea that the constant elasticity parameters in the Cobb-Douglas production function can be extended to functional forms. Applying such model to economic growth analysis, on the one hand, the dynamic contribution rate of each input factor can be measured, and, on the other hand, the contribution rate of the input factor can be decomposed into net factor contribution rate and interaction factor contribution rate, thus expanding the explanatory ability of growth rate equation. Using such model, the output elasticity of capital and labor in China’s rice yield growth are calculated from 1978 to 2020, and the dynamic characteristics of the contribution rate of capital, labor and generalized technological progress are analyzed. Next, the capital contribution rate is decomposed according to the composition of the total capital. The results show that: (1) The capital elasticity and labor elasticity are indeed not constant in different years. In China, from 1978 to 2020 the value of capital elasticity was between 0.3209 to 0.3589, with a mean of 0.3437, and the value of labor elasticity was between −0.1759 to −0.1640, with a mean of −0.1730. (2) Natural disasters do affect capital elasticity and labor elasticity in rice production. (3) When the annual proportion of crop disasters increases, the contribution rate of interaction between capital and natural disaster (KDR) value is negative, whereas the contribution rate of interaction between labor and natural disaster (LDR) value is positive. (4) Compared with 1978, the generalized technological progress contribution rate (GTPR) of the rice yield growth in China from 1979 to 2020 shows a declining trend in fluctuations, whereas the total capital contribution rate (TKR) shows a rising trend in fluctuations and the total labor contribution rate (TLR) is relatively stable in the same period. Since 2000, capital investment has become the main factor for the rice yield growth per unit area in China, of which machinery, chemical fertilizer, seed and pesticide are the four most important input factors.