Grazing exclusion has been widely implemented in degraded grassland. However, the changes of plant communities and soil nutrients in response to fencing are still controversial. Thus, the effects of free grazing, 17 and 36 years of fencing on the plant biomass and litter biomass, carbon (C), nitrogen (N) and phosphorus (P) concentrations and stocks of plant, litter and soil were investigated in the temperate steppe grasslands of northern China. The results indicated that fencing increased the aboveground live biomass and litter biomass. In addition, fencing increased C, N and P stocks of aboveground live biomass, litter biomass and soil. Although root biomass and its nutrient stocks were also significantly increased by 17 years of fencing, they were decreased with fencing extending from 17 to 36 years. Moreover, there were no significant differences in aboveground live biomass and soil N and P stocks between 17 and 36 years of fencing. Litter biomass and its C, N and P stocks were positively correlated with soil C, N and P stocks. Our results demonstrated that 17 years of fencing is an effective way to restore vegetation and soil nutrients in the temperate steppe of Inner Mongolia, but a longer fencing duration has no further positive effects on biomass production and soil nutrients accumulation.