Understanding the dominant variables driving soil respiration is critically important for predicting soil CO2 emission and assessing the carbon balance of forest ecosystems. In a small catchment of the semiarid Loess Plateau in China, soil respiration and soil biophysical factors were studied on sites of five forest types, comprising plots established in a pure Pinus tabulaeformis plantation, a pure Robinia pseudoacacia plantation, a mixed P. tabulaeformis and R. pseudoacacia plantation, a pure Platycladus orientalis plantation, and a natural Populus davidiana stand. Soil temperature at the 10 cm depth was found to be the most predominant factor controlling the temporal pattern of soil respiration, accounting for 11–40% seasonal variation in the rate of soil CO2 efflux across forest types. By applying an empirical model and the calculated temperature sensitivity of soil respiration (Q10) and the rate of basal soil respiration (R10), annual soil CO2 emission was estimated separately for each forest type using the automatically monitored data of soil temperature at the 10 cm depth. The annual soil CO2 emission varied greatly with forest types and ranged from 647.71 g C m−2 y−1 in the P. orientalis plantation to 1448.50 g C m−2 y−1 in the natural P. davidiana stand. Annual soil CO2 efflux is better predicted by soil organic carbon content and the amount of carbon in roots, litter and top soil than soil temperature when data are pooled for all plots of the five forest types. A first order exponential analysis indicates that about 77% of the variation in annual soil CO2 efflux is explained by root carbon stock, 63% by the combined carbon stock in roots, litter, and top soil, and 48% by the combined carbon stock in litter and top soil. We conclude that annual soil CO2 efflux can be predicted by carbon pools in roots and soils rather than by soil temperature in watersheds where spatial variation in soil temperature is relatively small in the semiarid Loess Plateau of China.