Climate change plays a critical role in crop yield variations, which has attracted a great deal of concern worldwide. However, the mechanisms of how climatic trend and fluctuations affect crop yields are not well understood and need to be further investigated. Thus, using the GIS-based Environmental Policy Integrated Climate (EPIC) model, we simulated the yields of major crops (i.e., wheat, maize, and rice) and evaluated the impacts of climatic factors on crop yields in the Agro-Pastoral Transitional Zone (APTZ) of northern China between 1980 and 2010. The partial least squares regression model was used to assess the contribution rates of climatic factors (i.e., precipitation, photosynthetically active radiation (PAR), minimum temperature (T min), maximum temperature (T max)) to the variation of crop yields. The Breaks for Additive Season and Trend (BFAST) model was adopted to decompose the climate factors into trend and fluctuation components, and the relative contributions of climate trend and fluctuation were then evaluated. The results indicated that the contributions of climatic factors to yield variations of wheat, maize, and rice were 31.7, 37.7, and 23.1%, respectively. That is, climate change had larger impacts on maize than wheat and rice. More cultivated areas were significantly and positively correlated with precipitation than with other climatic factors due to the limited precipitation in the APTZ. Also, climatic trend component had positive impacts on crop yields in the whole region, whereas the climate fluctuation was associated mainly with the areas where the crop yields decreased. This study helps improve our understanding of the mechanisms of climate change impacts on crop yields, and provides useful scientific information for designing regional-scale strategies of adaptation to climate change.