Exploring population dynamics and its driving factors has important practical significance for guiding reasonable population distribution. In view of this, this paper systematically analyzes the population dynamics and driving factors in China based on the latest three decennial censuses, using research methods such as the population concentration index, the center of gravity model, relative change in population density and multiple linear regression. The conclusions are as follows: (1) China’s population distribution is uneven, and the trend of polarization in population distribution is increasingly evident. The spatial differences in population growth are shifting from east–west to north–south. Under the influence of the “core–periphery effect”, more people are gathering in a few large cities. (2) The factors affecting population changes have obvious temporal variability: terrain and temperature have an increasing impact on China’s population changes. Temperature in particular has become an important factor in China’s population changes. Population changes are gradually shifting from being driven by a single economic factor to being driven jointly by social and economic factors. (3) The factors affecting population changes also have obvious spatial heterogeneity: temperature affects population changes in both the eastern and central–western regions, while terrain only affects population changes in the central–western regions. Currently, population changes in the economically developed eastern region are more driven by economic factors, while the central–western regions are driven by both economic and social factors. Central cities in the central–western regions are experiencing accelerated population agglomeration, while central cities in the eastern region are losing their ability to attract population agglomeration. The above conclusion basically clarifies the patterns and influencing factors of China’s population changes since the 21st century, which can provide a useful reference for future population development and regional planning.