Climate change during the Holocene is thought to have had a significant impact on human migration and development. In Japan, the earliest prehistoric culture based on hunting and gathering began about 16,000 years ago (the Jomon era). At least 3000 years ago, people with paddy rice cultivation skills started to migrate from mainland China, causing an explosion of population growth, lifestyle changes, and changes to the level of civilisation (Yayoi era). To clarify the climatic changes that potentially led to the movement of the Yayoi people to Japan, we analysed the palaeotemperature changes recorded in coastal marine sediment cores obtained from the Zhejiang coast in China, where the first paddy rice cultivation occurred, and northern Kyushu Island, where it was introduced to Japan. The reconstructed temperature fluctuations over the past 7000 years from both sites were similar on the scale of hundreds to thousands of years. The cold periods that occurred approximately 300 and 4200 years ago, which are related to the Little Ice Age and 4.2 ka events, respectively, may have been caused by modulations of the East Asian summer monsoon. Our study suggests that enhanced coastal upwelling may have amplified the cold climate of the 4.2 ka event on the Zhejiang coast. Our results support the recently proposed hypothesis that climate change prompted the collapse of the Yangtze civilisation and the migration of paddy rice farmers to other parts of Asia, including Japan. In contrast, the Jomon people, who still survived mainly based on hunting and gathering at that time, might have avoided a major population collapse.