Human activity has transformed the Earth’s landscapes pervasively for thousands of years, and the most important anthropogenic alteration was the clearing of forests and the establishment of agriculture. As a center of rice domestication and early population growth, the lower Yangtze River has been extensively transformed in the Holocene. However, the timing, extent, and process of deforestation and its relationship with the intensification of rice cultivation remain controversial. Here, four representative archaeological sites ranging from 8,100 to 4,300 cal a BP, that is, Jingtoushan, Hemudu, Yushan, and Xiawangdu sites, were selected for detailed palynological analysis, and evidence of anthropogenic deforestation and subsistence strategy were also synthesized to investigate dynamic human–forest interaction. Although natural vegetation had already been altered at the Jingtoushan site around 8,000 cal a BP, it was more likely to be the management of acorns by limiting burning to open habitats and increasing yield. As the subsistence shifted from acorn exploitation toward rice cultivation after 6,000 cal a BP, real deforestation for agriculture may occur at the Yushan and Xiawangdu sites due to conflict on labor input and land use. However, these deforestations were just confined to the archaeological sites at local scale, and no consistent vegetation change occurred at regional scales induced by human activities until the last 3,000 years.