Holocene temperature changes and their forcings serve as pivotal references for current and future warming trends. However, significant discrepancies exist between proxy reconstructions and model simulations of Holocene temperature evolution. Pollen evidence, often central to these discrepancies, have been criticized for potentially reflecting human influence rather than pure temperature variations, complicating our understanding of Holocene temperature changes. Our study focuses on southern China, a region with pronounced discrepancies between models and proxies. We introduce and validate a novel methodology to isolate genuine temperature signals from pollen data. This approach employs an arboreal pollen-based temperature index and correct biases inherent in raw pollen data using the Regional Estimates of Vegetation Abundance from Large Sites (REVEALS) model. Applying this method, we present a new winter/annual temperature record for the past 10,000 years based on two fossil pollen data from the Luoxiao Mountains. Simultaneously, we reconstruct the historical impact of human activities in the region. Our temperature records reveal a sustained warming trend during the Holocene, closely matching model-simulated mean annual temperatures (R = 0.97), and temperature reconstructions based on branched glycerol dialkyl glycerol tetraethers from regional terrestrial and marine archives. In contrast, uncorrected pollen data indicate a cooling trend during the late Holocene, coinciding with significant human impact since approximately 3 ka BP. Our analysis and regional comparison with existing temperature records indicate that such contrasting temperature trends stem from a human-induced cooling bias, particularly pronounced in uncorrected pollen data. We infer that the early to middle Holocene warming was due to various factors, while late Holocene warming was predominantly driven by local annual insolation changes. Our findings challenge previously widely identified late-Holocene cooling trends based on uncorrected pollen data, demonstrating that the correction of pollen data can effectively mitigate human-induced cooling biases in temperature reconstructions. This study confirms the accuracy of climate models in simulating a Holocene warming trend, both temporally and spatially, at least in southern China.