Alpine rockwall erosion studies suggest that deglaciated rockwalls in the European Alps are eroding slower today than at earlier times in the Holocene. Explanations for this have included a waned glacial debuttressing effect since the retreat of Last Glacial ice loads and the establishment of more moderate climates, but seldom have such explanations been robustly tested. We combine field data with modelling to reconstruct changes in rockwall erosion and associated climate drivers, in an alpine valley of the European Alps since the beginning of the Holocene. Paleo (Holocene to decadal-scale) erosion rates were calculated from talus accumulation below rockwalls, and compared with recent rates (2016–2019) measured from repeat laserscan surveys of the same rockwalls. We reconstructed the glacial retreat history in the valley, and modelled Holocene changes in permafrost distribution and frost cracking using calibrated rockwall temperature reconstructions. We found that rockwalls that have been free of glacier ice since ∼10 ka experienced higher Holocene-averaged erosion rates compared to recent erosion rates. Our modelling suggests this relates to periods of higher intensities of frost cracking and cycles of permafrost aggradation and degradation in the Holocene, relative to today. For a recently deglaciated high-elevation rockwall, erosion rates were 1–2 orders of magnitude greater than the lower-elevation sites, but decayed rapidly over time since deglaciation. A high, but rapidly decaying rate results from short-lived paraglacial adjustment, permafrost thaw and high frost cracking activity. Our findings suggest that periglacial activity strongly influences the rates and patterns of erosion of deglaciated alpine rockwalls. This helps to explain why paleo Holocene deglaciated rockwall erosion rates tend to exceed recent rates in the European Alps.