Many studies of Holocene relative sea-level (RSL) changes across Antarctica assume that their reconstructions record uplift from glacial isostatic adjustment caused by the demise of the Last Glacial Maximum (LGM) ice sheets. However, recent analysis of GPS observations suggests that mantle viscosity beneath the Antarctic Peninsula is weaker than previously thought, which would imply that solid Earth motion is not controlled by post-LGM ice-sheet retreat but instead by late Holocene ice-mass changes. If this hypothesis is correct, one might expect to find Holocene RSL records that do not reflect a monotonic decrease in the rate of RSL fall but show variations in the rate of RSL change through the Holocene. We present a new record of late Holocene RSL change from Torgersen Island near Palmer Station in the western Antarctic Peninsula that shows an increase in the rate of relative sea-level fall from 3.0 ± 1.2 mm/yr to 5.1 ± 1.8 mm/yr during the late Holocene. Independent studies of the glacial history of the region provide evidence of ice-sheet changes over similar time scales that may be driving this change. When our RSL records are corrected for sea-surface height changes associated with glacial isostatic adjustment (GIA), the rate of post-0.79 ka land uplift at Torgersen Island, 5.3 ± 1.8 mm/yr, is much higher than the rate of uplift recorded at a nearby GPS site at Palmer Station prior to the Larsen B breakup in 2002 AD (1998-2002 AD; <0.1 mm/yr), but similar to the rates observed after 2002 AD (2002-2013 AD; 6–9 mm/yr). This substantial variation in uplift rates further supports the hypothesis that Holocene RSL rates of change are recording responses to late Holocene and recent changes in local ice loading rather than a post-LGM signal across portions of the Antarctic Peninsula. Thus middle-to-late Holocene RSL data may not be an effective tool for constraining the size of the LGM ice sheet across portions of the Antarctic Peninsula underlain by weaker mantle. In addition, current global-scale GIA models are unable to predict our observed changes in late Holocene RSL. Complexities in Earth structure and neoglacial history need to be taken into consideration in GIA models used for correcting modern satellite-based observations of ice-mass loss.