Mangroves are ecosystems that are seriously threatened by climate change, particularly sea level rise and an increase of droughts. They are resilient to short droughts but their response to longer ones remains arguable. In this work, we studied the responses of R. mangle dominated forests on a carbonate island (El Rosario Island) to climate anomalies and their drivers at inter-annual, decadal and inter-decadal scales during the last centuries. Palynological analysis of a sediment core retrieved from an R. mangle dominated forest was used to reconstruct vegetation changes during the last 400 years. Time series, Multi-taper spectral, Fourier and correlation analyses of PDO, AMO and ENSO series were performed to identify the matches between these series and mangrove changes during this time. The pollen record showed the influence of coastal dynamics on mangrove vegetation changes on a local scale and the influence of climate on a regional one. Changes in coastal vegetation and mangroves were influenced by beach ridge formations and lagoon closing, which were, in turn, determined by shifts in wave and wind intensity. The simultaneous diminution of both types of vegetation were related to both natural (earthquake and extreme droughts) and anthropogenic disturbances. Our results also showed the replacement of the Rhizophora dominated forest by an Avicennia dominated forest when droughts became longer. The peaks of Rhizophora pollen percentages identified along the pollen record showed a 64 year recurrence cycle that dominated during the Little Ice Age (1610–1862) and a 32 year cycle during the Current Warm Period (1866–1954). While the first cycle was positively correlated with the warm phase of PDO and AMO time series, the second cycle showed a negative significant correlation with PDO, AMO and ENSO time series. During warm periods, mangroves were stressed by droughts, but as these cycles became shorter, mangroves could recover more rapidly. From 1954 to the present, no significant correlations were observed between vegetation changes and climate drivers, which is likely related to the increased anthropogenic disturbance beyond that period of time. Our results show the high resilience of mangroves to cyclic climate changes that involve droughts, winds and sea level rises but low recovery and species turnover with extreme and longer droughts and strong anthropogenic disturbances.