Dunes not only serve as natural landscapes but also act as crucial natural barriers protecting coastlines from storm surges. However, the stability of coastlines and dunes is further affected by climate change-induced sea level rise and increased storm activity. An effective evaluation of dune stability necessitates the collection of parameters such as wave height, changes in dune crest elevation, and dune erosion rates. This research employs wave flumes and high-speed camera technology in laboratory settings to observe and simulate coastal dynamics under overwash and collision regimes. The study investigates wave propagation under both regimes using the XBeach model. In the overwash regime, the model slightly overestimates the values of infragravity waves, whereas its accuracy improved under the collision regime. Moreover, in the overwash regime, higher wave skewness results in more sediment transport to the shore, exhibiting a linear relationship between sediment erosion volume and shoreline retreat distance. Through comparison with time series wave runup data obtained by cameras and traditional predictive formulas, the study validates the applicability of the formula proposed by Stockdon as a predictive tool for wave runup in this experiment. To evaluate dune stability, the study introduces the dimensionless overwash threshold parameter Cs, which is based on wave runup and dune crest elevation, to distinguish between dunes in a collision regime or an overwash regime. These findings help identify dune stability patterns, aiding in early detection of coastal erosion and assisting authorities in ecosystem management.