During the COVID-19 pandemic, the behavioral response to reported case numbers changed drastically over time. While a few dozen cases were enough to trigger government-induced and voluntary contact reduction in early 2020, less than a year later much higher case numbers were required to induce behavioral change. Little attention has been paid to understand and mathematically model this effect of decreasing risk perception over longer time-scales. Here, first we show that weighing the number of cases with a time-varying factor of the form ta,a<0 explains real-world mobility patterns from several European countries during 2020 when introduced into a very simple behavior model. Subsequently, we couple our behavior model with an SIR epidemic model. Remarkably, decreasing risk perception can produce complex dynamics, including multiple waves of infection. We find two regimes for the total number of infected individuals that are explained by the interplay of initial attention and the rate of attention decrease. Our results show that including adaption into non-equilibrium models is necessary to understand behavior change over long time scales and the emergence of non-trivial infection dynamics.