COVID-19 continues to threaten the world. Relaxing local travel behaviours on preventing the spread of COVID-19, may increase the infection risk in subsequent waves of SARS-CoV-2 transmission. In this study, we analysed changes in the travel behaviour of different population groups (adult, child, student, elderly) during four pandemic waves in Hong Kong before January 2021, by 4-billion second-by-second smartcard records of subway. A significant continuous relaxation in human travel behaviour was observed during the four waves of SARS-CoV-2 transmission. Residents sharply reduced their local travel by 51.9%, 50.1%, 27.6%, and 20.5% from the first to fourth pandemic waves, respectively. The population flow in residential areas, workplaces, schools, shopping areas, amusement areas and border areas, decreased on average by 30.3%, 33.5%, 41.9%, 58.1%, 85.4% and 99.6%, respectively, during the pandemic weeks. We also found that many other cities around the world experienced a similar relaxation trend in local travel behaviour, by comparing traffic congestion data during the pandemic with data from the same period in 2019. The quantitative pandemic fatigue in local travel behaviour could help governments partially predicting personal protective behaviours, and thus to suggest more accurate interventions during subsequent waves, especially for highly infectious virus variants such as Omicron.