The COVID-19 pandemic changed the future of work sustainably and led to a general increase in mental stress. A study conducted during the second and third pandemic wave with a retrospective survey of the first wave among 1,545 non-healthcare workers confirmed an increase in anxiety and depression symptoms and showed a correlation with the occupational SARS-CoV-2 infection risk. This online follow-up survey aims to examine changes in mental distress as the pandemic progressed in Germany and to identify factors influencing potential changes. Longitudinal data from 260 subjects were available for this analysis. Mental distress related to anxiety and depression symptoms, assessed by the Patient Health Questionnaire-4 (PHQ-4), and occupational risk factors were solicited at the end of 2022 and retrospectively at the fifth wave. Categorized PHQ-4 scores were modelled with mixed ordinal regression models and presented with odds ratios (OR) and 95% confidence intervals (95% CI). A previous diagnosis of a depressive or anxiety disorder was a strong risk factor for severe symptoms (OR 3.49, 95% CI 1.71-7.11). The impact of occupational SARS-CoV-2 infection risk on mental distress was increased, albeit failing to reach the formal level of statistical significance (high risk OR 1.83, 95% CI 0.59-5.63; probable risk OR 1.72, 95% CI 0.93-3.15). Mental distress was more pronounced in those with a previous diagnosis of anxiety and depression. Confirmed occupational risk factors were protective measures against occupational SARS-CoV-2 infection perceived as inadequate, chronic work-related stress, overcommitment, reduced interactions with fellow-workers, and work-privacy conflicts. The pandemic had a negative impact on anxiety and depression symptoms among the studied non-healthcare workers, particularly early in the pandemic, although this effect does not appear to be permanent. There are modifiable risk factors that can protect workers' mental health, including strengthening social interactions among employees and reducing work-privacy conflicts.