Psychological distress is commonly reported sequela in COVID-19-infected people. We investigated the association between experiencing COVID-19 infection and psychological distress in Japan. A total of 14,901 persons who participated in a follow-up survey in December 2022 were included in the analysis. Odds ratios and regression coefficients were estimated by statistical analysis, with history of COVID-19 infection as the independent variable and presence of psychological distress as the dependent variable. Experiencing COVID-19 infection was associated with psychological distress. In a model adjusted for "feeling treated unfairly," the association between infection experience and a high K6 score was significantly attenuated. The results showed that the experience of COVID-19 infection is associated with psychological distress. Moreover, most cases of psychological distress among those who experienced COVID-19 infection can be at least partly explained by a perception of unfair treatment.