Although modal shift from private motorized vehicles to public and active transport is regarded as crucially important for a sustainable society, such a shift has not been successful in Japan. One reason may be that measures have not satisfied the latent factors underlying people's transportation mode choices. Moreover, these factors may have changed during the COVID-19 pandemic. This study is aimed at ascertaining what workers have emphasized in transportation mode choice, and how those emphases have affected their actual mode choice. We also examined how changes in individuals' emphases during the pandemic have affected their actual mode choices. We used a four-time panel survey with 1377 respondents from Japan that asked participants about important items related to their transportation mode choice and activity patterns at four time points before and during the COVID-19 pandemic. After conducting a factor analysis to ascertain the latent factors behind mode choice, we applied a logistic regression model showing whether or not the respondents used each transportation mode at each time point. A conditional logistic regression model with individual fixed effect was applied to the relationship between an individual's change in emphasis and actual mode choice. Factor analysis results indicated four latent factors: usability and reliability, personal time–space, attitude toward the environment and health, and ease of use and convenience. A higher emphasis on personal time–space encouraged car use and discouraged public transport use at any time points. This trend became stronger during the pandemic. Changes in the emphasis level of this factor for an individual also affected the actual mode choice. A higher emphasis on usability and reliability also discouraged public transport use. Ease of use and convenience encouraged public transport use, and attitude toward the environment and health neither encouraged nor discouraged actual mode choice at any time point. Emphases related to transportation mode choice changed over time. However, changes in individuals' emphasis altered the actual mode choice at some time points but not at others. These results should be considered in transportation planning in order to prompt desired modal shifts by transportation users.