Data are limited on the protective role of the Omicron BA bivalent vaccine, previous infection, and their induced neutralizing antibodies against Omicron XBB.1.16 and EG.5.1 infection. We conducted a nested case-control analysis among tertiary hospital staff in Tokyo who had received ≥3 doses of COVID-19 vaccines and donated blood samples in June 2023 (1 month before the Omicron XBB.1.16 and EG.5.1 wave). We identified 206 symptomatic cases between June and September 2023 and selected their controls with 1:1 propensity score matching. We examined the association of vaccination, previous infection, and preinfection live virus neutralizing antibody titers against Omicron XBB.1.16 and EG.5.1 with the risk of COVID-19 infection. Previous infection during the Omicron BA- or XBB-dominant phase was associated with a significantly lower infection risk during the XBB.1.16 and EG.5.1-dominant phase than infection-naive status, with 70% and 100% protection, respectively, whereas Omicron BA bivalent vaccination showed no association. Preinfection neutralizing titers against XBB.1.16 and EG.5.1 were 39% (95% CI, 8%-60%) and 28% (95% CI, 8%-44%) lower in cases than matched controls. Neutralizing activity against XBB.1.16 and EG.5.1 was somewhat detectable in the sera of individuals with previous infection but barely detectable in those who were infection naive and received the Omicron bivalent vaccine. In the era when the Omicron XBB vaccine was unavailable, the Omicron BA bivalent vaccine did not confer the neutralizing activity and protection against Omicron XBB.1.16 and EG.5.1 symptomatic infection. The previous infection afforded neutralizing titers and protection against symptomatic infection with these variants.