Antigenic drift of influenza virus hemagglutinin (HA) is enabled by facile evolvability. However, HA antigenic site B, which has become immunodominant in recent human H3N2 influenza viruses, is also evolutionarily constrained by its involvement in receptor binding. Here, we employ deep mutational scanning to probe the local fitness landscape of HA antigenic site B in six different human H3N2 strains spanning from 1968 to 2016. We observe that the fitness landscape of HA antigenic site B can be very different between strains. Sequence variants that exhibit high fitness in one strain can be deleterious in another, indicating that the evolutionary constraints of antigenic site B have changed over time. Structural analysis suggests that the local fitness landscape of antigenic site B can be reshaped by natural mutations via modulation of the receptor-binding mode. Overall, these findings elucidate how influenza virus continues to explore new antigenic space despite strong functional constraints.