Using data of cosmic ray muon intensities observed with the 17 directional telescopes at Nagoya during the period 1971–1988, we have studied the long-term modulation of the cosmic ray anisotropy of solar origin responsible for the solar semidiurnal variation. It is found that the 17 semidiurnal harmonic vectors suddenly change their relative configuration from a proper type to another on the harmonic dial for the polarity reversal of the polar magnetic field of the Sun from “positive” to “negative” state, which occurred in the period of 1979–1980, and the new type of the configuration has continued until now for all the negative state, even in the period of the minimum solar activity ( 1986). Using the coupling coefficients of the semidurnal variation for various rigidity spectra, it is found that such a configuration change is due to the spectrum change of the anisotropy from “soft” to “hard”. According to the diffusion-convection theory, there are two causal factors producing the hardness change, one is the change of the power exponent (δ) of the rigidity spectrum ( p δ ) of the cosmic ray diffusion coefficient and the other is the difference of the drift motion of cosmic rays in the positive and negative state. The former can be excluded as δ is supposed to be independent of the polarity state, although it might depend on the solar activity. While, in the latter case, the hardness change is quite reversed, depending on the value of δ greater or less than unity. The observed hardness change suggests that δ is less than unity in the rigidity region between several and several tens gigavolts for all the periods.