Nitrogen (N) fertilizer application is one of the causes of soil acidification at tea plantations. However, the effect of N fertilizer application on the soil acidification characteristics of tea plantations with different acidities remains unclear. In this study, field experiments were conducted to investigate the effects of different nitrogen fertilizer application rates on the pH, pH buffer capacity (pHBC), exchangeable total acidity (ETA), exchangeable base cations (EBCs), and cation exchange capacity (CEC) in the topsoil of non-acidified (NA), mildly acidified (MA), and heavily acidified (HA) tea plantations. The results showed that the exchangeable Al3+ (E-Al) and CEC were HA > MA > NA in all tea plantations, whereas the EBCs and base saturation percentage (BSP) were HA < MA < NA. In the tea plantations with pH > 4.0, the pH, EBCs, and BSP showed decreasing trends with increasing N fertilizer application, whereas E-Al showed an increasing trend. In the tea plantations with pH < 4.0, the soil pH showed a small increasing trend with the increase in N fertilizer application, whereas the soil exchangeable H+ (E-H), E-Al, and CEC showed decreasing trends. Meanwhile, in the pH range of 4–6, the soil acid–base buffer curve rose sharply, and an excessive application of N fertilizer (N900) significantly reduced the pHBC. In addition, a stepwise regression analysis showed that the BSP, EBCs, and exchangeable Mg2+ (E-Mg) had significant direct effects on the soil pH, whereas the CEC and N application had significant direct effects on the soil pHBC. In conclusion, a decrease in the BSP and an increase in E-Al were the main mechanisms of acidification at tea plantations, whereas a decrease in the BSP caused by the application of N fertilizer was the main cause of exacerbated soil acidification in non-acidified tea plantations.