In the anammox process, maintaining a high anammox activity at low water temperatures for stable nitrogen removal is a challenge. In this study, to verify the adaptability of anammox to low water temperatures, we investigated effects of annual temperature fluctuations on nitrogen removal in a full-scale swine wastewater treatment plant, where anammox bacteria accumulated. Annual quarters were defined as L-1 (November–January), L-2 (February–April), H-1 (April–July), and H-2 (July–October). The total nitrogen removal rate was stable at 0.08–0.11 kg-N/m3/d, even during temperature fluctuations. Removal efficiencies of biochemical oxygen demand and total nitrogen were consistently high at 95–99% and 69–81%, respectively. The anammox activity and abundance of anammox bacteria were highest in granule L-1 and lowest in granule H-2. The optimal temperature for anammox activity shifted from 35 °C in granules H-1 and H-2 to 30 °C in granules L-1 and L-2, while the latter maintained a moderate activity compared to the former at low temperature. Candidatus Jettenia asiatica was predominant, especially in granule L-2, accounting for up to 54% of the microbial community composition at the genus level. The high specific anammox activity in granule L-2 was considered to be due to the abundance of anammox bacteria and the adaptation of Ca. Jettenia asiatica to low temperature. The anammox granules adapted well to low temperatures and demonstrated high efficiency in the simultaneous partial nitrification anammox and denitrification process without heating. Thus, constructing an energy-saving and cost-effective nitrogen removal system can be considered.