Pantana phyllostachysae Chao (PPC) is one of the deadliest defoliators of Moso bamboo. Accurately locating and evaluating PPC damage is essential for the management of bamboo forests. Moso bamboo has a unique biennial growth cycle, consisting of the on-year period (bamboo shoots are incubated and then produced) and the off-year period (old leaves are dropped and then new leaves are grown, and no bamboo shoots are produced in the coming year). The similar physiological characteristics of off-year bamboo and damaged on-year bamboo create difficulties in monitoring PPC damage using remote sensing data. In this study, we synergistically used Sentinel-1, Sentinel-2, and field inventory data to construct machine learning (extreme gradient boosting, XGBoost) models monitoring PPC damage. The results show that the single-time observation feature-based model (using images from October) outperformed the double-time observation feature-based model (using the differences between remote sensing signals from October and February or April) due to the interference from other disturbance agents (e.g., logging and weeding). The overall accuracy (OA) values of the single-time observation feature-based model were at least 3% and 10% higher than those for double-time observation feature-based models for on- and off-year samples, respectively. With the consideration of the on- and off-year phenological differences, OA was improved by over 4%. The model without differentiation of the phenological difference tended to underestimate the damaged area of on-year bamboo and overestimate that of off-year bamboo. We also found that the responses of optical and SAR (synthetic aperture radar) features to PPC damage were different. The optical features increased or decreased with increasing damage severity. SAR features decreased significantly at the initial stage of PPC damage and then changed marginally with the increase in damage severity. The addition of SAR features to optical features improved the model performance, mainly for healthy and mildly damaged samples. The methodology developed in this study provides technical and theoretical support for the pest monitoring of bamboo forests using remote sensing data.