In the operation and maintenance of planetary gearboxes, the growth of monitoring data is often faster than its analysis and classification. Careful data analysis is generally considered to require more expertise. Rendering the machine learning algorithm able to provide more information, not just the diagnosis conclusion, is promising work. This paper proposes an analysis and diagnosis two-stage framework based on time-frequency information analysis. In the first stage, a U-net model is used for the semantic segmentation of vibration time-frequency spectrum to highlight faulty feature regions. Shape features are then calculated to extract useful information from the segmented image. In the second stage, the decision tree algorithm completes the health state classification of the planetary gearboxes using the input of shape features. The real data of wind turbine planetary gearboxes and augmented data are utilized to verify the proposed framework’s effectiveness and superiority. The F1-score of segmentation and the classification accuracy reach 0.942 and 97.4%, respectively, while in the environmental robustness experiment, they reached 0.747 and 83.1%. Equipping the two-stage framework with different analytical methods and diagnostic algorithms can construct flexible diagnostic systems for similar problems in the community.