Tropical cyclones (TCs) are associated with severe weather phenomena, making accurate wind field retrieval crucial for TC monitoring. SAR’s high-resolution imaging capability provides detailed information for TC observation, and wind speed calculations require wind direction as prior information. Therefore, utilizing SAR images to retrieve TC wind fields is of significant importance. This study introduces a novel approach for retrieving wind direction from SAR images of TCs through the classification of TC sub-images. The method utilizes a transfer learning-based Inception V3 model to identify wind streaks (WSs) and rain bands in SAR images under TC conditions. For sub-images containing WSs, the Mexican-hat wavelet transform is applied, while for sub-images containing rain bands, an edge detection technique is used to locate the center of the TC eye and subsequently the tangent to the spiral rain bands is employed to determine the wind direction associated with the rain bands. Wind direction retrieval from 10 SAR TC images showed an RMSD of 19.52° and a correlation coefficient of 0.96 when compared with ECMWF and HRD observation wind directions, demonstrating satisfactory consistency and providing highly accurate TC wind directions. These results confirm the method’s potential applications in TC wind direction retrieval.