Damage detection of wind turbine blades can provide guidance for maintaining the wind turbine system and reduce maintenance costs. Although machine vision has made good progress in blade damage detection, the complex background information brings great challenges to blade damage detection. Existing methods treat all information or features of the image equally, which may result in insufficient attention to damage features. In this paper, a novel framework of channel-spatial attention convolutional neural networks, together with an adaptive learning rate scheme, is proposed for surface damage detection of wind turbine blades. It guides the attention of the feature extraction network to focus on the blade damage feature by embedding CBAM (Convolutional Block Attention Module) to enhance the blade damage features. To optimize the training process and make it reach saturation faster, a novel adaptive learning rate scheme is also proposed. The effectiveness of the proposed is verified on real wind turbine blade image database, containing three types of damage, manually collected from a commercial wind farm. Experimental results show that the proposed method can improve binary damage classification accuracy by 2.68% and multiple class damage classification accuracy by 5.36% in comparison with the compared state-of-the-art methods.