Improving the reliability and generalization of the conveyor belt damage detection method is crucial and challenging for its industrial application. In this study, a novel vision-based detector named Yolox-BTFPN is proposed. First, a biased texture feature pyramid network (BTFPN), which focuses on the texture feature of the conveyor belt is designed to more accurately capture the damage characteristic. Then the anchor-free decoupled detection heads are utilized to eliminate the hyperparameters design of anchors. Finally, the simplified optimal transport assignment (SimOTA) strategy is employed to solve the problem of samples imbalance and assignment of ambiguous feature points. The experimental results indicate that the proposed method has better reliability and faster convergence performance compared to the state-of-the-art methods. It was found that the mAPs of Yolox-BTFPN are 98.45% and 94.37% on our two test datasets, and that the Epoch, Params and FLOPs are 41, 7.9 M and 12.37 G, respectively.