Multi-level features of images have different contributions to image retrieval. Fixed weight feature fusion cannot give full play to the effectiveness of shallow visual features and semantic features in image retrieval of colored spun fabrics. In view of this fact, this paper proposes an algorithm of colored spun fabrics image retrieval on variable weight and semantic features. The algorithm constructs the similarity correlation coefficient based on the contribution of features at different levels to image retrieval, and obtains the fusion weight coefficient of multi-level features by retrieving the change rate of similarity score of the images and k neighbor images in the sample base. Experimental results show that the proposed algorithm on variable weight feature fusion can bring all features into full play. Compared with the retrieval method that only based on the shallow visual features fusion or the fixed weight feature fusion, the Recall rate of the proposed algorithm increases by 13.15% and 0.54% respectively, and the mean Average Precision of the whole class increases by 7.21% and 1.15% respectively. This research can provide technical support for digital management and product development of colored spun fabrics image, and it has important theoretical research value and broad application prospects.