In response to the challenges posed by complex background textures and limited hardware resources in fabric defect detection, this study proposes a lightweight fabric defect detection algorithm based on an improved GSL-YOLOv8n model. Firstly, to reduce the parameter count and complexity of the YOLOv8n network, the GhostNet concept is used to construct the C2fGhost module, replacing the conventional convolution layers in the YOLOv8n structure with Ghost convolutions. Secondly, the SimAM parameter-free attention mechanism is embedded at the end of the backbone network to eliminate redundant background, enhance semantic information for small targets, and improve the network’s feature extraction capability. Lastly, a lightweight shared convolution detection head is designed, employing the scale layer to adjust features, ensuring the lightweight nature of the model while minimizing precision loss. Compared to the original YOLOv8n model, the improved GSL-YOLOv8n algorithm increases the mAP@0.5 by 0.60% to 98.29% and reduces model size, computational load, and parameter count by 66.7%, 58.0%, and 67.4%, respectively, meeting the application requirements for fabric defect detection in textile industry production.