The generation of defects during fabric production impacts fabric quality, and fabric production factories can greatly benefit from the automatic detection of fabric defects. Although object detection algorithms based on convolutional neural networks can be quickly developed, fabric defect detection encounters several problems. Accordingly, a fabric defect detection model based on Cascade R-CNN was proposed in this study. We also proposed block recognition and detection box merging algorithms to achieve complete defect detection in high-resolution images. We implemented a Switchable Atrous Convolution layer to enhance the feature extraction ability of ResNet-50 and upgraded the Feature Pyramid Network to improve the detection accuracy of small defects. Experimental results demonstrated that the proposed model can efficiently perform defect detection in fabric.