Deep learning-based defect inspection has gained popularity in recent years. The dataset requirements for the supervised learning-based method are currently high, but the types of defects are numerous and difficult to gather. This work proposes a local image reconstruction-based unsupervised fabric defect segmentation method to address this problem. Cyclic structures make up the normal portion of the fabric image, whereas the defects are anomalous and minor in comparison. As a result, the defect will be recreated as a normal texture utilizing the information from its surrounding areas, and the defect information will be preserved in the residual image. By masking the same area with various shapes, different reconstruction outcomes and residual images can be achieved. The signal of the defect will be amplified and the noise will be decreased due to the random distribution when the generated residual pictures are fused, which can effectively identify the defect from the noise and lower the false detection rate. On the denim fabric dataset, the proposed unsupervised method can achieve high precision fabric defect segmentation, with the defect detection rate and detection precision reaching at least 85% and 89%, respectively, with high efficiency (approximately 60 m/min inspection speed), outperforming other fabric defect segmentation methods.