Defect detection is a crucial technology that is extensively employed in the manufacturing industry to monitor and ensure the quality of output. Deep learning models have shown remarkable potential for defect detection. However, the success of these models heavily relies on voluminous training data. Collecting substantial amounts of defect data is challenging in practical settings, and the tedious process of pixel-level defect annotation further complicates the task. Among the common defects encountered in manufacturing, scratches are particularly significant. To address these challenges, this study proposes a two-phase generative adversarial network (GAN) approach for synthesizing defect images and generating semi-automatic pixel-wise labels for anomaly detection. The first phase primarily focuses on synthesizing images, while the second phase involves the pixel-wise labeling of the images. The synthesized paired images generated by the GANs serve as input to the semantic network. Notably, the proposed methodology requires only a few real defect samples for training and a small amount of annotated data, making it practical and computationally efficient for implementation in the manufacturing industry. Experimental results indicate the effectiveness of the proposed deep-learning solution in defect detection, specifically in identifying scratches on various textured and patterned surfaces. A notably high detection accuracy is achieved, validating the potential of the approach in real-world manufacturing scenarios.