Zero-shot surface defect recognition with class knowledge graph

Abstract

Convolutional neural networks (CNNs)-based methods achieve excellent performance on surface defects that belong to the same base classes from a training set, which plays an essential role in ensuring product quality in manufacturing systems. However, in the real world, novel classes of defects always exit due to the complexity and changes of the manufacturing environment. These novel classes are never seen during the training stage but are required to be correctly classified. This paper proposes a class knowledge graph (ZS-CKG) method to address the zero-shot problem in real-world surface defect recognition. In the proposed ZS-CKG method, the class knowledge graph construction method (CKGC) is proposed to construct a class knowledge graph to establish the relationship between base and novel defect classes. Then learns class features by using a graph convolutional neural network. The ZS-CKG utilizes the transformer encoder with capturing long-range dependencies to extract features of defect samples to obtain discriminative defect image features, due to the fact that industrial defects have different shapes and sizes. The experimental results on the public NEU-CLS dataset and real engineering dataset printed circuit boards (PCB) surface defects collected from an actual manufacturing factory demonstrate that the proposed method can effectively address zero-shot surface defect recognition. The ZS-CKG achieve an accuracy of 60.91% and 50.53% on the NEU-CLS and PCB datasets, respectively, which increase by 33.82% and 2.36% compared to the best competing method.