AbstractFloat-Zone (FZ) crystal growth process allows for producing higher purity silicon crystal with much lower concentrations of impurities, in particular low oxygen content. Nevertheless, the FZ process occasionally faces the problem of small contamination from oxidation. This can come in the form of a thin oxide layer that may form on un-melted polysilicon surface. The appearance of the oxide layer indicates degraded machine performance and the need for machine maintenance. Therefore, oxide investigation is important for improving both the FZ process and FZ machines, and the first step is oxide recognition. In this study, we characterized oxide into mainly three varieties, according to their surface texture characteristics, which are: (i) spot (ii) shadow and (iii) ghost curtain. We leveraged FZ images captured from the vision system integrated on the FZ machine to establish an oxide dataset. Targeted for data imbalance problem in our dataset, a method based on transfer learning and asymmetric loss for multi-label oxide classification is presented in this work. The results showed that the pre-trained model and the asymmetric loss used for training outperformed the baseline models and improved the classification performance. Furthermore, this study deeply investigated the effectiveness of the components of asymmetric loss. Finally, Gradient-weighted Class Activation Mapping (Grad-CAM) was employed to explain decision process of the models in order to adopt them in the industry.