Training Deep Neural Networks Research Articles

Waterwave: A GPU Memory Flow Engine for Concurrent DNN Training

Training Deep Neural Networks (DNN) concurrently is becoming increasingly important for deep learning practitioners, e.g., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">hyperparameter optimization (HPO)</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">neural architecture search (NAS)</i> . The GPU memory capacity is the impediment that prohibits multiple DNNs from being trained on the same GPU due to the large memory usage during training. In this paper, we propose Waterwave a GPU memory flow engine for concurrent deep learning training. Firstly, to address the memory explosion brought by the long time lag between memory allocation and deallocation time, we develop an allocator tailored for multi-streams. By making the allocator aware of the stream information, a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">prioritized allocation</i> is conducted based on the chunk's <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">synchronization</i> attributes, allowing us to provide useable memory after scheduling rather than waiting it to be really released after GPU computation. Secondly, Waterwave partitions the compute graph to a set of continuous <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">node groups</i> and then performs finer-grained scheduling: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">NodeGroup pipeline execution</i> , to guarantee a proper memory requests order. Waterwave can accomplish up to 96.8% of the maximum batch size of solo training. Additionally, in scenarios with high memory demand, Waterwave can outperform existing spatial sharing and temporal sharing by up to 12x and 1.49x, respectively.

SmartDeal: Remodeling Deep Network Weights for Efficient Inference and Training.

The record-breaking performance of deep neural networks (DNNs) comes with heavy parameter budgets, which leads to external dynamic random access memory (DRAM) for storage. The prohibitive energy of DRAM accesses makes it nontrivial for DNN deployment on resource-constrained devices, calling for minimizing the movements of weights and data in order to improve the energy efficiency. Driven by this critical bottleneck, we present SmartDeal, a hardware-friendly algorithm framework to trade higher-cost memory storage/access for lower-cost computation, in order to aggressively boost the storage and energy efficiency, for both DNN inference and training. The core technique of SmartDeal is a novel DNN weight matrix decomposition framework with respective structural constraints on each matrix factor, carefully crafted to unleash the hardware-aware efficiency potential. Specifically, we decompose each weight tensor as the product of a small basis matrix and a large structurally sparse coefficient matrix whose nonzero elements are readily quantized to the power-of-2. The resulting sparse and readily quantized DNNs enjoy greatly reduced energy consumption in data movement as well as weight storage, while incurring minimal overhead to recover the original weights thanks to the required sparse bit-operations and cost-favorable computations. Beyond inference, we take another leap to embrace energy-efficient training, by introducing several customized techniques to address the unique roadblocks arising in training while preserving the SmartDeal structures. We also design a dedicated hardware accelerator to fully utilize the new weight structure to improve the real energy efficiency and latency performance. We conduct experiments on both vision and language tasks, with nine models, four datasets, and three settings (inference-only, adaptation, and fine-tuning). Our extensive results show that 1) being applied to inference, SmartDeal achieves up to 2.44x improvement in energy efficiency as evaluated using real hardware implementations and 2) being applied to training, SmartDeal can lead to 10.56x and 4.48x reduction in the storage and the training energy cost, respectively, with usually negligible accuracy loss, compared to state-of-the-art training baselines. Our source codes are available at: https://github.com/VITA-Group/SmartDeal.

Application of cluster repeated mini-batch training method to classify electroencephalography for grab and lift tasks.

Modern deep neural network training is based on mini-batch stochastic gradient optimization. While using extensive mini-batches improves the computational parallelism, the small batch training proved that it delivers improved generalization performance and allows a significantly smaller memory, which might also improve machine throughput. However, mini-batch size and characteristics, a key factor for training deep neural networks, has not been sufficiently investigated in training correlated group features and looping with highly complex ones. In addition, the unsupervised learning method clusters the data into different groups with similar properties to make the training process more stable and faster. Then, the supervised learning algorithm was applied with the cluster repeated mini-batch training (CRMT) methods. The CRMT algorithm changed the random minibatch characteristics in the training step into training in order of clusters. Specifically, the self-organizing maps (SOM) were used to cluster the information into n groups based on the dataset's labels Then, neural network models (ANN) were trained with each cluster using the cluster repeated mini-batch training method. Experiments conducted on EEG datasets demonstrate the survey of the proposed method and optimize it. In addition, the results in our research outperform other state-of-the-art methods.

Data augmentation using generative models for track intrusion detection.

The objective of this work is to address the problem of detecting track intruders in railway systems using deep learning-based algorithms. Unauthorized entry onto railway tracks poses a significant risk of collisions between trains and humans. However, intrusion discrimination algorithms often suffer from a lack of learning data and data imbalance issues. To overcome these challenges, this research proposes an algorithm that combines generative models and classification networks. Generative models are utilized to generate synthetic intrusion data by learning the underlying distribution of available data and creating new samples resembling the original data. The augmented intrusion data is then used to train deep neural networks to accurately identify intrusions. The proposed algorithm is evaluated using real data sets, demonstrating its effectiveness in overcoming limited learning data and data imbalance issues. By augmenting intrusion data using generative models, the algorithm achieves improved accuracy compared to traditional approaches. In conclusion, the algorithm presented in this work provides a solution for detecting track intruders in railway systems. By leveraging generative models to augment limited intrusion data and utilizing classification networks for intrusion discrimination, the algorithm demonstrates improved performance in accurately identifying intrusions. This research highlights the potential of deep learning-based approaches in enhancing railway safety and recommends further exploration and application of these methods in real-world settings.

Discriminative ensemble meta-learning with co-regularization for rare fundus diseases diagnosis.

Deep neural networks (DNNs) have been widely applied in the medical image community, contributing to automatic ophthalmic screening systems for some common diseases. However, the incidence of fundus diseases patterns exhibits a typical long-tailed distribution. In clinic, a small number of common fundus diseases have sufficient observed cases for large-scale analysis while most of the fundus diseases are infrequent. For these rare diseases with extremely low-data regimes, it is challenging to train DNNs to realize automatic diagnosis. In this work, we develop an automatic diagnosis system for rare fundus diseases, based on the meta-learning framework. The system incorporates a co-regularization loss and the ensemble-learning strategy into the meta-learning framework, fully leveraging the advantage of multi-scale hierarchical feature embedding. We initially conduct comparative experiments on our newly-constructed lightweight multi-disease fundus images dataset for the few-shot recognition task (namely, FundusData-FS). Moreover, we verify the cross-domain transferability from miniImageNet to FundusData-FS, and further confirm our method's good repeatability. Rigorous experiments demonstrate that our method can detect rare fundus diseases, and is superior to the state-of-the-art methods. These investigations demonstrate that the potential of our method for the real clinical practice is promising.

Deep learning-driven adaptive optics for single-molecule localization microscopy

The inhomogeneous refractive indices of biological tissues blur and distort single-molecule emission patterns generating image artifacts and decreasing the achievable resolution of single-molecule localization microscopy (SMLM). Conventional sensorless adaptive optics methods rely on iterative mirror changes and image-quality metrics. However, these metrics result in inconsistent metric responses and thus fundamentally limit their efficacy for aberration correction in tissues. To bypass iterative trial-then-evaluate processes, we developed deep learning-driven adaptive optics for SMLM to allow direct inference of wavefront distortion and near real-time compensation. Our trained deep neural network monitors the individual emission patterns from single-molecule experiments, infers their shared wavefront distortion, feeds the estimates through a dynamic filter and drives a deformable mirror to compensate sample-induced aberrations. We demonstrated that our method simultaneously estimates and compensates 28 wavefront deformation shapes and improves the resolution and fidelity of three-dimensional SMLM through >130-µm-thick brain tissue specimens.

Medimatrix: innovative pre-training of grayscale images for rheumatoid arthritis diagnosis revolutionises medical image classification.

Efficient and accurate medical image classification (MIC) methods face two major challenges: (1) high similarity between images of different disease classes; and (2) generating large medical image datasets for training deep neural networks is challenging due to privacy restrictions and the need for expert ground truth annotations. In this paper, we introduce a novel deep learning method called pre-training grayscale images with supervised learning for MIC (MediMatrix). Instead of pre-training on color ImageNet, our approach uses MediMatrix on grayscale ImageNet. To improve the performance of the network, we introduce ShuffleAttention (SA), a self-attention mechanism. By combining SA with the multiple residual structure (ResSA block) and replacing short-cut connections with dense residual connections between corresponding layers (densepath), our network can dynamically adjust channel attention weights and receive image inputs of different sizes, resulting in improved feature representation and better discrimination of similarities between different categories. MediMatrix effectively classifies X-ray images of rheumatoid arthritis (RA), enabling efficient screening without the need for expert analysis or invasive testing. Through extensive experiments, we demonstrate the superiority of MediMatrix over state-of-the-art methods and that color is not critical for rich natural image classification. Our results highlight the potential of computer-aided diagnosis combined with MediMatrix as a valuable screening tool for early detection and intervention in RA.

Reducing the Effect of Incorrect Lithology Labels on the Training of Deep Neural Networks for Lithology Identification

Reducing the Effect of Incorrect Lithology Labels on the Training of Deep Neural Networks for Lithology Identification

DNNShifter: An efficient DNN pruning system for edge computing

Deep neural networks (DNNs) underpin many machine learning applications. Production quality DNN models achieve high inference accuracy by training millions of DNN parameters which has a significant resource footprint. This presents a challenge for resources operating at the extreme edge of the network, such as mobile and embedded devices that have limited computational and memory resources. To address this, models are pruned to create lightweight, more suitable variants for these devices. Existing pruning methods are unable to provide similar quality models compared to their unpruned counterparts without significant time costs and overheads or are limited to offline use cases. Our work rapidly derives suitable model variants while maintaining the accuracy of the original model. The model variants can be swapped quickly when system and network conditions change to match workload demand. This paper presents DNNShifter, an end-to-end DNN training, spatial pruning, and model switching system that addresses the challenges mentioned above. At the heart of DNNShifter is a novel methodology that prunes sparse models using structured pruning - combining the accuracy-preserving benefits of unstructured pruning with runtime performance improvements of structured pruning. The pruned model variants generated by DNNShifter are smaller in size and thus faster than dense and sparse model predecessors, making them suitable for inference at the edge while retaining near similar accuracy as of the original dense model. DNNShifter generates a portfolio of model variants that can be swiftly interchanged depending on operational conditions. DNNShifter produces pruned model variants up to 93x faster than conventional training methods. Compared to sparse models, the pruned model variants are up to 5.14x smaller and have a 1.67x inference latency speedup, with no compromise to sparse model accuracy. In addition, DNNShifter has up to 11.9x lower overhead for switching models and up to 3.8x lower memory utilisation than existing approaches. DNNShifter is available for public use from https://github.com/blessonvar/DNNShifter.

Learned mappings for targeted free energy perturbation between peptide conformations.

Targeted free energy perturbation uses an invertible mapping to promote configuration space overlap and the convergence of free energy estimates. However, developing suitable mappings can be challenging. Wirnsberger et al. [J. Chem. Phys. 153, 144112 (2020)] demonstrated the use of machine learning to train deep neural networks that map between Boltzmann distributions for different thermodynamic states. Here, we adapt their approach to the free energy differences of a flexible bonded molecule, deca-alanine, with harmonic biases and different spring centers. When the neural network is trained until "early stopping"-when the loss value of the test set increases-we calculate accurate free energy differences between thermodynamic states with spring centers separated by 1 Å and sometimes 2 Å. For more distant thermodynamic states, the mapping does not produce structures representative of the target state, and the method does not reproduce reference calculations.

Fighting selection bias in statistical learning: application to visual recognition from biased image databases

In practice, and especially when training deep neural networks, visual recognition rules are often learned based on various sources of information. On the other hand, the recent deployment of facial recognition systems with uneven performances on different population segments has highlighted the representativeness issues induced by a naive aggregation of the datasets. In this paper, we show how biasing models can remedy these problems. Based on the (approximate) knowledge of the biasing mechanisms at work, our approach consists in reweighting the observations, so as to form a nearly debiased estimator of the target distribution. One key condition is that the supports of the biased distributions must partly overlap, and cover the support of the target distribution. In order to meet this requirement in practice, we propose to use a low dimensional image representation, shared across the image databases. Finally, we provide numerical experiments highlighting the relevance of our approach.

Hybrid morphological-convolutional neural networks for computer-aided diagnosis.

Training deep Convolutional Neural Networks (CNNs) presents challenges in terms of memory requirements and computational resources, often resulting in issues such as model overfitting and lack of generalization. These challenges can only be mitigated by using an excessive number of training images. However, medical image datasets commonly suffer from data scarcity due to the complexities involved in their acquisition, preparation, and curation. To address this issue, we propose a compact and hybrid machine learning architecture based on the Morphological and Convolutional Neural Network (MCNN), followed by a Random Forest classifier. Unlike deep CNN architectures, the MCNN was specifically designed to achieve effective performance with medical image datasets limited to a few hundred samples. It incorporates various morphological operations into a single layer and uses independent neural networks to extract information from each signal channel. The final classification is obtained by utilizing a Random Forest classifier on the outputs of the last neural network layer. We compare the classification performance of our proposed method with three popular deep CNN architectures (ResNet-18, ShuffleNet-V2, and MobileNet-V2) using two training approaches: full training and transfer learning. The evaluation was conducted on two distinct medical image datasets: the ISIC dataset for melanoma classification and the ORIGA dataset for glaucoma classification. Results demonstrate that the MCNN method exhibits reliable performance in melanoma classification, achieving an AUC of 0.94 (95% CI: 0.91 to 0.97), outperforming the popular CNN architectures. For the glaucoma dataset, the MCNN achieved an AUC of 0.65 (95% CI: 0.53 to 0.74), which was similar to the performance of the popular CNN architectures. This study contributes to the understanding of mathematical morphology in shallow neural networks for medical image classification and highlights the potential of hybrid architectures in effectively learning from medical image datasets that are limited by a small number of case samples.

A paradigm shift in the prevention and diagnosis of oral squamous cell carcinoma.

Oral squamous cell carcinoma (OSCC) is a widespread disease with only 50%-60% 5-year survival. Individuals with potentially malignant precursor lesions are at high risk. Survival could be increased by effective, affordable, and simple screening methods, along with a shift from incisional tissue biopsies to non-invasive brush biopsies for cytology diagnosis, which are easy to perform in primary care. Along with the explainable, fast, and objective artificial intelligence characterisation of cells through deep learning, an easy-to-use, rapid, and cost-effective methodology for finding high-risk lesions is achievable. The collection of cytology samples offers the further opportunity of explorative genomic analysis. Our prospective multicentre study of patients with leukoplakia yields a vast number of oral keratinocytes. In addition to cytopathological analysis, whole-slide imaging and the training of deep neural networks, samples are analysed according to a single-cell RNA sequencing protocol, enabling mapping of the entire keratinocyte transcriptome. Mapping the changes in the genetic profile, based on mRNA expression, facilitates the identification of biomarkers that predict cancer transformation. This position paper highlights non-invasive methods for identifying patients with oral mucosal lesions at risk of malignant transformation. Reliable non-invasive methods for screening at-risk individuals bring the early diagnosis of OSCC within reach. The use of biomarkers to decide on a targeted therapy is most likely to improve the outcome. With the large-scale collection of samples following patients over time, combined with genomic analysis and modern machine-learning-based approaches for finding patterns in data, this path holds great promise.

Range estimation in a weakly range-dependent waveguide in the South China Sea using single hydrophone and multi-task trained deep neural network

In this paper, an underwater source range estimation method using a multi-task trained deep neural network is proposed. Unlike existing machine learning frameworks, in this case, the physical quantities utilized for the training and testing are not identical. The training data parameter is the phase of the complex sound pressure, which is independent of the source and receiver depth and can be generated easily by an acoustic propagation model with only a few environmental parameters, decreasing computational costs significantly. The testing data parameter is the realistically-measured complex sound pressure. Instead of simply using the range as the only supervision signal, two other parameters — the phase mode and the bottom sound speed — are applied as auxiliary labels during the training process, embedding additional information about the sound field into the deep model. The projection between the sound field features and the range extracted from the training data by the deep model is applied to range estimation task on testing data. The effectiveness of the proposed approach is verified for a weakly range-dependent waveguide in the South China Sea by localizing several upsweep signals within a frequency band of 20–200 Hz. Testing results illustrate that the proposed method outperforms the classical matched impulse response approach and deep-learning-based single-label trained models, with the mean absolute percentage error (MAPE) decreasing by 24.63% and 59.46%, respectively. The mechanism of promotion is explained by comparing visualized hidden features of the neural network using t-distributed stochastic neighbor embedding (t-SNE) between the single-task and multi-task approach, showing that the latter helps to construct a more reasonable data manifold in high-dimensional space and thus ensures better performance of the deep model. We also fine-tune the network with a few realistic samples and further improve the performance by up to 25.34% against the baseline.

Multiscale modeling of woven composites by deep learning neural networks and its application in design optimization

Due to complex yarn geometries and multiscale nature, it is very challenging to predict mechanical properties and strength constants of woven composites in an accuracy and efficient way. In this paper, mechanics of structure genome (MSG) is employed to predict effective material properties and strength constants of woven composites. The microscale analysis is performed to compute the effective properties and strength constants of yarns based on a hexagonal packed microstructure. Then, the mesoscale analysis is performed to compute the effective material properties and strength constants of woven composites. The accuracy and efficiency of the MSG method are compared with the computational homogenization method. In addition, simulation data using the MSG model is generated to train deep learning neural networks (DNN) models. The trained models provide ultra-efficient surrogate models with high accuracy. The DNN models are further applied to a design optimization problem to tailor geometry parameters to enhance the mechanical performance of plain woven composites, and the optimization can be efficiently solved with gradients from the DNN models. The developed computational framework can be applied to general woven composites featuring complex microscale and mesoscale features.

Semi-supervised and weakly-supervised deep neural networks and dataset for fish detection in turbid underwater videos

Fish are key members of marine ecosystems, and they have a significant share in the healthy human diet. Besides, fish abundance is an excellent indicator of water quality, as they have adapted to various levels of oxygen, turbidity, nutrients, and pH. To detect various fish in underwater videos, Deep Neural Networks (DNNs) can be of great assistance. However, training DNNs is highly dependent on large, labeled datasets, while labeling fish in turbid underwater video frames is a laborious and time-consuming task, hindering the development of accurate and efficient models for fish detection. To address this problem, firstly, we have collected a dataset called FishInTurbidWater, which consists of a collection of video footage gathered from turbid waters, and quickly and weakly (i.e., giving higher priority to speed over accuracy) labeled them in a 4-times fast-forwarding software. Next, we designed and implemented a semi-supervised contrastive learning fish detection model that is self-supervised using unlabeled data, and then fine-tuned with a small fraction (20%) of our weakly labeled FishInTurbidWater data. At the next step, we trained, using our weakly labeled data, a novel weakly-supervised ensemble DNN with transfer learning from ImageNet. The results show that our semi-supervised contrastive model leads to more than 20 times faster turnaround time between dataset collection and result generation, with reasonably high accuracy (89%). At the same time, the proposed weakly-supervised ensemble model can detect fish in turbid waters with high (94%) accuracy, while still cutting the development time by a factor of four, compared to fully-supervised models trained on carefully labeled datasets. Our dataset and code are publicly available at the hyperlink FishInTurbidWater.

STADIA: Photonic Stochastic Gradient Descent for Neural Network Accelerators

Deep Neural Networks (DNNs) have demonstrated great success in many fields such as image recognition and text analysis. However, the ever-increasing sizes of both DNN models and training datasets make deep leaning extremely computation- and memory-intensive. Recently, photonic computing has emerged as a promising technology for accelerating DNNs. While the design of photonic accelerators for DNN inference and forward propagation of DNN training has been widely investigated, the architectural acceleration for equally important backpropagation of DNN training has not been well studied. In this paper, we propose a novel silicon photonic-based backpropagation accelerator for high performance DNN training. Specifically, a general-purpose photonic gradient descent unit named STADIA is designed to implement the multiplication, accumulation, and subtraction operations required for computing gradients using mature optical devices including Mach-Zehnder Interferometer (MZI) and Mircoring Resonator (MRR), which can significantly reduce the training latency and improve the energy efficiency of backpropagation. To demonstrate efficient parallel computing, we propose a STADIA-based backpropagation acceleration architecture and design a dataflow by using wavelength-division multiplexing (WDM). We analyze the precision of STADIA by quantifying the precision limitations imposed by losses and noises. Furthermore, we evaluate STADIA with different element sizes by analyzing the power, area and time delay for photonic accelerators based on DNN models such as AlexNet, VGG19 and ResNet. Simulation results show that the proposed architecture STADIA can achieve significant improvement by 9.7× in time efficiency and 147.2× in energy efficiency, compared with the most advanced optical-memristor based backpropagation accelerator.

Prediction and optimization of 3D-printed sandwich beams with chiral cores

This study pursues two primary objectives concerning sandwich structures with chiral cores. Firstly, it delves into the realm of machine learning-assisted predictions regarding the behavior of these structures under compressive loads. In this way, three key geometric parameters of the repetitive chiral unit cells-thickness, angle, and diameter-are considered as highly influential design factors. Accordingly, a custom method for designing experiments was employed, resulting in achieving 27 totally different sandwich beams. These beam structures were made up of Poly Lactic Acid (PLA) for both their face sheets and cores, employing Fused Deposition Modeling (FDM) printing technology. Subsequently, they underwent compressive loading, where the resulting mechanical responses forming the foundational dataset for training Deep Neural Networks (DNNs). Remarkably, the DNNs were trained utilizing both Bayesian and conjugate gradient algorithms. The outcomes notably demonstrated that DNNs trained with the Bayesian algorithm exhibited superior accuracy in predicting stress-strain responses of the beams. Transitioning to the study's second objective, Response Surface Methodology (RSM) was harnessed to optimize Young's modulus and Specific Energy Absorption (SEA). The application of RSM impressively showcased its robustness in attaining optimal design values for these mechanical properties over the defined range of design parameters. This comprehensive exploration effectively reveals the potential of machine learning predictions and optimization techniques, providing valuable insights into the intricate mechanical behaviors of sandwich structures featuring innovative cores.

Skeleton-guided generation of synthetic noisy point clouds from as-built BIM to improve indoor scene understanding

The limited amount of high-quality training data available in indoor understanding with deep learning is a major problem. A possible solution to this problem is to use synthetic data to improve network training. In this study, a fully automatic method to generate synthetic noisy point clouds from as-built building information modeling (BIM) models is presented and it assesses the potential of these synthetic point clouds to improve deep neural network training. Based on a skeleton-guided strategy, all hypothetical scanning sites are located along the central axis of the buildings, which are obtained through equidistant sampling. Then, the synthetic labeled point cloud is generated station-by-station, and data augmentation is achieved using a random combination of data from different stations. The proposed approach involves generating over 44 sets of synthetic noisy point clouds based on BIM models. The performance of state-of-the-art (SOTA) deep learning methods in understanding indoor scenes enhanced by the synthetic point clouds is thoroughly assessed, and the effectiveness of various combinations of real and synthetic datasets is investigated. The experimental results demonstrate that leveraging synthetic point clouds generated from BIM models leads to a remarkable 5%–10% improvement in 3D semantic segmentation accuracy. The research signifies the value of synthetic point clouds as an effective tool for improving deep neural network training. All simulation datasets are publicly available, including original BIM models, full synthetic point clouds, and point clouds after IHPR processing, accessible via the BIMSyn Dataset link. In future research, an exploration of how synthetic point clouds will be further improved by considering specific characteristics of objects such as color, material reflectance, and illumination.

Active learning with label quality control.

Training deep neural networks requires a large number of labeled samples, which are typically provided by crowdsourced workers or professionals at a high cost. To obtain qualified labels, samples need to be relabeled for inspection to control the quality of the labels, which further increases the cost. Active learning methods aim to select the most valuable samples for labeling to reduce labeling costs. We designed a practical active learning method that adaptively allocates labeling resources to the most valuable unlabeled samples and the most likely mislabeled labeled samples, thus significantly reducing the overall labeling cost. We prove that the probability of our proposed method labeling more than one sample from any redundant sample set in the same batch is less than 1/k, where k is the number of the k-fold experiment used in the method, thus significantly reducing the labeling resources wasted on redundant samples. Our proposed method achieves the best level of results on benchmark datasets, and it performs well in an industrial application of automatic optical inspection.