Generalization Ability Of Network Research Articles

Identification method of thyroid nodule ultrasonography based on self-supervised learning dual-branch attention learning framework.

Thyroid ultrasound is a widely used diagnostic technique for thyroid nodules in clinical practice. However, due to the characteristics of ultrasonic imaging, such as low image contrast, high noise levels, and heterogeneous features, detecting and identifying nodules remains challenging. In addition, high-quality labeled medical imaging datasets are rare, and thyroid ultrasound images are no exception, posing a significant challenge for machine learning applications in medical image analysis. In this study, we propose a Dual-branch Attention Learning (DBAL) convolutional neural network framework to enhance thyroid nodule detection by capturing contextual information. Leveraging jigsaw puzzles as a pretext task during network training, we improve the network's generalization ability with limited data. Our framework effectively captures intrinsic features in a global-to-local manner. Experimental results involve self-supervised pre-training on unlabeled ultrasound images and fine-tuning using 1216 clinical ultrasound images from a collaborating hospital. DBAL achieves accurate discrimination of thyroid nodules, with a 88.5% correct diagnosis rate for malignant and benign nodules and a 93.7% area under the ROC curve. This novel approach demonstrates promising potential in clinical applications for its accuracy and efficiency.

Contextual recovery network for low-light image enhancement with texture recovery

Low-light image enhancement has been a challenging topic in computer vision. In order to recover colors and detailed textures in images, several data-driven enhancement based methods have been developed and obtained encouraging results. However, the network generalization ability is not satisfactory due to the uncertainty of the collected data. In order to address this issue, we propose a network with texture enhancement (TEENet), which synergizes the image brightness recovery and recovers the texture information lost during the process. To recover image brightness, we propose a low-light image enhancement network with a squeeze-excitation operation and construct a texture-optimized network that combines contextual aggregation information to recover texture loss during the enhancement process. We conducted network performance tests using different data and ablation experiments to verify the performance of different components.

A lightweight residual network based on improved knowledge transfer and quantized distillation for cross-domain fault diagnosis of rolling bearings

Predictive maintenance advocates the use of artificial intelligence to analyze big data and provides support for monitoring health conditions and planning maintenance activities in smart manufacturing systems. However, deep networks with massive parameters incur high computational costs, hindering its practical applications. Knowledge distillation (KD) transfers knowledge from a deep network to a lightweight network at the cost of an evident decline in diagnostic accuracy. Instead of an accuracy-size trade-off, a framework for creating a lightweight residual network with higher diagnostic accuracy is proposed in this paper. First, the generalization ability of the teacher network is enhanced based on a domain-adversarial neural network (DANN) with normalization attention mechanism (NAM) and ResNet18. Then, a small student network, ResNet6, efficiently learns the knowledge from its well-learned teacher network through the improved probability KD (IPKD) and uniform quantization. The IPKD is designed to obtain the utmost knowledge, represented as probability distributions of labels and feature space, and narrow the gap between teacher and student networks. Meanwhile, the uniform quantization is incorporated into the distillation process for quantization and distillation co-optimization, further compressing the student network. Experiments on two open-source and one wheelset bearing sets are conducted for performance verification and comparison. The results demonstrate that the obtained lightweight student network has a similar accuracy to the deep network for cross-domain fault diagnosis of bearings with various damages and speeds, even bearing sources, and its quantized network has a much smaller size while retaining a comparatively similar accuracy, which makes it more practical for actual industrial equipment.

Short-circuit Current-based Parametrically Identification for Doubly Fed Induction Generator

Recently, deep learning has provided a new opportunity to achieve high precision and real-time parameter identification of the doubly-fed induction generator (DFIG) in the event of short-circuit fault. However, deep learning algorithms based on data training are facing the challenge of relying on a large amount of training data and poor generalization performance. In order to improve these shortcomings, we embed the forward calculation model of three-phase short-circuit current (SCC) into the neural network, and propose an unsupervised neural network which can realize high-precision parameter identification. The network only needs to convert the short circuit current curve into a two-dimensional gray level map to complete the precise training of the network without real labels, which effectively improves the fitting ability of the network for inverse problems. The simulation results show that the proposed method can achieve high precision identification of DFIG parameters both within and outside the domain, and verify the high precision identification and generalization ability of unsupervised networks.

Evaluating the Generalization Ability of Super-Resolution Networks.

Performance and generalization ability are two important aspects to evaluate the deep learning models. However, research on the generalization ability of Super-Resolution (SR) networks is currently absent. Assessing the generalization ability of deep models not only helps us to understand their intrinsic mechanisms, but also allows us to quantitatively measure their applicability boundaries, which is important for unrestricted real-world applications. To this end, we make the first attempt to propose a Generalization Assessment Index for SR networks, namely SRGA. SRGA exploits the statistical characteristics of the internal features of deep networks to measure the generalization ability. Specially, it is a non-parametric and non-learning metric. To better validate our method, we collect a patch-based image evaluation set (PIES) that includes both synthetic and real-world images, covering a wide range of degradations. With SRGA and PIES dataset, we benchmark existing SR models on the generalization ability. This work provides insights and tools for future research on model generalization in low-level vision.

Distributed deep reinforcement learning based on bi-objective framework for multi-robot formation

Improving generalization ability in multi-robot formation can reduce repetitive training and calculation. In this paper, we study the multi-robot formation problem with the ability to generalize the target position. Since the generalization ability of neural network is directly proportional to spatial dimension, we adopt the strategy of using different networks to solve different objectives, so that the network learning can focus on the learning of one objective to obtain better performance. In addition, this paper presents a distributed deep reinforcement learning method based on soft actor–critic algorithm for solving multi-robot formation problem. At the same time, the formation evaluation assignment function is designed to adapt to distributed training. Compared with the original algorithm, the improved algorithm can get higher reward cumulative values. The experimental results show that the proposed algorithm can better maintain the desired formation in the moving process, and the rotation design in the reward function makes the multi-robot system have better flexibility in formation. The comparison of control signal curve shows that the proposed algorithm is more stable. At the end of the experiments, the universality of the proposed algorithm in formation maintenance and formation variations is demonstrated.

Fault Diagnosis of Analog Circuits Based on The Sailfish Algorithm Optimized SKELM

In view of the long training time for the optimization of the network model parameters of the SELM and the uncertainty of the model generalization ability, this paper proposes an analog circuit fault diagnosis model based on the sailfish algorithm to optimize the stacked kernel extreme learning machine(SKELM). This model introduces a kernel function to build a multi-layer KELM, which can improve the generalization ability and learning speed of the feedforward neural network. The weights of each layer of SKELM are obtained through the automatic encoder training based on the KELM. Since KELM-AE does not need to set initial parameters, the training speed is improved. However, the kernel parameters and regularization coefficients of KELM-AE are set manually, so the sailfish optimizer (SFO) is used to optimize these two parameters, and then the optimal SKELM model is built through layer by layer training. Finally, the Leap frog filter circuit is used as the simulation experiment circuit, and further compared with the optimized SELM. The results show that KELM-AE has strong generalization ability, and it can map fault features to high-dimensional feature space through nonlinear mapping without extracting fault features separately, thus improving the classification accuracy.

A Comprehensively Improved Interval Type-2 Fuzzy Neural Network for NOx Emissions Prediction in MSWI Process

The accurate and timely prediction of nitrogen oxides (NOx) emissions ensures eco-friendly and efficient operations for municipal solid waste incineration (MSWI) plants. Due to the high nonlinearity and uncertainty in MSWI processes, constructing an efficient prediction model remains challenging. This work proposes a comprehensively improved interval type-2 fuzzy neural network (CI-IT2FNN) for NOx emissions prediction. First, the neighborhood rough set (NRS) is introduced to determine the structure of this fuzzy neural network automatically, including the number of fuzzy rules and their corresponding consequent parameters. Second, an adaptive shape factor is added to the fuzzy membership function to better cope with the uncertainty, which can help to improve the generalization ability of network. Furthermore, to reduce the computational complexity, the Begian-Melek-Mendel (BMM) method is utilized as the defuzzification method in this study. Then, by integrating the linear least square estimation (LSE) and gradient decent (GD), a hierarchical learning algorithm is applied to adjust the network parameters to further enhance the learning efficiency and accuracy. Finally, after being evaluated by a benchmark simulation, the proposed CI-IT2FNN demonstrates its effectiveness and superiority on NOx emissions prediction.

The Planning & Designing of Urban Building Water Supply and Drainage Engineering in AI Era

Abstract To explore how to improve the quality of urban building water supply and drainage engineering and further promote the economic construction of urban buildings, this paper combines artificial intelligence with water supply and drainage engineering of urban buildings. A detailed analysis of water supply and drainage engineering is unfolded, and the combination of BP neural network algorithm and particle swarm algorithm is used to improve the convergence accuracy of the algorithm and network generalization ability to optimize the BP neural network algorithm and construct the artificial intelligence algorithm model. Through experiments in the artificial intelligence algorithm model, the actual output quality parameter is 82, and the difference with the desired output quality parameter of 81 is 1, while in the ordinary design, the actual output quality parameter is 70, and the difference with the desired output quality parameter is 11, get in the artificial intelligence model the desired output and the expected output close. It shows that combining artificial intelligence with urban building water supply and drainage engineering can improve the quality of the project, which is more accurate and superior to the ordinary design and reduce the project cost.

An Adaptive Activation Transfer Learning Approach for Fault Diagnosis

For industrial scenarios with changing operating conditions, the vibration data of different operating conditions often have different data distributions. In this article, to make the deep learning framework perform more flexible nonlinear transformations for different input data, a new activation function, i.e., parameter-free adaptively Swish (PASwish), is developed. PASwish formulates different activation schemes for different input data so that vibration data under different operating conditions can carry out adaptive nonlinear transformation, and the generalization ability of the whole network is improved. In addition, this article proposes deep parameter-free cosine networks with PASwish on the basis of PASwish, which can help adjust the network weights of domain-specific features and domain-invariant features by constructing an attention module based on cosine adjustment. Finally, the reconstruction-based domain adaptation method is used to achieve cross-domain fault diagnosis. Experiments are carried out on the bearing fault experimental platform to verify the effectiveness and generalization of the proposed method. We have achieved 95.16 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$ \pm 1.76\%$</tex-math></inline-formula> ) average accuracy on 72 transfer tasks, which shows better performance than current studies.

Data Augmented of Mechanical Fault Sound Signal based on Generative Adversarial Networks

In this paper, a global average pooling convolutional neural network based on CNN is proposed for mechanical fault sound detection, which called as GCMD. To solve the data scarcity of mechanical fault sound data, a spectrum frame selection augmented method based on log Mel spectrum feature is proposed to augment the original data, that aim is to train GCMD and generate counter networks. In order to solve the unbalance problem of data set and further improve the generalization ability of GCMD, an augmented neural network model based on CapsuleGAN was proposed, which called MFS-CapsuleGAN. The model was evaluated on the augmented data set by training GCMD neural network. Compared with the original data set, the accurate recognition rate of the model was improved by 23.7%. The performance of this method is improved significantly, which proves the feasibility and effectiveness of MFS-CapsuleGAN data augmented. In addition, the data set with background noise was used to test the generalization ability of GCMD network. The fluctuation range was within 0.117, indicating the good robustness of GCMD network.

One-dimensional magnetotelluric parallel inversion using a ResNet1D-8 residual neural network

Deep Learning is an effective tool to invert the underground electrical conductivity structure. In this study, we build a new 8-layer residual neural network (ResNet1D-8) for audio magnetotelluric (AMT) data inversion based on the deep learning theory. In terms of the network structure, the degradation of the model with the increase of depth is effectively avoided by adding shortcut connections. Meanwhile, adding a batch normalization layer greatly improves the model training speed and network generalization ability. This study uses the parallel technology to quickly generate millions of samples, which effectively reduces the computational time and provides a large number of high-quality samples for deep learning model training. Compared with the simulated annealing algorithm, the neural network model in this paper has the advantages of high reliability, short inversion time, and strong model generalization ability. Moreover, we add Gaussian noise to the data of testing samples, and inversion results show that the model has good robustness. The inversion test is carried out on the field measured AMT data set collected in the Dachaidan area of Qinghai Province, China. The results show that the ResNet1D-8 residual neural network model established in this paper can effectively invert the underground electrical structure.

DilatedFormer: dilated granularity transformer network for placental maturity grading in ultrasound

Placental maturity grading (PMG) is often utilized for evaluating fetal growth and maternal health. Currently, PMG often relied on the subjective judgment of the clinician, which is time-consuming and tends to incur a wrong estimation due to redundancy and repeatability of the process. The existing methods often focus on designing diverse hand-crafted features or combining deep features and hand-crafted features to learn a hybrid feature with an SVM for grading the placental maturity of ultrasound images. Motivated by the dominated performance of end-to-end convolutional neural networks (CNNs) at diverse medical imaging tasks, we devise a dilated granularity transformer network for learning multi-scale global transformer features for boosting PMG. Our network first devises dilated transformer blocks to learn multi-scale transformer features at each convolutional layer and then integrates these obtained multi-scale transformer features for predicting the final result of PMG. We collect 500 ultrasound images to verify our network, and experimental results show that our network clearly outperforms state-of-the-art methods on PMG. In the future, we will strive to improve the computational complexity and generalization ability of deep neural networks for PMG.

AdvMask: A sparse adversarial attack-based data augmentation method for image classification

Data augmentation has been an essential technique for improving the generalization ability of deep neural networks in image classification tasks. However, intensive changes in appearance and different degrees of occlusion in images are the key factors that severely affect the generalization ability of image classification models. Therefore, in order to enhance the generalization performance and robustness of deep models, data augmentation approaches by providing models with more diverse training data in various scenarios are widely applied. Although many existing data augmentation methods simulate occlusion in the augmented images to enhance the generalization of models, these methods randomly delete some areas in images without considering the semantic information of images. In this work, we propose a novel data augmentation method named AdvMask for image classification based on sparse adversarial attack techniques. AdvMask first identifies the key points that have the greatest influence on the classification results via a proposed end-to-end sparse adversarial attack module. During the data augmentation process, AdvMask efficiently generates diverse augmented data with structured occlusions based on the key points. By doing so, AdvMask can force deep models to seek other relevant content while the most discriminative content is hidden. Extensive experimental results on various benchmark datasets and deep models demonstrate that our proposed method can effectively improve the generalization performance of deep models and significantly outperforms previous data augmentation methods. Code for reproducing our results is available at https://github.com/Jackbrocp/AdvMask.

Disparity refinement framework for learning-based stereo matching methods in cross-domain setting for laparoscopic images.

Stereo matching methods that enable depth estimation are crucial for visualization enhancement applications in computer-assisted surgery. Learning-based stereo matching methods have shown great promise in making accurate predictions on laparoscopic images. However, they require a large amount of training data, and their performance may be degraded due to domain shifts. Maintaining robustness and improving the accuracy of learning-based methods are still open problems. To overcome the limitations of learning-based methods, we propose a disparity refinement framework consisting of a local disparity refinement method and a global disparity refinement method to improve the results of learning-based stereo matching methods in a cross-domain setting. Those learning-based stereo matching methods are pre-trained on a large public dataset of natural images and are tested on two datasets of laparoscopic images. Qualitative and quantitative results suggest that our proposed disparity framework can effectively refine disparity maps when they are noise-corrupted on an unseen dataset, without compromising prediction accuracy when the network can generalize well on an unseen dataset. Our proposed disparity refinement framework could work with learning-based methods to achieve robust and accurate disparity prediction. Yet, as a large laparoscopic dataset for training learning-based methods does not exist and the generalization ability of networks remains to be improved, the incorporation of the proposed disparity refinement framework into existing networks will contribute to improving their overall accuracy and robustness associated with depth estimation.

Inverse-Reference Priors for Fisher Regularization of Bayesian Neural Networks

Recent studies have shown that the generalization ability of deep neural networks (DNNs) is closely related to the Fisher information matrix (FIM) calculated during the early training phase. Several methods have been proposed to regularize the FIM for increased generalization of DNNs. However, they cannot be used directly for Bayesian neural networks (BNNs) because the variable parameters of BNNs make it difficult to calculate the FIM. To address this problem, we achieve regularization of the FIM of BNNs by specifying a new suitable prior distribution called the inverse-reference (IR) prior. To regularize the FIM, the IR prior is derived as the inverse of the reference prior that imposes minimal prior knowledge on the parameters and maximizes the trace of the FIM. We demonstrate that the IR prior can enhance the generalization ability of BNNs for large-scale data over previously used priors while providing adequate uncertainty quantifications using various benchmark image datasets and BNN structures.

SGDAT: An optimization method for binary neural networks

Stochastic gradient descent (SGD), one of the most popular neural network optimization algorithms, has a solid theoretical foundation as well as good generalization performance. However vanilla SGD performs catastrophically in Binary Neural Networks (BNNs). Many studies have identified this phenomenon without explaining its causes in depth. In this paper, we try to experimentally understand the possible reasons for this and significantly improve the performance of vanilla SGD in BNNs training by adjusting the training strategy to be comparable to Adam. We subsequently propose a new optimization method for training deep neural networks (DNNs) with binary weights. In the proposed SGD with Adaptive Threshold, referred to as SGDAT, we suppress the frequency of weights flipping by thresholds and adjust the threshold of each parameter according to the number of flipping to further reduce the network noise, stabilize the network training, and improve the network generalization ability. Also, we present a complete ablation study of the hyperparameters space, as well as experimentally analyze the impact of using adaptive thresholds. Furthermore, we conduct image classification experiments over the CIFAR10, CIFAR100 and TinyImageNet datasets using BinaryNet and ResNet-18 network structure. The experiments show that SGDAT outperforms other binary optimizers. Code is available at: https://github.com/gushan/SGDAT.

AMC: Adaptive Multi-expert Collaborative Network for Text-guided Image Retrieval

Text-guided image retrieval integrates reference image and text feedback as a multimodal query to search the image corresponding to user intention. Recent approaches employ multi-level matching, multiple accesses, or multiple subnetworks for better performance regardless of the heavy burden of storage and computation in the deployment. Additionally, these models not only rely on expert knowledge to handcraft image-text composing modules but also do inference by the static computational graph. It limits the representation capability and generalization ability of networks in the face of challenges from complex and varied combinations of reference image and text feedback. To break the shackles of the static network concept, we introduce the dynamic router mechanism to achieve data-dependent expert activation and flexible collaboration of multiple experts to explore more implicit multimodal fusion patterns. Specifically, we construct AMC, our A daptive M ulti-expert C ollaborative network, by using the proposed router to activate the different experts with different levels of image-text interaction. Since routers can dynamically adjust the activation of experts for the current samples, AMC can achieve the adaptive fusion mode for the different reference image and text combinations and generate dynamic computational graphs according to varied multimodal queries. Extensive experiments on two benchmark datasets demonstrate that due to benefits from the image-text composing representation produced by an adaptive multi-expert collaboration mechanism, AMC has better retrieval performance and zero-shot generalization ability than the state-of-the-art method while keeping the lightweight model and fast retrieval speed. Moreover, we analyze the visualization of path activation, attention map, and retrieval results to further understand the routing decisions and semantic localization ability of AMC. The codes and pretrained models are available at https://github.com/KevinLight831/AMC .

Elasticity-mechanics-informed generative adversarial networks for predicting the thermal strain of thermal barrier coatings penetrated by CaO–MgO–Al2O3–SiO2

Air-plasma-sprayed thermal barrier coatings have complex and variable microstructures. As a result, they require repeated numerical or experimental analyses in each case, which increases the computation time and limits the optimization of the coating structure and processing parameters. In this work, a neural network-based framework for rapidly evaluating the thermal strain of thermal barrier coatings is proposed. The deformation compatibility condition and the generalized Hooke's law considering thermal expansion are applied to constrain the network. The training database is established using one high-resolution scanning electron microscopy structural image. The proposed model is validated using another image database with different microstructure characteristics and cooling temperature ranges. The model strains generated by the proposed model, the general pixel-to-pixel network, and finite-element models are compared. The results show that the constraint equations improve the generalization ability of the neural networks, and the proposed model exhibits higher prediction accuracy and confidence than general networks.

Predicting indoor particle concentration in mechanically ventilated classrooms using neural networks: Model development and generalization ability analysis

This study presents a comprehensive analysis of the model development (structure, parameter settings, and prediction accuracy) and generalization ability of neural networks in the classroom setting. A normalized database for model prediction and verification including indoor parameters from 27 high school classrooms over the course of four semesters was set up (a total of 14,112 sets of data for each variable). Three different models were compared: a back propagation artificial neural network (BP-ANN), a nonlinear autoregressive exogenous neural network (NARX), and a long-term and short-term memory neural network (LSTM). For the prediction of PM1 and PM2.5, which are impacted by outdoor PM concentrations, the best performance was achieved by the NARX model: the mean absolute percentage error (MAPE), root mean squared error (RMSE), and coefficient of determination (R2) were 41–55%, 0.45–1.27 μg/m3, and 0.81–0.87, respectively. The R2 of this model was 22% higher than that of the BP-ANN model and much higher than that of the LSTM model. For the prediction of indoor PM10, which are mostly emitted by indoor sources, relatively good performance was achieved by the LSTM model. The trained NARX model was used successfully to predict indoor time-series PM2.5 concentration and source strength in classrooms in five Chinese cities and five American cities. The calculated indoor source strength was 95.5–119.1 μg/h/P, within the range of the measured indoor source strength. The indoor PM2.5 concentrations calculated by a theoretical equation without the consideration of indoor source were 20.8–60.0% smaller than those calculated by the trained model.