Generalization Ability Of Network Research Articles

Stereo matching from monocular images using feature consistency

AbstractSynthetic images facilitate stereo matching. However, synthetic images may suffer from image distortion, domain bias, and stereo mismatch, which would significantly restrict the widespread use of stereo matching models in the real world. The first goal in this paper is to synthesize real‐looking images for minimizing the domain bias between the synthesized and real images. For this purpose, sharpened disparity maps are produced from a mono real image. Then, stereo image pairs are synthesized using these imperfect disparity maps and the single real image in the proposed pipeline. Although the synthesized images are as realistic as possible, the domain styles of the synthesized images are always very different from the real images. Thus, the second goal is to enhance the domain generalization ability of the stereo matching network. For that, the feature extraction layer is replaced with a teacher–student model. Then, a constraint of binocular contrast features is imposed on the output of the model. When tested on the KITTI, ETH3D, and Middlebury datasets, the accuracy of the method outperforms traditional methods by at least 30%. Experiments demonstrate that the approaches are general and can be conveniently embedded into existing stereo networks.

AAGCN: a graph convolutional neural network with adaptive feature and topology learning

In recent years, there has been a growing prevalence of deep learning in various domains, owing to advancements in information technology and computing power. Graph neural network methods within deep learning have shown remarkable capabilities in processing graph-structured data, such as social networks and traffic networks. As a result, they have garnered significant attention from researchers.However, real-world data often face challenges like data sparsity and missing labels, which can hinder the performance and generalization ability of graph convolutional neural networks. To overcome these challenges, our research aims to effectively extract the hidden features and topological information of graph convolutional neural networks. We propose an innovative model called Adaptive Feature and Topology Graph Convolutional Neural Network (AAGCN). By incorporating an adaptive layer, our model preprocesses the data and integrates the hidden features and topological information with the original data’s features and structure. These fused features are then utilized in the convolutional layer for training, significantly enhancing the expressive power of graph convolutional neural networks.To evaluate the effectiveness of the adaptive layer in the AAGCN model, we conducted node classification experiments on real datasets. The results validate its ability to address data sparsity and improve the classification performance of graph convolutional neural networks.In conclusion, our research primarily focuses on addressing data sparsity and missing labels in graph convolutional neural networks. The proposed AAGCN model, which incorporates an adaptive layer, effectively extracts hidden features and topological information, thereby enhancing the expressive power and classification performance of these networks.

A gradient descent algorithm for SNN with time-varying weights for reliable multiclass interpretation

Interpretation of the prediction is vital for mission critical tasks. Accurate interpretation relies upon the generalization accuracy of the model. In this paper, we propose a modified gradient descent learning algorithm to improve the generalization ability of a Spiking Neural Network with time-varying weights (SNN-t). This algorithm is referred to as GradST, can help towards improving the interpretation of multiclass classification problems. We have transformed the SNN-t to a Generalized Additive Model (GAM) to provide interpretation. The resultant Spiking Additive Model (SAM) has the generalization ability of SNN-t and the interpretable characteristics of GAM for multiclass problems. We also propose a post-processing method to enable better visualization of multiple shape functions of GAMs, towards better relative interpretation for multiclass classification problems. The post-processing method utilizes the properties of multiclass GAMs to visually modify the shape functions to establish the importance of the feature in multiclass setting. We first evaluate the performance of SNN-t, trained with GradST and the SAM generated from it, on large public datasets. The SNN-t trained with GradST has better generalization accuracy than other SNN-t classifier and consequently, the SAM generated from it has better generalization accuracy than other state-of-the-art multiclass GAMs. Improved accuracy in SAM implies more reliable interpretation. Then, we evaluate the proposed post-processing method for multiclass GAMs to provide relative interpretation. It is observed that relative interpretation of multiclass GAM is more meaningful and reliable.

A hybrid network with DNN and WGAN for supercontinum prediction

In this paper, a Hybrid Network is reported for predicting the output pulse shape parameters of the supercontinuum in optical fibers. The network incorporates the Wasserstein generative adversarial network (WGAN) and the deep neural network (DNN) with a TCN layer. The WGAN augments a limited set of original data, while the designed DNN accurately predicts supercontinuum. The simulation results show the spectra agreement (Agm) of the predictions by individual DNN is 89% and the root mean square error (RMSE) is 0.1092. After retraining with WGAN's extended dataset, the Agm of DNN predictions is improved to 98% and the RMSE is reduced to 0.0513. The input pump power corresponding to the appearance of the fourth soliton peak can be predicted by the retrained DNN. Numerical analysis demonstrates the effectiveness and generalization ability of a Hybrid Network in predicting pulse parameters. The network fits the propagation dynamics in the spectral domain of SC, which provides a promising tool for further research in this field.

Multi-View Synthesis of Sparse Projection of Absorption Spectra Based on Joint GRU and U-Net

Tunable diode laser absorption spectroscopy (TDLAS) technology, combined with chromatographic imaging algorithms, is commonly used for two-dimensional temperature and concentration measurements in combustion fields. However, obtaining critical temperature information from limited detection data is a challenging task in practical engineering applications due to the difficulty of deploying sufficient detection equipment and the lack of sufficient data to invert temperature and other distributions in the combustion field. Therefore, we propose a sparse projection multi-view synthesis model based on U-Net that incorporates the sequence learning properties of gated recurrent unit (GRU) and the generalization ability of residual networks, called GMResUNet. The datasets used for training all contain projection data with different degrees of sparsity. This study shows that the synthesized full projection data had an average relative error of 0.35%, a PSNR of 40.726, and a SSIM of 0.997 at a projection angle of 4. At projection angles of 2, 8, and 16, the average relative errors of the synthesized full projection data were 0.96%, 0.19%, and 0.18%, respectively. The temperature field reconstruction was performed separately for sparse and synthetic projections, showing that the application of the model can significantly improve the reconstruction accuracy of the temperature field of high-energy combustion.

Study on Gesture Recognition Method with Two-Stream Residual Network Fusing sEMG Signals and Acceleration Signals.

Currently, surface EMG signals have a wide range of applications in human-computer interaction systems. However, selecting features for gesture recognition models based on traditional machine learning can be challenging and may not yield satisfactory results. Considering the strong nonlinear generalization ability of neural networks, this paper proposes a two-stream residual network model with an attention mechanism for gesture recognition. One branch processes surface EMG signals, while the other processes hand acceleration signals. Segmented networks are utilized to fully extract the physiological and kinematic features of the hand. To enhance the model's capacity to learn crucial information, we introduce an attention mechanism after global average pooling. This mechanism strengthens relevant features and weakens irrelevant ones. Finally, the deep features obtained from the two branches of learning are fused to further improve the accuracy of multi-gesture recognition. The experiments conducted on the NinaPro DB2 public dataset resulted in a recognition accuracy of 88.25% for 49 gestures. This demonstrates that our network model can effectively capture gesture features, enhancing accuracy and robustness across various gestures. This approach to multi-source information fusion is expected to provide more accurate and real-time commands for exoskeleton robots and myoelectric prosthetic control systems, thereby enhancing the user experience and the naturalness of robot operation.

Multi-scale spatial pyramid attention mechanism for image recognition: An effective approach

Attention mechanisms have gradually become necessary to enhance the representational power of convolutional neural networks (CNNs). Despite recent progress in attention mechanism research, some open problems still exist. Most existing methods ignore modeling multi-scale feature representations, structural information, and long-range channel dependencies, which are essential for delivering more discriminative attention maps. This study proposes a novel, low-overhead, high-performance attention mechanism with strong generalization ability for various networks and datasets. This mechanism is called Multi-Scale Spatial Pyramid Attention (MSPA) and can be used to solve the limitations of other attention methods. For the critical components of MSPA, we not only develop the Hierarchical-Phantom Convolution (HPC) module, which can extract multi-scale spatial information at a more granular level utilizing hierarchical residual-like connections, but also design the Spatial Pyramid Recalibration (SPR) module, which can integrate structural regularization and structural information in an adaptive combination mechanism, while employing the Softmax operation to build long-range channel dependencies. The proposed MSPA is a powerful tool that can be conveniently embedded into various CNNs as a plug-and-play component. Correspondingly, using MSPA to replace the 3 × 3 convolution in the bottleneck residual blocks of ResNets, we created a series of simple and efficient backbones named MSPANet, which naturally inherit the advantages of MSPA. Without bells and whistles, our method substantially outperforms other state-of-the-art counterparts in all evaluation metrics based on extensive experimental results from CIFAR-100 and ImageNet-1K image recognition. When applying MSPA to ResNet-50, our model achieves top-1 classification accuracy of 81.74% and 78.40% on the CIFAR-100 and ImageNet-1K benchmarks, exceeding the corresponding baselines by 3.95% and 2.27%, respectively. We also obtained promising performance improvements of 1.15% and 0.91% compared to the competitive EPSANet-50. In addition, empirical research results in autonomous driving engineering applications also demonstrate that our method can significantly improve the accuracy and real-time performance of image recognition with cheaper overhead. Our code is publicly available at https://github.com/ndsclark/MSPANet.

Simplicity Bias in Overparameterized Machine Learning

A thorough theoretical understanding of the surprising generalization ability of deep networks (and other overparameterized models) is still lacking. Here we demonstrate that simplicity bias is a major phenomenon to be reckoned with in overparameterized machine learning. In addition to explaining the outcome of simplicity bias, we also study its source: following concrete rigorous examples, we argue that (i) simplicity bias can explain generalization in overparameterized learning models such as neural networks; (ii) simplicity bias and excellent generalization are optimizer-independent, as our example shows, and although the optimizer affects training, it is not the driving force behind simplicity bias; (iii) simplicity bias in pre-training models, and subsequent posteriors, is universal and stems from the subtle fact that uniformly-at-random constructed priors are not uniformly-at-random sampled ; and (iv) in neural network models, the biasing mechanism in wide (and shallow) networks is different from the biasing mechanism in deep (and narrow) networks.

An improved asynchronous batch gradient method for ridge polynomial neural network

The ridge polynomial neural network composed of pi-sigma modules is a typical higher-order feedforward network, which has good non-linear mapping capabilities. Due to the strong coupling of the network structure, the synchronous gradient method can easily result in significant fluctuations in updated weights. This will reduce the generalization ability of the ridge polynomial neural network for solving classification problems. In this paper, the proposed method is a novel improved asynchronous batch gradient method, and combined with the adaptive parameters of the activation function are trained synchronously. We strictly prove the convergence theorem of the improved method. The numerical experimental results also indicate that our method for training ridge polynomial neural networks can help the weight changes smoothly and the network has good generalization ability. The feasibility and effectiveness of our method are verified from both theoretical and experimental perspectives.

Diesel engine fault diagnosis for multiple industrial scenarios based on transfer learning

Fault diagnosis based on data-driven intelligence has recently attracted extensive interest owing to the rapid development of big data and deep-learning algorithms. However, when the amount of faulty data is limited, deep learning training is prone to overfitting. When the application scenario is changed, the generalization ability of the trained network is affected. In this study, a fault diagnosis architecture based on deep transfer learning is proposed to work with limited data and transfer between multiple scenarios. A wide convolution kernel convolutional long short-term memory neural network (WCL) was used to improve the feature extraction ability of fault data from a diesel engine with a low signal-to-noise ratio. A multiple transfer learning scheme based on WCL was further adopted to transfer the well-trained diagnostic knowledge of large-scale labeled source domain data to the target domain with limited samples. In addition, for diesel engines for various purposes, the knowledge transferability between different scenarios was studied. The algorithm evaluates the transfer performance of four different domains when the sample is insufficient, including the cross-fault type, cross-equipment type, cross-fault degree, and cross-working conditions. The results show the proposed method is proven with high noise immunity improves the accuracy of small sample cross-domain diagnosis and provides an optimal transfer scheme suitable for diesel engine fault signals.

A fault detection method for transmission line components based on synthetic dataset and improved YOLOv5

To address insufficient fault images for transmission line components (TLCs) and low accuracy in fault detection using deep learning techniques, we propose a fault detection method for TLCs based on synthetic datasets and improved YOLOv5. First, we introduce a synthetic image approach that uses prior information from the inspection process of a developed flying-walking power transmission line inspection robot (FPTLIR) to generate a synthetic dataset of fault components (SDFC). Second, we propose an improved YOLOv5 network called CSH-YOLOv5 to improve the accuracy of fault detection. The CSH-YOLOv5 network incorporates the convolutional block attention module (CBAM) and the latest SimCSPSPPF module to improve the detection accuracy of the network. In addition, a statistical analysis of small objects in the SDFC is performed and the neck and head of the YOLOv5 network are optimized accordingly to detect small objects. Finally, to address the lack of fault images in the unmanned aerial vehicle (UAV) inspection dataset, we implement a two-stage transfer learning strategy using the SDFC for training. We then experimentally evaluate the performance of the CSH-YOLOv5 network on a real test dataset. The results show that the CSH-YOLOv5 network achieves a mAP@[0.5] of 98.0% and a mAP@[0.5: 0.95] of 64.6% for fault detection, representing an improvement of 7.6% and 8.1%, respectively, over the YOLOv5 network. Comparative analysis indicates that the CSH-YOLOv5 outperforms other popular object detection networks, including Faster-RCNN, YOLOX, and YOLOv7 networks. The two-stage transfer learning strategy employed significantly enhances the network's generalization ability and detection accuracy on the UAV inspection dataset. The proposed method provides a technical reference for fault detection of TLCs, which can potentially benefit the power transmission industry.

Optimization of track and field training methods based on SSA-BP and its effect on athletes' explosive power

Digitalization and informationization are important trends in the development of the sports industry. The study first introduced the sparrow search algorithm to improve the generalization ability of traditional neural networks, optimizing the assignment of initial weights and thresholds of neural networks; Secondly, the chicken swarm algorithm is introduced to optimize the training combination, period, and intensity of athletes based on the evaluation results, improving the subjective limitations of traditional training methods. The results of model performance analysis show that the sparrow search algorithm is better than other intelligent optimization algorithms in finding fitted parameters, and the solution error is less than 0.50. The evaluation model performs well in terms of accuracy, recall, average relative error, and R2 evaluation indicators. The model has high repeatability and is suitable for evaluating track and field training methods. The accuracy and computational speed of the chicken swarm algorithm are relatively good; Compared with other optimization models, the chicken swarm algorithm has better optimization ability and accuracy. Friedman test found significant differences in the chicken swarm algorithm, and the optimized training method has a significant positive impact on the explosive power of athletes, and the training period and intensity arrangement are reasonable and more helpful to the improvement of athletic performance. This study improves the scientific rationality of the development of track and field training methods, which is conducive to optimizing the training effect of track and field sports, and facilitates the risk management and personalized training of athletes. At the same time, it greatly promotes the integration and development of sports and computer disciplines.

Identification method of thyroid nodule ultrasonography based on self-supervised learning dual-branch attention learning framework.

Thyroid ultrasound is a widely used diagnostic technique for thyroid nodules in clinical practice. However, due to the characteristics of ultrasonic imaging, such as low image contrast, high noise levels, and heterogeneous features, detecting and identifying nodules remains challenging. In addition, high-quality labeled medical imaging datasets are rare, and thyroid ultrasound images are no exception, posing a significant challenge for machine learning applications in medical image analysis. In this study, we propose a Dual-branch Attention Learning (DBAL) convolutional neural network framework to enhance thyroid nodule detection by capturing contextual information. Leveraging jigsaw puzzles as a pretext task during network training, we improve the network's generalization ability with limited data. Our framework effectively captures intrinsic features in a global-to-local manner. Experimental results involve self-supervised pre-training on unlabeled ultrasound images and fine-tuning using 1216 clinical ultrasound images from a collaborating hospital. DBAL achieves accurate discrimination of thyroid nodules, with a 88.5% correct diagnosis rate for malignant and benign nodules and a 93.7% area under the ROC curve. This novel approach demonstrates promising potential in clinical applications for its accuracy and efficiency.

Contextual recovery network for low-light image enhancement with texture recovery

Low-light image enhancement has been a challenging topic in computer vision. In order to recover colors and detailed textures in images, several data-driven enhancement based methods have been developed and obtained encouraging results. However, the network generalization ability is not satisfactory due to the uncertainty of the collected data. In order to address this issue, we propose a network with texture enhancement (TEENet), which synergizes the image brightness recovery and recovers the texture information lost during the process. To recover image brightness, we propose a low-light image enhancement network with a squeeze-excitation operation and construct a texture-optimized network that combines contextual aggregation information to recover texture loss during the enhancement process. We conducted network performance tests using different data and ablation experiments to verify the performance of different components.

A lightweight residual network based on improved knowledge transfer and quantized distillation for cross-domain fault diagnosis of rolling bearings

Predictive maintenance advocates the use of artificial intelligence to analyze big data and provides support for monitoring health conditions and planning maintenance activities in smart manufacturing systems. However, deep networks with massive parameters incur high computational costs, hindering its practical applications. Knowledge distillation (KD) transfers knowledge from a deep network to a lightweight network at the cost of an evident decline in diagnostic accuracy. Instead of an accuracy-size trade-off, a framework for creating a lightweight residual network with higher diagnostic accuracy is proposed in this paper. First, the generalization ability of the teacher network is enhanced based on a domain-adversarial neural network (DANN) with normalization attention mechanism (NAM) and ResNet18. Then, a small student network, ResNet6, efficiently learns the knowledge from its well-learned teacher network through the improved probability KD (IPKD) and uniform quantization. The IPKD is designed to obtain the utmost knowledge, represented as probability distributions of labels and feature space, and narrow the gap between teacher and student networks. Meanwhile, the uniform quantization is incorporated into the distillation process for quantization and distillation co-optimization, further compressing the student network. Experiments on two open-source and one wheelset bearing sets are conducted for performance verification and comparison. The results demonstrate that the obtained lightweight student network has a similar accuracy to the deep network for cross-domain fault diagnosis of bearings with various damages and speeds, even bearing sources, and its quantized network has a much smaller size while retaining a comparatively similar accuracy, which makes it more practical for actual industrial equipment.

Short-circuit Current-based Parametrically Identification for Doubly Fed Induction Generator

Recently, deep learning has provided a new opportunity to achieve high precision and real-time parameter identification of the doubly-fed induction generator (DFIG) in the event of short-circuit fault. However, deep learning algorithms based on data training are facing the challenge of relying on a large amount of training data and poor generalization performance. In order to improve these shortcomings, we embed the forward calculation model of three-phase short-circuit current (SCC) into the neural network, and propose an unsupervised neural network which can realize high-precision parameter identification. The network only needs to convert the short circuit current curve into a two-dimensional gray level map to complete the precise training of the network without real labels, which effectively improves the fitting ability of the network for inverse problems. The simulation results show that the proposed method can achieve high precision identification of DFIG parameters both within and outside the domain, and verify the high precision identification and generalization ability of unsupervised networks.

Evaluating the Generalization Ability of Super-Resolution Networks.

Performance and generalization ability are two important aspects to evaluate the deep learning models. However, research on the generalization ability of Super-Resolution (SR) networks is currently absent. Assessing the generalization ability of deep models not only helps us to understand their intrinsic mechanisms, but also allows us to quantitatively measure their applicability boundaries, which is important for unrestricted real-world applications. To this end, we make the first attempt to propose a Generalization Assessment Index for SR networks, namely SRGA. SRGA exploits the statistical characteristics of the internal features of deep networks to measure the generalization ability. Specially, it is a non-parametric and non-learning metric. To better validate our method, we collect a patch-based image evaluation set (PIES) that includes both synthetic and real-world images, covering a wide range of degradations. With SRGA and PIES dataset, we benchmark existing SR models on the generalization ability. This work provides insights and tools for future research on model generalization in low-level vision.

Distributed deep reinforcement learning based on bi-objective framework for multi-robot formation

Improving generalization ability in multi-robot formation can reduce repetitive training and calculation. In this paper, we study the multi-robot formation problem with the ability to generalize the target position. Since the generalization ability of neural network is directly proportional to spatial dimension, we adopt the strategy of using different networks to solve different objectives, so that the network learning can focus on the learning of one objective to obtain better performance. In addition, this paper presents a distributed deep reinforcement learning method based on soft actor–critic algorithm for solving multi-robot formation problem. At the same time, the formation evaluation assignment function is designed to adapt to distributed training. Compared with the original algorithm, the improved algorithm can get higher reward cumulative values. The experimental results show that the proposed algorithm can better maintain the desired formation in the moving process, and the rotation design in the reward function makes the multi-robot system have better flexibility in formation. The comparison of control signal curve shows that the proposed algorithm is more stable. At the end of the experiments, the universality of the proposed algorithm in formation maintenance and formation variations is demonstrated.

Fault Diagnosis of Analog Circuits Based on The Sailfish Algorithm Optimized SKELM

In view of the long training time for the optimization of the network model parameters of the SELM and the uncertainty of the model generalization ability, this paper proposes an analog circuit fault diagnosis model based on the sailfish algorithm to optimize the stacked kernel extreme learning machine(SKELM). This model introduces a kernel function to build a multi-layer KELM, which can improve the generalization ability and learning speed of the feedforward neural network. The weights of each layer of SKELM are obtained through the automatic encoder training based on the KELM. Since KELM-AE does not need to set initial parameters, the training speed is improved. However, the kernel parameters and regularization coefficients of KELM-AE are set manually, so the sailfish optimizer (SFO) is used to optimize these two parameters, and then the optimal SKELM model is built through layer by layer training. Finally, the Leap frog filter circuit is used as the simulation experiment circuit, and further compared with the optimized SELM. The results show that KELM-AE has strong generalization ability, and it can map fault features to high-dimensional feature space through nonlinear mapping without extracting fault features separately, thus improving the classification accuracy.

A Comprehensively Improved Interval Type-2 Fuzzy Neural Network for NOx Emissions Prediction in MSWI Process

The accurate and timely prediction of nitrogen oxides (NOx) emissions ensures eco-friendly and efficient operations for municipal solid waste incineration (MSWI) plants. Due to the high nonlinearity and uncertainty in MSWI processes, constructing an efficient prediction model remains challenging. This work proposes a comprehensively improved interval type-2 fuzzy neural network (CI-IT2FNN) for NOx emissions prediction. First, the neighborhood rough set (NRS) is introduced to determine the structure of this fuzzy neural network automatically, including the number of fuzzy rules and their corresponding consequent parameters. Second, an adaptive shape factor is added to the fuzzy membership function to better cope with the uncertainty, which can help to improve the generalization ability of network. Furthermore, to reduce the computational complexity, the Begian-Melek-Mendel (BMM) method is utilized as the defuzzification method in this study. Then, by integrating the linear least square estimation (LSE) and gradient decent (GD), a hierarchical learning algorithm is applied to adjust the network parameters to further enhance the learning efficiency and accuracy. Finally, after being evaluated by a benchmark simulation, the proposed CI-IT2FNN demonstrates its effectiveness and superiority on NOx emissions prediction.