Generalization Capability Of Networks Research Articles

Multidomain Object Detection Framework Using Feature Domain Knowledge Distillation.

Object detection techniques have been widely studied, utilized in various works, and have exhibited robust performance on images with sufficient luminance. However, these approaches typically struggle to extract valuable features from low-luminance images, which often exhibit blurriness and dim appearence, leading to detection failures. To overcome this issue, we introduce an innovative unsupervised feature domain knowledge distillation (KD) framework. The proposed framework enhances the generalization capability of neural networks across both low-and high-luminance domains without incurring additional computational costs during testing. This improvement is made possible through the integration of generative adversarial networks and our proposed unsupervised KD process. Furthermore, we introduce a region-based multiscale discriminator designed to discern feature domain discrepancies at the object level rather than from the global context. This bolsters the joint learning process of object detection and feature domain distillation tasks. Both qualitative and quantitative assessments shown that the proposed method, empowered by the region-based multiscale discriminator and the unsupervised feature domain distillation process, can effectively extract beneficial features from low-luminance images, outperforming other state-of-the-art approaches in both low-and sufficient-luminance domains.

SGDA: Towards 3D Universal Pulmonary Nodule Detection via Slice Grouped Domain Attention.

Lung cancer is the leading cause of cancer death worldwide. The best solution for lung cancer is to diagnose the pulmonary nodules in the early stage, which is usually accomplished with the aid of thoracic computed tomography (CT). As deep learning thrives, convolutional neural networks (CNNs) have been introduced into pulmonary nodule detection to help doctors in this labor-intensive task and demonstrated to be very effective. However, the current pulmonary nodule detection methods are usually domain-specific, and cannot satisfy the requirement of working in diverse real-world scenarios. To address this issue, we propose a slice grouped domain attention (SGDA) module to enhance the generalization capability of the pulmonary nodule detection networks. This attention module works in the axial, coronal, and sagittal directions. In each direction, we divide the input feature into groups, and for each group, we utilize a universal adapter bank to capture the feature subspaces of the domains spanned by all pulmonary nodule datasets. Then the bank outputs are combined from the perspective of domain to modulate the input group. Extensive experiments demonstrate that SGDA enables substantially better multi-domain pulmonary nodule detection performance compared with the state-of-the-art multi-domain learning methods.

On the generalization capability of artificial neural networks used to estimate fretting fatigue life

This study assesses the generalization capacity of artificial neural networks (ANNs) for predicting fretting fatigue of mechanical contacts using different materials and geometries. These ANN utilize as inputs, material properties and stress quantities that have been physically related to the fatigue crack initiation mechanism under multiaxial loading. Initially trained and validated on aeronautical aluminum alloys data, one tests their generalization performance by applying them to fretting fatigue data for Ti‐6Al‐4V, ASTM A743 CA6NM steel, and Al 7050-T745, employing both cylindrical and spherical pads under various loading conditions, including out-of-phase loading. The ANNs adeptly predict fatigue lives across this extensive dataset, surpassing classical multiaxial fatigue criteria in accuracy. This underscores the effectiveness of ANN-based methodologies in diverse fretting fatigue scenarios.

Feature-wise scaling and shifting: Improving the generalization capability of neural networks through capturing independent information of features

From the perspective of input features, information can be divided into independent information and correlation information. Current neural networks mainly concentrate on the capturing of correlation information through connection weight parameters supplemented by bias parameters. This paper introduces feature-wise scaling and shifting (FwSS) into neural networks for capturing independent information of features, and proposes a new neural network FwSSNet. In the network, a pair of scale and shift parameters is added before each input of each network layer, and bias is removed. The parameters are initialized as 1 and 0, respectively, and trained at separate learning rates, to guarantee the fully capturing of independence and correlation information. The learning rates of FwSS parameters depend on input data and the training speed ratios of adjacent FwSS and connection sublayers, meanwhile those of weight parameters remain unchanged as plain networks. Further, FwSS unifies the scaling and shifting operations in batch normalization (BN), and FwSSNet with BN is established through introducing a preprocessing layer. FwSS parameters except those in the last layer of the network can be simply trained at the same learning rate as weight parameters. Experiments show that FwSS is generally helpful in improving the generalization capability of both fully connected neural networks and deep convolutional neural networks, and FWSSNets achieve higher accuracies on UCI repository and CIFAR-10.

Decoupled knowledge distillation method based on meta-learning

With the advancement of deep learning techniques, the number of model parameters has been increasing, leading to significant memory consumption and limits in the deployment of such models in real-time applications. To reduce the number of model parameters and enhance the generalization capability of neural networks, we propose a method called Decoupled MetaDistil, which involves decoupled meta-distillation. This method utilizes meta-learning to guide the teacher model and dynamically adjusts the knowledge transfer strategy based on feedback from the student model, thereby improving the generalization ability. Furthermore, we introduce a decoupled loss method to explicitly transfer positive sample knowledge and explore the potential of negative samples knowledge. Extensive experiments demonstrate the effectiveness of our method.

Elastic Adaptively Parametric Compounded Units for Convolutional Neural Network

The activation function introduces nonlinearity into convolutional neural network, which greatly promotes the development of computer vision tasks. This paper proposes elastic adaptively parametric compounded units to improve the performance of convolutional neural networks for image recognition. The activation function takes the structural advantages of two mainstream functions as the function’s fundamental architecture. The SENet model is embedded in the proposed activation function to adaptively recalibrate the feature mapping weight in each channel, thereby enhancing the fitting capability of the activation function. In addition, the function has an elastic slope in the positive input region by simulating random noise to improve the generalization capability of neural networks. To prevent the generated noise from producing overly large variations during training, a special protection mechanism is adopted. In order to verify the effectiveness of the activation function, this paper uses CIFAR-10 and CIFAR-100 image datasets to conduct comparative experiments of the activation function under the exact same model. Experimental results show that the proposed activation function showed superior performance beyond other functions.

Improving generalization of convolutional neural network through contrastive augmentation

Data augmentation is widely used to improve the generalization ability of convolutional neural networks in the image domain. The conventional augmentation schemes, e.g., single augmentation (including RandAug, Mixup, and CutMix) or batch augmentation, only optimize the network from pairs of augmented images and corresponding labels, leading to a loss of discriminative representation learning. To this end, we propose contrastive augmentation (CA), to learn discriminative representation by capturing the contrastive semantics among augmented samples. Specifically, the proposed CA method explicitly regularizes the similarity of augmented sample pairs to enable the model learn contrastive semantics. Equipped with CA at the training stage, the model simultaneously learns classification representation and contrastive semantics, which makes representation discriminative and does not incur additional inference computational costs. On conventional and fine-grained classification tasks, the experimental results show that the proposed method can effectively improve the generalization capability of convolutional neural networks, including ResNet, EfficientNet, MobileNet, ShuffleNet, and VAN. The compatibility experiment shows that our contrastive augmentation can be combined with other augmentation techniques, e.g., Mixup and CutMix, to further improve the model’s performance.

Intelligent element: Coupling Green function approach and artificial intelligence to reduce discretization effort

This research work presents a method that modifies a classical numerical method using artificial intelligence (AI) and takes advantage of an analytical method to minimize the usual need for increasing discretization. Its formulation is based on the integration of two main concepts: the reciprocity theorem and the generalization capability of artificial neural networks (ANNs). The reciprocity theorem is used to formulate the mathematical expression governing the geomechanical problem, which is then discretized in space into intelligent elements. The behavior of the strain field inside these new elements is predicted using an ANN. To make these predictions, the neural network uses displacement boundary conditions, material properties, and the geometric shape of the element as input data. The comparison was performed for two examples, in which the first had a uniform depletion of the reservoir, while the second had a non-uniform variation of the pore pressure. For the same level of accuracy, the proposed method was 10 times faster than the traditional method for the first example and five times faster for the second example on a computer with 12 threads of 2.6 GHz and 32 GB RAM.

Comparison of Affine and DCGAN-based Data Augmentation Techniques for Chest X-Ray Classification

Data augmentation, also called implicit regularization, is one of the popular strategies to improve the generalization capability of deep neural networks. It is crucial in situations where there is a scarcity of high-quality ground-truth data. Also getting new samples is expensive and time consuming. This is a typical issue in the medical domain. Therefore, this study compares the performance of Affine and Generative Adversarial Networks (GAN)- based data augmentation techniques on the chest image X-Ray dataset. The Pneumonia dataset contains 5863 chest X-Ray images. The traditional Affine data augmentation technique is applied as a pre-processing technique to various deep learning-based CNN models like VGG16, Inception V3, InceptionResNetV2, DenseNet-169 and DenseNet-202 to compare their performance. On the other hand, DCGAN architecture is applied to the dataset for augmentation. Evaluation measures like accuracy, recall and AUC depict that DCGAN outperforms other traditional models. The most important advantage of DCGAN is that it is able to identify fake images with 100% accuracy. This is especially relevant for the medical domain as it deals with the life of individuals. Thus, it can be concluded that DCGAN has better performance as compared to affine transformations applied to traditional CNN models .

Covid-19 detection using chest X-rays: is lung segmentation important for generalization?

PurposeWe evaluated the generalization capability of deep neural networks (DNNs) in the task of classifying chest X-rays as Covid-19, normal or pneumonia, when trained in a relatively small and mixed datasets.MethodsWe proposed a DNN to perform lung segmentation and classification, stacking a segmentation module (U-Net), an original intermediate module and a classification module (DenseNet201). To evaluate generalization capability, we tested the network with an external dataset (from distinct localities) and used Bayesian inference to estimate the probability distributions of performance metrics. Furthermore, we introduce a novel evaluation technique, which uses layer-wise relevance propagation (LRP) and Brixia scores to compare the DNN grounds for decision with radiologists.ResultsThe proposed DNN achieved 0.917 AUC (area under the ROC curve) on the external test dataset, surpassing a DenseNet without segmentation, which showed 0.906 AUC. Bayesian inference indicated mean accuracy of 76.1% and [0.695, 0.826] 95% HDI (high-density interval, which concentrates 95% of the metric’s probability mass) with segmentation and, without segmentation, 71.7% and [0.646, 0.786].ConclusionEmploying an analysis based on LRP and Brixia scores, we discovered that areas where radiologists found strong Covid-19 symptoms are the most important for the stacked DNN classification. External validation showed smaller accuracies than internal, indicating difficulty in generalization, which is positively affected by lung segmentation. Finally, the performance on the external dataset and the analysis with LRP suggest that DNNs can successfully detect Covid-19 even when trained on small and mixed datasets.

Assessing the generalization capability of deep learning networks for aerial image classification using landscape metrics

Although deep learning models and GeoAI are showing great promise for the automatic interpretation of remotely sensed imagery, the accuracy and applicability of these models are dependent on the training dataset. Models trained on one area do not always perform well in new areas. From a research perspective, this has led to the investigation of the generalizability of supervised classification models; from a societal and regulatory perspective, this has led to a call for the development of methods which can help determine whether a trained model is relevant and applicable to a new dataset or study area. This work utilizes landscape metrics to define the similarity between the images used to train a Fully Convolutional Network (FCN) for building identification and an unseen testing image in a different area. The paper analyses similarity between various landscape metrics of an unseen image and the same metrics of the set of training images; and how that similarity is correlated to the performance of the FCN on the unseen image. Results demonstrate that some landscape metrics calculated from reference building outlines, but also from unsupervised clustering and segmentation methods, show strong and moderate correlations to the F1-score obtained by the classifier. It therefore indicates that using indicators such as landscape metrics is a promising research direction which may lead to the creation of training dataset descriptions which help users decide whether a trained model would be suitable for application to a new area.

SS R-CNN: Self-Supervised Learning Improving Mask R-CNN for Ship Detection in Remote Sensing Images

Due to the cost of acquiring and labeling remote sensing images, only a limited number of images with the target objects are obtained and labeled in some practical applications, which severely limits the generalization capability of typical deep learning networks. Self-supervised learning can learn the inherent feature representations of unlabeled instances and is a promising technique for marine ship detection. In this work, we design a more-way CutPaste self-supervised task to train a feature representation network using clean marine surface images with no ships, based on which a two-stage object detection model using Mask R-CNN is improved to detect marine ships. Experimental results show that with a limited number of labeled remote sensing images, the designed model achieves better detection performance than supervised baseline methods in terms of mAP. Particularly, the detection accuracy for small-sized marine ships is evidently improved.

Spiking Neural Network Regularization With Fixed and Adaptive Drop-Keep Probabilities.

Dropout and DropConnect are two techniques to facilitate the regularization of neural network models, having achieved the state-of-the-art results in several benchmarks. In this paper, to improve the generalization capability of spiking neural networks (SNNs), the two drop techniques are first applied to the state-of-the-art SpikeProp learning algorithm resulting in two improved learning algorithms called SPDO (SpikeProp with Dropout) and SPDC (SpikeProp with DropConnect). In view that a higher membrane potential of a biological neuron implies a higher probability of neural activation, three adaptive drop algorithms, SpikeProp with Adaptive Dropout (SPADO), SpikeProp with Adaptive DropConnect (SPADC), and SpikeProp with Group Adaptive Drop (SPGAD), are proposed by adaptively adjusting the keep probability for training SNNs. A convergence theorem for SPDC is proven under the assumptions of the bounded norm of connection weights and a finite number of equilibria. In addition, the five proposed algorithms are carried out in a collaborative neurodynamic optimization framework to improve the learning performance of SNNs. The experimental results on the four benchmark data sets demonstrate that the three adaptive algorithms converge faster than SpikeProp, SPDO, and SPDC, and the generalization errors of the five proposed algorithms are significantly smaller than that of SpikeProp. Furthermore, the experimental results also show that the five algorithms based on collaborative neurodynamic optimization can be improved in terms of several measures.

Multi-Head Modularization to Leverage Generalization Capability in Multi-Modal Networks

It has been crucial to leverage the rich information of multiple modalities in many tasks. Existing works have tried to design multi-modal networks with descent multi-modal fusion modules. Instead, we focus on improving generalization capability of multi-modal networks, especially the fusion module. Viewing the multi-modal data as different projections of information, we first observe that bad projection can cause poor generalization behaviors of multi-modal networks. Then, motivated by well-generalized network's low sensitivity to perturbation, we propose a novel multi-modal training method, multi-head modularization (MHM). We modularize a multi-modal network as a series of uni-modal embedding, multi-modal embedding, and task-specific head modules. Also, for training, we exploit multiple head modules learned with different datasets, swapping each other. From this, we can make the multi-modal embedding module robust to all the heads with different generalization behaviors. In testing phase, we select one of the head modules not to increase the computational cost. Owing to the perturbation of head modules, though including one selected head, the deployed network is more well-generalized compared to the simply end-to-end learned. We verify the effectiveness of MHM on various multi-modal tasks. We use the state-of-the-art methods as baselines, and show notable performance gain for all the baselines.

Hcdg: A Hierarchical Consistency Framework for Domain Generalization on Medical Image Segmentation

Modern deep neural networks struggle to transfer knowledge and generalize across domains when deploying to real-world applications. Domain generalization (DG) aims to learn a universal representation from multiple source domains to improve the network generalization ability on unseen target domains. Previous DG methods mostly focus on the data-level consistency scheme to advance the generalization capability of deep networks, without considering the synergistic regularization of different consistency schemes. In this paper, we present a novel Hierarchical Consistency framework for Domain Generalization (HCDG) by ensembling Extrinsic Consistency and Intrinsic Consistency. Particularly, for Extrinsic Consistency, we leverage the knowledge across multiple source domains to enforce data-level consistency. Also, we design a novel Amplitude Gaussian-mixing strategy for Fourier-based data augmentation to enhance such consistency. For Intrinsic Consistency, we perform task-level consistency for the same instance under the dual-task form. We evaluate the proposed HCDG framework on two medical image segmentation tasks, i.e., optic cup/disc segmentation on fundus images and prostate MRI segmentation. Extensive experimental results manifest the effectiveness and versatility of our HCDG framework. Code will be available once accept.

Performance Evaluation of Convolutional Neural Network Using Synthetic Medical Data Augmentation Generated by GAN

Data augmentation is widely used in image processing and pattern recognition problems in order to increase the richness in diversity of available data. It is commonly used to improve the classification accuracy of images when the available datasets are limited. Deep learning approaches have demonstrated an immense breakthrough in medical diagnostics over the last decade. A significant amount of datasets are needed for the effective training of deep neural networks. The appropriate use of data augmentation techniques prevents the model from over-fitting and thus increases the generalization capability of the network while testing afterward on unseen data. However, it remains a huge challenge to obtain such a large dataset from rare diseases in the medical field. This study presents the synthetic data augmentation technique using Generative Adversarial Networks to evaluate the generalization capability of neural networks using existing data more effectively. In this research, the convolutional neural network (CNN) model is used to classify the X-ray images of the human chest in both normal and pneumonia conditions; then, the synthetic images of the X-ray from the available dataset are generated by using the deep convolutional generative adversarial network (DCGAN) model. Finally, the CNN model is trained again with the original dataset and augmented data generated using the DCGAN model. The classification performance of the CNN model is improved by 3.2% when the augmented data were used along with the originally available dataset.

Generalization Capability of Convolutional Neural Networks for Progress Variable Variance and Reaction Rate Subgrid-Scale Modeling

Deep learning has recently emerged as a successful approach to produce accurate subgrid-scale (SGS) models for Large Eddy Simulations (LES) in combustion. However, the ability of these models to generalize to configurations far from their training distribution is still mainly unexplored, thus impeding their application to practical configurations. In this work, a convolutional neural network (CNN) model for the progress-variable SGS variance field is trained on a canonical premixed turbulent flame and evaluated a priori on a significantly more complex slot burner jet flame. Despite the extensive differences between the two configurations, the CNN generalizes well and outperforms existing algebraic models. Conditions for this successful generalization are discussed, including the effect of the filter size and flame–turbulence interaction parameters. The CNN is then integrated into an analytical reaction rate closure relying on a single-step chemical source term formulation and a presumed beta PDF (probability density function) approach. The proposed closure is able to accurately recover filtered reaction rate values on both training and generalization flames.

Scale-, Shift-, and Rotation-Invariant Diffractive Optical Networks

Recent research efforts in optical computing have gravitated towards developing optical neural networks that aim to benefit from the processing speed and parallelism of optics/photonics in machine learning applications. Among these endeavors, Diffractive Deep Neural Networks (D2NNs) harness light-matter interaction over a series of trainable surfaces, designed using deep learning, to compute a desired statistical inference task as the light waves propagate from the input plane to the output field-of-view. Although, earlier studies have demonstrated the generalization capability of diffractive optical networks to unseen data, achieving e.g., >98% image classification accuracy for handwritten digits, these previous designs are in general sensitive to the spatial scaling, translation and rotation of the input objects. Here, we demonstrate a new training strategy for diffractive networks that introduces input object translation, rotation and/or scaling during the training phase as uniformly distributed random variables to build resilience in their blind inference performance against such object transformations. This training strategy successfully guides the evolution of the diffractive optical network design towards a solution that is scale-, shift- and rotation-invariant, which is especially important and useful for dynamic machine vision applications in e.g., autonomous cars, in-vivo imaging of biomedical specimen, among others.

PnP-AdaNet: Plug-and-Play Adversarial Domain Adaptation Network at Unpaired Cross-Modality Cardiac Segmentation

Deep convolutional networks have demonstrated state-of-the-art performance on various challenging medical image processing tasks. Leveraging images from different modalities for the same analysis task holds large clinical benefits. However, the generalization capability of deep networks on test data sampled from different distribution remains as a major challenge. In this paper, we propose a plug-and-play adversarial domain adaptation network (PnP-AdaNet) for adapting segmentation networks between different modalities of medical images, e.g., MRI and CT. We tackle the significant domain shift by aligning the feature spaces of source and target domains at multiple scales in an unsupervised manner. With the adversarial loss, we learn a domain adaptation module which flexibly replaces the early encoder layers of the source network, and the higher layers are shared between two domains. We validate our domain adaptation method on cardiac segmentation in unpaired MRI and CT, with four different anatomical structures. The average Dice achieved 63.9%, which is a significant recover from the complete failure (Dice score of 13.2%) if we directly test an MRI segmentation network on CT data. In addition, our proposed PnP-AdaNet outperforms many state-of-the-art unsupervised domain adaptation approaches on the same dataset. The experimental results with comprehensive ablation studies have demonstrated the excellent efficacy of our proposed method for unsupervised cross-modality domain adaptation. Our code is publically available at https://github.com/carrenD/Medical-Cross-Modality-Domain-Adaptation

Investigating the Generalization Capability and Performance of Neural Networks and Neuro-Fuzzy Systems for Nonlinear Dynamics Modeling of Impaired Aircraft

One of the challenging problems in the case of aircraft failure is to determine the new altered dynamics of the impaired aircraft. Among various methods, neural networks and neuro-fuzzy systems can be used for high-fidelity modeling of the aircraft nonlinear dynamics with the aim of onboard applications in real time. However, the method with better generalization capability is more preferred specifically in the case of unpredicted aircraft failures. Generalization of a network is mainly dependent on the network's parameters, the employed training algorithm, and the amount of training data. In this paper, several neural networks and local model networks are trained using different training algorithms and different amounts of training data to model the nonlinear dynamics of an impaired aircraft with the damaged rudder. These networks are compared based on their generalizations to the new cases of rudder failure. The effect of using different amounts of training data on the generalization capability and performance of the networks has also been investigated. The results of this paper show that both network types have good performance but neural networks generalize better to the new failure cases than local model networks. Also based on the obtained results, a significant reduction in the number of training samples could be accomplished without a considerable decrease in the network's performance and generalization. Finally, a neural network-based sensitivity analysis method is proposed which utilizes the network's regression equation as an emulator for fast model evaluations and can be used as an advisory tool for choosing safer path planning strategies.