Adaptive Dropout Research Articles

Accelerating the neural network controller embedded implementation on FPGA with novel dropout techniques for a solar inverter

Accelerating neural network (NN) controllers is important for improving the performance, efficiency, scalability, and reliability of real-time systems, particularly in resource-constrained embedded systems. This paper introduces a novel weight-dropout method for training neural network controllers in real-time closed-loop systems, aimed at accelerating the embedded implementation for solar inverters. The core idea is to eliminate small-magnitude weights during training, thereby reducing the number of necessary connections while ensuring the network’s convergence. To maintain convergence, only non-diagonal elements of the weight matrices were dropped. This dropout technique was integrated into the Levenberg–Marquardt and Forward Accumulation Through Time algorithms, resulting in more efficient training for trajectory tracking. We executed the proposed training algorithm with dropout on the AWS cloud, observing a performance increase of approximately four times compared to local execution. Furthermore, implementing the neural network controller on the Intel Cyclone V Field Programmable Gate Array (FPGA) demonstrates significant improvements in computational and resource efficiency due to the proposed dropout technique leading to sparse weight matrices. This optimization enhances the suitability of the neural network controller for embedded environments. In comparison to Sturtz et al. (2023), which dropped 11 weights, our approach eliminated 18 weights, significantly boosting resource efficiency. This resulted in a 16.40% reduction in Adaptive Logic Modules (ALMs), decreasing the count to 47,426.5. Combinational Look-Up Tables (LUTs) and dedicated logic registers saw reductions of 17.80% and 15.55%, respectively. However, the impact on block memory bits is minimal, showing only a 1% improvement, indicating that memory resources are less affected by weight dropout. In contrast, the usage of Memory 10 Kilobits (MK10s) dropped from 97 to 87, marking a 10% improvement. We also propose an adaptive dropout technique to further improve the previous results.

Meta-learning of feature distribution alignment for enhanced feature sharing

Meta-learning aims to learn common feature representations between tasks to quickly adapt and perform well when facing new tasks. However, current meta-learning algorithms ignore the differences in feature distributions among tasks. The significant differences in task features may lead to an over-fitting of the model to specific features of certain tasks during the learning process, thereby causing a decrease in adaptability to other tasks. In this paper, we propose a novel method for feature distribution alignment. Specifically, we employ knowledge distillation (KD) to minimize the Kullback–Leibler (KL) divergence between task features, thus promoting greater similarity in the extracted feature distributions. This incentivizes the model to prioritize learning shared parameters and general features across tasks during the training process, rather than task-specific features. Furthermore, we also use an adaptive dropout strategy, which adjusts the training process for the next batch based on the feature distribution of the current task. This method reduces the impact of feature distribution differences during the training iterations. Extensive experiments and analyses demonstrate that our method improves the performance of various meta-learning models in few-shot classification, few-shot fine-grained classification, cross-domain few-shot classification, and non-mutually exclusive few-shot classification.

Reliable federated learning based on dual-reputation reverse auction mechanism in Internet of Things

Federated learning, a promising distributed machine learning paradigm, has been used in various Internet of Things (IoT) environments to solve isolated data island issues and protect data privacy. However, since the central server in federated learning cannot detect the local training process of the client, it is vulnerable to adversarial attacks against its security and privacy by malicious clients during the training process. To address this problem, this work proposes a federated learning system model based on dual-reputation reverse auction in IoT. Specifically, due to resource constraints, not all clients participate in the federated learning process, so we propose the reputation-bid ratio based greedy half-splitting algorithm to select some clients to participate in the federated learning, which can guarantee that each client has the chance to be selected while selecting as many honest and high-quality clients as possible. Then, we propose an adaptive dropout aggregation method based on a training quality score, which can effectively defend against malicious workers’ attacks. After the completion of federated learning, we put forward a subjective evaluation incentive mechanism based on a second reputation to realize the fair incentive. Furthermore, we store and manage reputations through blockchain in our model to ensure their availability. Theoretical analysis deduces the complexity and security of our proposed model. Finally, simulation results indicate that our proposed model can achieve the highest accuracy across all test datasets compared to the baseline. Particularly, on the CIFAR10 dataset, the accuracy of our proposed model surpasses the baseline by 5% to 30%. In responding to sudden attacks initiated by normally participating workers, our model exhibits the fastest reaction time, with accuracy surpassing the baseline by 8% to 40%.

Robust learning of parsimonious deep neural networks

We propose a simultaneous learning and pruning algorithm capable of identifying and eliminating irrelevant structures in a neural network during the early stages of training. Simultaneous learning and pruning presents serious challenges such as premature pruning of units that can lead to poor performance. Our method is capable of overcoming such challenges and is robust, i.e., it gives consistent pruning levels and prediction accuracy regardless of weight initialization or the size of the starting network. Thus, it allows for substantial computational cost savings during training, besides that of inference, and it can enable the training of very deep networks when transfer learning to obtain fully trained deep networks that can be pruned after training is not possible. Our approach is based on variational inference principles using Gaussian scale mixture priors on the neural network weights. The variational posterior distribution of Bernoulli random variables multiplying the units/filters is learned, similarly to adaptive dropout. We construct a novel hyper-prior distribution over the prior parameters to impose properties crucial for their optimal selection and the overall robustness of our algorithm. It is shown in the context of our algorithm that the parameters of the posterior distributions practically converge to either 0 or 1, establishing a deterministic final network. Convergence is proved analytically based on dynamical systems theory and from the theoretical results, practical pruning conditions are established. The proposed algorithm is evaluated on the MNIST, CIFAR-10 and ImageNet data sets and the commonly used fully connected, convolutional and residual architectures LeNet, VGG16 and ResNet. The simulations show that our method typically achieves better pruning levels while maintaining test-accuracy on par with state-of the-art methods for structured pruning in a manner robust with respect to network initialization and initial size.

Adaptive visual-tactile fusion recognition for robotic operation of multi-material system.

The use of robots in various industries is evolving from mechanization to intelligence and precision. These systems often comprise parts made of different materials and thus require accurate and comprehensive target identification. While humans perceive the world through a highly diverse perceptual system and can rapidly identify deformable objects through vision and touch to prevent slipping or excessive deformation during grasping, robot recognition technology mainly relies on visual sensors, which lack critical information such as object material, leading to incomplete cognition. Therefore, multimodal information fusion is believed to be key to the development of robot recognition. Firstly, a method of converting tactile sequences to images is proposed to deal with the obstacles of information exchange between different modalities for vision and touch, which overcomes the problems of the noise and instability of tactile data. Subsequently, a visual-tactile fusion network framework based on an adaptive dropout algorithm is constructed, together with an optimal joint mechanism between visual information and tactile information established, to solve the problem of mutual exclusion or unbalanced fusion in traditional fusion methods. Finally, experiments show that the proposed method effectively improves robot recognition ability, and the classification accuracy is as high as 99.3%.

Temporal augmented contrastive learning for micro-expression recognition

Micro-expressions (MEs) can reveal the hidden but real emotion and are usually caused spontaneously. However, the characteristics of subtlety and temporariness with the lack of sufficient ME datasets make it hard for recognition. In this paper, we propose an adaptively temporal augmented momentum contrastive learning to alleviate these problems. For the small scale, we pre-train the model on a new interpolated dataset via contrastive learning with momentum contrast (MoCo). For the subtle and rapid facial movements, we augment the temporal dynamics using an adaptive dropout operation to redundant frames. Specifically, we use a recursive way to create a new interpolated dataset from raw datasets firstly. Then we design a shallow model with an inflated inception module as the encoder of the contrastive learning. Afterward, we pre-train the model on the new dataset via momentum contrastive learning. During the pre-training, we propose adaptively temporal augmentation via generative adversarial learning. After the pre-training, we take the encoder out and finetune it for recognition. Finally, we perform extensive experiments and ablation studies on three ME datasets. The results demonstrate the effectiveness of the MoCo-like pre-training and the temporal augmentation for recognition. Moreover, the pre-trained model outperforms other state-of-the-art models based on optical flow.

Spiking Neural Network Regularization With Fixed and Adaptive Drop-Keep Probabilities.

Dropout and DropConnect are two techniques to facilitate the regularization of neural network models, having achieved the state-of-the-art results in several benchmarks. In this paper, to improve the generalization capability of spiking neural networks (SNNs), the two drop techniques are first applied to the state-of-the-art SpikeProp learning algorithm resulting in two improved learning algorithms called SPDO (SpikeProp with Dropout) and SPDC (SpikeProp with DropConnect). In view that a higher membrane potential of a biological neuron implies a higher probability of neural activation, three adaptive drop algorithms, SpikeProp with Adaptive Dropout (SPADO), SpikeProp with Adaptive DropConnect (SPADC), and SpikeProp with Group Adaptive Drop (SPGAD), are proposed by adaptively adjusting the keep probability for training SNNs. A convergence theorem for SPDC is proven under the assumptions of the bounded norm of connection weights and a finite number of equilibria. In addition, the five proposed algorithms are carried out in a collaborative neurodynamic optimization framework to improve the learning performance of SNNs. The experimental results on the four benchmark data sets demonstrate that the three adaptive algorithms converge faster than SpikeProp, SPDO, and SPDC, and the generalization errors of the five proposed algorithms are significantly smaller than that of SpikeProp. Furthermore, the experimental results also show that the five algorithms based on collaborative neurodynamic optimization can be improved in terms of several measures.

HLA: a novel hybrid model based on fixed structure and variable structure learning automata

ABSTRACT Learning Automata (LAs) are adaptive decision-making models designed to find an appropriate action in unknown environments. LAs can be classified into two classes: variable structure and fixed structure. To the best of our knowledge, there is no hybrid model based on both of these classes. In this paper, we propose a model that brings together the benefits of both classes of LAs. In the proposed model, called an HLA, the action switching phase of a fixed structure learning automaton is fused with a variable structure learning automaton. Several computer simulations are conducted to study the performance of the proposed model with respect to the total number of rewards and action switching in addition to the convergence rate. The proposed model is compared to both variable structure and fixed structure learning automata, and in most cases, the numerical results demonstrate its superiority. In order to show the applicability of the HLA, a novel adaptive dropout mechanism in deep neural networks was suggested. The results of the simulations show that the proposed mechanism performs better than the simple dropout mechanism with respect to network accuracy.

Early prediction of remaining useful life for Lithium-ion batteries based on a hybrid machine learning method

Early prediction of battery remaining useful life (RUL) is critical to ensure a steady energy supply and the effective usage of energy. To reduce the amount of degradation data required for accurate RUL prediction, this study proposed a deep-learning-based prediction algorithm, namely ADLSTM-MC, which combines the features of adaptive dropout long short-term memory (ADLSTM) and Monte Carlo (MC) simulation. The adaptive dropout method was optimized by the Bayesian optimization. MC simulation is combined to describe the uncertainties of prediction results. To evaluate the performance of the proposed method, we developed two data sets including LiFePO4/graphite and LiNi0.8Co0.15Al0.05O2/graphite batteries with different charge/discharge rates and discharge cut-off voltages. The ADLSTM-MC method is shown to precisely achieve early prediction with only 25% of the entire degradation data, while existing models typically require 40%–70% of the degradation data. The R2 of the proposed capacity prediction model is up to 0.957–0.982, and the prediction errors including root mean square error (RMSE) and mean absolute error (MAE) are less than 0.033 and 0.027, respectively. The results show that the proposed method outperforms the mainstream regression algorithms and several recent published hybrid methods in terms of data requirement and prediction accuracy.

Adaptive dropout for high-dimensional expensive multiobjective optimization

Various works have been proposed to solve expensive multiobjective optimization problems (EMOPs) using surrogate-assisted evolutionary algorithms (SAEAs) in recent decades. However, most existing methods focus on EMOPs with less than 30 decision variables, since a large number of training samples are required to build an accurate surrogate model for high-dimensional EMOPs, which is unrealistic for expensive multiobjective optimization. To address this issue, we propose an SAEA with an adaptive dropout mechanism. Specifically, this mechanism takes advantage of the statistical differences between different solution sets in the decision space to guide the selection of some crucial decision variables. A new infill criterion is then proposed to optimize the selected decision variables with the assistance of surrogate models. Moreover, the optimized decision variables are extended to new full-length solutions, and then the new candidate solutions are evaluated using expensive functions to update the archive. The proposed algorithm is tested on different benchmark problems with up to 200 decision variables compared to some state-of-the-art SAEAs. The experimental results have demonstrated the promising performance and computational efficiency of the proposed algorithm in high-dimensional expensive multiobjective optimization.

Adaptive Dropout Method Based on Biological Principles.

Dropout is one of the most widely used methods to avoid overfitting neural networks. However, it rigidly and randomly activates neurons according to a fixed probability, which is not consistent with the activation mode of neurons in the human cerebral cortex. Inspired by gene theory and the activation mechanism of brain neurons, we propose a more intelligent adaptive dropout, in which a variational self-encoder (VAE) overlaps to an existing neural network to regularize its hidden neurons by adaptively setting activities to zero. Through alternating iterative training, the discarding probability of each hidden neuron can be learned according to the weights and thus effectively avoid the shortcomings of the standard dropout method. The experimental results in multiple data sets illustrate that this method can better suppress overfitting in various neural networks than can the standard dropout. Additionally, this adaptive dropout technique can reduce the number of neurons and improve training efficiency.

Advanced Dropout: A Model-Free Methodology for Bayesian Dropout Optimization.

Due to lack of data, overfitting ubiquitously exists in real-world applications of deep neural networks (DNNs). We propose advanced dropout, a model-free methodology, to mitigate overfitting and improve the performance of DNNs. The advanced dropout technique applies a model-free and easily implemented distribution with parametric prior, and adaptively adjusts dropout rate. Specifically, the distribution parameters are optimized by stochastic gradient variational Bayes in order to carry out an end-to-end training. We evaluate the effectiveness of the advanced dropout against nine dropout techniques on seven computer vision datasets (five small-scale datasets and two large-scale datasets) with various base models. The advanced dropout outperforms all the referred techniques on all the datasets. We further compare the effectiveness ratios and find that advanced dropout achieves the highest one on most cases. Next, we conduct a set of analysis of dropout rate characteristics, including convergence of the adaptive dropout rate, the learned distributions of dropout masks, and a comparison with dropout rate generation without an explicit distribution. In addition, the ability of overfitting prevention is evaluated and confirmed. Finally, we extend the application of the advanced dropout to uncertainty inference, network pruning, text classification, and regression. The proposed advanced dropout is also superior to the corresponding referred methods. Codes are available at https://github.com/PRIS-CV/AdvancedDropout.

A GPU-based residual network for medical image classification in smart medicine

The recent advance of high-performance computing techniques like graphics processing unit (GPU) enables large-scale deep learning models for medical image analytics in smart medicine. Smart medicine has made great progress by applying convolutional neural networks (CNNs) like ResNet and VGG-16 to medical image classification. However, various CNN models achieve very limited accuracy in some cases where multiple diseases are revealed in an X-ray image. This paper presents a variant ResNet model by replacing the global average pooling with the adaptive dropout for medical image classification. In order for the presented model to recognize multiple diseases (i.e., multi-label classification), we convert the multi-label classification to N binary classification by training the parameters of the presented model for N times. Finally, experiments are conducted on a GPU Cluster to evaluate the presented model on three datasets, namely Montgomery County chest X-ray set, Shenzhen X-ray set, and NIH chest X-ray set. The results show the presented model achieves a great performance improvement for medical image classification without a significant efficiency reduction compared to the traditional architecture and VGG-16.

Medical Image Segmentation Algorithm Based on Optimized Convolutional Neural Network-Adaptive Dropout Depth Calculation

Medical image segmentation is a key technology for image guidance. Therefore, the advantages and disadvantages of image segmentation play an important role in image-guided surgery. Traditional machine learning methods have achieved certain beneficial effects in medical image segmentation, but they have problems such as low classification accuracy and poor robustness. Deep learning theory has good generalizability and feature extraction ability, which provides a new idea for solving medical image segmentation problems. However, deep learning has problems in terms of its application to medical image segmentation: one is that the deep learning network structure cannot be constructed according to medical image characteristics; the other is that the generalizability y of the deep learning model is weak. To address these issues, this paper first adapts a neural network to medical image features by adding cross-layer connections to a traditional convolutional neural network. In addition, an optimized convolutional neural network model is established. The optimized convolutional neural network model can segment medical images using the features of two scales simultaneously. At the same time, to solve the generalizability problem of the deep learning model, an adaptive distribution function is designed according to the position of the hidden layer, and then the activation probability of each layer of neurons is set. This enhances the generalizability of the dropout model, and an adaptive dropout model is proposed. This model better addresses the problem of the weak generalizability of deep learning models. Based on the above ideas, this paper proposes a medical image segmentation algorithm based on an optimized convolutional neural network with adaptive dropout depth calculation. An ultrasonic tomographic image and lumbar CT medical image were separately segmented by the method of this paper. The experimental results show that not only are the segmentation effects of the proposed method improved compared with those of the traditional machine learning and other deep learning methods but also the method has a high adaptive segmentation ability for various medical images. The research work in this paper provides a new perspective for research on medical image segmentation.

Group-Wise Dynamic Dropout Based on Latent Semantic Variations

Dropout regularization has been widely used in various deep neural networks to combat overfitting. It works by training a network to be more robust on information-degraded data points for better generalization. Conventional dropout and variants are often applied to individual hidden units in a layer to break up co-adaptations of feature detectors. In this paper, we propose an adaptive dropout to reduce the co-adaptations in a group-wise manner by coarse semantic information to improve feature discriminability. In particular, we showed that adjusting the dropout probability based on local feature densities can not only improve the classification performance significantly but also enhance the network robustness against adversarial examples in some cases. The proposed approach was evaluated in comparison with the baseline and several state-of-the-art adaptive dropouts over four public datasets of Fashion-MNIST, CIFAR-10, CIFAR-100 and SVHN.

Multi-Factor Operating Condition Recognition Using 1D Convolutional Long Short-Term Network.

For a diesel engine, operating conditions have extreme importance in fault detection and diagnosis. Limited to various special circumstances, the multi-factor operating conditions of a diesel engine are difficult to measure, and the demand of automatic condition recognition based on vibration signals is urgent. In this paper, multi-factor operating condition recognition using a one-dimensional (1D) convolutional long short-term network (1D-CLSTM) is proposed. Firstly, a deep neural network framework is proposed based on a 1D convolutional neural network (CNN) and long short-Term network (LSTM). According to the characteristics of vibration signals of a diesel engine, batch normalization is introduced to regulate the input of each convolutional layer by fixing the mean value and variance. Subsequently, adaptive dropout is proposed to improve the model sparsity and prevent overfitting in model training. Moreover, the vibration signals measured under 12 operating conditions were used to verify the performance of the trained 1D-CLSTM classifier. Lastly, the vibration signals measured from another kind of diesel engine were applied to verify the generalizability of the proposed approach. Experimental results show that the proposed method is an effective approach for multi-factor operating condition recognition. In addition, the adaptive dropout can achieve better training performance than the constant dropout ratio. Compared with some state-of-the-art methods, the trained 1D-CLSTM classifier can predict new data with higher generalization accuracy.

An adaptive deep learning model to differentiate syndromes of infectious fever in smart medicine

Recently, smart medicine has been considered as a promising technique to treat the intractable diseases by combining deep learning techniques with medical Internet of Things. As an important component in the integration of traditional and western medicine, smart medicine is particularly effective to treat infectious fever. Before the cause of infectious fever diseases is ascertained, the Chinese medicine intervention is able to alleviating symptoms and strive for time for the causes detection. However, accurate syndrome differentiation, a difficult issue in infectious fever, is the premise of the Chinese medicine intervention. This work presents a possible adaptive deep learning model by integrating an adaptive dropout function into the stacked auto-encoder for computer-aided syndrome differentiation in infective fever. Moreover, we summarize the main syndromes and prescriptions in infectious fever. This work is expected to further the development of smart medicine, especially smart Chinese medicine. More importantly, it points out a novel research direction and medical technique in the treatment of infectious fever in clinic.

Rademacher dropout: An adaptive dropout for deep neural network via optimizing generalization gap

Dropout plays an important role in improving the generalization ability in deep learning. However, the empirical and fixed choice of dropout rates in traditional dropout strategies may increase the generalization gap, which is counter to one of the principle aims of dropout. To handle this problem, in this paper, we propose a novel dropout method. By the theoretical analysis of Dropout Rademacher Complexity, we first prove that the generalization gap of a deep model is bounded by a constraint function related to dropout rates. Meanwhile, we derive a closed form solution via optimizing the constraint function, which is a distribution estimation of dropout rates. Based on the closed form solution, a lightweight complexity algorithm called Rademacher Dropout (RadDropout) is presented to achieve the adaptive adjustment of dropout rates. Moreover, as a verification of the effectiveness of our proposed method, the extensive experimental results on benchmark datasets show that RadDropout achieves improvement of both convergence rate and prediction accuracy.

An Adaptive Dropout Deep Computation Model for Industrial IoT Big Data Learning With Crowdsourcing to Cloud Computing

Deep computation, as an advanced machine learning model, has achieved the state-of-the-art performance for feature learning on big data in industrial Internet of Things (IoT). However, the current deep computation model usually suffers from overfitting due to the lack of public available labeled training samples, limiting its performance for big data feature learning. Motivated by the idea of active learning, an adaptive dropout deep computation model (ADDCM) with crowdsourcing to cloud is proposed for industrial IoT big data feature learning in this paper. First, a distribution function is designed to set the dropout rate for each hidden layer to prevent overfitting for the deep computation model. Furthermore, the outsourcing selection algorithm based on the maximum entropy is employed to choose appropriate samples from the training set to crowdsource on the cloud platform. Finally, an improved supervised learning from multiple experts scheme is presented to aggregate answers given by human workers and to update the parameters of the ADDCM simultaneously. Extensive experiments are conducted to evaluate the performance of the presented model by comparing with the dropout deep computation model and other state-of-the-art crowdsourcing algorithms. The results demonstrate that the proposed model can prevent overfitting effectively and aggregate the labeled samples to train the parameters of the deep computation model with crowdsouring for industrial IoT big data feature learning.

Convolutional Neural Network With Adaptive Regularization to Classify Driving Styles on Smartphones

Driving style evaluation by smartphones depends on the quality of the features extracted from sensors data. Typically, these features are extracted based on experiments, expertness, or heuristics. In more modern approaches, some automatic methods such as convolutional neural network (CNN) are used to extract features including obvious and hidden ones. We also used the CNN on acceleration data collected by smartphones to extract the knowledge regarding driving style, vehicle, environment, and human characteristics. We found that this novel idea was more successful for evaluating the driving style compared with the previous machine learning algorithms. However, we faced over-fitting in the training process of the CNN and to avoid this, we proposed the state-of-the-art learning method applying two adaptive regularization schemes called adaptive dropout and adaptive weight decay. To evaluate these techniques, first, we checked the results on three popular large-scale datasets. When we proved the efficiency, we utilized them on two transportation data sets. In transportation-modes dataset, the accuracy was at least 95.8%’; and regarding the driving-style dataset, the classification accuracy was 95%. Thus, the adaptive regularized CNN is an amazing option for driving style evaluation on smartphones.