Dropout Regularization Research Articles

LGST-Drop: label-guided structural dropout for spatial–temporal convolutional neural networks

Region dropout regularization strategies have proven to be highly effective at improving the generalization performance of convolutional neural networks (CNNs) in a variety of computer vision tasks including image classification, object detection, and semantic segmentation because these strategies enable models to focus on a wider range of image region information. However, for action recognition, models need to be able to extract not only useful spatial information but also important temporal and motion information, which cannot be satisfied by traditional regularization strategies. We propose a spatiotemporal dropout strategy to meet the need for regularization in spatial–temporal CNNs. We call it label guided spatial–temporal drop (LGST-Drop); it not only provides effectively structured dropout in the spatial dimension but also regularizes motion information in the temporal dimension. In addition, LGST-Drop’s mask is guided by the predicted categories of the model itself, which we called temporary labels. Extensive experiments on several standard datasets from action recognition domains show the usefulness of the proposed technique in comparison with the previous methods and theirstate-of-the-art variant algorithms.

An intelligent approach for Arabic handwritten letter recognition using convolutional neural network.

Currently, digital transformation has occurred in most countries in the world to varying degrees, but digitizing business processes are complex in terms of understanding the various aspects of manual documentation. The use of digital devices and intelligent systems is vital in the digital transformation of manual documentation from hardcopy to digital formats. The transformation of handwritten documents into electronic files is one of the principal aspects of digitization and represents a common need shared by today’s businesses. Generally, handwriting recognition poses a complex digitization challenge, and Arabic handwriting recognition, specifically, proves inordinately challenging due to the nature of Arabic scripts and the excessive diversity in human handwriting. This study presents an intelligent approach for recognizing handwritten Arabic letters. In this approach, a convolution neural network (CNN) model is proposed to recognize handwritten Arabic letters. The model is regularized using batch normalization and dropout operations. Moreover, the model was tested with and without dropout, resulting in a significant difference in the performance. Hence, the model overfitting has been prevented using dropout regularization. The proposed model was applied to the prominent, publicly-available Arabic handwritten characters (AHCD) dataset with 16,800 letters, and the performance was measured using several evaluation measures. The experimental results show the best fit of the proposed model in terms of higher accuracy results that reached 96.78%; additionally, other evaluation measures compared to popular domain-relevant approaches in the literature.

Front-End Model of Wireless Network Combined with Artificial Intelligence in Computer Information Management System

To address the security problem of computer information management, an artificial intelligence- (AI-) based information intrusion detection model is built in combination with wireless network. Firstly, the background and characteristics of wireless local area network (WLAN) technology are analyzed, and the relationship between AI technology and deep learning is introduced. Secondly, an intrusion detection model on account of long short-term memory (LSTM) neural network and gated recursive unit (GRU) is constructed after analysis of different neural network models. The L2 weight attenuation and dropout regularization strategies are combined with the neural network model. Finally, an intrusion detection front-end model combining wireless network and AI is established. From the comparison of intrusion detection experiments, the generalization ability of the model can be improved by using L2 weight attenuation and dropout regularization strategies. Nevertheless, the performance improvement is only slight, so the early stop method is adopted instead of the regularization strategy. Compared with the existing classification models, the overall performance of LSTM and GRU models is improved by about 17%. The performance of GRU model is not much different from that of LSTM model, but the amount of computation is reduced. Therefore, GRU model is the optimal choice to construct intrusion detection system. The intrusion detection models in WLAN and GRU can improve the security performance of computer information management system. To sum up, this work provides reference for the development of computer management system.

A Hybrid Spatiotemporal Deep Model Based on CNN and LSTM for Air Pollution Prediction

Nowadays, air pollution is an important problem with negative impacts on human health and on the environment. The air pollution forecast can provide important information to all affected sides, and allows appropriate measures to be taken. In order to address the problems of filling in the missing values in the time series used for air pollution forecasts, the automation of the allocation of optimal subset of input variables, the dependency of the air quality at a particular location on the conditions of the surrounding environment, as well as automation of the model’s optimization, this paper proposes a deep spatiotemporal model based on a 2D convolutional neural network and a long short-term memory network for predicting air pollution. The model utilizes the automatic selection of input variables and the optimization of hyperparameters by a genetic algorithm. A hybrid strategy for missing value imputation is used based on a combination of linear interpolation and a strategy of using the average between the previous value and the average value for the same time in other years. In order to determine the best architecture of the spatiotemporal model, the architecture hyperparameters are optimized by a genetic algorithm with a modified crossover operator for solutions with variable lengths. Additionally, the trained models are included in various ensembles in order to further improve the prediction performance—these include ensembles of models with the same architecture comprising the best architecture obtained by the evolutionary optimization, and ensembles of diverse models comprising the k best models of the evolutionary optimization. The experimental results for the Beijing Multi-Site Air-Quality Data Set show that the proposed spatiotemporal model for air pollution forecasting provides good and consistent prediction results. The comparison of the suggested model with other deep NN models shows satisfactory results, with the best performance according to MAE, based on the experimental results for the station at Wanliu (16.753 ± 0.384). Most of the model architectures obtained by the optimization of the model hyperparameters using the genetic algorithm have one convolutional layer with a small number of kernels and a small kernel size; the convolutional layers are followed by a max-pooling layer, and one or two LSTM layers are utilized with dropout regularization applied to the LSTM layer using small values of p (0.1, 0.2 and 0.3). The utilization of ensembles from the k best trained models further improves the prediction results and surpasses other deep learning models, according to MAE and RMSE metrics. The used hybrid strategy for missing value imputation enhances the results, especially for data with clear seasonality, and produces better MAE compared to the strategy using average values for the same hour of the same day and month in other years. The experimental results also reveal that random searching is a simple and effective strategy for selecting the input variables. Furthermore, the inclusion of spatial information in the model’s input data, based on the local neighborhood data, significantly improves the predictive results obtained with the model. The results obtained demonstrate the benefits of including spatial information from as many surrounding stations as possible, as well as using as much historical information as possible.

Synthetic Theft Attacks and Long Short Term Memory-Based Preprocessing for Electricity Theft Detection Using Gated Recurrent Unit

Electricity theft is one of the challenging problems in smart grids. The power utilities around the globe face huge economic loss due to ET. The traditional electricity theft detection (ETD) models confront several challenges, such as highly imbalance distribution of electricity consumption data, curse of dimensionality and inevitable effects of non-malicious factors. To cope with the aforementioned concerns, this paper presents a novel ETD strategy for smart grids based on theft attacks, long short-term memory (LSTM) and gated recurrent unit (GRU) called TLGRU. It includes three subunits: (1) synthetic theft attacks based data balancing, (2) LSTM based feature extraction, and (3) GRU based theft classification. GRU is used for drift identification. It stores and extracts the long-term dependency in the power consumption data. It is beneficial for drift identification. In this way, a minimum false positive rate (FPR) is obtained. Moreover, dropout regularization and Adam optimizer are added in GRU for tackling overfitting and trapping model in the local minima, respectively. The proposed TLGRU model uses the realistic EC profiles of the Chinese power utility state grid corporation of China for analysis and to solve the ETD problem. From the simulation results, it is exhibited that 1% FPR, 97.96% precision, 91.56% accuracy, and 91.68% area under curve for ETD are obtained by the proposed model. The proposed model outperforms the existing models in terms of ETD.

Fault diagnosis of bearings based on deep separable convolutional neural network and spatial dropout

Bearing pitting, one of the common faults in mechanical systems, is a research hotspot in both academia and industry. Traditional fault diagnosis methods for bearings are based on manual experience with low diagnostic efficiency. This study proposes a novel bearing fault diagnosis method based on deep separable convolution and spatial dropout regularization. Deep separable convolution extracts features from the raw bearing vibration signals, during which a 3 × 1 convolutional kernel with a one-step size selects effective features by adjusting its weights. The similarity pruning process of the channel convolution and point convolution can reduce the number of parameters and calculation quantities by evaluating the size of the weights and removing the feature maps of smaller weights. The spatial dropout regularization method focuses on bearing signal fault features, improving the independence between the bearing signal features and enhancing the robustness of the model. A batch normalization algorithm is added to the convolutional layer for gradient explosion control and network stability improvement. To validate the effectiveness of the proposed method, we collect raw vibration signals from bearings in eight different health states. The experimental results show that the proposed method can effectively distinguish different pitting faults in the bearings with a better accuracy than that of other typical deep learning methods.

Research on Strawberry Disease Diagnosis Based on Improved Residual Network Recognition Model

Considering the problems of high cost, inefficiency, and time consumption of manual diagnosis of strawberry diseases, G-ResNet50 is proposed based on transfer learning and deep residual network for strawberry disease identification and classification. The G-ResNet50 is based on the ResNet50, and the focal loss function is introduced in G-ResNet50 to make the model devote itself to disease images that are difficult to classify. During the training process of the G-ResNet50 model, its convolutional layer and pooling layer inherit the pre-trained weight parameters from the ResNet50 model on the PlantVillage dataset, while adding dropout regularization and batch regularization methods to optimize the network model. The strawberry disease dataset includes four sample images of healthy plants, powdery mildew, strawberry anthracnose, and leaf spot disease. The dataset is enhanced and expanded by operations including angle rotation, adjusting contrast and brightness, and adding Gaussian noise. Compared with existing models such as VGG16, ResNet50, InceptionV3, and MobileNetV2, the results of model training and testing on 7,525 four-category leaf datasets show that the G-ResNet50 model has faster convergence speed and better classification effect, and its average recognition accuracy rate reached 98.67%, which is significantly higher than other models. Through the three evaluation indicators of precision rate, recall rate, and confusion matrix, it is concluded that the G-ResNet50 has good robustness, high stability, and high recognition accuracy and can provide a feasible solution for strawberry disease detection in practical applications.

Diagnosis of Malaria Infected Blood Cell Digital Images using Deep Convolutional Neural Networks

Automated medical diagnosis is an important topic, especially in detection and classification of diseases. Malaria is one of the most widespread diseases, with more than 200 million cases, according to the 2016 WHO report. Malaria is usually diagnosed using thin and thick blood smears under a microscope. However, proper diagnosis is difficult, especially in poor countries where the disease is most widespread. Therefore, automatic diagnostics helps in identifying the disease through images of red blood cells, with the use of machine learning techniques and digital image processing. This paper presents an accurate model using a Deep Convolutional Neural Network build from scratch. The paper also proposed three CNN models each one trained on the Malaria RBC dataset with different architectures for handling the classification tasks. Furthermore, disadvantage of the traditional method of using transfer learning, and how to control model complexity to achieve better performance was discussed. The dropout regularization technique was used to avoid overfitting problems and minimize validation loss. Applying Data Augmentation technique to avoid the problem of small data in training of proposed models, which is a very common problem in medical dataset. Finally, removing noise in Malaria images using a Median blur filter, and studying how effects of that on training CNN models. According to the classification results, the proposed model achieved better classification results at accuracy 99.22 on the original Malaria RBCs dataset, and it has the best performance comparing with related work.

Categorical Perception: A Groundwork for Deep Learning.

Classification is one of the major tasks that deep learning is successfully tackling. Categorization is also a fundamental cognitive ability. A well-known perceptual consequence of categorization in humans and other animals, categorical perception, is notably characterized by a within-category compression and a between-category separation: two items, close in input space, are perceived closer if they belong to the same category than if they belong to different categories. Elaborating on experimental and theoretical results in cognitive science, here we study categorical effects in artificial neural networks. We combine a theoretical analysis that makes use of mutual and Fisher information quantities and a series of numerical simulations on networks of increasing complexity. These formal and numerical analyses provide insights into the geometry of the neural representation in deep layers, with expansion of space near category boundaries and contraction far from category boundaries. We investigate categorical representation by using two complementary approaches: one mimics experiments in psychophysics and cognitive neuroscience by means of morphed continua between stimuli of different categories, while the other introduces a categoricality index that, for each layer in the network, quantifies the separability of the categories at the neural population level. We show on both shallow and deep neural networks that category learning automatically induces categorical perception. We further show that the deeper a layer, the stronger the categorical effects. As an outcome of our study, we propose a coherent view of the efficacy of different heuristic practices of the dropout regularization technique. More generally, our view, which finds echoes in the neuroscience literature, insists on the differential impact of noise in any given layer depending on the geometry of the neural representation that is being learned, that is, on how this geometry reflects the structure of the categories.

Improved Arabic Alphabet Characters Classification Using Convolutional Neural Networks (CNN).

Handwritten characters recognition is a challenging research topic. A lot of works have been present to recognize letters of different languages. The availability of Arabic handwritten characters databases is limited. Motivated by this topic of research, we propose a convolution neural network for the classification of Arabic handwritten letters. Also, seven optimization algorithms are performed, and the best algorithm is reported. Faced with few available Arabic handwritten datasets, various data augmentation techniques are implemented to improve the robustness needed for the convolution neural network model. The proposed model is improved by using the dropout regularization method to avoid data overfitting problems. Moreover, suitable change is presented in the choice of optimization algorithms and data augmentation approaches to achieve a good performance. The model has been trained on two Arabic handwritten characters datasets AHCD and Hijja. The proposed algorithm achieved high recognition accuracy of 98.48% and 91.24% on AHCD and Hijja, respectively, outperforming other state-of-the-art models.

Revisiting Dropout Regularization for Cross-Modality Person Re-Identification

This paper investigated targeted dropout regularization for improving convolutional neural network classifier performance on cross-modality person re-identification problems. The dropout regularization is carefully applied on specific layers (targeted dropout regularization) or regions of the features map (spatially targeted dropout regularization). The intuition behind the spatially targeted dropout regularization is that the importantness of feature map regions is not equally the same, and the specific object is more likely placed at the center. We experimented heavily with PKU-Sketch-ReID and multi-modality person re-identification datasets with SwinTransformer deep neural network architecture. Three different targeted dropout regularizations are used for the experiments, including block-wise dropout, horizontal block-wise dropout, and vertical-horizontal block-wise dropout. Experiments on three different sketch re-identification datasets show that the proposed spatially targeted dropout regularization can improve the performance of the deep neural network classifiers with the best rank-1 of 73.20% on the PKU-Sketch-ReID, 52.73% on the SYSU-MM01 dataset, and 72.74% on the RegDB dataset.

Deep Convolution Neural Network with Dropout in Modeling Exchange Rate Volatility

Exchange rate data possesses time-series features such as a trend. Based on a convolutional neural network (CNN) deep learning algorithm, which has the advantages of detecting patterns, extracting effective features, finding interdependence of time series data, and its computational efficiency, this paper proposes a convolutional neural network with dropout model-based approach to model and forecast exchange rates. In the meantime, this paper uses the CNN to first model and predict exchange rates and the corresponding results of this model are compared with those of the CNN-WD. The experimental results showed that the CNN-WD is superior to the CNN model in terms of the error value, fitting degree and training time. The dataset used for this research is that of daily exchange rates for the period between December 1, 2003, and October 15, 2021, which is comprised of 6528 daily trading observations. Adjusted closing rates are chosen. First, this paper adopts a CNN to effectively identify patterns and extract relevant data features of the exchange rate dataset, making use of the past 21 days. Dropout regularization is then adopted to help prevent the CNN model from overfitting data by temporarily removing a neuron from the network along with all its incoming and outgoing connections during training if its generated random value is below the set dropout rate. This paper further evaluates the reducibility and identifiability of the CNN-WD. As an application, this paper uses the CNN-WD to forecast the next month’s average tea price in Kenya.

Skipout: An Adaptive Layer-Level Regularization Framework for Deep Neural Networks

Regularization methods can surprisingly improve the generalization ability of deep neural networks. Among numerous methods, the branch of Dropout regularization is very popular in practice. However, Dropout-like regularization variants have some deficiencies: blindness of regularization, dependency of regularization on layer types, and how to determine the parameters of regularization. In this paper, we propose Skipout framework to tackle these drawbacks by breaking the chain of co-adapted units in consecutive layers. Skipout is a layer-level regularization method that categorizes the layers of a deep network adaptively during training. It divides them into robust and critical layers based on the network architecture and the given task. Instead of turning off some units as in Dropout methods, Skipout recognizes the robust layers and skips them out untrained while sustaining to train the critical layers. The trick of Skipout is in backpropagating the cumulated residual errors via the robust layers, where the activation functions of their units are temporarily set to identity. The units of robust layers are dually activated: by their own functions in the forward pass and by identity in the backward pass. Therefore, Skipout takes advantage of vanishing gradient solutions via robust layers. Moreover, its implementation is simple and applicable to both convolutional and fully-connected layers. Experiments on diverse benchmark datasets and different deep models confirm that Skipout framework can improve the generalization performance of deep neural networks.

An Efficient Lightweight Neural Network for Remote Sensing Image Change Detection

Remote sensing (RS) image change detection (CD) is a critical technique of detecting land surface changes in earth observation. Deep learning (DL)-based approaches have gained popularity and have made remarkable progress in change detection. The recent advances in DL-based methods mainly focus on enhancing the feature representation ability for performance improvement. However, deeper networks incorporated with attention-based or multiscale context-based modules involve a large number of network parameters and require more inference time. In this paper, we first proposed an effective network called 3M-CDNet that requires about 3.12 M parameters for accuracy improvement. Furthermore, a lightweight variant called 1M-CDNet, which only requires about 1.26 M parameters, was proposed for computation efficiency with the limitation of computing power. 3M-CDNet and 1M-CDNet have the same backbone network architecture but different classifiers. Specifically, the application of deformable convolutions (DConv) in the lightweight backbone made the model gain a good geometric transformation modeling capacity for change detection. The two-level feature fusion strategy was applied to improve the feature representation. In addition, the classifier that has a plain design to facilitate the inference speed applied dropout regularization to improve generalization ability. Online data augmentation (DA) was also applied to alleviate overfitting during model training. Extensive experiments have been conducted on several public datasets for performance evaluation. Ablation studies have proved the effectiveness of the core components. Experiment results demonstrate that the proposed networks achieved performance improvements compared with the state-of-the-art methods. Specifically, 3M-CDNet achieved the best F1-score on two datasets, i.e., LEVIR-CD (0.9161) and Season-Varying (0.9749). Compared with existing methods, 1M-CDNet achieved a higher F1-score, i.e., LEVIR-CD (0.9118) and Season-Varying (0.9680). In addition, the runtime of 1M-CDNet is superior to most, which exhibits a better trade-off between accuracy and efficiency.

Dropout Regularization Method of Convolutional Neural Network Based on Constant False Alarm Rate

Dropout Regularization Method of Convolutional Neural Network Based on Constant False Alarm Rate

Sea ice detection from SSMI data based on an improved U-Net convolution neural network

ABSTRACT It is of great significance to accurately obtain the polar sea ice for the study of global hydrosphere and polar climate. The U-Net convolutional neural network (CNN) can achieve end-to-end pixel-level semantic segmentation, and it can learn comprehensive image information as much as possible through the training process. The vertical polarization at 37 GHz and 19 GHz of Special Sensor Microwave/Imager (SSM/I) is used as the input of the improved U-Net CNN, and the improved U-Net CNN is proposed to detect the sea ice. In order to improve the accuracy of sea ice detection, anew feature fusion structure based on the improved U-Net CNN is constructed, which further refines the features of different semantic levels. By using batch normalization (BN), dropout regularization and modifying the loss function to improve the U-Net convolutional neural network, the accuracy of sea ice detection is improved. The results were compared with the NASA Team algorithm, which showed that the improved U-Net CNN is afeasible solution. The sea ice obtained from MODIS data is real result, and our comparison with NASA team algorithm and U-Net CNN showed that the modified U-Net CNN improves the accuracy of sea ice detection to 75%. Therefore, the improved U-Net CNN significantly improves the accuracy of sea ice detection. The improved U-Net CNN had the ability to combine multiple nonlinear transformation functions to solve the problems of sea ice identification, which improved the accuracy and robustness of sea ice detection.

Phase prediction and new parametric approach for multi-component alloys with using deep learning API: Keras

In this work phase prediction was estimated by deep learning algorithm using Keras. 400 alloys were randomly selected from the literature. The valence electron concentration (VEC), atomic size difference (δ), electronegativity difference (∆χ) and κ values are calculated for each alloy and a dataset has been created. Here κ value was defined as a new parametric value to be used in phase estimation. The κ value defined in this study will be an important tool in phase estimations. 320 multi-components’ data was used to train and 80 multi-components’ data was used as validation data. Here three different adaptive optimizers were used: Adam, Adagrad and RMSprop. Most important key feature of adaptive optimizers is that do not require a tuning of the learning rate. The aim of the machine-learning is to develop the capability of the models’ generalisation. Estimation of the models must be succeeded in the data that never seen before. Critical point for machine-learning model is overfitting. In this study; simple hold-out validation was used to avoid overfitting. 320 data used for training and 80 data used for validation. A second check was applied using dropout regularization techniques to avoid overfitting. Adam, Adagrad and RMSprop estimation achieved 90%, 66% and 88% respectively. After all these processes, the physical parameters of the 58 multi-component alloy were entered, which are not in 400 data, and predicted that it had an amorphous or crystalline structure. The κ value defined in this study has shown that it is a simple and easily calculable limit value in phase estimation for multi-component alloys.

Performance of a Novel Chaotic Firefly Algorithm with Enhanced Exploration for Tackling Global Optimization Problems: Application for Dropout Regularization

Swarm intelligence techniques have been created to respond to theoretical and practical global optimization problems. This paper puts forward an enhanced version of the firefly algorithm that corrects the acknowledged drawbacks of the original method, by an explicit exploration mechanism and a chaotic local search strategy. The resulting augmented approach was theoretically tested on two sets of bound-constrained benchmark functions from the CEC suites and practically validated for automatically selecting the optimal dropout rate for the regularization of deep neural networks. Despite their successful applications in a wide spectrum of different fields, one important problem that deep learning algorithms face is overfitting. The traditional way of preventing overfitting is to apply regularization; the first option in this sense is the choice of an adequate value for the dropout parameter. In order to demonstrate its ability in finding an optimal dropout rate, the boosted version of the firefly algorithm has been validated for the deep learning subfield of convolutional neural networks, with respect to five standard benchmark datasets for image processing: MNIST, Fashion-MNIST, Semeion, USPS and CIFAR-10. The performance of the proposed approach in both types of experiments was compared with other recent state-of-the-art methods. To prove that there are significant improvements in results, statistical tests were conducted. Based on the experimental data, it can be concluded that the proposed algorithm clearly outperforms other approaches.

Artificial Neural Network Modelling Using Pre-Operative MRI Features to Predict Obesity in Pediatric Patients With Craniopharyngioma Treated With Proton Therapy

Survival for children and adolescents diagnosed with craniopharyngioma and treated with surgery and radiation therapy exceeds 90% when measured at 10 years. Quality of life is often impacted by obesity resulting from tumor and treatment. We hypothesized that obesity could be predicted by analyzing pre-operative magnetic resonance imaging (MRI) radiomic features. An artificial neural network (ANN) model was developed to predict increase in body mass index (BMI) after surgery and proton therapy using MRI features.Eighty-four patients diagnosed with craniopharyngioma (aged 1-20 years) were included in this study. Surgery and proton therapy were performed on a prospective therapeutic trial. Institutional review board approval was obtained for this retrospective study. BMI was measured every 3 months in the first follow-up year and every 6 months hereafter until 5 years after therapy. The endpoint of this study was defined as the percentile increase of the normalized BMI in the last follow-up year. Radiomics features were extracted from pre-operative MRI images with T1-, T2-weighted, and FLAIR sequences using radiomics software. Images were pre-processed to improve the uniformity of the multicenter MRI datasets. One hundred and five features were calculated, and 5 key features were selected using an embedded feature selection method. A 4-layer ANN with group-Lasso and dropout regularization was developed to predict BMI increase using TensorFlow (Google, USA) toolbox. Patients were divided into 3 groups based on their BMI percentile increase from pre-irradiation baseline with thresholds of 10% and 20%, respectively. Data augmentation and 5-fold cross-validation were used to improve the stability of the ANN model.The average accuracy of the testing dataset in the 5-fold cross validation was 82.3% ± 4.3%, 69.1% ± 3.0%, and 84.0% ± 5.0% for T1, T2, and FLAIR images, respectively. The macro average area under the ROC curve (AUC) was 0.92, 0.88, and 0.92 for T1, T2, and FLAIR images, respectively. The average confusion matrix principal elements were 0.55, 1, and 0.92 for the 3 groups in T1 images, 0.29, 0.96, and 0.82 in T2 images, and 0.59, 0.98, and 0.96 in FLAIR images.A predictive model of post-treatment BMI increase for pediatric patients with craniopharyngioma was developed with relatively high accuracy and AUC. This model could facilitate early identification of patients at risk for obesity after treatment for close surveillance and intervention.

DNNAttention: A deep neural network and attention based architecture for cross project defect number prediction

Software defect prediction (SDP) is the process of detecting fault-prone classes or modules in a software system. It helps in allocating resources before the testing phase more optimally. Due to a lack of an adequate dataset, defects can be predicted by employing data from different projects to train the classifier called cross-project defect prediction (CPDP). Cross-project defect number prediction (CPDNP) is one step ahead of CPDP, in which we can also estimate the number of defects in each module of a software system; we contemplate it as a regression problem. This article dealt with the CPDNP mechanism and suggested a CPDNP architecture by employing a deep neural network and attention layer called DNNAttention. We syntheses substantial data named cross-heap by utilizing an amalgamation of 44 projects from the PROMISE repository. We fed the cross-heap into DNNAttention to train and evaluate the performance over 44 datasets by applying transfer learning. We have also address class imbalance (CI) and overfitting problems by employing multi-label random over-sampling and dropout regularization, respectively. We compared the performance of the DNNAttention using mean squared error (MSE), mean absolute error (MAE), and accuracy over eight baseline methods. We found out of 44 projects, 19 and 20 have minimum MSE and MAE, respectively, and in 19 projects, accuracy yields by the proposed model surpasses exiting techniques. We also compared the performance in terms of Kendall and Fault-Percentile-Average with the recent unsupervised method and found DNNAttention significantly outperforms this method. Moreover, we found the improvement of the DNNAttention over other baseline methods in terms of MAE, MSE, and accuracy by inspecting 20% line of code are substantial. In most situations, the improvements are significant, and they have a large effect size across all 44 projects.