Simple Deep Neural Network Research Articles

Adaptive Deep Neural Network for Click-Through Rate estimation

Over the past few years, deep learning networks have been applied in the estimation of the Click-Through Rate (CTR). The main task of the CTR model lies in predicting a user’s positive/negative response to an item. Most of existing CTR models are constituted of two parts: a deep neural network (DNN) and a wide model (e.g., deep cross network). The wide models are generally designed to learn the feature interaction and contain much more parameters than the DNN. As a consequence, it is quite difficult for existing CTR models to work on the resource-limited devices. Blindly removing the wide model will significantly encumber the predictive performance of the algorithm. In this paper, we adopt the knowledge distillation to train a single multi-branch network and assemble all branches as a teacher network. Each branch network is a simple DNN which not only matches the ground-truth label distribution but also aligns the prediction distribution of the teacher network. One of branch networks, termed as Adaptive Deep Neural Network (ADNN), is trained independently and further combined with a wide model to learn feature interactions. Our method does not require pre-training any high-capacity teacher models, which endows our method with higher efficiency compared with existing ones. The experimental results tested on Criteo and Avazu datasets show that the hybrid model outperforms state-of-the-art methods, and the light model ADNN also has a considerable performance accuracy over certain modern complex models, demonstrating the superiority of the methodology.

Generating Daily High-Resolution Regional XCO2 by Deep Neural Network and Multi-Source Data

CO2 is one of the primary greenhouse gases impacting global climate change, making it crucial to understand the spatiotemporal variations of CO2. Currently, commonly used satellites serve as the primary means of CO2 observation, but they often suffer from striping issues and fail to achieve complete coverage. This paper proposes a method for constructing a comprehensive high-spatiotemporal-resolution XCO2 dataset based on multiple auxiliary data sources and satellite observations, utilizing multiple simple deep neural network (DNN) models. Global validation results against ground-based TCCON data demonstrate the excellent accuracy of the constructed XCO2 dataset (R is 0.94, RMSE is 0.98 ppm). Using this method, we analyze the spatiotemporal variations of CO2 in China and its surroundings (region: 0°–60° N, 70°–140° E) from 2019 to 2020. The gapless and fine-scale CO2 generation method enhances people’s understanding of CO2 spatiotemporal variations, supporting carbon-related research.

T-distributed Stochastic Neighbor Network for unsupervised representation learning

Unsupervised representation learning (URL) is still lack of a reasonable operator (e.g. convolution kernel) for exploring meaningful structural information from generic data including vector, image and tabular data. In this paper, we propose a simple end-to-end T-distributed Stochastic Neighbor Network (TsNet) for URL with clustering downstream task. Concretely, our TsNet model has three major components: (1) an adaptive connectivity distribution learning module is presented to construct a pairwise graph for preserving the local structure of generic data; (2) a T-distributed stochastic neighbor embedding based loss function is designed to learn a transformation between embeddings and original data, which improves the discrimination of representations; (3) a nonlinear parametric mapping is learned via our TsNet on an unsupervised generalized manner, which can address the “out-of-sample” issue. By combining these components, our method is able to considerably outperform previous related unsupervised learning approaches on visualization and clustering of generic data. A simple deep neural network equipped on our model respectively achieves 74.90%, 76.56% ACC and NMI, which is 8% relative improvement over previous state-of-the-art on real single-cell RNA-sequencing (scRNA-seq) datasets clustering.

Assessment of Bone Age Based on Hand Radiographs Using Regression-Based Multi-Modal Deep Learning.

(1) Objective: In this study, a regression-based multi-modal deep learning model was developed for use in bone age assessment (BAA) utilizing hand radiographic images and clinical data, including patient gender and chronological age, as input data. (2) Methods: A dataset of hand radiographic images from 2974 pediatric patients was used to develop a regression-based multi-modal BAA model. This model integrates hand radiographs using EfficientNetV2S convolutional neural networks (CNNs) and clinical data (gender and chronological age) processed by a simple deep neural network (DNN). This approach enhances the model's robustness and diagnostic precision, addressing challenges related to imbalanced data distribution and limited sample sizes. (3) Results: The model exhibited good performance on BAA, with an overall mean absolute error (MAE) of 0.410, root mean square error (RMSE) of 0.637, and accuracy of 91.1%. Subgroup analysis revealed higher accuracy in females ≤ 11 years (MAE: 0.267, RMSE: 0.453, accuracy: 95.0%) and >11 years (MAE: 0.402, RMSE: 0.634, accuracy 92.4%) compared to males ≤ 13 years (MAE: 0.665, RMSE: 0.912, accuracy: 79.7%) and >13 years (MAE: 0.647, RMSE: 1.302, accuracy: 84.6%). (4) Conclusion: This model showed a generally good performance on BAA, showing a better performance in female pediatrics compared to male pediatrics and an especially robust performance in female pediatrics ≤ 11 years.

RN‐Net: Reservoir Nodes‐Enabled Neuromorphic Vision Sensing Network

Neuromorphic computing systems promise high energy efficiency and low latency. In particular, when integrated with neuromorphic sensors, they can be used to produce intelligent systems for a broad range of applications. An event‐based camera is such a neuromorphic sensor, inspired by the sparse and asynchronous spike representation of the biological visual system. However, processing the event data requires either using expensive feature descriptors to transform spikes into frames, or using spiking neural networks (SNNs) that are expensive to train. In this work, a neural network architecture is proposed, reservoir nodes‐enabled neuromorphic vision sensing network (RN‐Net), based on dynamic temporal encoding by on‐sensor reservoirs and simple deep neural network (DNN) blocks. The reservoir nodes enable efficient temporal processing of asynchronous events by leveraging the native dynamics of the node devices, while the DNN blocks enable spatial feature processing. Combining these blocks in a hierarchical structure, the RN‐Net offers efficient processing for both local and global spatiotemporal features. RN‐Net executes dynamic vision tasks created by event‐based cameras at the highest accuracy reported to date at one order of magnitude smaller network size. The use of simple DNN and standard backpropagation‐based training rules further reduces implementation and training costs.

Comparative Analysis of Machine Learning Models for Predicting Diabetes: Unveiling the Superiority of Advanced Ensemble Methods

This study presents a comprehensive analysis of various machine learning models to predict diabetes. The research evaluates and compares the predictive performance of advanced ensemble techniques Extra Trees Classifier and LightGBM with traditional machine learning algorithms and simpler deep neural network (DNN) architectures. The dataset comprises numerous features pertinent to diabetes diagnosis, such as glucose concentration, BMI, and insulin levels, among others. A methodology, including polynomial feature transformation and ten-fold cross-validation, was employed to ensure the study's reliability and the models' capability to generalize. The advanced ensemble models, Extra Trees and LightGBM, achieved stellar predictive metrics, with the former attaining a near-perfect ROC AUC, accuracy, precision, and an F1 score close to 1. LightGBM followed closely, demonstrating the high efficacy of ensemble methods in complex data settings. These results were contrasted with significantly lower performance metrics from DNNs and respectable, albeit lower, scores from traditional models like Decision Trees, Random Forest, KNN, and XGBoost.

Predicting the Number of Software Faults using Deep Learning

The software testing phase requires considerable time, effort, and cost, particularly when there are many faults. Thus, developers focus on the evolution of Software Fault Prediction (SFP) to predict faulty units in advance, therefore, improving software quality significantly. Forecasting the number of faults in software units can efficiently direct software testing efforts. Previous studies have employed several machine learning models to determine whether a software unit is faulty. In this study, a new, simple deep neural network approach that can adapt to the type of input data was designed, utilizing Convolutional Neural Networks (CNNs) and Multi-Layer Perceptron (MLP), to predict the number of software faults. Twelve open-source software project datasets from the PROMISE repository were used for testing and validation. As data imbalance can negatively impact prediction accuracy, the new version of synthetic minority over-sampling technique (SMOTEND) was used to resolve data imbalance. In experimental results, a lower error rate was obtained for MLP, compared to CNN, reaching 0.195, indicating the accuracy of this prediction model. The proposed approach proved to be effective when compared with two of the best machine learning models in the field of prediction. The code will be available on GitHub.

Cocrystal Prediction Based on Deep Forest Model—A Case Study of Febuxostat

To aid cocrystal screening, a deep forest-based cocrystal prediction model was developed in this study using data from the Cambridge Structural Database (CSD). The positive samples in the experiment came from the CSD. The negative samples were partly from the failure records in other papers, and some were randomly generated according to specific rules, resulting in a total of 8576 pairs. Compared with the models of traditional machine learning methods and simple deep neural networks models, the deep forest model has better performance and faster training speed. The accuracy is about 95% on the test set. Febuxostat cocrystal screening was also tested to verify the validity of the model. Our model correctly predicted the formation of cocrystal. It shows that our model is practically useful in practice.

Abstract 65: Automated Detection of Facial Droop: Ml/AI Based Tool for Early Detection of Stroke

Introduction: Facial droop (FD) is a common symptom (45-60%) of stroke so early detection is critical for timely treatment. FD is a part of almost all stroke scales. Our study is to develop a machine learning (ML)/AI based tool that uses an automated detection of FD as one of the components. Our approach uses a normalized value of minor facial expressions called a blendshape. Methods: Google’s Mediapipe was used to create 52 different blendshape values (e.g., left eyebrow raised, right mouth lowered, etc.) from an input image, which were then used as inputs for a simple deep neural network (DNN). We trained this DNN using the blendshapes from 872 images of people with symptoms of facial palsy (FP) and 872 images of people without symptoms exhibiting multiple emotions (neutral, smiling, frowning, etc.). Results: The proposed approach was evaluated on a dataset of facial images of patients with FD and strokes. Our model showed promising results with over 91% test accuracy. This result suggests that the proposed approach is a promising new method for the early detection of FP with high accuracy. The approach is integrated in our ML/AI based tool to replicate different stroke scales. Conclusions: Our model shows that the use of Blendshapes as features in a ML classifier can produce great results in the detection of FP. For greater accuracy, a better dataset with higher quality images as well as unilateral facial expressions (winking, smirks, etc.) should be used. In a future study, a ML model that uses Blendshapes as inputs could be used to classify FP on a scale.

Consumption prediction with privacy concern: Application and evaluation of Federated Learning

In this paper, we propose privacy-friendly electricity consumption prediction models based on Federated Learning (FL). Federated Learning provides a novel framework for Artificial Neural Network training. It decouples data storage from model training from decentralized data sources. A central server does not need to collect raw training data from individual clients but only the model parameters, like the gradients or weights updates. The sensitive raw data can be safely stored and used by end-users themselves. We improve the accuracy of the FL model by first clustering households, and training a personalized model for each cluster. We also analyse the Deep Leakage from Gradients (DLG) attack in our study case with three scenarios. Our simulations suggest that the DLG attack can barely succeed in consumption prediction. In this way, the FL model can guarantee clients’ privacy by design. We use a large-scale dataset that contains 3590 households’ 1.5 years of consumption to test the FL model’s performance. Several clustering algorithms are tested for the following experiments. To comprehensively test the FL model’s performance, we propose several popular neural network models: the simple Deep Neural Network, Long Short-Term Memory, Convolutional Neural Network, and WaveNet. These models are both trained under the centralized and federated framework. The federated trained models slightly sacrifice the model’s accuracy while guaranteeing the client’s data privacy. Meanwhile, the federated model shows remarkable scalability. Under the FL framework, new clients can obtain their prediction 6 times faster. Moreover, the federated model has strong robustness against missing or damaged training data. With a certain percentage of missing data in the training set, the centralized model’s accuracy gets 27% worse, while the federated model’s accuracy only gets 2% worse.

Speech Emotion Recognition Using Convolutional Neural Networks with Attention Mechanism

Speech emotion recognition (SER) is an interesting and difficult problem to handle. In this paper, we deal with it through the implementation of deep learning networks. We have designed and implemented six different deep learning networks, a deep belief network (DBN), a simple deep neural network (SDNN), an LSTM network (LSTM), an LSTM network with the addition of an attention mechanism (LSTM-ATN), a convolutional neural network (CNN), and a convolutional neural network with the addition of an attention mechanism (CNN-ATN), having in mind, apart from solving the SER problem, to test the impact of the attention mechanism on the results. Dropout and batch normalization techniques are also used to improve the generalization ability (prevention of overfitting) of the models as well as to speed up the training process. The Surrey Audio–Visual Expressed Emotion (SAVEE) database and the Ryerson Audio–Visual Database (RAVDESS) were used for the training and evaluation of our models. The results showed that the networks with the addition of the attention mechanism did better than the others. Furthermore, they showed that the CNN-ATN was the best among the tested networks, achieving an accuracy of 74% for the SAVEE database and 77% for the RAVDESS, and exceeding existing state-of-the-art systems for the same datasets.

Online machine learning approach for system marginal price forecasting using multiple economic indicators: A novel model for real-time decision making

In comparison to other countries, South Korea has a high reliance on energy, with the majority of its electricity being generated by a government-run company to ensure a stable and affordable supply. Unlike the "pay-as-bid" pricing approach, South Korea utilizes the system marginal price (SMP), also known as "pay-as-clear." Accurate SMP forecasting is crucial for guaranteeing steady economic growth because manufacturing is South Korea's flagship industry and accounts for 50 % of the nation's power consumption. In this study, a combination of machine learning-based batch learning and online learning techniques were employed to forecast the SMP in South Korea, utilizing a dataset consisting of five energy sectors, two financial sectors, and one transportation sector. The analysis of the F-value revealed that the coal sector, which is one of the energy sectors, had the most significant influence on SMP indicating the greatest score of 2,328. In this study, three machine learning models, namely support vector regression, simple deep neural network, and deep neural network, were suggested and compared for batch learning to determine the best-trained model. The evaluation metrics were used to assess the performance of these models. Based on the results obtained, the simple deep neural network was found to outperform the other models in terms of accuracy. Furthermore, two methods such as weight modification and time interval updating between inputs and output were employed for online learning based on the trained batch model. Upon the implementation of model updates, an ongoing assessment of its performance transpired utilizing the metrics of coefficient of determination, root mean square error, mean absolute error, and mean absolute percentage error. The average values for these metrics were observed to be 0.924, 7.991, 5.035, and 0.052, respectively. This study is expected to provide direct assistance in the formulation of energy plans for decision-makers in the industrial sector.

Divide-and-conquer DNN approach for the inverse point source problem using a few single frequency measurements

We consider the inverse problem to determine the number and locations of acoustic point sources from single low-frequency partial data. The problem is particularly challenging in the sense that the data is available only at a few locations which span a small aperture. Integrating the deep neural networks (DNNs) and Bayesian inversion, we propose a divide-and-conquer approach by dividing the inverse problem into three subproblems. The first subproblem is to determine the number of point sources, which is formulated as a common machine learning task—classification. A simple DNN is proposed and trained to predict the numbers of the point sources. The second subproblem is to reconstruct the (approximate) locations of the point sources. We formulate the problem as a nonlinear function with the input being the measured data and the output being a carefully elaborated location vector. Then a second DNN is proposed to learn the mapping and predict the location vector effectively. The location vector is post-processed to provide an indicator (image) function for the (approximate) locations of the point sources. The third subproblem is to improve the accuracy of the location prediction, for which we employ a Bayesian inversion algorithm. This divide-and-conquer approach can effectively treat both phase and phaseless data as demonstrated by various examples.

Helium focused ion beam induced subsurface damage on Si and SiC substrates: experiments and generative deep neural network modeling via position-dependent input

Deep neural networks (DNNs) are reshaping many fields due to their end-to-end ability to learn directly from data, leading to outstanding performance, especially in image processing. However, their training requires large image datasets while image preparation can be expensive/time-consuming. If the underlying behavior can be formulated as spatial-dependent, we propose that every pixel from every image can be considered as a different data source, thus enabling a truly large pixel-based dataset with millions of elements (‘position-dependent input’), even when obtained from a few images. The approach is applied to helium focused ion beam nanofabrication, where the cross-section of the helium-damaged region essentially resembles that of a lightbulb, while close inspection reveals the presence of (i) an outer defective region in direct contact with the bulk substrate and (ii) an inner amorphous region filled with helium bubbles of gradually increasing size. Interestingly, the amorphous phase may swell upwards to form a protruding mesa, depending on the beam energy and dose, a feature that has resisted modeling so far. Through dedicated experiments on both Si and SiC substrates, and careful image analysis (segmentation), we describe the transformations that take place in the defective and amorphous regions with increasing energy and dose, and use the outlined position-dependent input together with a simple DNN of 4 hidden layers and 16 neurons per layer to describe all damage features realistically, inherently demonstrating generative behavior. This is the first time that any model predicts swelling satisfactorily. Generalization is surprisingly smooth and accurate.

Pre-processed and combined EEG data for epileptic seizure classification using deep learning

Epilepsy is a neurological disease that affects nearly 60 million people around the world. Electroencephalography (EEG) and Machine Learning (ML) can be used to detect epileptic seizures, where ML algorithms depend on hand crafted features. Recently, deep learning (DL) methods became popular due to automatic feature learning. However, current DL based approaches mainly focuses on different DL architecture design, especially deeper DL networks. Only few works experiment modified EEG data as an input to DL models. In this work, we propose modified, pre-processed and combined EEG signals as an input to different DL models for epilepsy seizure classification rather than using raw EEG signals. A variety of pre-processed and combined EEG signals have been evaluated with three different DL architectures: A simple Deep Neural Network (DNN) model and a moderately complex 1D-Convolutional Neural Network (CNN) model and a complex 1D-CNN model. We perform several experiments for two-class and three-class epileptic seizure classification on UCI-Bonn dataset using the proposed EEG signals: (1)Original EEG signal of UCI-Bonn dataset, (2)standardized EEG signal, (3)original EEG signal combined with squared EEG signal, and then standardized, (4)original EEG signal combined with differentiated EEG signal, and then standardized, and (5)original EEG signal combined with magnitude of Fast Fourier Transform (FFT) of EEG signal, and then standardized (original + FFT). Extensive evaluations and comparisons show that the best results are achieved with original + FFT combination on all DL architectures. Using original + FFT EEG signal, competitive results can be achieved with simple and moderately complex DL models, by utilizing less computational resources.

A data-driven deep learning network for massive MIMO detection with high-order QAM

Massive multiple-input multiple-output (MIMO) can provide higher spectral efficiency and energy efficiency compared to conventional MIMO systems. Unfortunately, as the numbers of modulation orders and antennas increase, the computational complexity of conventional symbol detection algorithms becomes unaffordable and their performance deteriorates. However, deep learning (DL) techniques can provide flexibility, nonlinearity and computational parallelism for massive MIMO detection to address these challenges. We propose an efficient data-driven detection network, i.e., accelerated multiuser interference cancellation network (AMIC-Net), for uplink massive MIMO systems. Specifically, we first introduce an extrapolation factor regarded as a learnable parameter into the multiuser interference cancellation (MIC) algorithm for iterative sequential detection (ISD) detector through extrapolation technique to enhance the convergence performance. Then we unfold the above accelerated iterative algorithm and adopt a sparsely connected approach, instead of fully connected, to obtain a relatively simple deep neural network (DNN) structure to enhance the detection performance through the data-driven DL approach. Furthermore, in order to accommodate communication scenarios with higher-order modulation, a novel activation function is proposed, which is composed of multiple softsign activation functions with additional learnable parameters to implement a multi-segment mapping of the set of constellation points with different modulations. Numerical results show that the proposed DL network can bring significant performance gain to ISD detector with various massive antenna settings and outperform the existing detectors with the same or lower computational complexity, especially in high-order QAM modulation scenarios.

Low-sample classification in NIDS using the EC-GAN method

Numerous advanced methods have been applied throughout the years for the use in Network Intrusion Detection Systems (NIDS). Among these are various Deep Learning models, which have shown great success for attack classification. Nevertheless, false positive rate and detection rate of these systems remains a concern. This is mostly because of the low-sample, imbalanced nature of realistic datasets, which make models challenging to train. Considering this, we applied a novel semi-supervised EC-GAN method for network flow classifi- cation of CIC-IDS-2017 dataset. EC-GAN uses synthetic data to aid the training of a supervised classifier on low-sample data. To achieve this, we modified the original EC-GAN to work with tabular data. In our approach, WCGAN-GP is used for synthetic tabular data generation, while  a simple deep neural network is used for classification. The conditional nature of WCGAN-GP diminishes the class imbalance problem, while GAN itself solves the low-sample problem. This approach was successful in generating believable synthetic data, which was consequently used for training and testing the EC-GAN. To obtain our results, we trained a classifier on progressively smaller versions of the CIC-DIS-2017 dataset, first via a novel EC-GAN method and then in the conventional way, without the help of synthetic data. We then compared these two sets of results with another author’s results using accuracy, false positive rate, detection rate and macro F1 score as metrics. Our results showed that supervised classifier trained with EC-GAN can achieve significant results even when trained on as little as 25% of the original imbalanced dataset.

Simple dilated convolutional neural network for quantitative modeling based on near infrared spectroscopy techniques

A simple deep convolutional neural network architecture had been developed and applied to the establishment of a quantitative model based on near infrared spectroscopy techniques. The network architecture only contained general convolutional layers and dilated convolutional layers. The determination coefficient was used as a criterion to stop the training process of the neural network. Three data sets were used to test the performance of the neural network. The major results are (1) quantitative model can be established within one million iterations; (2) the quantitative model established on one spectrometer can be applied to other spectrometers of same manufacturer and even the same kind of spectrometer of different manufacturer; (3) simultaneous quantitative analysis of four components in grain samples. This study provided a new strategy to apply techniques of deep neural network in quantitative analysis.

Effects of sliding window variation in the performance of acceleration-based human activity recognition using deep learning models.

Deep learning (DL) models are very useful for human activity recognition (HAR); these methods present better accuracy for HAR when compared to traditional, among other advantages. DL learns from unlabeled data and extracts features from raw data, as for the case of time-series acceleration. Sliding windows is a feature extraction technique. When used for preprocessing time-series data, it provides an improvement in accuracy, latency, and cost of processing. The time and cost of preprocessing can be beneficial especially if the window size is small, but how small can this window be to keep good accuracy? The objective of this research was to analyze the performance of four DL models: a simple deep neural network (DNN); a convolutional neural network (CNN); a long short-term memory network (LSTM); and a hybrid model (CNN-LSTM), when variating the sliding window size using fixed overlapped windows to identify an optimal window size for HAR. We compare the effects in two acceleration sources’: wearable inertial measurement unit sensors (IMU) and motion caption systems (MOCAP). Moreover, short sliding windows of sizes 5, 10, 15, 20, and 25 frames to long ones of sizes 50, 75, 100, and 200 frames were compared. The models were fed using raw acceleration data acquired in experimental conditions for three activities: walking, sit-to-stand, and squatting. Results show that the most optimal window is from 20–25 frames (0.20–0.25s) for both sources, providing an accuracy of 99,07% and F1-score of 87,08% in the (CNN-LSTM) using the wearable sensors data, and accuracy of 98,8% and F1-score of 82,80% using MOCAP data; similar accurate results were obtained with the LSTM model. There is almost no difference in accuracy in larger frames (100, 200). However, smaller windows present a decrease in the F1-score. In regard to inference time, data with a sliding window of 20 frames can be preprocessed around 4x (LSTM) and 2x (CNN-LSTM) times faster than data using 100 frames.

Exploring Accurate Potential Energy Surfaces via Integrating Variational Quantum Eigensolver with Machine Learning.

The potential energy surface (PES) is crucial for interpreting a variety of chemical reaction processes. However, predicting accurate PESs with high-level electronic structure methods is a challenging task due to the high computational cost. As an appealing application of quantum computing, we show in this work that variational quantum algorithms can be integrated with machine learning (ML) techniques as a promising scheme for exploring accurate PESs. Different from using a ML model to represent the potential energy, we encode the molecular geometry information into a deep neural network (DNN) to represent parameters of the variational quantum eigensolver (VQE), leaving the PES to the wave function ansatz. Once the DNN model is trained, the variational optimization procedure that hinders the application of the VQE to complex systems is avoided, and thus the evaluation of PESs is significantly accelerated. Numerical results demonstrate that a simple DNN model is able to reproduce accurate PESs for small molecules.