Deep Neural Network Structure Research Articles

Research on disease diagnosis based on teacher-student network and Raman spectroscopy.

Diabetic nephropathy is a serious complication of diabetes, and primary Sjögren's syndrome is a disease that poses a major threat to women's health. Therefore, studying these two diseases is of practical significance. In the field of spectral analysis, although common Raman spectral feature selection models can effectively extract features, they have the problem of changing the characteristics of the original data. The teacher-student network combined with Raman spectroscopy can perform feature selection while retaining the original features, and transfer the performance of the complex deep neural network structure to another lightweight network structure model. This study selects five flow learning models as the teacher network, builds a neural network as the student network, uses multi-layer perceptron for classification, and selects the optimal features based on the evaluation indicators accuracy, precision, recall, and F1-score. After five-fold cross-validation, the research results show that in the diagnosis of diabetic nephropathy, the optimal accuracy rate can reach 98.3%, which is 14.02% higher than the existing research; in the diagnosis of primary Sjögren's syndrome, the optimal accuracy rate can be reached 100%, which is 10.48% higher than the existing research. This study proved the feasibility of Raman spectroscopy combined with teacher-student network in the field of disease diagnosis by producing good experimental results in the diagnosis of diabetic nephropathy and primary Sjögren's syndrome.

Effective Monoaural Speech Separation through Convolutional Top-Down Multi-View Network

Speech separation, sometimes known as the “cocktail party problem”, is the process of separating individual speech signals from an audio mixture that includes ambient noises and several speakers. The goal is to extract the target speech in this complicated sound scenario and either make it easier to understand or increase its quality so that it may be used in subsequent processing. Speech separation on overlapping audio data is important for many speech-processing tasks, including natural language processing, automatic speech recognition, and intelligent personal assistants. New speech separation algorithms are often built on a deep neural network (DNN) structure, which seeks to learn the complex relationship between the speech mixture and any specific speech source of interest. DNN-based speech separation algorithms outperform conventional statistics-based methods, although they typically need a lot of processing and/or a larger model size. This study presents a new end-to-end speech separation network called ESC-MASD-Net (effective speaker separation through convolutional multi-view attention and SuDoRM-RF network), which has relatively fewer model parameters compared with the state-of-the-art speech separation architectures. The network is partly inspired by the SuDoRM-RF++ network, which uses multiple time-resolution features with downsampling and resampling for effective speech separation. ESC-MASD-Net incorporates the multi-view attention and residual conformer modules into SuDoRM-RF++. Additionally, the U-Convolutional block in ESC-MASD-Net is refined with a conformer layer. Experiments conducted on the WHAM! dataset show that ESC-MASD-Net outperforms SuDoRM-RF++ significantly in the SI-SDRi metric. Furthermore, the use of the conformer layer has also improved the performance of ESC-MASD-Net.

BConvLSTM: a deep learning-based technique for severity prediction of a traffic crash

Predicting the severity of crashes has become a significant issue in research on road accidents. Traffic accident severity prediction is essential for protecting vulnerable road users and preventing traffic accidents. For practitioners to identify significant risk variables and set appropriate countermeasures in place, explainability of the forecast is also essential. Most previous research ignores the severity of property loss caused by traffic accidents and cannot differentiate between different levels of fatalities and property loss severity. Additionally, while an understandable structure of deep neural networks (DNN) is significantly lacking in existing works, understanding traditional systems is quite simple. An inability to use structural data when describing forecasting and the many attempts to incorporate neural networks afflict the absence of hidden layers. We propose a Deep Learning (DL) framework for forecasting traffic crash severity to overcome the accident severity prediction. It has three steps to process. Initially, collected input data are cleaned. Data cleaning is performed in a preprocessing step. We conduct experiments on two datasets, A Countrywide (US) Traffic Accident Dataset and UK Road Accident Dataset. The outcomes of the experiments demonstrate that the proposed technique outperformed other approaches and produced the best accuracy.

Implementation of singularity-free inverse kinematics for humanoid robotic arm using Bayesian optimized deep neural network

This study presents the efficacy of deep learning techniques in controlling the arm of a humanoid robot without resorting to inverse kinematic analysis. Emphasizing real-time applicability, a straightforward deep neural network (DNN) structure is developed and optimized using hyperparameter Bayesian optimization. The Keras Bayesian Optimization Tuner with Gaussian process fine-tunes the DNN architecture. Utilizing a simulation environment and generating around 10 billion datasets, the optimized DNN is evaluated on both simulated and real robots. Impressively, the trained DNN exhibits a notable ability to predict robot control within a 25 ms timeframe, achieving a total Mean Absolute Error (MAE) of 0.02 for the xyz-axes. These results underscore the potential of deep learning-based approaches in humanoid robot arm control, eliminating the need for inverse kinematic equations and providing valuable insights for future robotic system development.

Modified Structure of Deep Neural Network for Training Multi-Fidelity Data With Non-Common Input Variables

Abstract Multi-fidelity surrogate (MFS) modeling technology, which efficiently constructs surrogate models using low-fidelity (LF) and high-fidelity (HF) data, has been studied to enhance the predictive capability of engineering performances. In addition, several neural network (NN) structures for MFS modeling have been introduced, benefiting from recent developments in deep learning research. However, existing multi-fidelity (MF) NNs have been developed assuming identical sets of input variables for LF and HF data, a condition that is often not met in practical engineering systems. Therefore, this study proposes a new structure of composite NN designed for MF data with different input variables. The proposed network structure includes an input mapping network that connects the LF and HF data's input variables. Even when the physical relationship between these variables is unknown, the input mapping network can be concurrently trained during the process of training the whole network model. Customized loss functions and activation variables are suggested in this study to facilitate forward and backward propagation for the proposed NN structures when training MF data with different inputs. The effectiveness of the proposed method, in terms of prediction accuracy, is demonstrated through mathematical examples and practical engineering problems related to tire performances. The results confirm that the proposed method offers better accuracy than existing surrogate models in most problems. Moreover, the proposed method proves advantageous for surrogate modeling of nonlinear or discrete functions, a characteristic feature of NN-based methods.

Ultra-short-term wind power probabilistic forecasting based on an evolutionary non-crossing multi-output quantile regression deep neural network

Ultra-short-term wind power probabilistic forecasting is of significance for stable power grid operation; however, it is still challenging due to the inherent nonlinearity and uncertainty. Most state-of-the-art methods have focused on achieving quantile prediction using a combination of linear quantile regression and nonlinear complex deep neural networks. Yet, these methods struggle with several dilemmas. Quantile regression deep neural networks require a complete training once for each quantile. The multi-training mode and complex structure of quantile regression deep neural network can lead to extremely high computational complexity. Most of the training of quantile regression deep neural networks are guided by the loss of each quantile, and the weights are adjusted by gradient descent in which the gradient explosion and quantile crossover may be encountered. Therefore, this paper proposes a non-crossing multi-output quantile regression deep neural network optimized by chaotic particle swarm optimization. It designs a multi-output deep neural network to output all quantile estimations simultaneously through one training, effectively solving the structural complexity problem of traditional quantile regression deep neural networks. Since quantile regression produces a non-differentiable loss function which significantly hinders model training, the proposed neural network is trained by chaotic particle swarm optimization. It not only achieves the effect of optimizing all quantile losses simultaneously, but also can significantly alleviate the dilemma of training in traditional neural network weight optimization. In addition, several non-crossing constraints are designed for avoiding quantile crossover. The proposed model is trained and tested on two real-world wind power case studies. The experiment results show that the proposed model shows superiority in performance criteria, training speed, and avoiding quantile crossover.

Prioritizing rear-end crash explanatory factors for injury severity level using deep learning and global sensitivity analysis

Traffic accidents are usually unique events with unpredictable geographical and temporal dimensions; thus, accident injury severity level (INJ-SL) analysis presents formidable categorization and data stability problems. Classical statistical models are limited in their ability to correctly model INJ-SL, whilst sophisticated machine learning approaches do not appear to have any equations to prioritize/analyze multiple contributing factors to forecast accidents accompanying INJ-SLs. In addition, the intercorrelations between the input variables may render the conclusions of a formal sensitivity analysis incorrectly. Rear-end collisions are the most common form of traffic accidents; consequently, their linked INJ-SL requires more research. This paper provides a complex technique based on a deep learning paradigm paired with different indicators of Global Sensitivity Analysis to address all of these concerns. Unlike existing neural network designs, this technique presents a deep residual neural network structure that employs residual shortcuts (i.e., connections). The connections enable the DRNNs to sidestep a few levels of the deep network architecture, evading the regular training with high accuracy issues. Using the trained DRNNs model, a Latin Hypercube sampling simulation was undertaken to determine each explanatory component's influence on the resulting INJ-SL. The latest available data from 2011 to 2018 is used to assess all rear-end collisions in North Carolina. A comparison was made between the performance of two different schemes of data categorization using a set of global sensitivity metrics. It was determined that the devised technique overcame the data heterogeneity problems to achieve an accuracy of 87%. In addition, the proposed sensitivity analysis identified the most relevant factors associated with INJ-SL rear-end collisions.

Unambiguous and High-Fidelity Backdoor Watermarking for Deep Neural Networks.

The unprecedented success of deep learning could not be achieved without the synergy of big data, computing power, and human knowledge, among which none is free. This calls for the copyright protection of deep neural networks (DNNs), which has been tackled via DNN watermarking. Due to the special structure of DNNs, backdoor watermarks have been one of the popular solutions. In this article, we first present a big picture of DNN watermarking scenarios with rigorous definitions unifying the black-and white-box concepts across watermark embedding, attack, and verification phases. Then, from the perspective of data diversity, especially adversarial and open set examples overlooked in the existing works, we rigorously reveal the vulnerability of backdoor watermarks against black-box ambiguity attacks. To solve this problem, we propose an unambiguous backdoor watermarking scheme via the design of deterministically dependent trigger samples and labels, showing that the cost of ambiguity attacks will increase from the existing linear complexity to exponential complexity. Furthermore, noting that the existing definition of backdoor fidelity is solely concerned with classification accuracy, we propose to more rigorously evaluate fidelity via examining training data feature distributions and decision boundaries before and after backdoor embedding. Incorporating the proposed prototype guided regularizer (PGR) and fine-tune all layers (FTAL) strategy, we show that backdoor fidelity can be substantially improved. Experimental results using two versions of the basic ResNet18, advanced wide residual network (WRN28_10) and EfficientNet-B0, on MNIST, CIFAR-10, CIFAR-100, and FOOD-101 classification tasks, respectively, illustrate the advantages of the proposed method.

DeepKriging: Spatially Dependent Deep Neural Networks for Spatial Prediction

In spatial statistics, a common objective is to predict values of a spatial process at unobserved locations by exploiting spatial dependence. Kriging provides the best linear unbiased predictor using covariance functions and is often associated with Gaussian processes. However, when considering non-linear prediction for non-Gaussian and categorical data, the Kriging prediction is no longer optimal, and the associated variance is often overly optimistic. Although deep neural networks (DNNs) are widely used for general classification and prediction, they have not been studied thoroughly for data with spatial dependence. In this work, we propose a novel DNN structure for spatial prediction, where the spatial dependence is captured by adding an embedding layer of spatial coordinates with basis functions. We show in theory and simulation studies that the proposed DeepKriging method has a direct link to Kriging in the Gaussian case, and it has multiple advantages over Kriging for non-Gaussian and non-stationary data, i.e., it provides non-linear predictions and thus has smaller approximation errors, it does not require operations on covariance matrices and thus is scalable for large datasets, and with sufficiently many hidden neurons, it provides the optimal prediction in terms of model capacity. We further explore the possibility of quantifying prediction uncertainties based on density prediction without assuming any data distribution. Finally, we apply the method to predicting PM2.5 concentrations across the continental United States.

Self-Supervised Medical Image Denoising Based on WISTA-Net for Human Healthcare in Metaverse.

Medical image processing plays an important role in the interaction of real world and metaverse for healthcare. Self-supervised denoising based on sparse coding methods, without any prerequisite on large-scale training samples, has been attracting extensive attention for medical image processing. Whereas, existing self-supervised methods suffer from poor performance and low efficiency. In this paper, to achieve state-of-the-art denoising performance on the one hand, we present a self-supervised sparse coding method, named the weighted iterative shrinkage thresholding algorithm (WISTA). It does not rely on noisy-clean ground-truth image pairs to learn from only a single noisy image. On the other hand, to further improve denoising efficiency, we unfold the WISTA to construct a deep neural network (DNN) structured WISTA, named WISTA-Net. Specifically, in WISTA, motivated by the merit of the lp-norm, WISTA-Net has better denoising performance than the classical orthogonal matching pursuit (OMP) algorithm and the ISTA. Moreover, leveraging the high-efficiency of DNN structure in parameter updating, WISTA-Net outperforms the compared methods in denoising efficiency. In detail, for a 256 by 256 noisy image, the running time of WISTA-Net is 4.72 s on the CPU, which is much faster than WISTA, OMP, and ISTA by 32.88 s, 13.06 s, and 6.17 s, respectively.

Transfer learning approach to reduce similar IOT sensor data for industrial applications

The Industrial Internet of Things (IIoT) is a developing technology that has the potential to advance industrial intelligence, achieve targeted effectiveness, and lower manufacturing costs. Data fusion, a procedure that enables the collection, analysis, and processing of the enormous amounts of Internet of Things (IoT) information created by industrialized machinery and applications, is essential to IIoT's advancement and improvement of industrial products and applications. A real-time, efficient, and private information image fusion mechanism is required by the IIoT. Moreover, the current efforts call for such training of multiple systems in data analysis that also fall short of meeting real-time IIoT needs. The ineffectiveness of internal defenses and the challenge of finding the right balance between system stability and data confidentiality, however, undermine the efficiency and privacy preservation of a data fusion process. In this paper, we propose a Transfer learning-based Secure Data Fusion (TSDF) for the IIoT to address the aforementioned issues. In addition to the proposed network, we developed and deployed a defect estimation method that makes use of sensor data fusion to evaluate the manufacturing plant's operational state. The efficiency outcomes of evaluating the conceptual model are included in the original study, including a comparison to a conventional deep neural network structure. The study's findings demonstrate that TSDF may provide privacy, preserving information fusion in diverse IIoT application environments while reaching higher system capacity and low latency.

LGNN: a novel linear graph neural network algorithm.

The emergence of deep learning has not only brought great changes in the field of image recognition, but also achieved excellent node classification performance in graph neural networks. However, the existing graph neural network framework often uses methods based on spatial domain or spectral domain to capture network structure features. This process captures the local structural characteristics of graph data, and the convolution process has a large amount of calculation. It is necessary to use multi-channel or deep neural network structure to achieve the goal of modeling the high-order structural characteristics of the network. Therefore, this paper proposes a linear graph neural network framework [Linear Graph Neural Network (LGNN)] with superior performance. The model first preprocesses the input graph, and uses symmetric normalization and feature normalization to remove deviations in the structure and features. Then, by designing a high-order adjacency matrix propagation mechanism, LGNN enables nodes to iteratively aggregate and learn the feature information of high-order neighbors. After obtaining the node representation of the network structure, LGNN uses a simple linear mapping to maintain computational efficiency and obtain the final node representation. The experimental results show that the performance of the LGNN algorithm in some tasks is slightly worse than that of the existing mainstream graph neural network algorithms, but it shows or exceeds the machine learning performance of the existing algorithms in most graph neural network performance evaluation tasks, especially on sparse networks.

Inertia matrix identification of combined spacecraft using a deep neural network with optimized network structure

After the space robot captures the out-of-control spacecraft, they form a combined spacecraft. For the parameter identification problem of combined spacecraft with complex angular rate measurement noise, deep learning could provide a new solution, due to its ability to extract robust features from noisy data. However, the deep neural network (DNN) has substantial computing resource requirements, which restricts its application in spacecraft systems. To solve the above problems, an inertia matrix identification approach combining a structure optimization strategy of DNN is proposed to identify the inertia matrix in the presence of complex measurement noise using the DNN with optimized structure. The attitude dynamics of combined spacecraft are firstly derived, and the identification problem considering complex measurement noise is described. Based on the dynamics model, the construction process of DNN for accurately identifying the inertia matrix is proposed. A structure optimization strategy of DNN is then designed to optimize the structure of DNN for increasing computing efficiency. This strategy provides a whole set of solutions including the structure selection of the initial DNN, the redundant neurons pruning process, and the end-of-optimization condition. By combining the proposed construction process of DNN, the structure of the DNN is finally optimized by iterating the structured pruning and retraining step. The performance of the proposed DNN with optimized network structure is verified on a testing set with complex measurement noise. The results show that the proposed approach can accurately identify the inertia matrix of combined spacecraft, and meanwhile greatly improve the computing efficiency of DNN.

Data-driven model for predicting the current cycle count of power batteries based on model stacking

Electric vehicles have been heavily promoted in recent years to reduce carbon emissions. With rising fuel prices, more and more electric vehicles are being chosen by consumers. The safety performance of electric vehicle batteries is an indicator of great concern to the new energy vehicle industry and consumer.Many researchers have used machine learning to train on data to obtain models that can predict the current or the remaining number of battery cycles to achieve battery safety management. To further improve the model's prediction accuracy, this work designs an algorithm structure combining a neural network and a gradient-boosting decision tree (GBDT) class. The neural network structure is in two layers, the first layer is a parallel network composed of deep neural networks (DNN) and long short-term memory (LSTM), and the other layer is a network with a DNN structure. The trained neural network model is used to transform the input features into new features, and then the CatBoost algorithm is used to train the old and new features to obtain the prediction model. The results show that the method reduces the mean absolute error (MAE) by 41 % on the original basis. And it provides an idea for the research of related problems.

Photovoltaic power prediction based on sky images and tokens-to-token vision transformer

Photovoltaic (PV) power generation has high uncertainties due to the randomness and imbalance nature of solar energy and meteorological parameters. Hence, accurate PV power forecasts are essential in the operation of PV power plants (PVPP) for short-term dispatches and power generation schedules. In this paper, a new deep neural network structure based on vision transformer is proposed to combine sky images and Tokens-To-Token（T2T） for photovoltaic power prediction. The method uses an incremental tokenization module to aggregate neighboring image patches into tokens, which capture the local structural information of the clouds. Then, an efficient T2T-ViT backbone network is used to extract the global attentional relationships of the tokens for power prediction. In order to evaluate the performance of the proposed model, the method was compared with several deep learning architectures such as ResNet and GoogleNet on a dataset collected by the National Renewable Energy Laboratory in Colorado, USA. The results of power prediction were analysed using training loss, prediction error, and linear regression, and they show that the proposed method achieves higher prediction accuracy and lower error compared to the existing methods, especially in short- and ultra-short-term prediction. The paper demonstrates the potential of applying Transformer models to computer vision tasks for renewable energy forecasting. The results show that the proposed method achieves higher prediction accuracy and lower error than several deep learning architectures, such as ResNet and GoogleNet, especially in short- and ultra-short-term prediction.

Learning-based intrusion detection for high-dimensional imbalanced traffic

With the rapid development of Industry 4.0, industrial big data has become a hot topic in the field of smart manufacturing. However, the large-scale data flow generated by industrial IoT also has serious security challenges. This paper proposes a new multi-module intrusion detection system: DWGF-IDS, which consists of three modules: feature extraction, imbalance processing and traffic anomaly detection. Firstly, a deep denoising autoencoder is used to extract the deep feature representation of the data and improve the generalization performance of the detection model by adding noise to the autoencoder. Secondly, a Wasserstein Generative Adversarial Network - Gradient Penalty optimized based on the self-attention mechanism is used to generate a few classes in the anomalous traffic. Finally, the weights and bias values in the deep denoising autoencoder are transferred to the deep neural network structure, and a DNN improved based on focal loss is used to implement multi-classification detection on the reduced dimensional balanced traffic data. The system performance was evaluated using two datasets, namely NSL-KDD and CSE-CIC-IDS-2018. The multi-classification accuracy achieved on these datasets were 85.05% and 99.57%, respectively. The experimental results show that DWGF-IDS effectively copes with the high dimensionality and imbalance of IoT data, improves the detection rate of unknown attacks, and improves the misclassification of rare classes of attack traffic.

Towards practical single-shot phase retrieval with physics-driven deep neural network.

Phase retrieval (PR), a long-established challenge for recovering a complex-valued signal from its Fourier intensity-only measurements, has attracted considerable attention due to its widespread applications in optical imaging. Recently, deep learning-based approaches were developed and allowed single-shot PR. However, due to the substantial disparity between the input and output domains of the PR problems, the performance of these approaches using vanilla deep neural networks (DNN) still has much room to improve. To increase the reconstruction accuracy, physics-informed approaches were suggested to incorporate the Fourier intensity measurements into an iterative estimation procedure. Since the approach is iterative, they require a lengthy computation process, and the accuracy is still not satisfactory for images with complex structures. Besides, many of these approaches work on simulation data that ignore some common problems such as saturation and quantization errors in practical optical PR systems. In this paper, a novel physics-driven multi-scale DNN structure dubbed PPRNet is proposed. Similar to other deep learning-based PR methods, PPRNet requires only a single Fourier intensity measurement. It is physics-driven that the network is guided to follow the Fourier intensity measurement at different scales to enhance the reconstruction accuracy. PPRNet has a feedforward structure and can be end-to-end trained. Thus, it is much faster and more accurate than the traditional physics-driven PR approaches. Extensive simulations and experiments on an optical platform were conducted. The results demonstrate the superiority and practicality of the proposed PPRNet over the traditional learning-based PR methods.

Enhancing Traffic Intelligence in Smart Cities Using Sustainable Deep Radial Function

Smart cities have revolutionized urban living by incorporating sophisticated technologies to optimize various aspects of urban infrastructure, such as transportation systems. Effective traffic management is a crucial component of smart cities, as it has a direct impact on the quality of life of residents and tourists. Utilizing deep radial basis function (RBF) networks, this paper describes a novel strategy for enhancing traffic intelligence in smart cities. Traditional methods of traffic analysis frequently rely on simplistic models that are incapable of capturing the intricate patterns and dynamics of urban traffic systems. Deep learning techniques, such as deep RBF networks, have the potential to extract valuable insights from traffic data and enable more precise predictions and decisions. In this paper, we propose an RBF-based method for enhancing smart city traffic intelligence. Deep RBF networks combine the adaptability and generalization capabilities of deep learning with the discriminative capability of radial basis functions. The proposed method can effectively learn intricate relationships and nonlinear patterns in traffic data by leveraging the hierarchical structure of deep neural networks. The deep RBF model can learn to predict traffic conditions, identify congestion patterns, and make informed recommendations for optimizing traffic management strategies by incorporating these rich and diverse data. To evaluate the efficacy of our proposed method, extensive experiments and comparisons with real-world traffic datasets from a smart city environment were conducted. In terms of prediction accuracy and efficiency, the results demonstrate that the deep RBF-based approach outperforms conventional traffic analysis methods. Smart city traffic intelligence is enhanced by the model capacity to capture nonlinear relationships and manage large-scale data sets.

Detection of backdoor attacks using targeted universal adversarial perturbations for deep neural networks

Backdoor attacks on deep neural networks (DNNs) using targeted universal adversarial perturbations (TUAPs) do not require training datasets and model tampering, and triggers based on TUAPs can make DNNs output any class the adversary wants. Retraining DNNs using adversarial training for security is time-consuming and does not apply to DNNs in runtime. We want to detect backdoors using a black-box testing approach. We observe that after superimposing random noise on the input of a backdoor attack, the output still tends to remain the same, so we propose Sequential Analysis method based on the Metamorphosis Testing (SAMT). We designed two metamorphic relations for test case generation. Using sequential sampling, we calculate the label stability rate (LSR) and infer whether the image to be verified contains a trigger based on the sequential probability ratio change. The experimental results show that our method has a higher backdoor detection success rate (dsr) than the state-of-the-art detection algorithms. Moreover, our method does not need to use the model structure of DNNs, which has more adaptability and generalization ability. Based on our proposed method, we can simply add a backdoor detection layer to detect backdoors as early as possible, which can eventually alleviate the harm of such backdoors.

DNN modeling of partial differential equations with incomplete data

We present a computational technique for modeling the evolution of partial differential equations (PDEs) with incomplete data. It is a significant extension of the recent work of data driven learning of PDEs, in the sense that we consider two forms of partial data: data are observed only on a subset of the domain, and data are observed only on a subset of the state variables. Both cases resemble more realistic data collection scenarios in real-world applications. Leveraging the recent work on modeling partially-observed dynamical systems, we present a deep neural network (DNN) structure that is suitable for PDE modeling with such kinds of incomplete data. In addition to the mathematical motivation for the DNN structure, we present an extensive set of numerical examples in both one- and two-dimensions to demonstrate the effectiveness of the proposed DNN modeling. In one example, the method can accurately predict the solution when data are only available in less than half (40%) of the domain.