Trained Deep Neural Network Model Research Articles

Neural network architecture search model for thermal radiation in dense particulate systems

In the CFD-DEM simulation of the many dense particulate systems, particle-scale thermal radiation is an important heat transfer mode under high temperatures. In this work, neural network architecture search model with different layer connection matrix is applied to the view factor regression of the thermal radiation in dense particulate systems. Deep neural networks with 3346 feasible architectures are evaluated by the HyperBand algorithm to find the local optimal solution. Neural architecture search model trained by the big data of the view factor gives a good prediction of the macroscopic radiative properties and it is in general agreement with the empirical correlations and experimental data. The particle–wall radiation decreases strongly with the distance and the maximum interaction depth is about 2.0 times the sphere diameter. The trained deep neural network model provides an efficient data-driven closure to discuss the thermal radiation of the particle–particle and particle–wall interactions in particle bed.

Robustness analysis and experimental validation of a deep neural network for acoustic source imaging

Deep Neural Network (DNN) models offer an attractive alternative to existing acoustic source imaging techniques, such as acoustic beamforming, due to their ever-growing potential with increasing computational power. Source resolution of acoustic beamforming methods is limited at lower frequencies and their source maps may possess sidelobes at higher frequencies. However, acoustic beamforming methods are typically robust over a wide range of simulation and experimental conditions, such as (i) the number of sources present, (ii) source frequency and (ii) extraneous noise sources. The performance of DNN models, when these conditions are varied from their specific design criteria, is yet to be investigated and much work is needed in this area before DNN models can be utilized in experiments, such as wind tunnel tests. Furthermore, few studies have been conducted on experimental validation of DNN models, predominately due to the difficulty of large sets of experimentally obtained data needed for DNN model training and the sensitivity of DNN model performance when any of the aforementioned experimental conditions are varied. In this paper, a series of studies on the robustness of DNN models based on numerical data and experimental data are presented. Numerical DNN (NDNN) models are trained using in-phase and random-phase pressure data generated from six sources over design frequencies from 500 Hz to 20,000 Hz. The robustness of the NDNN models is tested via (1) inclusion of extraneous Gaussian white noise, (2) inclusion of extraneous tonal noise near the design frequency, (3) using source frequencies that slightly differ from the design frequencies and (4) using a number of sources that differs from the design source number. DNN model performance metrics are introduced that present a promising framework for future DNN model studies and bridging the gap between NDNN and experimentally trained DNN models. A preliminary experimental validation was conducted using a single speaker that was systematically placed over a speaker grid to generate training data via acoustic superposition, from which an experimentally trained DNN (EDNN) model was produced. The EDNN model yields exceptional noise source localization capability of the DNN model, revealing a promising start for a more sophisticated EDNN model.

Spectrally tunable light source based on deep neural network model

A spectrally tunable light source based on deep neural network (DNN) model is proposed in this work, which can reproduce arbitrary spectra accurately and rapidly. After calculating the scale factors using the trained DNN model, the target spectrum can be reproduced by regulating the combined monochromatic LEDs based on the pulse width modulation (PWM) signal with corresponding duty cycle. The standard solar spectrum and measured natural spectra at different times are reproduced using the proposed spectrally tunable light source. It is demonstrated that there is a good agreement between the target spectra and the corresponding reproduced spectra, with a fitted correlation index of above 0.9. Furthermore, the proposed spectrally tunable light source is able to transform spectrum at a frequency higher than 25 Hz and occupies only 750 KB memory space. As a result, the proposed spectrally tunable light source system can offer a high accuracy, fast speed and low cost approach to the development of specially light sources.

Artificial Intelligence in Intellectual Property Protection: Application of Deep Learning Model

To create and train a deep learning model costs a lot in comparison to ascertain a trained model. So, a trained model is considered as the intellectual property (IP) of the person who creates such model. However, there is every chance of illegal copying, redistributing and abusing of any of these high-performance models by the malicious users. To protect against such menaces, a few numbers of deep neural networks (DNN) IP security techniques have been developed recently. The present study aims at examining the existing DNN IP security activities. In the first instance, there is a proposal of taxonomy in favor of DNN IP protection techniques from the perspective of six aspects such as scenario, method, size, category, function, and target models. Afterwards, this paper focuses on the challenges faced by these methods and their capability of resisting the malicious attacks at different levels by providing proactive protection. An analysis is also made regarding the potential threats to DNN IP security techniques from various perspectives like modification of models, evasion and active attacks.
 Apart from that this paper look into the methodical assessment. The study explores the future research possibilities on DNN IP security by considering different challenges it would confront in the process of its operations.
 Result Statement: A high-performance deep neural Networks (DNN) model is costlier than the trained DNN model. It is considered as an intellectual property (IP) of the person who is responsible for creating DNN model. The infringement of the Intellectual Property of DNN model is a grave concern in recent years. This article summarizes current DNN IP security works by focusing on the limitations/ challenges they confront. It also considers the model in question's capacity for protection and resistance against various stages of attacks.

An Efficient Checkpoint Strategy for Federated Learning on Heterogeneous Fault-Prone Nodes

Federated learning (FL) is a distributed machine learning method in which client nodes train deep neural network models locally using their own training data and then send that trained model to a server, which then aggregates all of the trained models into a globally trained model. This protects personal information while enabling machine learning with vast amounts of data through parallel learning. Nodes that train local models are typically mobile or edge devices from which data can be easily obtained. These devices typically run on batteries and use wireless communication, which limits their power, making their computing performance and reliability significantly lower than that of high-performance computing servers. Therefore, training takes a long time, and if something goes wrong, the client may have to start training again from the beginning. If this happens frequently, the training of the global model may slow down and the final performance may deteriorate. In a general computing system, a checkpointing method can be used to solve this problem, but applying an existing checkpointing method to FL may result in excessive overheads. This paper proposes a new FL method for situations with many fault-prone nodes that efficiently utilizes checkpoints.

The Implementation of a Deep Neural Network (DNN) Approach in a Case Study Predicting the Distribution of Carbon Dioxide (CO2) Gas Saturation

Predicting the distribution of CO2 gas saturation is one example of how multiphase flow might be evaluated in Carbon Capture and Storage (CCS). The TOUGH2 simulator is one of the numerical simulations commonly used for multiphase flow simulation. Ordinary numerical simulations have several issues, including high grid spatial resolution and high processing costs. One of the most effective deep learning approaches to predicting the distribution of CO2 gas saturation is the deep neural network (DNN). A deep neural network is a network with three interconnected layers, there are input, hidden, and output layers. DNN learns about the previously constructed architecture from the input data. DNN requires a large quantity of data as input. Thus, in this study, we use 700 data points for each of the train_a and train_b variables. The distribution of CO2 gas saturation will be predicted automatically by the trained DNN model. This technique can handle complex data patterns, such as gas saturation in multiphase flow problems. The reconstruction loss findings show that the loss value decreases as the number of epochs increases. Furthermore, we used 3 and 4 epochs to determine the difference in results between the two. As a result, the model with 4 epochs and 10−3 regularization weight obtained the lowest error value of 0.4305. In summary, this model is capable of predicting CO2 gas saturation distribution, but more research is needed to produce more optimal results. This research hopes to help monitor multiphase flow in CCS systems in the future by forecasting the distribution of CO2 gas saturation.

Federated Learning: Centralized and P2P for a Siamese Deep Learning Model for Diabetes Foot Ulcer Classification

It is a known fact that AI models need massive amounts of data for training. In the medical field, the data are not necessarily available at a single site but are distributed over several sites. In the field of medical data sharing, particularly among healthcare institutions, the need to maintain the confidentiality of sensitive information often restricts the comprehensive utilization of real-world data in machine learning. To address this challenge, our study experiments with an innovative approach using federated learning to enable collaborative model training without compromising data confidentiality and privacy. We present an adaptation of the federated averaging algorithm, a predominant centralized learning algorithm, to a peer-to-peer federated learning environment. This adaptation led to the development of two extended algorithms: Federated Averaging Peer-to-Peer and Federated Stochastic Gradient Descent Peer-to-Peer. These algorithms were applied to train deep neural network models for the detection and monitoring of diabetic foot ulcers, a critical health condition among diabetic patients. This study compares the performance of Federated Averaging Peer-to-Peer and Federated Stochastic Gradient Descent Peer-to-Peer with their centralized counterparts in terms of model convergence and communication costs. Additionally, we explore enhancements to these algorithms using targeted heuristics based on client identities and f1-scores for each class. The results indicate that models utilizing peer-to-peer federated averaging achieve a level of convergence that is comparable to that of models trained via conventional centralized federated learning approaches. This represents a notable progression in the field of ensuring the confidentiality and privacy of medical data for training machine learning models.

Deep neural network-based phase calibration in integrated optical phased arrays

Calibrating the phase in integrated optical phased arrays (OPAs) is a crucial procedure for addressing phase errors and achieving the desired beamforming results. In this paper, we introduce a novel phase calibration methodology based on a deep neural network (DNN) architecture to enhance beamforming in integrated OPAs. Our methodology focuses on precise phase control, individually tailored to each of the 64 OPA channels, incorporating electro-optic phase shifters. To effectively handle the inherent complexity arising from the numerous voltage set combinations required for phase control across the 64 channels, we employ a tandem network architecture, further optimizing it through selective data sorting and hyperparameter tuning. To validate the effectiveness of the trained DNN model, we compared its performance with 20 reference beams obtained through the hill climbing algorithm. Despite an average intensity reduction of 0.84 dB in the peak values of the beams compared to the reference beams, our experimental results demonstrate substantial agreements between the DNN-predicted beams and the reference beams, accompanied by a slight decrease of 0.06 dB in the side-mode-suppression-ratio. These results underscore the practical effectiveness of the DNN model in OPA beamforming, highlighting its potential in scenarios that necessitate the intelligent and time-efficient calibration of multiple beams.

Using a Deep Neural Network with Small Datasets to Predict the Initial Production of Tight Oil Horizontal Wells

Due to its abundant reserves, tight oil has emerged as a significant substitute for conventional petroleum resources. It has become one of the focal points of exploration and research, and a new hot spot in global unconventional oil and gas exploration and development. This has led to a significant increase in the demand for forecasting the production capacity of tight oil horizontal wells. The deep neural network (DNN), as a mature model, has demonstrated significant advantages in many fields. However, due to the confidentiality and uniqueness of oilfield data, acquiring large datasets has become a challenge. Traditional methods using small datasets for training DNN models result in low accuracy and overfitting issues, which hinders the development of neural networks in the petroleum industry. This study aims to predict the initial production capacity of tight oil horizontal wells by using a small dataset of 650 data points through a DNN model. The research results indicate that pre-trained and fine-tuned DNNs outperform shallow neural networks, supporting vector machines, and DNN trained with traditional methods in terms of better generalization performance. Their accuracy reached 91.3%, demonstrating that it is reasonable to use a small dataset with pre-trained and fine-tuned DNN models.

Attacks on Recent DNN IP Protection Techniques and Their Mitigation

With the rapid increase in the development of Deep Learning methodologies, Deep Neural Networks (DNNs) are now being commonly deployed in smart systems (e.g. autonomous vehicles) and high-end security applications (e.g. face recognition, biometric authentication, etc.). The training of such DNN models often requires exclusive valuable training datasets, enormous computational resources, and expert fine-tuning skills. Hence, a trained DNN model can be regarded as valuable proprietary Intellectual Property (IP). Piracy of such DNN IPs has emerged as a major concern, with increasing trends of illegal copying and redistribution. A number of mitigation approaches targeting DNN IP protection have been proposed in recent years. In this work, we target two recently proposed DNN IP protection schemes: (a) Chaotic Map theory based encryption of the weight parameters, and (b) traditional block cipher based encryption of the weights. We demonstrate attacks on two recent DNN IP protection techniques, with one technique each belonging to the above-mentioned schemes, under a pragmatic attack model. We also propose a novel DNN IP protection technique based on selective encryption of the weight parameters, termed LEWIP to mitigate the exposed weaknesses, while having low implementation and performance overheads. Finally, we demonstrate the effectiveness of the LEWIP technique against state-of-the-art DNN implementations.

Optimizing process parameters for hot forging of Ti-6242 alloy: A machine learning and FEM simulation approach

In this study, we investigated the hot deformation behavior of Ti–6Al–2Sn–4Zr–2Mo (Ti-6242) alloy and propose a method to derive optimal hot process parameters for grain refinement and avoidance of flow instability. Microstructural Risk Index (MRI) was introduced as a microstructural evaluation index consisting of grain size, standard deviation of grain size, and flow instability. The initial temperature of the material and the stroke speed of the die were selected as design variables. Finite element analysis (FEA) was used to calculate grain size and flow instability and determine MRI of the forgings. The grain size model coefficient and flow instability were calculated based on the flow stress curve and verified with optical microscope (OM) and electron backscatter diffraction (EBSD) analysis results. The Deep Neural Network (DNN) model was used to determine the optimal process parameters for the forging process. The MRI prediction accuracy of the trained DNN models showed excellent performance at 97.01 %. The MRI of the optimized process variables was improved by 7.95 % compared to the minimum MRI of the training data set. Optimized process parameters can improve the quality of forgings through grain refinement and avoidance of flow instability regions.

Generating synthetic gait patterns based on benchmark datasets for controlling prosthetic legs

BackgroundProsthetic legs help individuals with an amputation regain locomotion. Recently, deep neural network (DNN)-based control methods, which take advantage of the end-to-end learning capability of the network, have been proposed. One prominent challenge for these learning-based approaches is obtaining data for the training, particularly for the training of a mid-level controller. In this study, we propose a method for generating synthetic gait patterns (vertical load and lower limb joint angles) using a generative adversarial network (GAN). This approach enables a mid-level controller to execute ambulation modes that are not included in the training datasets.MethodsThe conditional GAN is trained on benchmark datasets that contain the gait data of individuals without amputation; synthetic gait patterns are generated from the user input. Further, a DNN-based controller for the generation of impedance parameters is trained using the synthetic gait pattern and the corresponding synthetic stiffness and damping coefficients.ResultsThe trained GAN generated synthetic gait patterns with a coefficient of determination of 0.97 and a structural similarity index of 0.94 relative to benchmark data that were not included in the training datasets. We trained a DNN-based controller using the GAN-generated synthetic gait patterns for level-ground walking, standing-to-sitting motion, and sitting-to-standing motion. Four individuals without amputation participated in bypass testing and demonstrated the ambulation modes. The model successfully generated control parameters for the knee and ankle based on thigh angle and vertical load.ConclusionsThis study demonstrates that synthetic gait patterns can be used to train DNN models for impedance control. We believe a conditional GAN trained on benchmark datasets can provide reliable gait data for ambulation modes that are not included in its training datasets. Thus, designing gait data using a conditional GAN could facilitate the efficient and effective training of controllers for prosthetic legs.

Overfit deep neural network for predicting drug-target interactions

Drug-target interactions (DTIs) prediction is an important step in drug discovery. As traditional biological experiments or high-throughput screening are high cost and time-consuming, many deep learning models have been developed. Overfitting must be avoided when training deep learning models. We propose a simple framework, called OverfitDTI, for DTI prediction. In OverfitDTI, a deep neural network (DNN) model is overfit to sufficiently learn the features of the chemical space of drugs and the biological space of targets. The weights of trained DNN model form an implicit representation of the nonlinear relationship between drugs and targets. Performance of OverfitDTI on three public datasets showed that the overfit DNN models fit the nonlinear relationship with high accuracy. We identified fifteen compounds that interacted with TEK, a receptor tyrosine kinase contributing to vascular homeostasis, and the predicted AT9283 and dorsomorphin were experimentally demonstrated as inhibitors of TEK in human umbilical vein endothelial cells (HUVECs).

Prediction of particle-laden pipe flows using deep neural network models

An accurate and fast prediction of particle-laden flow fields is of particular relevance for a wide variety of industrial applications. The motivation for this research is to evaluate the applicability of deep learning methods for providing statistical properties of the carrier and dispersed phases in a particle-laden vertical pipe flow. Deep neural network (DNN) models are trained for different dependent variables using 756 high-fidelity datasets acquired from point-particle large-eddy simulations for different values of Stokes number, St, bulk particle volume fraction, Φ¯v, and wall roughness, Δγ, for the range St=10−500, Φ¯v=5×10−5−10−3, and Δγ=1°−6°. The considered parameter space corresponds to the inertia-dominated regime and covers a large extent of the typical conditions in powder-based laser metal deposition. We find that the DNN models capture the nonlinear dynamics of the system and recreate the statistical properties of the particle-laden pipe flow. However, DNN predictions of the particle statistics are of higher accuracy compared to the fluid statistics, which is attributed to the highly non-monotonic dependence of the fluid statistics on the control parameters. Owing to significantly decreased time-to-solution, the trained DNN models are promising as surrogate models to expedite model development and design process of various industrial applications.

Characterizing the Performance of Accelerated Jetson Edge Devices for Training Deep Learning Models

Deep Neural Network (DNN) models are becoming ubiquitous in a variety of contemporary domains such as Autonomous Vehicles, Smart cities and Healthcare. They help drones to navigate, identify suspicious activities from safety cameras, and perform diagnostics over medical imaging. Fast DNN inferencing close to the data source is enabled by a growing class of accelerated edge devices such as NVIDIA Jetson and Google Coral which host low-power Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs) along with ARM CPUs in a compact form-factor to offer a superior performance-to-energy ratio. E.g., the NVIDIA Jetson AGX Xavier kit has a 512-core Volta GPU, an 8-core ARM CPU and 32GB LPDDR4x memory, that operates within 65W of power, costs US999 and is smaller than a paperback novel. Recently, there has been a push towards training DNN models on the edge [2]. This is driven by the massive growth in data collected from edge devices in Cyber-Physical Systems (CPS) and Internet of Things (IoT), the need to refresh the models periodically, the bandwidth constraints in moving all this data to Cloud data centers for training, and a heightened emphasis on privacy by retaining data on the edge. This has led to techniques like federated and geo-distributed learning that train DNN models locally on data on an edge device and aggregate them centrally. In this abstract, we summarise and highlight key results from our full paper [5].

A Weight-Generating Approach of a Deep Neural Network for the Parameter Identification of Dynamic Systems

The general learning process of deep learning is extremely time-consuming. Unlike the traditional learning process, a weight-generating approach to quickly generate the weight vectors of a deep neural network model is proposed, which can be used for parameter identification of a dynamic system. Based on the analysis of three trained deep neural network models, which are used to identify the parameters of three different dynamic systems, the statistical relationships between the weight vectors of each hidden layer and its inputs are revealed. Then, the statistical patterns of the weight vectors are imitated by exploiting the statistical patterns of the inputs and these relationships. Then, a weight-generating approach is designed to quickly generate the weight vectors of a deep neural network model. The effectiveness of the weight-generating approach is tested on the tasks of parameter identification for the three dynamic systems. The numerical results are provided to demonstrate the validity and high efficiency of the proposed weight-generating approach.

A reconfigurable graphene patch antenna inverse design at terahertz frequencies

This article investigates the inverse design of a reconfigurable multi-band patch antenna based on graphene for terahertz applications to operate frequency range (2–5THz). In the first step, this article evaluates the dependence of the antenna radiation characteristics on its geometric parameters and the graphene properties. The simulation results show that it is possible to achieve up to 8.8 dB gain, 13 frequency bands, and 360^circ beam steering. Then and due to the complexity of the design of graphene antenna, a deep neural network (DNN) is used to predict the antenna parameters by given inputs like desired realized gain, main lobe direction, half power beam width, and return loss in each resonance frequency. The trained DNN model predicts almost with 93% accuracy and 3% mean square error in the shortest time. Then, this network was used to design five-band and three-band antennas, and it has been shown that the desired antenna parameters are achieved with negligible errors. Therefore, the proposed antenna finds many potential applications in the THz frequency band.

Denoising digital breast tomosynthesis projections using deep learning with synthetic data as training set.

Image denoising based on deep neural networks (DNN) needs a big dataset containing digital breast tomosynthesis (DBT) projections acquired in different radiation doses to be trained, which is impracticable. Therefore, we propose extensively investigating the use of synthetic data generated by software for training DNNs to denoise DBT real data. The approach consists of generating a synthetic dataset representative of the DBT sample space by software, containing noisy and original images. Synthetic data were generated in two different ways: (a)virtual DBT projections generated by OpenVCT and (b)noisy images synthesized from photography regarding noise models used in DBT (e.g., Poisson-Gaussian noise). Then, DNN-based denoising techniques were trained using a synthetic dataset and tested for denoising physical DBT data. Results were evaluated in quantitative (PSNR and SSIM measures) and qualitative (visual analysis) terms. Furthermore, a dimensionality reduction technique (t-SNE) was used for visualization of sample spaces of synthetic and real datasets. The experiments showed that training DNN models with synthetic data could denoise DBT real data, achieving competitive results to traditional methods in quantitative terms but showing a better balance between noise filtering and detail preservation in a visual analysis. T-SNE enables us to visualize if synthetic and real noises are in the same sample space. We propose a solution for the lack of suitable training data to train DNN models for denoising DBT projections, showing that we just need the synthesized noise to be in the same sample space as the target image.

Evaluation of deploying data-driven predictive controls in buildings on a large scale for greenhouse gas emission reduction

Buildings consume more than 70% of electricity in the U.S. In order to reduce building energy consumption, advanced building controls have been developed. However, most building controls are using physics-based models and lack of scalability. Recent development of data-driven control models could overcome this challenge and be automatically developed and implemented on large scale. The purpose of this study was to evaluate the effectiveness, robustness, and scalability of automatic and systematic data-driven predictive control (DDPC) for a large-scale real-world deployment. We first used collected data from 78 buildings in RTEM database to train deep neural network models. Then we applied the models to optimize the HVAC control for energy savings. We focused on over 1000 HVAC units in five different commonly used types, including air handling units, rooftop units, variable air volume systems, fan coil units, and unit ventilators. Next, we evaluated the energy-saving potential and the reduction of greenhouse gas emissions of the proposed method. We found that DDPC was robust and scalable in buildings, with an average energy saving of 65% and peak load reduction of 15% compared to current control systems. The average reduction of GHG emissions for CO2, CH4, and N2O was 15.18 kg, 5.76e-4 kg, and 5.48e-5 kg per m2 per year, respectively. New York State can benefit 11% reduction in carbon emission from DDPC in buildings. For scalability, we also identified and categorized the challenging conditions when DDPC may not work properly and summarized the lessons learned from large-scale DDPC deployment.

ADU-Net: An Attention Dense U-Net based deep supervised DNN for automated lesion segmentation of COVID-19 from chest CT images

An automatic method for qualitative and quantitative evaluation of chest Computed Tomography (CT) images is essential for diagnosing COVID-19 patients. We aim to develop an automated COVID-19 prediction framework using deep learning.We put forth a novel Deep Neural Network (DNN) composed of an attention-based dense U-Net with deep supervision for COVID-19 lung lesion segmentation from chest CT images. We incorporate dense U-Net where convolution kernel size 5×5 is used instead of 3×3. The dense and transition blocks are introduced to implement a densely connected network on each encoder level. Also, the attention mechanism is applied between the encoder, skip connection, and decoder. These are used to keep both the high and low-level features efficiently. The deep supervision mechanism creates secondary segmentation maps from the features. Deep supervision combines secondary supervision maps from various resolution levels and produces a better final segmentation map. The trained artificial DNN model takes the test data at its input and generates a prediction output for COVID-19 lesion segmentation. The proposed model has been applied to the MedSeg COVID-19 chest CT segmentation dataset. Data pre-processing methods help the training process and improve performance.We compare the performance of the proposed DNN model with state-of-the-art models by computing the well-known metrics: dice coefficient, Jaccard coefficient, accuracy, specificity, sensitivity, and precision. As a result, the proposed model outperforms the state-of-the-art models.This new model may be considered an efficient automated screening system for COVID-19 diagnosis and can potentially improve patient health care and management system.