Neural Network Training Research Articles

Unsupervised representation learning of Kohn–Sham states and consequences for downstream predictions of many-body effects

Representation learning for the electronic structure problem is a major challenge of machine learning in computational condensed matter and materials physics. Within quantum mechanical first principles approaches, density functional theory (DFT) is the preeminent tool for understanding electronic structure, and the high-dimensional DFT wavefunctions serve as building blocks for downstream calculations of correlated many-body excitations and related physical observables. Here, we use variational autoencoders (VAE) for the unsupervised learning of DFT wavefunctions and show that these wavefunctions lie in a low-dimensional manifold within latent space. Our model autonomously determines the optimal representation of the electronic structure, avoiding limitations due to manual feature engineering. To demonstrate the utility of the latent space representation of the DFT wavefunction, we use it for the supervised training of neural networks (NN) for downstream prediction of quasiparticle bandstructures within the GW formalism. The GW prediction achieves a low error of 0.11 eV for a combined test set of two-dimensional metals and semiconductors, suggesting that the latent space representation captures key physical information from the original data. Finally, we explore the generative ability and interpretability of the VAE representation.

An Automated Penetration Testing Framework Based on Hierarchical Reinforcement Learning

Given the large action space and state space involved in penetration testing, reinforcement learning is widely applied to enhance testing efficiency. This paper proposes an automatic penetration testing scheme based on hierarchical reinforcement learning to reduce both action space and state space. The scheme includes a network intelligence responsible for specifying the penetration host and a host intelligence designated to perform penetration testing on the selected host. Specifically, within the network intelligence, an action-masking mechanism is adopted to shield unenabled actions, thereby reducing the explorable action space and improving the penetration testing efficiency. Additionally, the host intelligence employs an invalid discrimination mechanism, terminating testing after actions that do not alter system states, thereby preventing sudden increases in the number of neural network training steps for an action. An optimistic estimation mechanism is also introduced to select penetration strategies suited to various hosts, preventing training crashes due to value confusion between different hosts. Ablation experiments demonstrate that the host intelligence can learn different penetration strategies for varying penetration depths without significant fluctuations in training steps, and the network intelligence can coordinate with the host intelligence to perform network penetration steadily. This hierarchical reinforcement learning framework can detect network vulnerabilities more quickly and accurately, significantly reducing the cost of security policy updates.

Locally linearized physics-informed neural networks for Riemann problems of hyperbolic conservation laws

In this work, we demonstrate that physics-informed neural networks (PINNs) tend to propagate predicted shock wave information bidirectionally in time, which does not align with the actual evolution direction of solutions to hyperbolic conservation laws. This mismatch results in instability and hinders the reduction of the loss of governing equations, as well as the initial condition loss by the deep neural network. In order to tackle this problem, we simplify the complexity of the problem by constructing equivalent linear transport equations in the region of shock wave generation. The speeds of these linearized waves are governed by the Rankine–Hugoniot relations of conservation laws. This approach is termed the Locally Linearized PINNs method. Specifically, an appropriate shock wave detector is initially designed to identify domains where shock waves occur. Near shock waves, the original nonlinear equations are transformed into their linearized forms, thereby modifying the residual terms of the partial differential equations. Additionally, an equilibrium factor is introduced in fluid compression regions to reduce prediction errors and stabilize the training of deep neural networks. Numerical examples illustrate that Locally Linearized PINNs effectively address the challenge of predicting global solutions in PINNs and significantly improve shock-capturing performance for hyperbolic conservation laws.

A fast deployable model for crack identification with laser thermography testing

This paper presents a novel feature extraction method, enabling efficient training and deployment of neural networks for rapid identification of crack defects in Laser Array Spot Thermography (LAST). We trained the crack defect identification model based on pixel-level features, encoding each pixel as a feature vector using Frangi filter, and classifying them using a neural network. Experimental results demonstrate that Frangi features are an effective method for distinguishing cracks, speckles, and background noise interference in the experiment. Furthermore, the model only requires a small region of interest (ROI) as training samples to achieve effective training and efficient crack identification under the same detection conditions, allowing for rapid deployment in practical inspections.

Research on trajectory planning based on deep reinforcement learning

Abstract For trajectory planning in a three-dimensional continuous space with multiple complex constraints, traditional optimal control methods face significant challenges, such as long computation times and poor dynamic performance. Using deep reinforcement learning algorithms for training agents in flight trajectory planning can effectively improve the dynamic performance of the planning algorithms. This paper focuses on training agents for UAV emergency landing and obstacle avoidance trajectory planning based on deep reinforcement learning algorithms. Within the reinforcement learning framework, we design the training environment, state space, action space, reward function, and neural network structure. To address common issues in reinforcement learning, such as poor generalization and slow training speed, we improve the training process using the concept of curriculum learning. Comparisons with pseudospectral methods demonstrate that deep reinforcement learning can effectively handle trajectory planning tasks in real flight environments, achieving trajectory quality close to that of pseudospectral methods while offering superior real-time performance.

Hessian-based mixed-precision quantization with transition aware training for neural networks

Hessian-based mixed-precision quantization with transition aware training for neural networks

Scalable Multi-Round Multi-Party Privacy-Preserving Neural Network Training

Scalable Multi-Round Multi-Party Privacy-Preserving Neural Network Training

Extracting fiber paths from the optimized lamination parameters of variable-stiffness laminated shells based on physic-informed neural network

This paper presents a novel approach for extracting fiber paths from the optimized lamination parameters (LPs) of variable stiffness laminated shells, utilizing the framework of physics-informed neural network (PINN). In this methodology, each fiber layer is associated with a specific stream function, which is approximated by an independent neural network. The stream function is governed by a partial differential equation (PDE) derived from the fiber orientation field in the parameter space. Moreover, the isocontours of the stream function are transformed into the actual fiber paths in the physical space. To account for manufacturing constraints, Riemannian geometry serves as a computational tool to determine the intrinsic distance between adjacent fiber paths and the geodesic curvature of the isocontours. By incorporating regularization terms into the loss function based on the physical relationships, the constrained optimization problem is converted into an unconstrained one, making it more suitable for neural network training. Meanwhile, a fiber path extraction (FPE) algorithm is used to minimize the loss function at randomly sampled points through gradient descent. The numerical results suggest that the extraction of fiber paths using PINN can achieve satisfactory levels of accuracy while effectively satisfying the imposed constraints.

Physics-based parameters selection and machine learning based prediction of pool boiling bubble departure diameter

For forecasting the bubble departure diameter in pool boiling heat transfer, this study creates a deep-learning neural network based on physics and a variety of working fluids, manufactured surfaces, and materials subjected to different testing situations. This work analyzes nearly 5,000 data points of bubble departure diameters ranging from 0.2-28.7 mm using neural network input parameters such as saturation temperature, pressure, contact angle, surface roughness, surface tension, liquid density, gas density, wall superheat and heat flux, and other thermophysical properties, predicting their impact on the bubble departure diameter, and also uses them for training neural networks. The best Neural network which is designated as Case-4 is selected on the basis of coefficient of determination(R2), mean absolute error (MAE), and mean-square error (MSE) was used to understand the degree of influence of each input parameter and it was found that θ and Q have the greatest impact on the model. A comparison was also done to correlations proposed by researchers and it was found that neural networks have much better efficiency and accuracy than empirical formulas and thus can be an essential tool to predict the bubble diameter in the future.

Inertial Measurement Unit Self-Calibration by Quantization-Aware and Memory-Parsimonious Neural Networks

This paper introduces a methodology to compensate inertial Micro-Electro-Mechanical System (IMU-MEMS) time-varying calibration loss, induced by stress and aging. The approach relies on a periodic assessment of the sensor through specific stimuli, producing outputs which are compared with the response of a high-precision sensor, used as ground truth. At any re-calibration iteration, differences with respect to the ground truth are approximated by quantization-aware trained tiny neural networks, allowing calibration-loss compensations. Due to the unavailability of aging IMU-MEMS datasets, a synthetic dataset has been produced, taking into account aging effects with both linear and nonlinear calibration loss. Also, field-collected data in conditions of thermal stress have been used. A model relying on Dense and 1D Convolution layers was devised and compensated for an average of 1.97 g and a variance of 1.07 g2, with only 903 represented with 16 bit parameters. The proposed model can be executed on an intelligent signal processing inertial sensor in 126.4 ms. This work represents a step forward toward in-sensor machine learning computing through integrating the computing capabilities into the sensor package that hosts the accelerometer and gyroscope sensing elements.

Mapping Leaf Mass Per Area and Equivalent Water Thickness from PRISMA and EnMAP

With the continued advancement of spaceborne hyperspectral sensors, hyperspectral remote sensing is evolving as an increasingly pivotal tool for high-precision global monitoring applications. Novel image spectroscopy data, e.g., the PRecursore IperSpettrale della Missione Applicativa (PRISMA) and Environmental Mapping and Analysis Program (EnMAP), can rapidly and non-invasively capture subtle spectral information of terrestrial vegetation, facilitating the precise retrieval of the required vegetation parameters. As critical vegetation traits, Leaf Mass per Area (LMA) and Equivalent Water Thickness (EWT) hold significant importance for comprehending ecosystem functionality and the physiological status of plants. To address the demand for high-precision vegetation parameter datasets, a hybrid modeling approach was proposed in this study, integrating the radiative transfer model PROSAIL and neural network models to retrieve LMA and EWT from PRISMA and EnMAP images. To achieve this objective, canopy reflectance was simulated via PROSAIL, and the optimal band combinations for LMA and EWT were selected as inputs to train neural networks. The evaluation of the hybrid inversion models over field measurements showed that the RMSE values for the LMA and EWT were 4.11 mg·cm−2 and 9.08 mg·cm−2, respectively. The hybrid models were applied to PRISMA and EnMAP images, resulting in LMA and EWT maps displaying adequate spatial consistency, along with cross-validation results showing high accuracy (RMSELMA = 5.78 mg·cm−2, RMSEEWT = 6.84 mg·cm−2). The results demonstrated the hybrid inversion model’s universality and applicability, enabling the retrieval of vegetation parameters from image spectroscopy data and offering a valuable contribution to hyperspectral remote sensing for vegetation monitoring, though the availability of field measurement data remained a significant challenge.

Fredholm Integral Equations for Function Approximation and the Training of Neural Networks

Fredholm Integral Equations for Function Approximation and the Training of Neural Networks

Algorithms and Data Structures for Numerical Computations with Automatic Precision Estimation

We introduce data structures and algorithms to count numerical inaccuracies arising from usage of floating numbers described in IEEE 754. Here we describe how to estimate precision for some collection of functions most commonly used for array manipulations and training of neural networks. For highly optimized functions like matrix multiplication, we provide a fast estimation of precision and some hint how the estimation can be strengthened.

Artificial intelligence-based automated determination in breast and colon cancer and distinction between atypical and typical mitosis using a cloud-based platform.

Artificial intelligence (AI) technology in pathology has been utilized in many areas and requires supervised machine learning. Notably, the annotations that define the ground truth for the identification of different confusing process pathologies, vary from study to study. In this study, we present our findings in the detection of invasive breast cancer for the IHC/ISH assessment system, along with the automated analysis of each tissue layer, cancer type, etc. in colorectal specimens. Additionally, models for the detection of atypical and typical mitosis in several organs were developed using existing whole-slide image (WSI) sets from other AI projects. All H&E slides were scanned by different scanners with a resolution of 0.12-0.50μm/pixel, and then uploaded to a cloud-based AI platform. Convolutional neural networks (CNN) training sets consisted of invasive carcinoma, atypical and typical mitosis, and colonic tissue elements (mucosa-epithelium, lamina propria, muscularis mucosa, submucosa, muscularis propria, subserosa, vessels, and lymph nodes). In total, 59 WSIs from 59 breast cases, 217 WSIs from 54 colon cases, and 28 WSIs from 23 different types of tumor cases with relatively higher amounts of mitosis were annotated for the training. The harmonic average of precision and sensitivity was scored as F1 by AI. The final AI models of the Breast Project showed an F1 score of 94.49% for Invasive carcinoma. The mitosis project showed F1 scores of 80.18%, 97.40%, and 97.68% for mitosis, atypical, and typical mitosis layers, respectively. Overall F1 scores for the current results of the colon project were 90.02% for invasive carcinoma, 94.81% for the submucosa layer, and 98.02% for vessels and lymph nodes. After the training and optimization of the AI models and validation of each model, external validators evaluated the results of the AI models via blind-reader tasks. The AI models developed in this study were able to identify tumor foci, distinguish in situ areas, define colonic layers, detect vessels and lymph nodes, and catch the difference between atypical and typical mitosis. All results were exported for integration into our in-house applications for breast cancer and AI model development for both whole-block and whole-slide image-based 3D imaging assessment.

Les Houches lectures on deep learning at large and infinite width*

Abstract These lectures, presented at the 2022 Les Houches Summer School on Statistical Physics and Machine Learning, focus on the infinite-width limit and large-width regime of deep neural networks. Topics covered include the various statistical and dynamical properties of these networks. In particular, the lecturers discuss properties of random deep neural networks, connections between trained deep neural networks, linear models, kernels and Gaussian processes that arise in the infinite-width limit, and perturbative and non-perturbative treatments of large but finite-width networks, at initialization and after training.

Accelerating Deep Learning in Radar Systems: A Simulation Framework for 60 GHz Indoor Radar

FMCW radar systems are increasingly used in diverse applications, and emerging technologies like JCAS offer new opportunities. However, machine learning for radar faces challenges due to limited application-specific datasets, often requiring advanced simulations to supplement real-world data. This paper presents a setup for generating synthetic radar data for indoor environments, evaluated using CNNs. The setup involves comprehensive modeling, including far-field antenna simulations, variations in human radar cross-section, and detailed representations of indoor environments with their corresponding propagation channel properties. These synthetic data are used to train CNNs, and their performance is assessed on real measurement data. The results demonstrate that CNNs trained on synthetic data can perform well when tested on real measurement data. Specifically, the models trained with synthetic data showed performance comparable to models trained with real measurement data, which required a minimum of 300 samples to reach similar levels of accuracy. This result demonstrates that synthetic data can effectively train neural networks, providing an alternative to real measurement data, particularly when collecting sufficient real-world samples is difficult or costly. This approach significantly reduces the time required for generating datasets, and the ability to quickly label data in simulations simplifies and accelerates post-processing. Additionally, the generated datasets can be made more heterogeneous by introducing varying signal conditions, enhancing the diversity and robustness of the training data.

RECOMMENDATIONS FOR THE DEPLOYMENT OF EQUIPMENT FOR TRAINING NEURAL NETWORKS

The paper presents recommendations for choosing a hardware resource when training neural networks. Analytical expressions are considered for calculating the computational complexity when deploying direct propagation neural networks of various dimensions. Some problems arising in the selection and configuration of hardware on which neural networks are trained are formulated. An assessment of the use of libraries/frameworks of various neural networks is given. A set of recommendations for the mutual arrangement of hardware modules is developed.

A student trained convolutional neural network competing with a commercial AI software and experts in organ at risk segmentation

This retrospective, multi-centered study aimed to improve high-quality radiation treatment (RT) planning workflows by training and testing a Convolutional Neural Network (CNN) to perform auto segmentations of organs at risk (OAR) for prostate cancer (PCa) patients, specifically the bladder and rectum. The objective of this project was to develop a clinically applicable and robust artificial intelligence (AI) system to assist radiation oncologists in OAR segmentation. The CNN was trained using manual contours in CT-datasets from diagnostic 68Ga-PSMA-PET/CTs by a student, then validated (n = 30, PET/CTs) and tested (n = 16, planning CTs). Further segmentations were generated by a commercial artificial intelligence (cAI) software. The ground truth were manual contours from expert radiation oncologists. The performance was evaluated using the Dice-Sørensen Coefficient (DSC), visual analysis and a Turing test. The CNN yielded excellent results in both cohorts and OARs with a DSCmedian > 0.87, the cAI resulted in a DSC > 0.78. In the visual assessment, 67% (bladder) and 75% (rectum) of the segmentations were rated as acceptable for treatment planning. With a misclassification rate of 45.5% (bladder) and 51.1% (rectum), the CNN passed the Turing test. The metrics, visual assessment and the Turing test confirmed the clinical applicability and therefore the support in clinical routine.

Scientific publications and large language models: will neural network understand Russian science?

The impact of artificial intelligence technologies, neural networks, and chatbots on science and education has induced widespread discussion in the academic community. It is no longer possible to contain the use of neural networks such as ChatGPT for writing texts, including scientific texts. The present study is done in a qualitative approach. The aim of the study is to analyze the application of large language models on the example of ChatGPT in the scientific publishing activities of Russian scientists. The practice of using chatbots does not always satisfy the user and the scientific community as a whole. On the one hand, the user is confronted with the lack of the requested information. On the other hand, the scientific community, and especially editors and readers of scientific journals, question the feasibility of neural networks due to the shortcomings of large language models that have been widely disputed in scientific publications. This study shows that there is another reason to distrust neural networks. Incompleteness and opacity of the information produced by artificial intelligence is related to the texts on which neural networks are trained. For Russian science, this problem poses a serious threat, since popular artificial intelligence companies use mostly English-language texts to teach their neural nets. The author puts forward the opinion that the social and humanitarian knowledge produced in modern Russia remains outside the scope of texts used for training neural networks. This point of view is supported by research by Russian scientists on Arctic governance. The data is absent in the English-language texts of ChatGPT but reflected in Russian-language publications.

An Improved Neural Network Algorithm for Energy Consumption Forecasting

Accurate and efficient forecasting of energy consumption is a crucial prerequisite for effective energy planning and policymaking. The BP neural network has been widely used in forecasting, machine learning, and various other fields due to its nonlinear fitting ability. In order to improve the prediction accuracy of the BP neural network, this paper introduces the concept of forecast lead time and establishes a mathematical model accordingly. Prior to training the neural network, the input layer data are preprocessed based on the forecast lead time model. The training and forecasting results of the BP neural network when and when not considering forecast lead time are compared and verified. The findings demonstrate that the forecast lead time model can significantly improve the prediction speed and accuracy, proving to be highly applicable for short-term energy consumption forecasting.