Number Of Training Data Points Research Articles

Performance of Fourier-based activation function in physics-informed neural networks for patient-specific cardiovascular flows

Background and ObjectivesPhysics-informed neural networks (PINNs) can be used to inversely model complex physical systems by encoding the governing partial differential equations and training data into the neural network. However, neural networks are known to be biased towards learning less complex functions, called spectral bias. This has important implications in modeling cardiovascular flows, where spatial frequencies can vary substantially across anatomies and pathologies (e.g., aneurysms or stenoses). Recent evidence suggests that Fourier-based activation functions have desirable properties, and can potentially reduce spectral bias; however, the performance and adequacy of such Fourier activation functions have not yet been evaluated in patient-specific cardiovascular flow applications. MethodsThe performance of sine activation function was evaluated against tanh and swish activation functions in a 1D advection-diffusion problem, an eccentric 2D stenosis model (Re=5000), and a patient-specific 3D aortic model (Re=823) under pulsatile flow conditions. CFD simulations were performed at high spatio-temporal resolution and data points were extracted for training the neural network. The number of training data points were normalized by L/D. The performance of the PINNs framework was evaluated with increasing number of training data points and across all three activation functions. ResultsOur results demonstrate that sine activation function presents desirable characteristics, such as monotonic reduction in errors, relatively faster convergence, and accurate eigen spectra at higher modes, compared to tanh and swish activation functions. Interestingly, for all activation functions, the domain-averaged errors tended to asymptote at ≈15−20% despite substantial increase in training point density. For 2D eccentric stenosis, errors asymptoted at a sensor point density of 40L/D. For 3D patient-specific aorta, this asymptote was achieved at 180L/D for all three activation functions with an error of ≈15% although sine activation function demonstrated relatively faster convergence. ConclusionsWe have demonstrated that Fourier-based activation functions have higher performance in terms of accuracy and convergence properties for cardiovascular flow applications; however, inherent challenges of neural networks (e.g., spectral bias) can limit the accuracy to ≈15% under physiological, 3D patient-specific blood flow conditions.

Damage Location Determination with Data Augmentation of Guided Ultrasonic Wave Features and Explainable Neural Network Approach for Integrated Sensor Systems

Machine learning techniques such as deep learning have already been successfully applied in Structural Health Monitoring (SHM) for damage localization using Ultrasonic Guided Waves (UGW) at various temperatures. However, a common issue arises due to the time-consuming nature of collecting guided wave measurements at different temperatures, resulting in an insufficient amount of training data. Since SHM systems are predominantly employed in sensitive structures, there is a significant interest in utilizing methods and algorithms that are transparent and comprehensible. In this study, a method is presented to augment feature data by generating a large number of training features from a relatively limited set of measurements. In addition, robustness to environmental changes, e.g., temperature fluctuations, is improved. This is achieved by utilizing a known temperature compensation method called temperature scaling to determine the function of signal features as a function of temperature. These functions can then be used for data generation. To gain a better understanding of how the damage localization predictions are made, a known explainable neural network (XANN) architecture is employed and trained with the generated data. The trained XANN model was then used to examine and validate the artificially generated signal features and to improve the augmentation process. The presented method demonstrates a significant increase in the number of training data points. Furthermore, the use of the XANN architecture as a predictor model enables a deeper interpretation of the prediction methods employed by the network.

Physics-informed neural networks for approximating dynamic (hyperbolic) PDEs of second order in time: Error analysis and algorithms

We consider the approximation of a class of dynamic partial differential equations (PDEs) of second order in time by the physics-informed neural network (PINN) approach, and provide an error analysis of PINN for the wave equation, the nonlinear Klein-Gordon equation and the linear elastodynamic equation. Our analyses show that, with feed-forward neural networks having two hidden layers and the tanh activation function, the PINN approximation errors for the solution field, its time derivative and its gradient field can be effectively bounded by the training loss and the number of training data points (quadrature points). Our analyses further suggest new forms for the training loss function, which contain certain residuals that are crucial to the error estimate but would be absent from the canonical PINN loss formulation. Adopting these new forms for the loss function leads to a variant PINN algorithm. We present ample numerical experiments with the new PINN algorithm for the wave equation, the Sine-Gordon equation and the linear elastodynamic equation, which show that the method can capture the solution well.

Stochastic Mirror Descent on Overparameterized Nonlinear Models.

Most modern learning problems are highly overparameterized, i.e., have many more model parameters than the number of training data points. As a result, the training loss may have infinitely many global minima (parameter vectors that perfectly "interpolate" the training data). It is therefore imperative to understand which interpolating solutions we converge to, how they depend on the initialization and learning algorithm, and whether they yield different test errors. In this article, we study these questions for the family of stochastic mirror descent (SMD) algorithms, of which stochastic gradient descent (SGD) is a special case. Recently, it has been shown that for overparameterized linear models, SMD converges to the closest global minimum to the initialization point, where closeness is in terms of the Bregman divergence corresponding to the potential function of the mirror descent. With appropriate initialization, this yields convergence to the minimum-potential interpolating solution, a phenomenon referred to as implicit regularization. On the theory side, we show that for sufficiently-overparameterized nonlinear models, SMD with a (small enough) fixed step size converges to a global minimum that is "very close" (in Bregman divergence) to the minimum-potential interpolating solution, thus attaining approximate implicit regularization. On the empirical side, our experiments on the MNIST and CIFAR-10 datasets consistently confirm that the above phenomenon occurs in practical scenarios. They further indicate a clear difference in the generalization performances of different SMD algorithms: experiments on the CIFAR-10 dataset with different regularizers, l1 to encourage sparsity, l2 (SGD) to encourage small Euclidean norm, and l∞ to discourage large components, surprisingly show that the l∞ norm consistently yields better generalization performance than SGD, which in turn generalizes better than the l1 norm.

Finite-Data Error Bounds for Koopman-Based Prediction and Control

The Koopman operator has become an essential tool for data-driven approximation of dynamical (control) systems, e.g., via extended dynamic mode decomposition. Despite its popularity, convergence results and, in particular, error bounds are still scarce. In this paper, we derive probabilistic bounds for the approximation error and the prediction error depending on the number of training data points, for both ordinary and stochastic differential equations while using either ergodic trajectories or i.i.d. samples. We illustrate these bounds by means of an example with the Ornstein–Uhlenbeck process. Moreover, we extend our analysis to (stochastic) nonlinear control-affine systems. We prove error estimates for a previously proposed approach that exploits the linearity of the Koopman generator to obtain a bilinear surrogate control system and, thus, circumvents the curse of dimensionality since the system is not autonomized by augmenting the state by the control inputs. To the best of our knowledge, this is the first finite-data error analysis in the stochastic and/or control setting. Finally, we demonstrate the effectiveness of the bilinear approach by comparing it with state-of-the-art techniques showing its superiority whenever state and control are coupled.

Optimized 1D-CNN model for medicinal Psyllium Husk crop mapping with temporal optical satellite data

Deep Learning models are preferred for complex image analysis-based solutions to application-oriented problems. However, the architecture of such models largely influences the results which includes several hyperparameters that need to be tuned. This study aims at developing an optimized 1D-CNN model for medicinal Psyllium Husk crop mapping using open source temporal optical Sentinel-2A/2B satellite data. In this study, a sequential 1D-CNN model architecture was developed by optimizing hyperparameters which includes convolution layers, number of neurons, activation function, and batch size. Psyllium Husk crop fields were mapped in the Jalore district of Rajasthan using Sentinel 2A/ 2B (10 m) optical data. For spectral dimensionality reduction of the data, Modified Soil Adjusted Vegetation Index (MSAVI2) was used to maintain the data dimensionality since temporal data was utilized. The dataset was subsequently refined to include the target crop's specific phenological stages that distinguish it from other closely resembling species. The information corresponding to these specific crop stages was fed to the 1D-CNN model to carry out the classification. A range of training sample sizes were explored to determine the optimal number of training data points. As the output from the model, fractional images are obtained consisting of values proportional to the probability of a pixel lying in the target class. Accuracy assessment was carried out using fuzzy error matrix (FERM) by generating fractional output images from temporal optical PlanetScope data (3m) which was used as a reference. The best overall accuracy among the test cases came out to be 89.85% using conventional MSAVI2 with 1000 training samples.

Using Euler’s Method to Prove the Convergence of Neural Networks

In Du, it is shown that, for a fully connected neural network (NN), the gradient descent algorithm converges to zero. Motivated by that work, we provide here general conditions under which we can derive the convergence of the gradient descent algorithm from the convergence of the gradient flow, in the case of NNs, in a systematic way. Our approach is based on an analysis of the error in Euler’s method in the case of NNs, and relies on the concept of local strong convexity. Unlike Du, our approach allows to provide convergence guarantees without making any assumptions on the number of hidden nodes of the NN or the number of training data points. A numerical example is proposed, showing the merits of our approach.

Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations

We present a neural network-based method for solving linear and nonlinear partial differential equations, by combining the ideas of extreme learning machines (ELM), domain decomposition and local neural networks. The field solution on each sub-domain is represented by a local feed-forward neural network, and Ck continuity conditions are imposed on the sub-domain boundaries. Each local neural network consists of a small number of hidden layers, while its last hidden layer can be wide. The weight/bias coefficients in all the hidden layers of the local neural networks are pre-set to random values and fixed throughout the computation, and only the weight coefficients in the output layers of the local neural networks are training parameters. The overall neural network is trained by a linear or nonlinear least squares computation, not by the back-propagation type algorithms. We introduce a block time-marching scheme together with the presented method for long-time simulations of time-dependent linear/nonlinear partial differential equations. The current method exhibits a clear sense of convergence with respect to the degrees of freedom in the neural network. Its numerical errors typically decrease exponentially or nearly exponentially as the number of training parameters, or the number of training data points, or the number of sub-domains increases. Extensive numerical experiments have been performed to demonstrate the computational performance of the presented method. We also demonstrate its capability for long-time dynamic simulations with some test problems. We compare the presented method with the deep Galerkin method (DGM) and the physics-informed neural network (PINN) method in terms of the accuracy and computational cost. The current method exhibits a clear superiority, with its numerical errors and network training time considerably smaller (typically by orders of magnitude) than those of DGM and PINN. We also compare the current method with the classical finite element method (FEM). The computational performance of the current method is on par with, and often exceeds, the FEM performance in terms of the accuracy and computational cost.

Imitation Learning System Design with Small Training Data for Flexible Tool Manipulation

Population aging has become a major problem in developed countries. As the labor force declines, robot arms are expected to replace human labor for simple tasks. A robotic arm attaches a tool specialized for a task and acquires the movement through teaching by an engineer with expert knowledge. However, the number of such engineers is limited; therefore, a teaching method that can be used by non-technical personnel is necessitated. As a teaching method, deep learning can be used to imitate human behavior and tool usage. However, deep learning requires a large amount of training data for learning. In this study, the target task of the robot is to sweep multiple pieces of dirt using a broom. The proposed learning system can estimate the initial parameters for deep learning based on experience, as well as the shape and physical properties of the tools. It can reduce the number of training data points when learning a new tool. A virtual reality system is used to move the robot arm easily and safely, as well as to create training data for imitation. In this study, cleaning experiments are conducted to evaluate the effectiveness of the proposed method. The experimental results confirm that the proposed method can accelerate the learning speed of deep learning and acquire cleaning ability using a small amount of training data.

Passive Indoor Visible Light Positioning System Using Deep Learning

A passive indoor visible light positioning system is proposed that does not require active participation from the user and is suitable for IoT sensor networks. This approach does not require a line-of-sight path and measures the impulse response (IR) between sources and receivers installed in the room. The presence of an object of interest (OI), i.e., a person to be localized, disrupts the IRs among the source–receiver pairs, which can be related to its position. A deep learning framework is developed that learns the relationship between changes in sets of IRs and the OI position through a set of training data obtained by placing the OI at random locations in the room. This approach shows that the OI can be localized using a very limited set of training data under a wide range of illumination levels. In order to represent a realistic scenario, a room with furniture is modeled in the optical system design software. The ray trace information of the modeled room is used to construct IR measurements among different source–receiver pairs that include multiorder reflections. The results show that localization performance is crucially related to the signal-to-noise ratio and the number of training data points used in the learning process. A root-mean-square error (RMSE) near 30 cm is possible in the case of high SNR and a large training set. However, even with a very limited training set and over a range of dimming levels, RMSEs of near 80 cm were obtained without the need for explicit user involvement.

Gaussian Process Modelling of the F-16 Ground Vibration Test Benchmark: Data Selection Case Study

Abstract We observe the effects of training data sample selection in modelling of a physical system with Gaussian process nonlinear autoregressive models with exogenous input. Gaussian process modelling limits the number of training data points and we use a big nonlinear benchmark data set. The combination calls for training data sample selection. We compare a ‘smart’ method based on Euclidean distance between training data points with decimation. We use the training data samples obtained by both methods to train the models, test model predictions on a test data set, and calculate two figures of merit, eRMSt and mean standardised log loss (MSLL). The model trained on the ‘smartly’ selected training data points is better in eRMSt while the one with the decimated data is superior in MSLL. The direct conclusion is that the purpose of the model determines which training data sample selection method is better, as the relevant figure of merit depends on the model purpose. We notice that the predicted variance is more sensitive to the training data sample than the predicted mean. We warn that training data sample selection may have unexpected consequences.

Universal deep learning representation of effective refractive index for photonics channel waveguides

An optical waveguide is the fundamental element in a photonic integrated circuit. This paper establishes a universal deep learning representation for the effective refractive index of an optical channel waveguide for the entire and usual parameter space for applications in photonics. The deep learning model is able to make precise predictions for wide spectrum optical wavelengths, dielectric materials of refractive indices varying from 1.45 to 3.8, and a wide range of feasible geometrical parameters of the waveguides. The deep learning model consists of fully connected feedforward neural networks, and rigorous optimization of neural network architecture is carried out. Deep learning models with two and three hidden layers provide rapid convergence with a minimal number of training data points and offer unprecedented precisions that are a few orders better in magnitude than the conventional interpolation techniques.

Machine learning classification for tool life modeling using production shop-floor tool wear data

Abstract Tool wear is an important limitation to machining productivity. Tool wear in machining is difficult to predict due to large number of influencing variables and tool-to-tool performance variation. As a result, empirical models (such as Taylor’s tool life equation) or physics-based models require experimentation to calibrate model coefficients, which is infeasible in an industrial setting due to large number of tool-material combinations. In this paper, machine learning classification methods for modeling tool life using production shop-floor tool wear data are presented. Tool wear data is simulated using a known ‘true’ tool life curve as a function of cutting speed. Two machine learning classification methods are evaluated, Support Vector Machines, and logistic, and the results are compared to the true tool life curve. Results show good agreement with the true tool life curve using the radial basis function, and polynomial kernels of Support Vector Machine and logistic classification with log transformation of input variables. The classification accuracy increases with the number of training data points. A method to generate synthetic data to augment sparse and unbalanced datasets is presented. Results show that domain knowledge and experience can be effectively used to generate synthetic data to improve tool life classification. The proposed machine learning classification approach offers a powerful and practical approach for modeling tool life in machining in an industrial setting.

MS-SVM: Minimally Spanned Support Vector Machine

For a Support Vector Machine (SVM) algorithm, the time required for classifying an unknown data point is proportional to the number of support vectors. For some real time applications, use of SVM could be a problem if the number of support vectors is high. Depending on the complexity of the class structure, sometimes the number of support vectors of a SVM model increases with the number of training data points. Here our objective is to reduce the number of support vectors, yet, maintaining more of less the same level of accuracy as that of a normal SVM that does not use any reduction of support vectors. An SVM finds a separating hyperplane maximizing the margin of separation and hence, the location of the hyperplane is primarily dependent on a set of “boundary points”. Here, we first identify some boundary points using a minimum spanning tree on the training data to obtain a reduced training set. The SVM algorithm is then applied on the reduced training data to generate the classification model. We call this algorithm, Minimally Spanned Support Vector Machine (MS-SVM). We also assess the performance by relaxing the definition of boundary points. Moreover we extend the algorithm to a feature space using a kernel transformation. In this case, an MST is generated in the feature space using the associated kernel matrix. Our experimental results demonstrate that the proposed algorithm can considerably reduce the number of support vectors without affecting the overall classification accuracy. This is true irrespective of whether the MST is generated in the input space or in the feature space. Thus the MS-SVM algorithm can be used instead of SVM for efficient classification.

On the Monotonicity of Interval Type-2 Fuzzy Logic Systems

Qualitative knowledge is very useful for system modeling and control problems, especially when specific physical structure knowledge is unavailable and the number of training data points is small. This paper studies the incorporation of one common qualitative knowledge-monotonicity into interval type-2 (IT2) fuzzy logic systems (FLSs). Sufficient conditions on the antecedent and consequent parts of fuzzy rules are derived to guarantee the monotonicity between inputs and outputs. We take into account five type-reduction and defuzzification methods (the Karnik-Mendel method, the Du-Ying method, the Begian-Melek-Mendel method, the Wu-Tan method, and the Nie-Tan method). We show that IT2 FLSs are monotonic if the antecedent and consequents parts of their fuzzy rules are arranged according to the proposed monotonicity conditions. The derived monotonicity conditions are valid for the IT2 FLSs using any kind of IT2 fuzzy sets (FSs) (e.g., Trapezoidal IT2 FSs and Gaussian IT2 FSs) and stand for type-1 FLSs as well. Guidelines for applying the proposed conditions to modeling and control problems are also given. Our results will be useful in the design of monotonic IT2 FLSs for engineering applications when the monotonicity property is desired.

Recurrent sparse support vector regression machines trained by active learning in the time-domain

A method for the sparse solution of recurrent support vector regression machines is presented. The proposed method achieves a high accuracy versus complexity and allows the user to adjust the complexity of the resulting model. The sparse representation is guaranteed by limiting the number of training data points for the support vector regression method. Each training data point is selected based on the accuracy of the fully recurrent model using the active learning principle applied to the successive time-domain data. The user can adjust the training time by selecting how often the hyper-parameters of the algorithm should be optimised. The advantages of the proposed method are illustrated on several examples, and the experiments clearly show that it is possible to reduce the number of support vectors and to significantly improve the accuracy versus complexity of recurrent support vector regression machines.

Sparse multikernel support vector regression machines trained by active learning

A method for the sparse multikernel support vector regression machines is presented. The proposed method achieves a high accuracy versus complexity ratio and allows the user to adjust the complexity of the resulting models. The sparse representation is guaranteed by limiting the number of training data points for the support vector regression method. Each training data point is selected based on its influence on the accuracy of the model using the active learning principle. A different kernel function is attributed to each training data point, yielding multikernel regressor. The advantages of the proposed method are illustrated on several examples and the experiments show the advantages of the proposed method.

Prediction of diesel fuel cold properties using artificial neural networks

In this paper, two neural networks, multilayer perceptron and networks with radial-basis function, were used to predict important cold properties of commercial diesel fuels, namely cloud point and cold filter plugging point. The developed models predict the named properties using cetane number, density, viscosity, contents of total aromatics, and distillation temperatures at 10, 50, and 90 vol. % recovery as input data. The training algorithms, number of hidden layer neurons, and number of training data points were optimized in order to obtain a model with optimal predictive ability. The results indicated better prediction of cloud and cold filter plugging points in the case of multilayer perceptron networks. The obtained absolute error mean for the optimal neural network models (0.58°C for the cloud point and 1.46°C for the cold filter plugging point) are within the range of repeatability of standard cold properties determination methods.

Spatiotemporal pattern validation of chlorophyll-a concentrations in Lake Okeechobee, Florida, using a comparative MODIS image mining approach

A comparative analysis was conducted using three types of data-mining models produced from Moderate Resolution Imaging Spectroradiometer (MODIS) Terra Surface Reflectance 1-day or 8-day composite images to estimate chlorophyll-a (chl-a) concentrations in Lake Okeechobee, Florida. To understand the pros and cons of these three models, a genetic programming (GP) model was compared to an artificial neural network (ANN) model and multiple linear regression (MLR) model with respect to two different data sets related to model formulation. The first data set included the MODIS Terra bands from 1 to 7; the second data set extended the first data set by adding environmental parameters such as Secchi disc depth (SDD), total suspended solids (TSS), wind speed, water level, rainfall and air temperature collected around the lake in 2003 and 2004. The GP algorithm, which has an advantage in machine learning allowing us to select the appropriate input parameters that significantly impact the prediction accuracy, outperformed the other two models based on four statistical indices. Specifically, the GP modelling outputs revealed interesting determinations of chl-a concentrations for MODIS bands 3, 5, 6 and 7, corresponding to wavelengths 459–479, 1230–1250, 1628–1652 and 2105–2155 nm, respectively. The number of training data points is limited; therefore, the inclusion of additional environmental variables cannot improve the prediction accuracy of the GP-derived chl-a concentrations.

An adaptive location estimator using tracking algorithms for indoor WLANs

This paper presents adaptive algorithms for estimating the location of a mobile terminal (MT) based on radio propagation modeling (RPM), Kalman filtering (KF), and radio-frequency identification (RFID) assisting for indoor wireless local area networks (WLANs). The location of the MT of the extended KF positioning algorithm is extracted from the constant-speed trajectory and the radio propagation model. The observation information of the KF tracker is extracted from the empirical and RPM positioning methods. Specifically, a sensor-assisted method employs an RFID system to adapt the sequential selection cluster algorithm. As compared with the empirical method, not only can the RPM algorithm reduce the number of training data points and perform on-line calibration in the signal space, but the RPM and KF algorithms can alleviate the problem of aliasing. In addition, the KF tracker with the RFID-assisted scheme can calibrate the location estimation and improve the corner effect. Experimental results demonstrate that the proposed location-tracking algorithm using KF with the RFID-assisted scheme can achieve a high degree of location accuracy (i.e., more than 90% of the estimated positions have error distances of less than 2.1 m).