Training Error Research Articles

Efficient predictive modeling of resilient modulus in stabilized clayey soil using automated machine learning

Pavement subgrade design relies on the resilient modulus (Mr) to analyze structural response to vehicle-like loading. Adding stabilizers to the soil subgrade makes estimating Mr difficult and resource-intensive. This study uses an automated machine learning (ML) strategy to predict the Mr of stabilized clayey soil using recycled plastic waste. The proposed method automates model selection and hyperparameter tuning, making it a feasible alternative to tedious ML modeling and costly laboratory testing. From extensive laboratory investigation involving 3285 experimental data points, the automated ML model using Bayesian optimization evaluates ensembles, support vector machine (SVM), neural network (NET), decision trees, (TREE), and Gaussian process (GP) regression models, identifying the best model based on cross-validation mean squared error (MSE). Bayesian optimization explores hyperparameter spaces to find optimal configurations, enhancing the accuracy, scalability, and reliability of the prediction model. The optimization process yielded the best results for the ensemble least square boost (LSBoost) model with a cross-validation mean squared error (MSE) value of 6.723×10−29. The optimized ML model’s performance is measured using R2 and adjusted R2. The LSboost model’s R2 and adjusted R2 values of 0.9999 suggested overfitting, prompting further investigations using performance metrics like root mean squared error (RMSE), mean absolute error (MSE), and probability density function (PDF) for normalized absolute error (NAE) for training and testing datasets for the predictive ML model. The small RMSE and MAE (0.0049 and 0.0005) values and symmetrical NAE distribution of the proposed ML model demonstrate its high accuracy and generalization capabilities. The proposed model was subsequently tested on the new, unseen data and achieved predictions with an error rate of 0.24 %. This confirms the proposed ML model’s superiority over conventional deformation models, making it ideal for reliable geotechnical engineering applications.

Prediction of global ionospheric TEC using attention based bidirectional long short-term memory and gated recurrent unit

An accurate prediction of ionospheric total electron content (TEC) at the primary stage is essential for applications related to global navigation satellite systems (GNSS) under varying weather conditions. The previous TEC prediction schemes contribute for each time step that increases the prediction time. The eye contact phenomenon establishes a metaphorical connection which intends to capture and emphasize the attention worthy elements in a sequence. This research introduces a deep learning approach which is a combination of attention-based bidirectional long short-term memory and gated recurrent unit (Bi-LSTM GRU) to predict TEC in the ionosphere. Bidirectional LSTM is the better option for achieving durability when combined with a gated recurrent unit (GRU) to predict TEC in the ionosphere. The proposed approach is evaluated with the existing LSTM approach for root mean square error (RMSE) during training and validation. The RMSE while predicting the global ionospheric delay using the existing LSTM for 20 epochs is seen to be 0.004, whereas the existing approach achieves a training error of 0.003.

Remote Sensing of Chlorophyll-a and Water Quality over Inland Lakes: How to Alleviate Geo-Location Error and Temporal Discrepancy in Model Training

Remote Sensing of Chlorophyll-a and Water Quality over Inland Lakes: How to Alleviate Geo-Location Error and Temporal Discrepancy in Model Training

A Recurrent Stochastic Configuration Network for Dynamic System Modeling and Its Application

In order to model nonlinear dynamic systems quickly and accurately, a recurrent stochastic configuration network (RSCN) with universal approximation ability is proposed in this paper. The method consists of network parameter learning and network structure determination. The network parameter learning strategy consists of input weights and the biases of hidden nodes generated randomly through a supervision mechanism, while the output weights of hidden nodes are determined by solving the least-squares problem. The network structure is determined incrementally by the training error of a single-layer recurrent neural network. The effectiveness of RSCN is tested and evaluated by using three nonlinear dynamic functions and historical data from a municipal solid waste incineration (MSWI) process. The experimental results show that the proposed method can improve modeling accuracy and reduce training time, which is valuable for applications in the field of industrial process modeling.

A new fault location method for high‐voltage transmission lines based on ICEEMDAN‐MSA‐ConvGRU model

AbstractGiven the complex form of distribution line faults, the accuracy of fault location using traditional artificial intelligence networks needs to be further improved. Here, a combined fault location method is proposed for a 110 kV distribution line based on the improved complete ensemble empirical mode decomposition with adaptive noise (ICEEMDAN), mantis search algorithm (MSA), and convolutional gate recurrent unit (ConvGRU). Firstly, the study used the ICEEMDAN algorithm to decompose the signals and discard the high‐frequency signals with low correlation so as to achieve the purpose of noise cancellation. Then, the study used the root mean square error (RMSE) of the ConvGRU model training as the adaptation value, optimized the internal parameters of the model using the MSA algorithm, and obtained a combined fault locating model. By using the proposed model, the effects of the fault form and transition impedance changes on the location accuracy were analysed, and the location accuracy was compared with other artificial intelligence methods. The location accuracy index showed that the proposed model had a better convergence speed of training error than the traditional model. Also, the RMSE of the localization results was reduced by 50%, with a higher fault location accuracy.

Pinball-Huber boosted extreme learning machine regression: a multiobjective approach to accurate power load forecasting

Power load data frequently display outliers and an uneven distribution of noise. To tackle this issue, we present a forecasting model based on an improved extreme learning machine (ELM). Specifically, we introduce the novel Pinball-Huber robust loss function as the objective function in training. The loss function enhances the precision by assigning distinct penalties to errors based on their directions. We employ a genetic algorithm, combined with a swift nondominated sorting technique, for multiobjective optimization in the ELM-Pinball-Huber context. This method simultaneously reduces training errors while streamlining model structure. We practically apply the integrated model to forecast power load data in Taixing City, which is situated in the southern part of Jiangsu Province. The empirical findings confirm the method’s effectiveness.

Machine learning without a processor: Emergent learning in a nonlinear analog network

Standard deep learning algorithms require differentiating large nonlinear networks, a process that is slow and power-hungry. Electronic contrastive local learning networks (CLLNs) offer potentially fast, efficient, and fault-tolerant hardware for analog machine learning, but existing implementations are linear, severely limiting their capabilities. These systems differ significantly from artificial neural networks as well as the brain, so the feasibility and utility of incorporating nonlinear elements have not been explored. Here, we introduce a nonlinear CLLN-an analog electronic network made of self-adjusting nonlinear resistive elements based on transistors. We demonstrate that the system learns tasks unachievable in linear systems, including XOR (exclusive or) and nonlinear regression, without a computer. We find our decentralized system reduces modes of training error in order (mean, slope, curvature), similar to spectral bias in artificial neural networks. The circuitry is robust to damage, retrainable in seconds, and performs learned tasks in microseconds while dissipating only picojoules of energy across each transistor. This suggests enormous potential for fast, low-power computing in edge systems like sensors, robotic controllers, and medical devices, as well as manufacturability at scale for performing and studying emergent learning.

Multi-scale keypoints detection and motion features extraction in dairy cows using ResNet101-ASPP network

Detecting Keypoints in dairy cows aims to locate and track the motion trajectories of the body's joints, which plays a crucial role in behavior analysis and lameness detection. However, real farming scenarios, characterized by occlusions and large variations in object scale may result in poor detection results. Therefore we introduce the atrous spatial pyramid pooling (ASPP) module into the shallow network of ResNet101, designed to improve the multi-scale feature extraction capability of the model. The ASPP module enhances the robustness of recognition for different dimensional sizes and occluded keypoints using different dilatation rates in the parallel atrous convolutional layers to expand the model's receptive field. Furthermore, seven types of motion features, including tracking up, gait symmetry, step height balance, motion speed variability, head swing amplitude, head-neck slope and back curvature are extracted simultaneously by monitoring and tracking the motion trajectory of distinct keypoints. Several of these features represent innovative extraction models and attributes, first proposed in this study. Multiple models are trained and tested on datasets containing 2385 frames for ablation experiments. The experiments show that, in comparison with the ResNet50, MobileNet_v2_1.0, and EfficientNet-b0 backbone networks, the training error and test error of ResNet101 improve by 4.04–30.12 pixels and 3.81–28.14 pixels. Therefore, ResNet101 is used as the benchmark for subsequent model improvement by adding the ASPP module. The training error and test error of the ResNet101-ASPP network are improved by 0.27 and 0.24 pixels, respectively, compared to the benchmark network. The prediction confidence improves by 1.65-2.50% at three different dairy cow object scales, In addition, the keypoints under different occlusion conditions improve considerably, especially for small-scale keypoints, demonstrating the capability of the ASPP module for multi-scale feature extraction. By analyzing the distribution between the seven features and health, mild lameness, and severe lameness in dairy cows, it is shown that all the different features play an important role in distinguishing between different levels of lameness.

Evaluating the performance of the Anwaralardh photovoltaic power generation plant in Jordan: Comparative analysis using artificial neural networks and multiple linear regression modeling

The global energy demand is rising, driven by population growth, economic development, and industrialization. Shifting towards renewable energy, like solar energy, is gaining momentum worldwide because of ecological concerns and resource depletion. This paper aims to utilize Artificial Neural Networks (ANNs) and multiple linear regression (MLR) modeling techniques to evaluate the productivity of 11 MW photovoltaic (PV) solar power plant currently operational in Jordan. The case study reveals that both models can be used to predict the daily, monthly, and yearly average power produced and system efficiency with reasonable accuracy. The ANN model exhibited promising results, where the best value for the coefficient of determination (R2) and mean absolute percentage error (MAPE) for training were 95.85% and 0.59%, respectively. However, R2 was 93.7%, and MAPE was 1.27% for validation tests. All these results were achieved using a 7-6-1 model, with a sample ratio of 1:1 for the data allocated in training and validation. When using multiple linear regression, the R2 and standard error values were 93.42% and 0.17%. On the other hand, the results showed that the yearly output power for actual and predicted by both models over the year was 24,399 MWh, 24,538 MWh, and 24,401 MWh, respectively. This research showed valuable results in the monthly output power for solar cells at the Anwaralardh PV power system project, contributing to a better understanding of solar energy generation in arid desert climates and emphasizing the potential of solar power plants to play a crucial role in achieving SDG 7 objectives.

Predicting pressure fields from incomplete velocity fields based on deep convolutional neural network

Acquisition of an instantaneous pressure field in a flow field is essential for experimental fluid dynamics research because pressure is associated with various flow phenomena. Machine learning has performed well in various fluid mechanics studies in recent years due to its nonlinearity. In this paper, we use the machine learning method to predict the pressure fields from incomplete velocity fields while reconstructing the missing velocity information, as exemplified by the stationary circular cylinder flow and circular cylinder undergoing forced oscillating problems, in which the incompleteness of the velocity field is caused by the optical path occlusion in experiments. We propose a network model based on convolutional neural networks. The experimental results show that the error of machine learning methods is around 4%, and even if the resolution of the training set is reduced, it does not affect the prediction accuracy of the network. For statistical averaging of prediction results, the network has high accuracy in predicting first-order quantities, but its ability to predict second-order quantities is limited. We also explore the effects of the amount of noise on the performance of networks. When applying the model to the forced oscillating cylinder problem, we use transfer learning to train the network given the similarity of the two problems, and the training error yields faster convergence and more minor convergence results. In addition to this, we use flow field data with a single vibration frequency to construct a single dataset, and flow field data with multiple vibration frequencies to construct a mixed dataset, and study the effect of different datasets on the prediction accuracy, which shows that it is difficult for neural networks to achieve both accuracy and generalization.

Predicting Alzheimer's disease CSF core biomarkers: a multimodal Machine Learning approach.

Alzheimer's disease (AD) is a progressive neurodegenerative disorder. Current core cerebrospinal fluid (CSF) AD biomarkers, widely employed for diagnosis, require a lumbar puncture to be performed, making them impractical as screening tools. Considering the role of sleep disturbances in AD, recent research suggests quantitative sleep electroencephalography features as potential non-invasive biomarkers of AD pathology. However, quantitative analysis of comprehensive polysomnography (PSG) signals remains relatively understudied. PSG is a non-invasive test enabling qualitative and quantitative analysis of a wide range of parameters, offering additional insights alongside other biomarkers. Machine Learning (ML) gained interest for its ability to discern intricate patterns within complex datasets, offering promise in AD neuropathology detection. Therefore, this study aims to evaluate the effectiveness of a multimodal ML approach in predicting core AD CSF biomarkers. Mild-moderate AD patients were prospectively recruited for PSG, followed by testing of CSF and blood samples for biomarkers. PSG signals underwent preprocessing to extract non-linear, time domain and frequency domain statistics quantitative features. Multiple ML algorithms were trained using four subsets of input features: clinical variables (CLINVAR), conventional PSG parameters (SLEEPVAR), quantitative PSG signal features (PSGVAR) and a combination of all subsets (ALL). Cross-validation techniques were employed to evaluate model performance and ensure generalizability. Regression models were developed to determine the most effective variable combinations for explaining variance in the biomarkers. On 49 subjects, Gradient Boosting Regressors achieved the best results in estimating biomarkers levels, using different loss functions for each biomarker: least absolute deviation (LAD) for the Aβ42, least squares (LS) for p-tau and Huber for t-tau. The ALL subset demonstrated the lowest training errors for all three biomarkers, albeit with varying test performance. Specifically, the SLEEPVAR subset yielded the best test performance in predicting Aβ42, while the ALL subset most accurately predicted p-tau and t-tau due to the lowest test errors. Multimodal ML can help predict the outcome of CSF biomarkers in early AD by utilizing non-invasive and economically feasible variables. The integration of computational models into medical practice offers a promising tool for the screening of patients at risk of AD, potentially guiding clinical decisions.

Data Validation Utilizing Expert Knowledge and Shape Constraints

Data validation is a primary concern in any data-driven application, as undetected data errors may negatively affect machine learning models and lead to suboptimal decisions. Data quality issues are usually detected manually by experts, which becomes infeasible and uneconomical for large volumes of data. To enable automated data validation, we propose “shape constraint-based data validation,” a novel approach based on machine learning that incorporates expert knowledge in the form of shape constraints. Shape constraints can be used to describe expected (multivariate and nonlinear) patterns in valid data and enable the detection of invalid data that deviates from these expected patterns. Our approach can be divided into two steps: (1) shape-constrained prediction models are trained on data, and (2) their training error is analyzed to identify invalid data. The training error can be used as an indicator for invalid data because shape-constrained models can fit valid data better than invalid data. We evaluate the approach on a benchmark suite consisting of synthetic datasets, which we have published for benchmarking similar data validation approaches. Additionally, we demonstrate the capabilities of the proposed approach with a real-world dataset consisting of measurements from a friction test bench in an industrial setting. Our approach detects subtle data errors that are difficult to identify even for domain experts.

Inf-sup neural networks for high-dimensional elliptic PDE problems

Solving high dimensional partial differential equations (PDEs) has historically posed a considerable challenge when utilizing conventional numerical methods, such as those involving domain meshes. Recent advancements in the field have seen the emergence of neural PDE solvers, leveraging deep networks to effectively tackle high dimensional PDE problems. This study introduces Inf-SupNet, a model-based unsupervised learning approach designed to acquire solutions for a specific category of elliptic PDEs. The fundamental concept behind Inf-SupNet involves incorporating the inf-sup formulation of the underlying PDE into the loss function. The analysis reveals that the global solution error can be bounded by the sum of three distinct errors: the numerical integration error, the duality gap of the loss function (training error), and the neural network approximation error for functions within Sobolev spaces. To validate the efficacy of the proposed method, numerical experiments conducted in high dimensions demonstrate its stability and accuracy across various boundary conditions, as well as for both semi-linear and nonlinear PDEs.

Establishing the relationship between generalized crystallographic texture and macroscopic yield surfaces using partial input convex neural networks

In this study, we present a methodology to predict the macroscopic yield surface of metals and metallic alloys with general crystallographic textures. In previous work, we have established the use of partially input convex neural networks (pICNN) as macroscopic yield functions of crystal plasticity simulations. However, this work was performed with an over-abundance of data, and on limited crystallographic textures. Here, we extend this study to approach more realistic material states (i.e., complex crystallographic textures), and consider data-availability as a major driver for our approach. We present our modified framework capable of handling generalized material states and demonstrate its effectiveness on samples with multi-modal textures deformed under plane stress conditions. We further describe an adaptive algorithm for the generation of training data as informed by the shape of yield surfaces to reduce the time for both the generation of training data as well as pICNN training. Finally, we will discuss errors in both training and test datasets, limitations, and future extensibility.

An intelligent threshold selection method to improve orbital angular momentum-encoded quantum key distribution under turbulence

High-dimensional quantum key distribution (HD-QKD) encoded by orbital angular momentum (OAM) presents significant advantages in terms of information capacity. However, perturbations caused by free-space atmospheric turbulence decrease the performance of the system by introducing random fluctuations in the transmittance of OAM photons. Currently, the theoretical performance analysis of OAM-encoded QKD systems exists a gap when concerning the statistical distribution under the free-space link. In this article, we analyzed the security of QKD systems by combining probability distribution of transmission coefficient (PDTC) of OAM with decoy-state BB84 method. To address the problem that the invalid key rate is calculated in the part transmittance interval of the post-processing process, an intelligent threshold method based on neural network is proposed to improve OAM-encoded QKD, which aims to conserve computing resources and enhance system efficiency. Our findings reveal that the ratio of root mean square (RMS) OAM-beam radius to Fried constant plays a crucial role in ensuring secure key generation. Meanwhile, the training error of neural network is at the magnitude around 10−3, indicating the ability to predict optimization parameters quickly and accurately. Our work contributes to the advancement of parameter optimization and prediction for free-space OAM-encoded HD-QKD systems. Furthermore, it provides valuable theoretical insights to support the development of free-space experimental setups.

Stochastic configuration networks with particle swarm optimisation search

While building stochastic configuration networks (SCNs), there is no guarantee that the randomly generated weights will satisfy the supervisory mechanism and that the adopted weights will significantly reduce the training error. This paper extends SCN by applying the particle swarm optimisation (PSO) technique for one-step optimising of the set of random weights generated. These optimised weights provide a higher likelihood of satisfying the supervisory mechanism and improving the learning rate. Simulations are carried out over five regression and three classification datasets. Results demonstrate that an improved training rate and generalisation can be achieved.

Identification of Leaf Diseases of Different Crops using modified ResNet50

Every country’s income depends heavily on the agriculture sector. Agriculture output is influenced by a wide range of factors, including crop health, weather, water availability, and type of land. Low output is primarily caused by crop disease. The Identification and treatment of diseases early on are very crucial. Traditional approaches don't offer a reliable evaluation. The spread of the disease must be determined using automated techniques. One of the cutting-edge technologies that aids in automating disease detection is deep learning. Deep learning has been used in numerous studies to detect crop diseases. The goal of this research is to effectively diagnose leaf diseases in all major crops using a novel deep learning model using a modified ResNet50. The research employed different deep learning models like VGG16, Inception V3 and modified ResNet50 on publicly available dataset of plant village. A novel approach has been proposed by extracting features from different models. The ResNet50 uses the skip connection which is considered to training and test error. The modified ResNet50 model helps add diversity in different crops for providing accuracy across all crops on public dataset of plant village. The dataset was obtained from a publicly available database on Kaggle. The dataset contains 13 different crops and a total of 38 diseases classes. On modified plant villages the proposed model generated a very good accuracy of 99.49% with precision of 0.966 and f score of 0.995. The proposed model is reliable for the classification of various leaf diseases in all the major crops.

African Vulture Optimization-Based Decision Tree (AVO-DT): An Innovative Method for Malware Identification and Evaluation through the Application of Meta-Heuristic Optimization Algorithm

Abstract Malware remains a big threat to cyber security, calling for machine learning-based malware detection. Malware variations exhibit common behavioral patterns indicative of their source and intended use to enhance the existing framework’s usefulness. Here we present a novel model, i.e., African Vulture Optimization-based Decision Tree (AVO-DT) to increase the overall optimization. The datasets from Android apps and malware software train the AVO-DT model. After training, the datasets are pre-processed by removing training errors. The DT algorithm is used by the developed AVO model to carry out the detection procedure and predict malware activity. To detect malware activities and improve accuracy, such an AVO-DT model technique employs both static and dynamic methodologies. The other measurements on Android applications might be either malicious or benign. Here we also developed malware prevention and detection systems to address ambiguous search spaces in multidimensionality difficulties and resolve optimization challenges.

Vegetation change detection and recovery assessment based on post-fire satellite imagery using deep learning

Wildfires are uncontrolled fires fuelled by dry conditions, high winds, and flammable materials that profoundly impact vegetation, leading to significant consequences including noteworthy changes to ecosystems. In this study, we provide a novel methodology to understand and evaluate post-fire effects on vegetation. In regions affected by wildfires, earth-observation data from various satellite sources can be vital in monitoring vegetation and assessing its impact. These effects can be understood by detecting vegetation change over the years using a novel unsupervised method termed Deep Embedded Clustering (DEC), which enables us to classify regions based on whether there has been a change in vegetation after the fire. Our model achieves an impressive accuracy of 96.17%. Appropriate vegetation indices can be used to evaluate the evolution of vegetation patterns over the years; for this study, we utilized Enhanced Vegetation Index (EVI) based trend analysis showing the greening fraction, which ranges from 0.1 to 22.4 km2 while the browning fraction ranges from 0.1 to 18.1 km2 over the years. Vegetation recovery maps can be created to assess re-vegetation in regions affected by the fire, which is performed via a deep learning-based unsupervised method, Adaptive Generative Adversarial Neural Network Model (AdaptiGAN) on post-fire data collected from various regions affected by wildfire with a training error of 0.075 proving its capability. Based on the results obtained from the study, our approach tends to have notable merits when compared to pre-existing works.

Formulation of Common Spatial Patterns for Multi-Task Hyperscanning BCI.

This work proposes a new formulation for common spatial patterns (CSP), often used as a powerful feature extraction technique in brain-computer interfacing (BCI) and other neurological studies. In this approach, applied to multiple subjects' data and named as hyperCSP, the individual covariance and mutual correlation matrices between multiple simultaneously recorded subjects' electroencephalograms are exploited in the CSP formulation. This method aims at effectively isolating the common motor task between multiple heads and alleviate the effects of other spurious or undesired tasks inherently or intentionally performed by the subjects. This technique can provide a satisfactory classification performance while using small data size and low computational complexity. By using the proposed hyperCSP followed by support vector machines classifier, we obtained a classification accuracy of 81.82% over 8 trials in the presence of strong undesired tasks. We hope that this method could reduce the training error in multi-task BCI scenarios. The recorded valuable motor-related hyperscanning dataset is available for public use to promote the research in this area.