Training Error Research Articles

Exploring optical descriptors for rapid estimation of coastal sediment organic carbon and nearby land-use classifications via machine learning models

This study utilizes ultraviolet and fluorescence spectroscopic indices of dissolved organic matter (DOM) from sediments, combined with machine learning (ML) models, to develop an optimized predictive model for estimating sediment total organic carbon (TOC) and identifying adjacent land-use types in coastal sediments from the Yellow and Bohai Seas. Our results indicate that ML models surpass traditional regression techniques in estimating TOC and classifying land-use types. Penalized Least Squares Regression (PLR) and Cubist models show exceptional TOC estimation capabilities, with PLR exhibiting the lowest training error and Cubist achieving a correlation coefficient 0.79. In land-use classification, Support Vector Machines achieved 85.6 % accuracy in training and 92.2 % in testing. Maximum fluorescence intensity and ultraviolet absorbance at 254 nm were crucial factors influencing TOC variations in coastal sediments. This study underscores the efficacy of ML models utilizing DOM optical indices for near real-time estimation of marine sediment TOC and land-use classification.

Neural network fusion optimization for photovoltaic power forecasting

This paper aims to establish a comprehensive photovoltaic power generation prediction model. By collecting photovoltaic power generation data and weather data for a year, we analyzed the photovoltaic output characteristics in different seasons and found that the output characteristics in different seasons are also different. This article uses three neural network models, Long Short Term Memory Network, Recurrent Neural Network, and Dense Neural Network, to analyze the output characteristics of different seasons. Training, prediction, and prediction error analysis found that different models have different prediction accuracy in different seasons. Therefore, this paper proposes a weighted ensemble model add weights model based on the Nelder-Mead method to train and predict different seasons respectively. By analyzing the prediction error, the prediction accuracy needs to be better than a single model. We add noise to the data set to simulate unstable lighting conditions such as rainy days, and train and predict the data set after adding noise. The prediction results show that the comprehensive model has higher prediction accuracy than a single model in extreme weather. In order to verify the reliability of the model, this article uses a sliding window to extract the confidence interval of the prediction results, and uses the Bootstrap method to calculate the confidence interval. By analyzing and comparing each model’s Average Coverage, Root Mean Squared Length, and Mean Width, the prediction accuracy and reliability of add weights model are better than those of a single model.

Improving Neural Network Generalization on Data-Limited Regression with Doubly-Robust Boosting

Enhancing the generalization performance of neural networks given limited data availability remains a formidable challenge, due to the model selection trade-off between training error and generalization gap. To handle this challenge, we present a posterior optimization issue, specifically designed to reduce the generalization error of trained neural networks. To operationalize this concept, we propose a Doubly-Robust Boosting machine (DRBoost) which consists of a statistical learner and a zero-order optimizer. The statistical learner reduces the model capacity and thus the generalization gap; the zero-order optimizer minimizes the training error in a gradient-free manner. The two components cooperate to reduce the generalization error of a fully trained neural network in a doubly robust manner. Furthermore, the statistical learner alleviates the multicollinearity in the discriminative layer and enhances the generalization performance. The zero-order optimizer eliminates the reliance on gradient calculation and offers more flexibility in learning objective selection. Experiments demonstrate that DRBoost improves the generalization performance of various prevalent neural network backbones effectively.

Prediction of the impact of ecological restoration technology on the restoration of heavy metal pollution in agricultural soil

ABSTRACT Heavy metal in agricultural soils increases its contamination and infertility in a due course of time. The vast expansion of industrialization along the agricultural soils has led to such adverse effects impacting vegetation. Several existing learning techniques are utilized to analyze the contamination in agricultural soil; however, the learning process consumes a high restoration error rate. This article introduces a Concentration-specific Restoration Prediction Technique (CRPT) for heavy metal-exposed agricultural soils. This technique relies on the heavy metal level present in different soil types for assessing its adverse impact. The vegetation failures and infertility progressions are accounted for by the continuous recurrent learning for predicting its restoration age. The restoration procedures preferred in the earlier stages are used for training the recurrent neural network. Based on the more feasible restoration method, the failure and infertility are validated for achieving a maximum prediction value. The learning process accounts for soil type, crop resistance, and climatic factors for holding restoration procedures. Therefore, the learning process performs restoration prediction along with resistance analysis for progressive restoration prediction. The least possible prediction value is validated using different training inputs for matching outputs. This technique is reliable in detecting pollution possibility with maximum restoration prediction (9.41%) and less training error (14.53%).

Design and Development of Hypertuned Deep learning Frameworks for Detection and Severity Grading of Brain Tumor using Medical Brain MR images.

Brain tumor is a grave illness causing worldwide fatalities. The current detection methods for brain tumors are manual, invasive, and rely on histopathological analysis. Determining the type of brain tumor after its detection relies on biopsy measures and involves human subjectivity. The use of automated CAD techniques for brain tumor detection and classification can overcome these drawbacks. The paper aims to create two deep learning-based CAD frameworks for automatic detection and severity grading of brain tumors - the first model for brain tumor detection in brain MR images and model 2 for the classification of tumors into three types: Glioma, Meningioma, and Pituitary based on severity grading. The novelty of the research work includes the architectural design of deep learning frameworks for detection and classification of brain tumor using brain MR images. The hyperparameter tuning of the proposed models is done to achieve the optimal parameters that result in maximizing the models' performance and minimizing losses. The proposed CNN models outperform the existing state of the art models in terms of accuracy and complexity of the models. The proposed model developed for detection of brain tumors achieved an accuracy of 98.56% and CNN Model developed for severity grading of brain tumor achieved an accuracy of 92.36% on BraTs dataset. The proposed models have an edge over the existing CNN models in terms of less complexity of the structure and appreciable accuracy with low training and test errors. The proposed CNN Models can be employed for clinical diagnostic purposes to aid the medical fraternity in validating their initial screening for brain tumor detection and its multi-classification.

Electroosmotic microchannel flow of blood conveying copper and cupric nanoparticles: Ciliary motion experiencing entropy generation using backpropagated networks

AbstractA novel mathematical model for a hybrid (Cu–CuO/blood) Jeffrey nanofluid passing a vertical symmetric microchannel along with an electroosmosis pump is presented. The focuses on the advancement of mathematical modeling techniques, its comprehensive analysis of microfluidic system dynamics, and its potential to inform the optimal design of devices using nanofluids with broad applications in various fields. Arrhenius's law is used to analyze endothermic–exothermic reactions and activation energy. The governing partial differential equations of the fixed frame are transformed into ordinary differential equations of the wave frame using self‐similarity transformations. Low Reynolds number and long‐wavelength approximations helped to find solutions of the equations by applying a suitable BVP solver in MATLAB. The fluid's velocity, temperature, concentration, and electroosmosis properties are studied graphically. Two‐dimensional contour plots of fluid velocity and three‐dimensional surface plots of fluid properties are discussed. Physically significant quantities of mass transfer rate, skin friction coefficient, entropy generation, and heat transfer rate are studied using contour plots. Artificial neural network simulation using Bayesian regularization backpropagation algorithms is analyzed for training state, error histogram, fit, performance, and regression plots. Conclusively, the comprehensive analysis of the fluid dynamics, entropy generation, mass and heat transfer, and in the microchannel, coupled with the successful implementation of artificial neural network simulation, contributes to an improved understanding of the system's behavior. Entropy generation was raised for enhanced Brownian motion number and reduced values of thermophoresis, activation energy, and endothermic–exothermic reaction parameters. This study's results can be used to improve the efficiency and effectiveness of microfluidic devices used in fields as diverse as electrical cooling and medicine delivery.

Bellman Neural Networks for the Class of Optimal Control Problems With Integral Quadratic Cost

This work introduces Bellman Neural Networks (BeNNs) and employs them to learn the optimal control actions for the class of optimal control problems (OCPs) with integral quadratic cost. BeNNs represent a particular family of Physics-Informed Neural Networks (PINNs) specifically designed and trained to tackle OCPs via applying the Bellman Principle of Optimality (BPO). The BPO provides necessary and sufficient optimality conditions, which result in a nonlinear partial differential equation known as the Hamilton-Jacobi-Bellman (HJB) equation. BeNNs learn the optimal control actions from the unknown solution of the arising HJB equation (i.e., the value function), where the unknown solution is modeled using a Neural Network. Additionally, the paper shows how to estimate the upper bounds on the generalization error of BeNNs while learning the solutions for the OCP class under consideration. The generalization error estimate is provided in terms of the choice and number of the training points as well as the training error. Numerical studies show that BeNNs can be successfully applied to learn the feedback control actions for the class of optimal control problems considered and, after the training is completed, deployed to control the system in a closed-loop fashion. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Impact Statement</i> —The proposed research improves our understanding of how to solve optimal control problems with closed-loop solutions and has potentially a countless number of applications in several different areas. The study is at the intersection between optimal control theory and artificial intelligence connected with mathematical tools for functional interpolation. This advances the ability to implement a higher level of autonomy in decision-making for practical applications with a beneficial impact on our society.

Ovarian Cancer Detection Based on Elman Recurrent Neural Network

The early detection of cancers increases the possibility of health recovery and prevents the disease from becoming a silent killer. This study introduces an effective method for identifying ovarian cancer (OC) using Elman Recurrent Neural Network (ERNN), which can recognize cancer via mass spectrometry data. The network has a topology of 100 input neurons for receiving data, five neurons for hidden and context layers, and two output nodes to indicate the status. The proposed method uses reduced-size features, including ion concentration levels at specific mass/charge values, which are trained using various learning algorithms to determine the suitable one that achieves the best results. The experimental results show that all the training algorithms achieve about 100% performance rate, with the Levenberg Marquardt (LM) being the most accurate and fastest algorithm, which converges after six epochs and achieves 0.0035, 0.0045 and 0.0045 mean square errors for training, validation, and test performances, respectively. Based on comparative results, the proposed LM-ERNN method outperforms other OC detection methods and holds promise for detecting other types of cancer.

Stress fractures of the foot - current evidence on management

Stress fractures are a consequence of repeated submaximal loads with inadequate time for recovery and biologic repair or remodelling. The foot and ankle complex (FAC) represents a common site for development of stress fractures. Whilst the overall incidence of stress fractures is low, they are prevalent in athletes and military personnel causing significant time away from sports or work. Within these populations, certain stress fractures directly correlate to specific activities.Factors that commonly influence these fractures include an acute increase in new repetitive physical activity combined with muscle fatigue, training errors or improper athletic techniques, which challenge the regenerative and remodelling capacity of bone. Depending on the site that is subject to repetitive loading, various biomechanical factors can result in abnormal concentration of forces to specific areas of the FAC resulting in stress fracture. Decreased bone marrow density (BMD) is a major biologic cause for developing stress fractures. The female athlete triad comprising eating disorder, amenorrhea and osteoporosis in competitive athletes also predisposes to stress fractures. Vitamin D deficiency is also postulated to be the cause of these fractures and may contribute to poor healing.Clinical presentation is usually with vague pain of insidious onset which worsens with activity and improves with rest. Diffuse tenderness over the affected bone is common with only a minority having any visible swelling. Plain radiographs are the first line of investigation but rarely reveal an obvious fracture. MRI scans aid in diagnosis and CT scans help in treatment and characterisation of the fracture and monitor healing.Management relates to the site of injury, which stratifies them into high or low-risk. Stress fractures of the calcaneus, cuboid and cuneiforms are classed as low-risk fractures as they usually heal with simple activity modification or short duration of non-weight bearing. Stress fractures of the navicular, talus and hallucal sesamoids are classed as high-risk fractures due to higher rates of non-union and prolonged recovery time. Metatarsal fractures can be considered high or low-risk depending on location. These warrant aggressive management, often requiring surgical intervention. Adjuncts such as vitamin D supplements, external shockwave therapy, low-intensity pulsed ultrasound therapy have been used with varying success but there remains little supportive evidence of superiority in the available literature.

On Optimizing Ensemble Models using Column Generation

In recent years, an interest appeared in integrating various optimization algorithms in machine learning. We study the potential of ensemble learning in classification tasks and how to efficiently decompose the underlying optimization problem. Ensemble learning has become popular for machine learning applications and it is particularly interesting from an optimization perspective due to its resemblance to column generation. The challenge for learning is not only to obtain a good fit for the training data set, but also good generalization, such that the classifier is generally applicable. Deep networks have the drawback that they require a lot of computational effort to get to an accurate classification. Ensemble learning can combine various weak learners, which individually require less computational time. We consider binary classification problems studying a three-phase algorithm. After initializing a set of base learners refined by a bootstrapping approach, base learners are generated using the solution of an linear programming (LP) master problem and then solving a machine learning sub-problem regarding a reduced data set, which can be viewed as a so-called pricing problem. We theoretically show that the algorithm computes an optimal ensemble model in the convex hull of a given model space. The implementation of the algorithm is part of an ensemble learning framework called decolearn. Numerical experiments with CIFAR-10 data set show that the base learners are diverse and that both the training and generalization error are reduced after several iterations.

Deep hybrid modeling of a HEK293 process: Combining long short-term memory networks with first principles equations.

The combination of physical equations with deep learning is becoming a promising methodology for bioprocess digitalization. In this paper, we investigate for the first time the combination of long short-term memory (LSTM) networks with first principles equations in a hybrid workflow to describe human embryonic kidney 293 (HEK293) culture dynamics. Experimental data of 27 extracellular state variables in 20 fed-batch HEK293 cultures were collected in a parallel high throughput 250 mL cultivation system in an industrial process development setting. The adaptive moment estimation method with stochastic regularization and cross-validation were employed for deep learning. A total of 784 hybrid models with varying deep neural network architectures, depths, layers sizes and node activation functions were compared. In most scenarios, hybrid LSTM models outperformed classical hybrid Feedforward Neural Network (FFNN) models in terms of training and testing error. Hybrid LSTM models revealed to be less sensitive to data resampling than FFNN hybrid models. As disadvantages, Hybrid LSTM models are in general more complex (higher number of parameters) and have a higher computation cost than FFNN hybrid models. The hybrid model with the highest prediction accuracy consisted in a LSTM network with seven internal states connected in series with dynamic material balance equations. This hybrid model correctly predicted the dynamics of the 27 state variables (R2 = 0.93 in the test data set), including biomass, key substrates, amino acids and metabolic by-products for around 10 cultivation days.

Quantitative evaluation method of operation reliability of substation relay protection device based on improved neural network algorithm

With the increase in power demand and the increasing complexity of the power grid, the operation of relay protection devices in substations is often unreasonable. Therefore, a quantitative evaluation method of operation reliability of relay protection devices in substations based on an improved neural network algorithm is proposed. The time-varying probability of misoperation of relay protection in substation is calculated based on the improved neural network algorithm, identifying misoperation behavior of relay protection device, obtaining the optimal hyperplane decision function, selecting kernel function, and construct a quantitative evaluation function of operation reliability of relay protection device. In the evaluation process, two stages of the evaluation process are carried out, namely, reliability block diagram analysis, failure mode evaluation, and verification of relay protection device operation state, so as to complete the reliability quantitative evaluation. The experimental results show that this method can minimize the error of evaluation training and improve evaluation efficiency. At the same time, the error rate is small, which can be controlled within 0.5%.

Semantic Linear Genetic Programming for Symbolic Regression.

Symbolic regression (SR) is an important problem with many applications, such as automatic programming tasks and data mining. Genetic programming (GP) is a commonly used technique for SR. In the past decade, a branch of GP that utilizes the program behavior to guide the search, called semantic GP (SGP), has achieved great success in solving SR problems. However, existing SGP methods only focus on the tree-based chromosome representation and usually encounter the bloat issue and unsatisfactory generalization ability. To address these issues, we propose a new semantic linear GP (SLGP) algorithm. In SLGP, we design a new chromosome representation to encode the programs and semantic information in a linear fashion. To utilize the semantic information more effectively, we further propose a novel semantic genetic operator, namely, mutate-and-divide propagation, to recursively propagate the semantic error within the linear program. The empirical results show that the proposed method has better training and test errors than the state-of-the-art algorithms in solving SR problems and can achieve a much smaller program size.

A double‐layer neural network wind speed prediction framework based on training set segmentation and error correction

AbstractHighly accurate wind speed prediction is one of the effective ways to improve wind energy utilization. Therefore, this study proposes a novel double‐layer neural network (D‐NN) framework based on feedforward neural networks. First, a feedforward neural network with an extremely short training time is used as the basic learner to ensure that the D‐NN framework has a short training and optimization time. Then, the double‐layer neural network automatically segments and adjusts the length of the training set using an improved error correction method, thus improving the model's ability to capture sequence features and enhancing the model's generalization performance in a non‐directly‐connected superposition manner. Finally, a genetic algorithm is used to automatically optimize the hyperparameters of each sub‐neural network in the D‐NN and the optimal training set length of each sub‐neural network, resulting in the final optimized prediction model. The accuracy and reliability of the prediction models designed based on the D‐NN framework are verified by comparing several different models. The results of the five different prediction evaluation metrics show that the D‐NN framework can greatly improve the accuracy of wind speed prediction.

Infinitesimal gradient boosting

We define infinitesimal gradient boosting as a limit of the popular tree-based gradient boosting algorithm from machine learning. The limit is considered in the vanishing-learning-rate asymptotic, that is when the learning rate tends to zero and the number of gradient trees is rescaled accordingly. For this purpose, we introduce a new class of randomized regression trees bridging totally randomized trees and Extra Trees and using a softmax distribution for binary splitting. Our main result is the convergence of the associated stochastic algorithm and the characterization of the limiting procedure as the unique solution of a nonlinear ordinary differential equation in a infinite dimensional function space. Infinitesimal gradient boosting defines a smooth path in the space of continuous functions along which the training error decreases, the residuals remain centered and the total variation is well controlled.

Multi-resolution partial differential equations preserved learning framework for spatiotemporal dynamics

Traditional data-driven deep learning models often struggle with high training costs, error accumulation, and poor generalizability in complex physical processes. Physics-informed deep learning (PiDL) addresses these challenges by incorporating physical principles into the model. Most PiDL approaches regularize training by embedding governing equations into the loss function, yet this depends heavily on extensive hyperparameter tuning to weigh each loss term. To this end, we propose to leverage physics prior knowledge by “baking” the discretized governing equations into the neural network architecture via the connection between the partial differential equations (PDE) operators and network structures, resulting in a PDE-preserved neural network (PPNN). This method, embedding discretized PDEs through convolutional residual networks in a multi-resolution setting, largely improves the generalizability and long-term prediction accuracy, outperforming conventional black-box models. The effectiveness and merit of the proposed methods have been demonstrated across various spatiotemporal dynamical systems governed by spatiotemporal PDEs, including reaction-diffusion, Burgers’, and Navier-Stokes equations.

Many-objective coevolutionary learning algorithm with extreme learning machine auto-encoder for ensemble classifier of feedforward neural networks

In artificial neural network (ANN) learning, empirical risk can be expressed by training error, while structural risk can be expressed by network complexity. Learning from data is often seen as a tradeoff between these two risks. Additionally, balancing training error and validation error can overcome the overfitting problem to some extent. It is known that the network performance is also related to regularization term. In order to consider four factors, i.e. training error, validation error, network complexity and regularization term simultaneously in the training process of a single-hidden layer feedforword neural network (SLFN), a many-objective coevolutionary learning algorithm (MOCELA) integrated with extreme learning machine auto-encoder (ELMAE), called MOCELA-ELMAE is presented. In MOCELA, the non-dominated sorting genetic algorithm III (NSGA-III) is improved for handling the many-objective optimization model with hybrid variables, where binary coding is used for structure learning and real coding is utilized for representing input parameters, referring to all input weights and hidden biases of the AE network. Output parameters of AE, i.e. output weights are analytically calculated by the non-iterative learning rule. The network structure and connection parameters of SLFN are determined based on those of AE. A set of Pareto optimal solutions are eventually collected by the MOCELA-ELMAE, which represents multiple optimal SLFNs. To make the final decision, three best SLFNs with minimum validation errors are selected as the base classifiers for selective ensemble learning. Extensive experiments are implemented on the benchmark classification data sets of UCI machine learning repository, and obvious improvements have been observed when the proposed MOCELA-ELMAE is compared with the NSGA-III based on hybrid coding and completely non-iterative learning of SLFN respectively. The experimental results also illustrate that the MOCELA-ELMAE performs much better than other state-of-the-art learning algorithms on many data sets.

Calf stretch effectiveness in plantar fasciitis cases: A case series

Plantar fasciitis in Indonesia, occur after 30 years of age, 77% of sufferers between 40 to 69 years old, men to women is 1:2. There are risk factors for plantar fasciitis include structural abnormalities, being overweight, age-related degenerative changes, activities that require prolonged standing and/or ambulation, and errors in training, especially in athletes. We present 20 case series of all patients diagnosed with plantar fasciitis who went to the Orthopedic Polyclinic in 2018 and 2019, they received conservative management with Calf Stretch workouts for 30 days and were evaluated with AOFAS Score. Calf stretch exercises in which the patient stands with the sore leg behind and both hands pushing against a wall, also known as the wall stretch. This exercise is performed 2 times a day with a duration of 10-15 minutes each session. This exercise is done routinely for 30 days. A simple, progressive training protocol consisting of strength training with heavy weights will produce better results in the first three months when compared to plantar-focused stretching and may aid in more rapid relief of symptoms on pain. Conservative management with calf exercises still provides a good outcome in patients with plantar fasciitis, based on the AOFAS score.

Application of Nonlinear Big Data Analysis Techniques in Computer Software Reliability Prediction

This research addresses the difficulties of underfitting, overfitting, and convergence to local minima in artificial neural networks for software dependability prediction. The work specifically focuses on enhancing the performance of the conventional PSO-SVM model for software reliability prediction. The analysis of the conventional PSO-SVM model and the special features of software reliability prediction serve as the foundation. An improved PSO-SVM software reliability prediction model is developed and the PSO-SVM model and a Backpropagation (BP) prediction model are compared experimentally. The critical metrics assessed include training error, and efficiency. The experimental results reveal that the training error of the enhanced PSO-LSSVM prediction model diminishes rapidly, levelling off after approximately 200 training generations. The BP prediction model requires 1,733 generations to meet training requirements. Furthermore, the improved PSO-LSSVM prediction model demonstrates significantly higher training efficiency than the BP prediction model. The optimized prediction model exhibits superior adaptability to small sample sizes, swift training, and high prediction accuracy, making it a more suitable choice for software reliability prediction applications.

Ultra-Short-Term Wind Power Prediction Based on eEEMD-LSTM

The intermittent and random nature of wind brings great challenges to the accurate prediction of wind power; a single model is insufficient to meet the requirements of ultra-short-term wind power prediction. Although ensemble empirical mode decomposition (EEMD) can be used to extract the time series features of the original wind power data, the number of its modes will increase with the complexity of the original data. Too many modes are unnecessary, making the prediction model constructed based on the sub-models too complex. An entropy ensemble empirical mode decomposition (eEEMD) method based on information entropy is proposed in this work. Fewer components with significant feature differences are obtained using information entropy to reconstruct sub-sequences. The long short-term memory (LSTM) model is suitable for prediction after the decomposition of time series. All the modes are trained with the same deep learning framework LSTM. In view of the different features of each mode, models should be trained differentially for each mode; a rule is designed to determine the training error of each mode according to its average value. In this way, the model prediction accuracy and efficiency can make better tradeoffs. The predictions of different modes are reconstructed to obtain the final prediction results. The test results from a wind power unit show that the proposed eEEMD-LSTM has higher prediction accuracy compared with single LSTM and EEMD-LSTM, and the results based on Bayesian ridge regression (BR) and support vector regression (SVR) are the same; eEEMD-LSTM exhibits better performance.