Adaptive Optimization Method Research Articles

An Adaptive Deep Learning Optimization Method Based on Radius of Curvature.

An adaptive clamping method (SGD-MS) based on the radius of curvature is designed to alleviate the local optimal oscillation problem in deep neural network, which combines the radius of curvature of the objective function and the gradient descent of the optimizer. The radius of curvature is considered as the threshold to separate the momentum term or the future gradient moving average term adaptively. In addition, on this basis, we propose an accelerated version (SGD-MA), which further improves the convergence speed by using the method of aggregated momentum. Experimental results on several datasets show that the proposed methods effectively alleviate the local optimal oscillation problem and greatly improve the convergence speed and accuracy. A novel parameter updating algorithm is also provided in this paper for deep neural network.

Robustness of Adaptive Neural Network Optimization Under Training Noise

Adaptive gradient methods such as adaptive moment estimation (Adam), RMSProp, and adaptive gradient (AdaGrad) use the temporal history of the gradient updates to improve the speed of convergence and reduce reliance on manual learning rate tuning, making them a popular choice for off-the-shelf Deep Neural Network (DNN) optimizers. In this article, we study the robustness of neural network optimizers in the presence of training perturbations. We show that popular adaptive optimization methods exhibit poor generalization while learning from noisy training data, compared to vanilla Stochastic Gradient Descent (SGD) and its variants, which manifest better implicit regularization properties. We construct an illustrative example of a family of two-class linearly separable toy-data such that models trained under noise using adaptive optimizers show only 52% test accuracy (random classifier), whereas SGD-based methods can achieve 100% test accuracy. We strengthen our hypothesis by empirical analysis using Convolutional Neural Networks (CNNs) on publicly available image datasets. For this purpose, our method trains neural network models with various optimizers on noisy training data, and we compute test accuracy on clean test data. Our results further highlight the robustness of SGD optimization against such noisy training data compared to its adaptive counterparts. Based on the results, our paper suggests a reconsideration of the extensive use of adaptive gradient methods for neural network optimization, especially when the training data is noisy.

Adaptive powered descent guidance based on multi-phase pseudospectral convex optimization

The real-time and fuel-optimal powered descent guidance is one of the critical technologies for planetary landing. This work focuses on the powered descent guidance algorithm based on pseudospectral convex optimization and aims to increase the accuracy and adaptability of the algorithm. The entire powered descent process is divided into phases according to the thrust switching times, and the pseudospectral method is adopted to discretize the optimization problem. Convexification process is performed to transform the optimization problem into a convex programming problem, and a successive solving process algorithm is proposed. For the established multi-phase pseudospectral convex optimization formulation, the indirect method is adopted to derive and determine the optimal phase division from switching function, and the adaptive multi-phase pseudospectral convex optimization method is proposed. Numerical examples demonstrate that the proposed method has excellent accuracy and adaptability.

Hybrid Design Optimization of Heating Furnace Temperature using ANFIS-PSO

-- Intelligent control design for industrial heating furnace temperature control is indispensable. PID, Fuzzy, and ANFIS controllers have been proven reliable and have been widely used. However, it is constrained in choosing a better gain controller. Then an approach method is given to determine the most appropriate controller gain value using the artificial intelligence tuning method. The artificial intelligence method used is a combination of the Adaptive Neuro Fuzzy Inference System and Particle Swarm Optimization (ANFIS-PSO) methods. As a comparison, several methods were used, namely; Conventional PID (PID-Konv), Matlab Auto tuning PID (PID-Auto), PSO tuned PID (PID-PSO), and Hybrid ANFIS-PSO. The ANFIS-PSO controller is the best choice compared to conventional single loop control systems, conventional PID, and matlab 2013a auto tuning methods to control this nonlinear process. The simulation results show that the ANFIS-PSO design is the best method with overshot = 0.0722, undershot 0.0085, and settling time at 18.8789 seconds which can produce a fast response with strong dynamic performance.

Adaptive Optimization Method in Digital Twin Conveyor Systems via Range-Inspection Control

The automated conveyor system, as the core component in the modern manufacturing world, has gained lots of attention from researchers. To optimize the operation of the conveyor system, range-inspection control (RIC) has been considered an efficient strategy to bring this conventional technology to an intelligent level. Various algorithms have been put into use to achieve optimal control. However, the current methodologies are only focusing on control optimization, not scaled into the smart manufacturing framework. The schema of alignment and corporation between the physical and virtual spaces for the system remains an important problem. Therefore, the work in this article aims for an effective framework of implementation between the physical and virtual stations in an automated conveyor system. Since increasingly more application scenarios rely on the digital twin (DT) technology to realize the integration of physical and virtual systems, we proposed the DT automated conveyor system (DT-ACS) that constructs the road map to implement the RIC-based conveyor system under the background of a smart factory. Besides, profit-sharing-based deep Q-networks (PDQNs) have been proposed to cope with the RIC optimization problem. The robustness and efficiency of the proposed PDQN were evaluated via sets of experiments. The discussion and conclusion are presented at last accordingly. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article aims to propose a strategy of control optimization for conveyor-based manufacturing systems under the digital twin (DT) framework. The conveyor system can be flexible to control the running flows to avoid overloading workstations. Due to the complex environment in the production line, the range that is able to be inspected and the capacity of the reserve area can be considerably diverse among the workstations. To maximally evaluate our framework, we set a comparatively complex environment for the experiments. Nevertheless, to obtain practically ideal performance under other circumstances, the parameters should be precisely tested and fine-tuned with simulation in advance.

Robust Transmission Expansion Planning Based on Adaptive Uncertainty Set Optimization Under High-Penetration Wind Power Generation

Previous robust transmission expansion planning (RTEP) studies have rarely considered the important question of how large the uncertainty set should be, and hence, how the uncertainty budget can be determined objectively. This study addresses this issue by proposing a novel adaptive optimization method that optimizes the value of the uncertainty budget to minimize the size of the uncertainty set while considering the underlying risk for wind power generation (WPG) fluctuations residing outside of the proposed uncertainty set. As such, the proposed method ensures a good tradeoff between the robustness and costs of RTEP solutions. In addition, the proposed method optimizes investment strategies under forecasted WPG scenarios, while providing a security guarantee under extreme WPG scenarios. The variable-limit integral terms introduced by the proposed method are addressed in the solution process by applying the piecewise linearization approximation method combined with the quadratic Newton-Gregory interpolating polynomial technique, which allows the solution process to be cast as a mixed integer linear programming problem. The good performance and effectiveness of the proposed adaptive uncertainty set optimization method are verified by numerical results.

Accelerating full waveform inversion through Adaptive Gradient Optimization methods and Dynamic Simultaneous Sources

Summary Full-Waveform Inversion (FWI) is a procedure based on the minimization of a misfit (or cost) function applied to the difference between synthetic waveforms and real seismic traces that derives high-resolution velocity models. This is achieved through the iterative adjustment of the velocity model and/or some other physical parameters of the Earth’s subsurface, which generally implies large computational effort. In order to minimize this cost function we explore the use of Adaptive Gradient Optimization (AGO), a variant of Stochastic Gradient Descent (SGD) methods, combining them with a dynamic simultaneous sources strategy that allow us to reduce the computational cost involved in this process. AGO methods are computationally efficient, have little memory requirements and have the capability of adapting the step-length according to the optimization process’ evolution. Since a precise calibration of the step-length is needed to ensure efficiency, the AGOs are well-suited for this task because they are able to adapt the step-length according to the optimization’s development. In this work, we propose a simple non-linear relationship that allows an adjustment of the step-length with respect to the frequencies used in the multiscale FWI, avoiding the line-search strategy’s high computational burden. Additionally, the application of this new step-length rule into the AGO methods with a dynamic simultaneous sources strategy, allow us to concurrently accelerate and significantly improve the FWI’s numerical performance and results. We compare the performance and final results of seven AGO methods, using two different FWI misfit functionals (based on L1 and L2 norms) applied to estimate the final velocity models of two benchmark acoustic models: the Marmousi and the Canadian overthrust BP velocity models.

Adaptive uncertainty sets‐based two‐stage robust optimisation for economic dispatch of microgrid with demand response

It is an effective way to regard the electric vehicles as the demand response for reducing the negative impact of large-scale introduction on the power system. Aiming at the microgrid with demand response, the adaptive uncertainty sets-based two-stage robust optimisation method is established in this study. The coordination of micro-gas turbine, energy storage, and demand response etc. are considered in the economic dispatch model. To effectively consider the uncertain variable contained in the microgrid, the concept of adaptive uncertainty sets is proposed in this study. The uncertainty sets are achieved by the long short-term memory network and modified fuzzy information granulation. To handle the adaptive uncertainty sets-based robust optimisation model, the column and constraint generation algorithm and strong duality theory are introduced to decompose the model into a master problem and a subproblem with mixed-integer linear structure. To verify the performance of the proposed adaptive uncertainty sets-based two-stage robust optimisation method, measured data from a plateau city of China are introduced in the simulation test. The simulation results demonstrate the effectiveness of the model and solution strategy.

Rolling bearing remaining useful life prediction via weight tracking relevance vector machine

The application scenarios of rotating machinery are becoming increasingly complicated due to the rapid development of the manufacturing industry. The remaining useful life (RUL) prediction of rolling bearings has gradually been considered in many industry fields for ensuring the safety and reliability of whole systems. As an effective way to analyze data, the relevance vector machine (RVM) approach holds great potential for RUL prediction. However, the redundant features of rolling bearing vibration signals can easily lead to overfitting and low accuracy of the RVM model for RUL prediction. To conquer these issues, inspired by the idea of the boosting algorithm and ensemble learning, this paper proposes a new RVM model, called the weight-tracking relevance vector machine (WTRVM). Within the proposed WTRVM model, an adaptive sequential optimal feature selection method is designed to avoid overfitting by selecting the best features. The error between the prediction value of the RVM model and the true value is counted for the RVM model training and weight tracking. The most accurate model can be obtained when all selected features have been trained. Finally, the proposed WTRVM algorithm is experimentally demonstrated to be effective for the RUL prediction of rolling bearings.

Adaptive clustering‐based hierarchical layout optimisation for large‐scale integrated energy systems

Different energy systems are generally planned and operated independently, which result in the low energy utilisation, weak self-healing ability and low system reliability. Therefore, an adaptive clustering-based hierarchical layout optimisation method is proposed for a large-scale integrated energy system, considering energy balance, transmission losses and construction costs. First, an adaptive clustering partition method based on energy balance and load moments is proposed to determine the optimal location of energy hubs and to allocate each distributed generation and load to different energy hubs, forming multiple regional integrated energy systems adaptively. Then, the proposed hierarchical layout optimisation model is formulated as to find the modified minimum spanning tree of the regional integrated energy system and multi-regional integrated energy systems, respectively, to construct an economical and reliable interconnection network. Finally, the effectiveness of the optimisation model and strategy is verified by simulations.

Self-adaptive process optimisation method for SBS cloud application based on reinforcement learning

To solve the problem of poor comprehensive performance of the traditional SBS cloud application adaptive process optimisation method, a new SBS cloud application adaptive process optimisation method based on reinforcement learning was proposed. Establish the adaptive action type selection model, realise the optimal choice of operation type, build the cloud application adaptive process optimisation model for resource cost, convert the problem into the corresponding mathematical model, solve and make the mathematical model of the algorithm, and obtain the best adaptive process optimisation scheme SBS cloud application. The simulation results show that the predicted load value of the method is the closest to the actual load value, the relative error of the prediction is less than 21.03, and the average time is less than 2.3 s, indicating that the method has good performance and high practical application value.

An adaptive adjustment strategy for bolt posture errors based on an improved reinforcement learning algorithm

Designing an intelligent and autonomous system remains a great challenge in the assembly field. Most reinforcement learning (RL) methods are applied to experiments with relatively small state spaces. However, the complicated situation and high-dimensional spaces of the assembly environment cause traditional RL methods to behave poorly in terms of their efficiency and accuracy. In this paper, a model-driven adaptive proximal proximity optimization (MAPPO) method was presented to make the assembly system autonomously rectify the bolt posture error. In the MAPPO method, a probabilistic tree and adaptive reward mechanism were used to improve the calculation efficiency and accuracy of the traditional PPO method. The size of the action space was reduced by establishing a hierarchical logical relationship for each parameter with a probabilistic tree. Based on an adaptive reward mechanism, the phenomenon that the algorithm easily falls into local minima could be improved. Finally, the proposed method was verified based on the Unity simulation engine. The advancement and robustness of the proposed model were also validated by comparing different cases in simulations and experiments. The results revealed that MAPPO has better algorithm efficiency and accuracy compared with other state-of-the-art algorithms.

A kriging-based adaptive global optimization method with generalized expected improvement and its application in numerical simulation and crop evapotranspiration

The generalized effective global optimization (EGO) method based on Kriging model can sequentially solve the expensive black-box problems. However, it can only obtain one sampling point in a cycle, which will result in more time spent on expensive function evaluations and affect the global convergence. To this end, A Kriging-based adaptive global optimization method with generalized expected improvement (KAGO-GEI) is proposed. It divides the enhanced generalized expected improvement (GEI) criterion which recursively changes in the iterative process into double objectives, and then uses multi-objective PSO method to optimize the two objectives to produce the Pareto frontier. Further, more valuable sampling points from Pareto frontier are screened and corrected as the expensive-evaluation points for updating Kriging model. Test results on eighteen benchmark functions and crop evapotranspiration calculation example show that the proposed method is superior to other classical optimization methods in terms of convergence and accuracy in most cases.

An adaptive intelligent collaborative optimization method based on inconsistent information

Multidisciplinary Design Optimization (MDO) is a design optimization method for dealing with large-scale and multi coupling complex engineering systems. The collaborative optimization (CO) has the characteristics of high degree of discipline autonomy, multi-level optimization and distributed computing. It can effectively solve the design optimization problems of large-scale complex engineering systems, and has been widely used in aerospace, ship, automobile, machinery and other fields. Because of its own optimization model and principle, the CO method has the defects of low computational efficiency and difficult convergence. In this paper, in order to overcome the convergence difficulty caused by the internal definition defects of the CO method, combined with the adaptive mechanism, the position relationship between the system level optimization points and the constraint conditions is analyzed, and the adaptive penalty function is constructed based on the inconsistent information of the system. The system level optimization model of the CO method is reconstructed by transforming the system level constraints. Finally, an example is given to demonstrate the effectiveness of the proposed method.

Adaptive Ensemble-Based Optimisation for Petrophysical Inversion

This article introduces a new optimisation algorithm designed for sampling the solution space of non-linear, non-convex quadratic problems. This method has been specifically designed for inversion problems where multiple distinct scenarios must be explored, which would not be achievable with a standard ensemble-based optimisation method (EnOpt). A prior ensemble is used to sample the uncertainty of the parameters before updating each member of the ensemble independently in order to find the minimum of an objective function. This update is given by a Gauss–Newton-like approach, where the first-derivative matrix is adaptively estimated from sub-ensembles of parameters and their corresponding forward model responses. As the first-derivative matrix is statistically computed from an ensemble of realisations, the forward model does not need to be known and the algorithm is independent of it. The final ensemble provides an estimation of the uncertainty after the inversion process. The efficiency of this adaptive ensemble optimisation (A-EnOpt) method is first tested on simple two-dimensional problems where known mathematical functions are used as forward models. The results show that the method minimises the objective functions and samples the final uncertainties of the problems to a better degree than a standard EnOpt. The A-EnOpt method is then applied to a real heavy-oil field petrophysical inversion. Porosity, shale fraction and fluid saturations are inverted under continuity and Lagrange constraints, conditioned by P-impedance models from stochastic seismic inversion. The updated properties are incorporated in a fluid flow model of the field. The simulation results produce a better match to historical production data at one well than equivalent flow simulations using the properties before inversion. This shows that conditioning by seismic data improves the quality of the geological models.

Event-Triggered Adaptive Optimal Control With Output Feedback: An Adaptive Dynamic Programming Approach.

This article presents an event-triggered output-feedback adaptive optimal control method for continuous-time linear systems. First, it is shown that the unmeasurable states can be reconstructed by using the measured input and output data. An event-based feedback strategy is then proposed to reduce the number of controller updates and save communication resources. The discrete-time algebraic Riccati equation is iteratively solved through event-triggered adaptive dynamic programming based on both policy iteration (PI) and value iteration (VI) methods. The convergence of the proposed algorithm and the closed-loop stability is carried out by using the Lyapunov techniques. Two numerical examples are employed to verify the effectiveness of the design methodology.

An optimal filter length selection method for MED based on autocorrelation energy and genetic algorithms

This paper proposed a method for exact selecting the optimal filter length of minimal entropy deconvolution (MED) to solve it recovering a single random pulse when the filter length is not improper. The energy ratio of autocorrelation between the filtered signal and the residual signal is adopted to measure the salience of periodic impulses. Then this index is used as an objective function of genetic algorithms (GA) to form an adaptive optimal selection method of filter length. The proposed method is verified by two different rolling bearing fault experiments. The results show that the proposed method reveals the periodic impulses successfully from the casing signals. Compared with other MED-based methods, the proposed method has better performance in detecting the weak fault signal.

Vague Adaptive Optimization Based Research on Yunnan Favorable Natural Rubber Species

Based on Vague adaptive optimization method, taking traditional natural rubber species Indonesia PR107 and Malaysia RRIM600 as reference objects to optimize new natural rubber species. The conclusion of this study shows that it is advisable to promote and plant the cold-resistant and high-yield natural rubber species, including Yunyan 77-2, Yunyan 77-4 and Indian RRII 208.

Secure Image Communication Through Adaptive Deer Hunting Optimization Based Vector Quantization Coding of Perceptually Encrypted Images

Communication fields are growing rapidly in the recent era, so transmitting the multimedia contents through an open channel becomes a challenging task. The multimedia contents that are transmitted through this channel are highly prone to vulnerabilities and attacks. Therefore, secure and efficient data communication is considered as a major concern in the multimedia communication systems. So, major efforts are taken by researchers to safeguard the originality of each image. In a conventional system, the secure image communication process was achieved by compressing the content first, and then encryption is performed over the compressed data. Even though it met the required security and compression ratio, but some applications may require the reverse system. In this method, the encryption process is conducted prior to compression to improve the privacy of user data. Moreover, the initial concentration is given for improving content privacy rather than concentrating on size reduction. This paper proposes a reversed system that uses block based perceptual encryption algorithm for encryption and vector quantization (VQ) with hybrid Lloyd–Buzo–Gray (LBG)-Adaptive Deer Hunting Optimization (ADHO) algorithm (VQ-LBG-ADHO) for compression. So, the content secrecy gets improved. The involvement of this adaptive optimization method enhances the performance of VQ in the compression process. This method highly concentrates on secure communication, so the reverse process is followed in this method. It not only improves the image secrecy, however, it further enhances the image quality. The performance of this secure communication process is compared with state-of-the-art algorithms, and the results reveal that the proposed method outperforms the other existing methods.

Adaptive Stochastic Optimization: A Framework for Analyzing Stochastic Optimization Algorithms

Optimization lies at the heart of machine learning (ML) and signal processing (SP). Contemporary approaches based on the stochastic gradient (SG) method are nonadaptive in the sense that their implementation employs prescribed parameter values that need to be tuned for each application. This article summarizes recent research and motivates future work on adaptive stochastic optimization methods, which have the potential to offer significant computational savings when training largescale systems.