Adaptive Optimization Method Research Articles

AdaDiff: Adaptive Gradient Descent with the Differential of Gradient

Optimization methods are crucial to train deep neural networks. Adaptive optimization methods, especially Adam, are wildly used because they aren’t sensitive to the selection of learning rate and converge fast. Recent work point out Adam has a performance gap with SGD and even not converge because of the unstable and extreme learning rates. Many variants of Adam are proposed to solve the problem, such as AMSGrad, AdaBound and AdaBelief. In this paper, we propose a new variant of Adam, called AdaDiff. AdaDiff computing gradient descent step size by the exponential moving average(EMA) of gradient and differential of gradients aiming to make the training of networks more stable. We compare our method with other optimizers on various tasks. The results show AdaDiff outperforms Adam and minimizes the performance gap with SGD.

Sliding-mode surface-based adaptive actor-critic optimal control for switched nonlinear systems with average dwell time

This paper studies the problem of sliding-mode surface (SMS)-based adaptive optimal control for a class of continuous-time switched nonlinear systems with average dwell time (ADT) via using an actor-critic (AC) reinforcement learning (RL) strategy. By developing a specific cost function related to SMS, the original control problem is equivalently transformed into the problem of finding a series of optimal control policies. Then, the error terms separated from the Hamilton-Jacobi-Bellman (HJB) equation are integrated into a function, which effectively reduces the influence of some repeated magnifications caused by approximation errors. Besides, the solution to the HJB equation is identified by the SMS-based AC neural networks (NNs), where the actor and critic NNs are developed to carry out the RL strategy simultaneously. The actor updating law is to implement control actions based on the system output, while the critic updating law is required to assess the current control action and feedback to the actor. Based on the Lyapunov stability theory, the applicability of the proposed adaptive AC optimal control method is verified to guarantee the boundedness of all signals in the considered closed-loop switched nonlinear systems. Finally, a simulation examples is given to illustrate the effectiveness of the proposed adaptive optimal control method.

A Novel Framework for Anomaly Detection for Satellite Momentum Wheel Based on Optimized SVM and Huffman-Multi-Scale Entropy

The health status of the momentum wheel is vital for a satellite. Recently, research on anomaly detection for satellites has become more and more extensive. Previous research mostly required simulation models for key components. However, the physical models are difficult to construct, and the simulation data does not match the telemetry data in engineering applications. To overcome the above problem, this paper proposes a new anomaly detection framework based on real telemetry data. First, the time-domain and frequency-domain features of the preprocessed telemetry signal are calculated, and the effective features are selected through evaluation. Second, a new Huffman-multi-scale entropy (HMSE) system is proposed, which can effectively improve the discrimination between different data types. Third, this paper adopts a multi-class SVM model based on the directed acyclic graph (DAG) principle and proposes an improved adaptive particle swarm optimization (APSO) method to train the SVM model. The proposed method is applied to anomaly detection for satellite momentum wheel voltage telemetry data. The recognition accuracy and detection rate of the method proposed in this paper can reach 99.60% and 99.87%. Compared with other methods, the proposed method can effectively improve the recognition accuracy and detection rate, and it can also effectively reduce the false alarm rate and the missed alarm rate.

Adaptive topology optimization under suitably graded THB‐spline refinement and coarsening

AbstractIn the present work, we propose an adaptive topology optimization (TO) method under suitably graded truncated hierarchical B‐spline refinement and coarsening (SGTHB‐TO). A series of algorithms have been devised to implement admissible adaptivity of SGTHB‐TO by resorting to the definitions of suitably graded isogeometric hierarchical meshes. We apply the proposed SGTHB‐TO method to two‐dimensional and three‐dimensional TO problems of compliance and compliant mechanism. According to the numerical results, SGTHB‐TOs with or without admissible restriction share similar optimization processes, except that numerical artifact, such as edge blurring and islanding gray regions occurring in SGTHB‐TO without admissible restriction, can be eliminated by SGTHB‐TO with admissible parameter equaling to 2. Besides, the convergence rate can be largely accelerated by imposing an admissible constraint on SGTHB‐TO for TO problems with geometric nonlinearity existing in the design domain, which is attributed to the improved numerical accuracy by the admissible adaptive discretization scheme of isogeometric analysis (IGA). Therefore, the SGTHB‐TO method with the admissible parameter equaling to 2 is a more effective adaptive TO method than SGTHB‐TO without considering admissible restriction, in terms of both numerical stability and convergence rate.

RETRACTED ARTICLE: Training an innovative physics-learned deep neural network by the Adam optimization method to simulate the responses of the hybrid multilayer system

This article presents the high-exactitude analysis of hygro-thermo-mechanical vibration of the three-phase multi-scale hybrid composite angle-ply laminated rectangular plate (MHCALRP) for different couples of boundary conditions, within the context of the refined higher-order shear deformation theory with fifteen variables (RHOSDT15). Defining the system's kinematics according to RHOSDT15 is the essential step toward achieving sufficient accuracy in the elucidation of vibrational behavior of the moderately thick plates. The numerical solution is acquired by implementing the generalized differential quadrature method (GDQM). The accuracy of the employed approach is evaluated by comparing the results with the published studies. As a practical conclusion, it is revealed that the orientation angle of the macrofibers can effectively compensate for the lack of nano-reinforcement and the absence of extensional load to overcome the negative impacts of hygro-thermal environment in an affordable cost. The influence of extensional load, nano-/macro-scale reinforcements on the vibrational response of the MHCALRP is completely elucidated. Additionally, a comparative study is performed based on deep neural network (DNN) and its learning parameters are calculated according to adaptive Adam optimization method. Importance of this comparative study is related to providing a non-model-based system by means of a cutting-edge artificial intelligence method to predict vibrational response of the structure.

A multi-objective residential load management based on self-adapting differential evolution

• An adaptive multi-objective optimization method for demand response is proposed. • Epsilon constrains method is used to get a single point lies within Pareto front. • A tiered incentive can encourage customers to participate in demand response program. • The results indicate significant cost reduction (53.34%) and peak load (43.04%). The current grid system is unable to meet the ever-increasing demands for energy due to annual population growth and the increasing number of energy-consuming devices that are being used in the modern era. One solution that helps to balance available generation capacity with required demand, and increases the grid reliability is the demand response (DR). For this purpose, this paper presents a residential load scheduling algorithm to manage the operation time of household appliances depending on the time of use (ToU) tariff. A self-adapting multi-objective differential evolution (SaMODE) algorithm was developed to find the optimal operation points of home energy appliances, with the aim of minimizing the customer electricity bill and peak load while maintaining the customer's convenience. The proposed tri-objective problem is solved by using the Epsilon-constraint optimization method. In addition, a prior approach is adopted to manage the tradeoff among objectives according to the customer's preferences. The desirable customer preferences are ensured through the prior approach by restricting two objectives to specific values, then finding an optimal corresponding solution for them. The findings and discussion demonstrate that the proposed SaMODE strategy benefits the customers by reducing their energy bill for up to 53.34% with an acceptable level of inconvenience. On the other hand, the power suppliers have also received benefits by decreasing the peak energy demand by up to 43.04%, which in turn increases the stability of the power system and lessen the burden on the utility company.

Adaptive selection multi-objective optimization method for hybrid flow shop green scheduling under finite variable parameter constraints: case study

The energy consumption loss is high particularly in manufacturing processes involving heating furnaces. Moreover, the mandatory constraints in continuous heating stage bring difficult challenges to production scheduling. To improve the production efficiency and reduce the energy consumption in a hybrid flow shop with continuous and discrete processing stages, this study developed an adaptive selection multi-objective optimization algorithm with preference (ASMOAP). The mandatory constraints of continuous processing stage are transformed into one of the optimization objectives, which is defined as maximum excess value of adjustment time in this paper. A multi-objective optimization scheduling model with the makespan, energy consumption, and maximum excess of adjustment time is established. The optimization preference is designed in the proposed multi-objective optimization algorithm. The maximum excess of adjustment time is set as the multi-objective optimization preference. Three adaptive selection strategies are designed for the algorithm based on the proportions of outstanding and preference individuals to eliminate constraint conflicts. Presented results prove that the proposed algorithm could effectively solve hybrid flow shop scheduling problem considering discrete and continuous processing stages with limited time. It can be applied to obtain a better feasible solution while improving the efficiency and reducing the energy consumed in practical production processes.

ACMo: Angle-Calibrated Moment Methods for Stochastic Optimization

Stochastic gradient descent (SGD) is a widely used method for its outstanding generalization ability and simplicity. Adaptive gradient methods have been proposed to further accelerate the optimization process. In this paper, we revisit existing adaptive gradient optimization methods with a new interpretation. Such new perspective leads to a refreshed understanding of the roles of second moments in stochastic optimization. Based on this, we propose Angle-Calibration Moment method (ACMo), a novel stochastic optimization method. It enjoys the benefits of second moments with only first moment updates. Theoretical analysis shows that ACMo is able to achieve the same convergence rate as mainstream adaptive methods. Experiments on a variety of CV and NLP tasks demonstrate that ACMo has a comparable convergence to state-of-the-art Adam-type optimizers, and even a better generalization performance in most cases. The code is available at https://github.com/Xunpeng746/ACMo.

Signal estimation and filtering from quantized observations via adaptive stochastic resonance.

Using a gradient-based algorithm, we investigate signal estimation and filtering in a large-scale summing network of single-bit quantizers. Besides adjusting weights, the proposed learning algorithm also adaptively updates the level of added noise components that are intentionally injected into quantizers. Experimental results show that minimization of the mean-squared error requires a nonzero optimal level of the added noise. The process adaptively achieves in this way a form of stochastic resonance or noise-aided signal processing. This adaptive optimization method of the level of added noise extends the application of adaptive stochastic resonance to some complex nonlinear signal processing tasks.

Intelligent Wavelet Based Pre-Filtering Method for Ultrasonic Sensor Fusion Inverse Algorithm Performance Optimization

Certain obstacle mapping applications require the live evaluation of the measured data to prevent collision with obstacles. The fusion of different or similar sensors usually has a high calculation demand, which increases significantly with the area to be evaluated and the number of sensors. In the present considerations, we propose a wavelet-based adaptive optimization method, which can greatly decrease the number of grid points to be evaluated, and thus the necessary computation time. The basis of the method is to use the fact that the areas to be evaluated mostly face a rather small number of obstacles, which cover a smaller percentage of the whole environment. The first step in a pre-filtering process is the determination of the zones where no obstacles are present. This step can already result in a considerable decrease in the computation time, however with the transformation to polar coordinates, the method will not only be more fitted to the problem to be solved, but the area of the evaluation can also be increased with the same number of grid points. As a last step, we applied wavelet transformation to identify the regions of interest, where the application of a refined raster is necessary, and thus further decreasing the number of grid points where the calculation has to be carried out. We used our previously developed probability-based ultrasonic sensor fusion inverse algorithm to demonstrate the efficiency of the proposed method.

Path planning of surgical needle: A new adaptive intelligent particle swarm optimization method

Percutaneous puncture interventional therapy is an important method for pathological examination, local anesthesia, and local drug delivery in modern clinics. Due to the existence of complex obstacles such as nerves, arteries, bones and so on in the puncture path, it is a challenging work to design the optimal path for surgical needle. In this paper, we propose a new path planning method based on the adaptive intelligent particle swarm optimization (PSO) algorithm with parameter adjustment mechanism. First, force and motion analysis are carried out on the bevel-tip flexible needle after piercing into human tissues, the motion model of the needle and the spatial transformation model of puncture route in three-dimensional space are obtained, respectively. Then, a multi-objective function is established, which includes puncture path length function, puncture error function and collision detection function. Finally, the optimal puncture path is obtained based on the adaptive intelligent PSO algorithm. The simulation results show that the newly proposed path planning method has higher efficiency, better adaptability to complex environments and higher accuracy than other path planning methods in literature.

Intelligent injection molding: Parameters self‐learning optimization using iterative gradient‐approximation adaptive method

AbstractIntelligent injection molding consists of three aspects: intelligent parameter optimization, process monitoring and control. The optimal process parameters are critical to guarantee product quality. Injection molding is a typical batch process and has the property that previous runs can provide feedback to optimize subsequent runs. This study proposes a self‐learning parameter optimization method named iterative gradient‐approximation adaptive optimization (IGAO) method, which adopts the batch‐to‐batch information to remove the need to establish optimization model with large numbers of experiments. The analysis and the optimization of the parameters can be performed simultaneously. The IGAO method approximates the gradient iteratively and assigns an adaptive step size to each parameter according to gradient accumulation. The experiments conducted in both simulation software and injection molding machine prove that the method has fast convergence speed. Standard product weight can be obtained within 11 runs from three different starting process parameters. Experiment results show that 25% less steps are needed compared with the traditional gradient descent method. The method also has good stability to resist disturbances during the optimization procedure. In general, the proposed IGAO method is fast, stable and robust, and it has good prospects for parameter optimization in batch processes.

Inverse Optimal Design of Direct Adaptive Fuzzy Controllers for Uncertain Nonlinear Systems

Optimized performance obtained from existing adaptive fuzzy optimal control methods comes at the cost of a intricate design procedure and a heavy computation of online parameter learning, and it is an under-explored problem on how to remove such a restriction. In this article, we tackle this problem and ensure the optimized performance using only one adaptive parameter. To this end, a direct adaptive fuzzy inverse approach is first proposed to design a switching-type inverse optimal controller and a one-parameter learning mechanism. It is proved that the proposed approach ensures the input-to-state stability of the control system and besides, the inverse optimality in regard to a meaningful cost functional is achieved. Illustrative examples verify the approach developed.

Simulation physics-informed deep neural network by adaptive Adam optimization method to perform a comparative study of the system

Taking the marvelous advantages of artificial intelligence (AI) in accelerating the procedure of finding a solution to different engineering analyses is the main motivation of this article to establish a non-model-based mechanism on the basics of fully connected deep neural networks (FC-DNN) to analyze the hygro-thermomechanical buckling response of the multiscale hybrid composite MHC doubly curved panel. First, the system's buckling response at its design points is obtained by applying DQM to motion equations developed based upon the refined-form of third-order shear deformation theory (TSDT). Then the obtained information would be transferred to DNN to acquire the regressor system. Finding the optimal values of weights and biases of the DNN is the key factor to provide an AI system with high-accuracy prediction. For this reason, the adaptive Adam optimization approach is chosen due to its phenomenal speed as well as lower computational costs.

Vague Adaptive Optimization Based Research on Western Guangdong Province Favorable Natural Rubber Species

Apply vague adaptive optimization method to find new natural rubber species. It is advisable to accelerate the promotion of wind-resistant and high-yield natural rubber species’ plantation in the dominant areas of western Guangdong Province, which are Reyan 7-33-97, Yunyan 77-2, Dafeng-95, Yunyan 77-4, Haiken 1, PR 107, RRIM 600, RRII 208 and other species.

An Adaptive Optimization Method Based on Learning Rate Schedule for Neural Networks

Artificial intelligence (AI) is achieved by optimizing the cost function constructed from learning data. Changing the parameters in the cost function is an AI learning process (or AI learning for convenience). If AI learning is well performed, then the value of the cost function is the global minimum. In order to obtain the well-learned AI learning, the parameter should be no change in the value of the cost function at the global minimum. One useful optimization method is the momentum method; however, the momentum method has difficulty stopping the parameter when the value of the cost function satisfies the global minimum (non-stop problem). The proposed method is based on the momentum method. In order to solve the non-stop problem of the momentum method, we use the value of the cost function to our method. Therefore, as the learning method processes, the mechanism in our method reduces the amount of change in the parameter by the effect of the value of the cost function. We verified the method through proof of convergence and numerical experiments with existing methods to ensure that the learning works well.

Multi-objective two-stage adaptive robust planning method for an integrated energy system considering load uncertainty

An integrated energy system (IES) is considered as an effective approach to reduce carbon emissions, lower energy supply costs and increase system flexibility. As numerous energy conversion and storage devices with various features have been developed, determining their types and capacity size and optimizing the synergic action of all selected energy devices under load uncertainty have become challenging issues during the planning and designing of a new IES. To address these issues, a load uncertainty model is constructed by integrating the regional equivalent standard building-integrated load prediction method, the robust uncertainty set method and the fuzzy C-means clustering algorithm. Based on this uncertainty model, this paper proposes a planning method that incorporates the fuzzy multi-objective decision and two-stage adaptive robust optimization methods for handling these challenging issues. The multi-objective decision method enables comprehensive optimization based on the system's economic performance, carbon emissions and grid integration level. The two-stage adaptive robust optimization method aims to optimize the single-objective problem transformed by the multi-objective decision method. The planning method is implemented in an industrial park in the Baoding City of China. The influence of load budget uncertainty, annual load growth rate and energy purchase price is then investigated. The results show that the load budget uncertainty and annual load growth rate only affect the optimal capacity size of the selected devices, whereas the energy purchase price influences both the optimal device types and their capacity size. The optimal capacity size remains constant when the load budget uncertainty exceeds 3. This study provides a methodology to select the suitable device types and determine their capacity size for IES under load uncertainty, which will benefit the design of a new IES.

An adaptive RBF neural network–based multi-objective optimization method for lightweight and crashworthiness design of cab floor rails using fuzzy subtractive clustering algorithm

To improve the computational cost and accuracy of approximate-based multi-objective optimization problems in engineering, an adaptive RBF neural network (ARBFNN) method integrating RBF neural network model, fuzzy subtractive clustering (FSC) sequence sampling method and non-dominated sorting genetic algorithm (NSGA-II) is proposed. To solve the problem that the number of new sample points is difficult to determine, FSC sequence sampling is proposed to select the clustering center points as newly added sample points. First, the ARBFNN method is verified by the five test functions. The results show that the ARBFNN method is significantly better than static RBFNN model based on multi-objective optimization (SRBFNN) in terms of global convergence and efficiency performance, and the other two adaptive approximate optimization methods in terms of overall efficiency. Based on the above, the accuracy and efficiency of ARBFNN method are very high. Finally, the method is applied to an engineering example: the lightweight and crashworthiness design of floor rails. The cab model coupled with implicit parameterized floor rails model is built using the SFE-CONCEPT software to achieve collaborative optimization design of shape-size-material. The optimization results show that the ARBFNN method can guarantee the error of the approximate optimal solution and the finite element solution (expensive solution) within 2%, so the accuracy of highly nonlinear (finite element analysis of collision conditions) approximate optimization is improved. Hence, the proposed ARBFNN method is feasible and effective in solving complex and expensive engineering optimization problems.

Crude palm oil price prediction using multilayer perceptron and long short-term memory

Crude Palm Oil is a leading commodity from Indonesia. Accurate prediction of Crude Palm Oil prices is very important to ensure future prices and help decision making. Study on crude palm oil prices is needed to anticipate fluctuations. In this study, prediction model was made using Multilayer Perceptron and Long Short-Term Memory. The optimization methods in this study are Stochastic Gradient Descent, Root Mean Square Propagation, and Adaptive Moment Estimation. The best model is selected based on Mean Square Error. Based on the results, Long Short-Term Memory model with Adaptive Moment Estimation optimization method is more optimal than Long Short-Term Memory with Stochastic Gradient Descent and Long Short-Term Memory with Root Mean Square Propagation. The prediction results using Long Short-Term Memory with Adam optimization show that the predicted value is not different from the actual value and Mean Absolute Percentage Error is 2.11%. This model has high forecasting accuracy because Mean Absolute Percentage Error is less than 10%.

A Two-stage Adaptive Robust Model for Residential Micro-CHP Expansion Planning

This paper addresses the planning problem of residential micro combined heat and power (micro-CHP) systems (including micro-generation units, auxiliary boilers, and thermal storage tanks) considering the associated technical and economic factors. Since the accurate values of the thermal and electrical loads of this system cannot be exactly predicted for the planning horizon, the thermal and electrical load uncertainties are modeled using a two-stage adaptive robust optimization method based on a polyhedral uncertainty set. A solution method, which is composed of column-and-constraint generation (C&CG) algorithm and block coordinate descent (BCD) method, is proposed to efficiently solve this adaptive robust optimization model. Numerical results from a practical case study show the effective performance of the proposed adaptive robust model for residential micro-CHP planning and its solution method.