Gradient Search Method Research Articles

E2SCNet: Efficient Multiobjective Evolutionary Automatic Search for Remote Sensing Image Scene Classification Network Architecture.

Remote sensing image scene classification methods based on deep learning have been widely studied and discussed. However, most of the network architectures are directly reliant on natural image processing methods and are fixed. A few studies have focused on automatic search mechanisms, but they cannot weigh the interpretation accuracy and the parameter quantity for practical application. As a result, automatic global search methods based on multiobjective evolutionary computation have more advantages. However, in the ranking process, the network individuals with large parameter quantities are easy to eliminate, but a higher accuracy may be obtained after full training. In addition, evolutionary neural architecture search methods often take several days. In this article, in order to solve the above concerns, we propose an efficient multiobjective evolutionary automatic search framework for remote sensing image scene classification deep learning network architectures (E2SCNet). In E2SCNet, eight kinds of lightweight operators are used to build a diversified search space, and the coding connection mode is flexible. In the search process, a large model retention mechanism is implemented through two-step multiobjective modeling and evolutionary search, where one step involves the "parameter quantity and accuracy," and the other step involves the "parameter quantity and accuracy growth quantity." Moreover, a super network is constructed to share the weight in the process of individual network evaluation and promote the search speed. The effectiveness of E2SCNet is proven by comparison with several networks designed by human experts and networks obtained by gradient and evolutionary computing-based search methods.

GBRUN: A Gradient Search-based Binary Runge Kutta Optimizer for Feature Selection

<p>Feature selection (FS) is a pre-processing technique for data dimensionality reduction in machine learning and data mining algorithms. FS technique reduces the number of features and improves the model generalization ability. This study presents a Gradient Search-based Binary Runge Kutta Optimizer (GBRUN) for solving the FS problem of high-dimensional. First, the proposed method converts the continuous Runge Kutta optimizer (RUN) into a binary version through S-, V-, and U-shaped transfer functions. Second, a gradient search method is introduced to improve the exploration capability of the algorithm. Five standard datasets provided by Arizona State University’s Data Mining and Machine Learning Lab were selected to verify the performance of the GBRUN algorithm. The experimental results show that GBRUN has better performance than other advanced algorithms regarding classification accuracy and the number of selected features. Moreover, the GBRUN algorithm is also combined with EfficientNet in this manuscript, using the GBRUN algorithm to select the features extracted by EfficientNet. The results show that the V-shaped (GBRUN-V) and U-shaped (GBRUN-U) algorithms have better performance than other algorithms.</p> <p> </p>

Performance Optimization for 5GWireless Network

Fifth Generation (5th G) wireless network is a challengeable research area. As it is intended to serve growing number of subscribers, and to handle various types of data. Therefore, the need for more number of channels and higher performance is increasing. In this paper, we focus on the adopted modulation scheme by 5th G (256 QAM), where the signals set have been arranged among the constellation diagram in order to reach minimum Bit Error Rate (BER) using Gradient Search Method [1]. The novelty of this study comes from adding permutation loop, which is applied on the resulting signals set of Gradient Algorithm, in order to achieve better performance (lower BER).The results show that acceptable BER under given Signal to Noise Ratio (SNR) could be reached with the relatively large amount of data that is required by this digital modulation scheme.

A Quality Prediction and Parameter Optimization Approach for Turbine Blade Multistage Manufacturing

As core components of an aircraft engine, turbine blades are featured by complex curved surface with strict precision requirements in multistage manufacturing processes (MMPs). Hence, reliable quality prediction and parameter optimization strategy on MMPs have long been in great demand. However, most techniques focus on the precision improving of single-stage single-product manufacturing systems; these improvements are difficult to implement in MMPs with coordinate transformations among multiple machines. In this article, we propose a noninvasive feature-based quality prediction and parameter optimization framework for high-precision MMPs. In particular, rather than retrofitting sensors onto machinery after installation, this study presents an executable and noninvasive solution by using coordinate measuring machine (CMM) data as input features for prediction. Based on features of different stages of machining, a sequential multistep deep learning architecture is developed to predict the final product quality. A preheating optimization algorithm, modified from a local gradient search method, is used for the optimization of key initial machining features, which results are further improved by the L-BFGS method, thus increasing the final production quality. To estimate the effectiveness of the proposed framework, experiments are carried out on two series of aviation turbine blades with complex curved surfaces from the Wuxi Turbine Blade Company Ltd. China. CMM data at key stages (i.e., blade root milling and comprehensive accurate milling) are used as inputs to predict the final geometrical errors. Comparison results verify that the proposed framework achieves the smallest root mean square error at 0.030 mm and 0.032 mm for two blades, respectively.

A new conjugate gradient for unconstrained optimization problems and its applications in neural networks

We introduce a novel efficient and effective conjugate gradient approach for large-scale unconstrained optimization problems. The primary goal is to improve the conjugate gradient method's search direction in order to propose a new, more active method based on the modified vector , which is dependent on the step size of Barzilai and Borwein. The suggested algorithm features the following traits: (i) The ability to achieve global convergence; (ii) numerical results for large-scale functions show that the proposed algorithm is superior to other comparable optimization methods according to the number of iterations (NI) and the number of functions evaluated (NF); and (iii) training neural networks is done to improve their performance.

Optimal nonlinear system identification using fractional delay second-order volterra system

The aim of this work is to design a fractional delay second order Volterra filter that takes a discrete time sequence as input and its output is as close as possible to the output of a given nonlinear unknown system which may have higher degree nonlinearities in the least square sense. The basic reason for such a design is that rather than including higher than second degree nonlinearities in the designed system, we use the fractional delay degrees of freedom to approximate the given system. The advantage is in terms of obtaining a better approximation of the given nonlinear system than is possible by using only integer delays ( since we are giving more degrees of freedom via the fractional delays ) and simultaneously it does not require to incorporate higher degree nonlinearities than two. This work hinges around the fact that if the input signal is a decimated version of another signal by a factor of M, then fractional delays can be regarded as delays by integers less than M. Using the well known formula for calculating the discrete time Fourier transform ( TFT ) of a decimated signal, we then arrive at an expression for the DTFT of the output of a fractional delay system in terms of the unknown first and second order Volterra system coefficients and the fractional delays. The final energy function to be minimized is the norm square of the difference between the DTFT of the given output and the DTFT of the output of the fractional delay system. Minimization over the filter coefficients is a linear problem and thus the final problem is to minimize a highly nonlinear function of the fractional delays which is accomplished using search techniques like the gradient-search and nature inspired optimization algorithms. The effectiveness of the proposed method is demonstrated using two nonlinear benchmark systems tested with five different input signals. The accuracy of the stated models using the globally convergent metaheuristic, cuckoo-search algorithm ( CSA ) are observed to be superior when compared with other techniques such as real-coded genetic algorithm ( RGA ), particle swarm optimization ( PSO ) and gradient-search ( GS ) methods. Finally, statistical analysis affirms the potential of the proposed designs for its successful implementation.

Distributed Nash Equilibrium Seeking Strategies Under Quantized Communication

This paper is concerned with distributed Nash equilibrium seeking strategies under quantized communication. In the proposed seeking strategy, a projection operator is synthesized with a gradient search method to achieve the optimization of players' objective functions while restricting their actions within required non-empty, convex and compact domains. In addition, a leader-following consensus protocol, in which quantized information flows are utilized, is employed for information sharing among players. More specifically, logarithmic quantizers and uniform quantizers are investigated under both undirected and connected communication graphs and strongly connected digraphs, respectively. Through Lyapunov stability analysis, it is shown that players' actions can be steered to a neighborhood of the Nash equilibrium with logarithmic and uniform quantizers, and the quantified convergence error depends on the parameter of the quantizer for both undirected and directed cases. A numerical example is given to verify the theoretical results.

GRAPE optimization for open quantum systems with time-dependent decoherence rates driven by coherent and incoherent controls

The GRadient Ascent Pulse Engineering (GRAPE) method is widely used for optimization in quantum control. GRAPE is gradient search method based on exact expressions for gradient of the control objective. It has been applied to various coherently controlled closed and open quantum systems. In this work, we adopt the GRAPE method for optimizing objective functionals in open quantum systems driven by both coherent and incoherent controls. In our case, a tailored or engineered environment acts on the controlled system as control via time-dependent decoherence rates or, equivalently, via spectral density of the environment. To develop the GRAPE approach for this problem, we compute gradient of various objectives for general N-level open quantum systems for the piecewise constant class of control. The case of a single qubit is considered in details and solved analytically. For this case, an explicit analytical expression for evolution and objective gradient is obtained via diagonalization of a matrix determining the system’s dynamics in the Bloch ball. The diagonalization is obtained by solving a cubic equation via Cardano’s method. The efficiency of the algorithm is demonstrated through numerical simulations for the state-to-state transition problem and its complexity is estimated. Robustness of the optimal controls is also studied.

Network revenue management game in the railway industry: Stackelberg equilibrium, global optimality, and mechanism design

Many countries have adopted the vertical separation governance structure in the railway freight industry over the past decades. Under this governance structure, an Infrastructure Manager (IM), which might be an independent company or a government agency, sells train itineraries to Freight Operating Companies (FOCs). After purchasing the itineraries, a FOC will have the rights to run trains on the designated paths at the designated times and thus can provide transport service to shippers. In the process, an IM needs to determine a list of prices for their train itineraries; and a FOC needs to determine which train itineraries to purchase to serve uncertain customer demands based on the IM's price list. This study considers the interaction between an IM and a FOC as a network-based Stackelberg game. Our study first formulates a bi-level optimisation model to determine the equilibrium prices that the IM would charge to maximise its own profits unilaterally without collaboration. A method involving gradient and local search has been developed to solve the bi-level model. Secondly, an inverse optimisation model is proposed to determine the prices leading to global optimality. A Fenchel cutting plane-based algorithm is developed to solve the inverse optimisation model. Thirdly, a subsidy contract is designed for the game to coordinate the players’ decisions. A two-layer gradient search method is developed to determine the optimal subsidy rate. Numerical cases based on the UK rail freight industry data are provided to validate the models and algorithms.

Big data analytics in flexible supply chain networks

Supply chain responsiveness and Big Data Analytics (BDA) have incited an ample amount of interest in academia and among practitioners. This work is concerned with improving responsiveness in supply chain networks by extending production capacity to cope with changes and variations in demand. BDA helps researchers make sense of the current challenges of data: high volume, high velocity, and high variety. In this work, we will look at sales data and at large warehouses, which envelop all the said three characteristics of Big Data (BD). This is quite important as demand market data is increasingly shared with supply chain managers. Here, a working architecture is introduced to handle the challenges of BD. The work uses a neural network to detect patterns within the demand. The work combines deep learning with nonlinear programming to enable flexibility at supply chain production facilities to respond to the forecasted demand. The parameters in the neural network are analyzed and studied for each different product type. We see significant prediction improvements when the parameters are better tuned. Further, the work introduces a BD architecture that automates the acquisition of the data, data mining, and the storage of input and output files. Overall, the work utilizes a gradient search method, a genetic algorithm, ARIMA, a deep learning algorithm, and a mixed-integer nonlinear program.

Tourism Service Scheduling in Smart City Based on Hybrid Genetic Algorithm Simulated Annealing Algorithm

The disruptions in this era have caused a leap forward in information technology being applied in organizations to create a competitive advantage. In particular, we see this in tourism services, as they provide the best solution and prompt responses to create value in experiences and enhance the sustainability of tourism. Since scheduling is required in tourism service applications, it is regarded as a crucial topic in production management and combinatorial optimization. Since workshop scheduling difficulties are regarded as extremely difficult and complex, efforts to discover optimal or near-ideal solutions are vital. The aim of this study was to develop a hybrid genetic algorithm by combining a genetic algorithm and a simulated annealing algorithm with a gradient search method to the optimize complex processes involved in solving tourism service problems, as well as to compare the traditional genetic algorithms employed in smart city case studies in Thailand. A hybrid genetic algorithm was developed, and the results could assist in solving scheduling issues related to the sustainability of the tourism industry with the goal of lowering production requirements. An operation-based representation was employed to create workable schedules that can more effectively handle the given challenge. Additionally, a new knowledge-based operator was created within the context of function evaluation, which focuses on the features of the problem to utilize machine downtime to enhance the quality of the solution. To produce the offspring, a machine-based crossover with order-based precedence preservation was suggested. Additionally, a neighborhood search strategy based on simulated annealing was utilized to enhance the algorithm’s capacity for local exploitation, and to broaden its usability. Numerous examples were gathered from the Thailand Tourism Department to demonstrate the effectiveness and efficiency of the proposed approach. The proposed hybrid genetic algorithm’s computational results show good performance. We found that the hybrid genetic algorithm can effectively generate a satisfactory tourism service, and its performance is better than that of the genetic algorithm.

Structural damage detection based on cross-correlation function with data fusion of various dynamic measurements

To take advantage of various types of dynamic measurement data for structural damage detection, an identification strategy based on cross-correlation function with data fusion of various dynamic measurements was proposed to improve the accuracy of damage identification in the present study. First of all, the cross-correlation functions among the easily acquired strain, acceleration, and displacement responses were theoretically derived when the structure was subjected to ambient excitations. Furthermore, an identification strategy was proposed for structural damage detection with the objective function of minimizing the difference between the measured and computed cross-correlation function under multiple unknown ambient excitations. In the proposed strategy, four optimization methods, namely, gradient search, genetic algorithm, particle swarm optimization, and the hybrid method of particle optimization method and gradient search were applied as the search engine to identify structural unknown damage index. Moreover, the performance of the proposed identification strategy was examined by the numerical studies on a two-dimensional and a three-dimensional truss as well as the experimental study on a cantilever beam. These results showed that the cross-correlation function among different types of vibration measurements could significantly improve the accuracy of the identification results, meanwhile, the proposed strategy exhibited excellent robustness to the measurement noise. In addition, the performance of the proposed strategy with different combinations of vibration data and the influence of reference data on the accuracy of damage identification results were further investigated.

Path Learning in Human–Robot Collaboration Tasks Using Iterative Learning Methods

In a repetitive human–robot collaboration (HRC) task, robots typically are required to learn the intended motion of the human user to improve the collaboration efficiency. However, the human user’s trajectory is of uncertainty when repeating the same task (e.g., human hand tremor and uncertain movement speed), which may directly deteriorate the learning performance. To address this issue, a path characterized by spatial correlation parameters, is of necessity to be learned by robots so that the aforementioned time-related uncertainty will be avoided. In this article, based on the path parameterization, a gradient-based iterative path learning (IPL) strategy is designed for the robot to learn the desired path of human. The proposed IPL strategy draws on the iterative learning methods with a properly designed performance index. Since the gradient of the performance index is hard to directly obtain in HRC, two learning methods with gradient search (GS) and gradient estimation (GE) are developed. The GS estimates the gradient online and has an advantage of easy implementation. By contrast, the advantage of GS is more obvious when the number of learned parameters is considerable due to its high learning efficiency. With these two methods, the unknown path parameters can be iteratively updated toward the desired values. To verify the effectiveness of the proposed IPL algorithm, experiments are carried out. In the experiment, a comparison between GS and GE methods is made to display their respective advantages. Besides, the proposed IPL is compared with an existing trajectory learning method subject to two different kinds of uncertainties and its better learning performance verifies its stability and capability in dealing with uncertainty.

Memory efficient constrained optimization of scanning-beam lithography.

This article describes a memory efficient method for solving large-scale optimization problems that arise when planning scanning-beam lithography processes. These processes require the identification of an exposure pattern that minimizes the difference between a desired and predicted output image, subject to constraints. The number of free variables is equal to the number of pixels, which can be on the order of millions or billions in practical applications. The proposed method splits the problem domain into a number of smaller overlapping subdomains with constrained boundary conditions, which are then solved sequentially using a constrained gradient search method (L-BFGS-B). Computational time is reduced by exploiting natural sparsity in the problem and employing the fast Fourier transform for efficient gradient calculation. When it comes to the trade-off between memory usage and computational time we can make a different trade-off compared to previous methods, where the required memory is reduced by approximately the number of subdomains at the cost of more computations. In an example problem with 30 million variables, the proposed method reduces memory requirements by 67% but increases computation time by 27%. Variations of the proposed method are expected to find applications in the planning of processes such as scanning laser lithography, scanning electron beam lithography, and focused ion beam deposition, for example.

Continuous-variable optimization with neural network quantum states

Inspired by proposals for continuous-variable quantum approximate optimization (CV-QAOA), we investigate the utility of continuous-variable neural network quantum states (CV-NQS) for performing continuous optimization, focusing on the ground state optimization of the classical antiferromagnetic rotor model. Numerical experiments conducted using variational Monte Carlo with CV-NQS indicate that although the non-local algorithm succeeds in finding ground states competitive with the local gradient search methods, the proposal suffers from unfavorable scaling. A number of proposed extensions are put forward which may help alleviate the scaling difficulty.

Policy search with rare significant events: Choosing the right partner to cooperate with.

This paper focuses on a class of reinforcement learning problems where significant events are rare and limited to a single positive reward per episode. A typical example is that of an agent who has to choose a partner to cooperate with, while a large number of partners are simply not interested in cooperating, regardless of what the agent has to offer. We address this problem in a continuous state and action space with two different kinds of search methods: a gradient policy search method and a direct policy search method using an evolution strategy. We show that when significant events are rare, gradient information is also scarce, making it difficult for policy gradient search methods to find an optimal policy, with or without a deep neural architecture. On the other hand, we show that direct policy search methods are invariant to the rarity of significant events, which is yet another confirmation of the unique role evolutionary algorithms has to play as a reinforcement learning method.

Градієнтна оптимізація двопараметричної віконної функції Найквіста для зменшення позасмугогово випромінювання в OFDM системі

The relationship between Nyquist pulses in the time domain and Nyquist window functions used in orthogonal frequency division multiplexing (OFDM) technology has been investigated. One of the ways to approximate the transition region of Nyquist window functions using piecewise linear functions has been analyzed. This method allowed increasing the number of degrees of signal freedom through the introduction of two additional parameters. The spectral density of the selected two-parameter Nyquist window function has been determined. Optimization of the synthesized signal by the criterion of minimum energy in the first side lobes of its spectral density has been performed. To solve the optimization problem, the gradient descent method with minimization of the function at each stage of the iteration by the golden section method is chosen. In the MATLAB programming environment, program has been created that works according to the selected optimization algorithm. With the maximum allowable relative error ε =10–3, the number of iterations of the gradient search method is 2 iterations. It is established that at the optimal values of the signal parameters the amount of energy concentrated in the first three side lobes of its spectral density is 12.6 dB less than for the rectangular window function. The graphs for the synthesized window function with optimal parameters in time and frequency domain are given in this paper. The use of the two-parameter Nyquist window function with reduced level of spectral density of the side lobes allows reducing the level of out-of-band radiation of the OFDM signal and increasing the resistance of the system with many subcarriers to frequency offsets caused by the mutual movement of the transmitter and receiver. It should be noted that the chosen gradient search algorithm is universal and can be used to optimize the Nyquist window functions by different criteria and regardless of the number of signal parameters.

Explorative gradient method for active drag reduction of the fluidic pinball and slanted Ahmed body

We address a challenge of active flow control: the optimization of many actuation parameters guaranteeing fast convergence and avoiding suboptimal local minima. This challenge is addressed by a new optimizer, called the explorative gradient method (EGM). EGM alternatively performs one exploitive downhill simplex step and an explorative Latin hypercube sampling iteration. Thus, the convergence rate of a gradient based method is guaranteed while, at the same time, better minima are explored. For an analytical multi-modal test function, EGM is shown to significantly outperform the downhill simplex method, the random restart simplex, Latin hypercube sampling, Monte Carlo sampling and the genetic algorithm. EGM is applied to minimize the net drag power of the two-dimensional fluidic pinball benchmark with three cylinder rotations as actuation parameters. The net drag power is reduced by 29 % employing direct numerical simulations at a Reynolds number of $100$ based on the cylinder diameter. This optimal actuation leads to 52 % drag reduction employing Coanda forcing for boat tailing and partial stabilization of vortex shedding. The price is an actuation energy corresponding to 23 % of the unforced parasitic drag power. EGM is also used to minimize drag of the $35^\circ$ slanted Ahmed body employing distributed steady blowing with 10 inputs. 17 % drag reduction are achieved using Reynolds-averaged Navier–Stokes simulations at the Reynolds number $Re_H=1.9 \times 10^5$ based on the height of the Ahmed body. The wake is controlled with seven local jet-slot actuators at all trailing edges. Symmetric operation corresponds to five independent actuator groups at top, middle, bottom, top sides and bottom sides. Each slot actuator produces a uniform jet with the velocity and angle as free parameters, yielding 10 actuation parameters as free inputs. The optimal actuation emulates boat tailing by inward-directed blowing with velocities which are comparable to the oncoming velocity. We expect that EGM will be employed as efficient optimizer in many future active flow control plants as alternative or augmentation to pure gradient search or explorative methods.

Layout Optimization of Stiffeners in Heavy-Duty Thin-Plate Box Grider

The optimization of layout and sizes of the stiffeners in heavy-duty box girder could make this kind of the structure more compact and reasonable, which has certain engineering values. In this study, on the basis of the establishment of parametric finite element model, the function approaching method and gradient search method are combined to form a high-precision optimization algorithm, which makes structural analysis be integrated into the optimization process. The optimization takes the type and location of longitudinal stiffening ribs and the thickness and hole position of transverse diaphragms as design variables, the box girder structural behaviors as constraint conditions, and the total volume as objective function. Finally, the weight is reduced by nearly 7%. More importantly, the new asymmetrical layout of the stiffeners is obtained, the distance between the longitudinal stiffening ribs on the main web and the neutral layer is longer than the distance between the longitudinal stiffening ribs on the secondary web and the neutral layer, and the hole position of transverse diaphragms is close to the secondary web. Compared with the current production of symmetrical layout structure, this layout provides a new idea for the design of stiffeners in the bias-rail box girder.

Data‐driven Learning Algorithm of Neural Fuzzy Based Hammerstein‐Wiener System

A novel data‐driven learning approach of nonlinear system represented by neural fuzzy Hammerstein‐Wiener model is presented. The Hammerstein‐Wiener system has two static nonlinear blocks represented by two independent neural fuzzy models surrounding a dynamic linear block described by finite impulse response model. The multisignal theory is designed for employing Hammerstein‐Wiener system to separate parameter learning issues. To begin with, the output nonlinearity parameters are learned utilizing separable signal with different amplitudes. Furthermore, correlation analysis method is implemented for estimating linear block parameters using separable signal inputs and outputs; thereby, the interference of process noise is effectively handled. Finally, multi‐innovation learning technology is introduced to improve system learning accuracy, and then, multi‐innovation extended stochastic gradient algorithm is obtained for optimizing input nonlinearity and noise model using multi‐innovation technique and gradient search method. The simulation results display that presented data‐driven learning approach has the availability of learning Hammerstein‐Wiener system.