Gradient-based Search Method Research Articles

HyGloadAttack: Hard-label black-box textual adversarial attacks via hybrid optimization

Hard-label black-box textual adversarial attacks present a highly challenging task due to the discrete and non-differentiable nature of text data and the lack of direct access to the model’s predictions. Research in this issue is still in its early stages, and the performance and efficiency of existing methods has potential for improvement. For instance, exchange-based and gradient-based attacks may become trapped in local optima and require excessive queries, hindering the generation of adversarial examples with high semantic similarity and low perturbation under limited query conditions. To address these issues, we propose a novel framework called HyGloadAttack (adversarial Attacks via Hybrid optimization and Global random initialization) for crafting high-quality adversarial examples. HyGloadAttack utilizes a perturbation matrix in the word embedding space to find nearby adversarial examples after global initialization and selects synonyms that maximize similarity while maintaining adversarial properties. Furthermore, we introduce a gradient-based quick search method to accelerate the search process of optimization. Extensive experiments on five datasets of text classification and natural language inference, as well as two real APIs, demonstrate the significant superiority of our proposed HyGloadAttack method over state-of-the-art baseline methods.

Towards Efficient Verification of Quantized Neural Networks

Quantization replaces floating point arithmetic with integer arithmetic in deep neural network models, providing more efficient on-device inference with less power and memory. In this work, we propose a framework for formally verifying the properties of quantized neural networks. Our baseline technique is based on integer linear programming which guarantees both soundness and completeness. We then show how efficiency can be improved by utilizing gradient-based heuristic search methods and also bound-propagation techniques. We evaluate our approach on perception networks quantized with PyTorch. Our results show that we can verify quantized networks with better scalability and efficiency than the previous state of the art.

Reusing Convolutional Neural Network Models through Modularization and Composition

With the widespread success of deep learning technologies, many trained deep neural network (DNN) models are now publicly available. However, directly reusing the public DNN models for new tasks often fails due to mismatching functionality or performance. Inspired by the notion of modularization and composition in software reuse, we investigate the possibility of improving the reusability of DNN models in a more fine-grained manner. Specifically, we propose two modularization approaches named CNNSplitter and GradSplitter, which can decompose a trained convolutional neural network (CNN) model for N -class classification into N small reusable modules. Each module recognizes one of the N classes and contains a part of the convolution kernels of the trained CNN model. Then, the resulting modules can be reused to patch existing CNN models or build new CNN models through composition. The main difference between CNNSplitter and GradSplitter lies in their search methods: the former relies on a genetic algorithm to explore search space, while the latter utilizes a gradient-based search method. Our experiments with three representative CNNs on three widely used public datasets demonstrate the effectiveness of the proposed approaches. Compared with CNNSplitter, GradSplitter incurs less accuracy loss, produces much smaller modules (19.88% fewer kernels), and achieves better results on patching weak models. In particular, experiments on GradSplitter show that (1) by patching weak models, the average improvement in terms of precision, recall, and F1-score is 17.13%, 4.95%, and 11.47%, respectively, and (2) for a new task, compared with the models trained from scratch, reusing modules achieves similar accuracy (the average loss of accuracy is only 2.46%) without a costly training process. Our approaches provide a viable solution to the rapid development and improvement of CNN models.

Artificial intelligence for novel fuel design

Co-optimization of fuels and engines can help decarbonize the transportation sector. Traditional methods for fuel-engine co-optimization rely on experimental or theoretical approaches for determining relationships between fuel molecular properties and engine performance. However, these methods can be expensive and time consuming to develop and are prone to errors when there are complex non-linear relationships between molecular structure and combustion properties. Data-driven approaches that rely on artificial intelligence (AI) and machine learning (ML) algorithms are gaining interest for their versatility, adaptability, and ease of use. This paper provides an overview of AI and ML methods and their applications in fuel design. ML algorithms are now routinely developed to predict quantitative structure property relationships (QSPR) for combustion applications. However, molecular representations relying on descriptors are limited to iterative screening of molecules or mixtures for fuel design; they do not enable inverse fuel design wherein a fuel's structure or composition is predicted based on desired target properties. Inverse fuel design is enabled by using AI/ML methods to train QSPR models using molecular graphs and neural networks for a fully differentiable mapping of molecular structure to fuel properties. Gradient-based search methods can then be employed on the latent space representation to generate novel fuel components or mixtures. While AI for fuel design is an evolving domain, the need for uncertainty quantification in training data and model parameters cannot be overlooked. Methods for addressing aleatoric and epistemic are discussed herein together with outlooks for their applications in fuel design.

A Review on the Preliminary Design of Axial and Radial Turbines for Small-Scale Organic Rankine Cycle

Organic Rankine cycle (ORC) is an effective technology to harness low-grade energy. Turbine, as a key component of ORC, takes advantages of its high efficiency and compact size compared with other expanders. Currently, developing suitable turbines with a high performance and a low cost is one of the bottlenecks for wide applications of various ORCs. In this context, technical progress on radial inflow turbines (RITs), axial turbines (ATs), and radial outflow turbines (ROTs) is introduced, and loss models used in the preliminary design are compared, especially for small-scale ORCs. RIT is recommended for medium and small ORCs with an expansion pressure ratio of <10. The power outs and rotational speeds of the designed RITs spanned the ranges of 9.3–684 kW and 3000–114,000 r/min with an efficiency of 56.1–91.75%. In comparison, the power outputs and speeds of ATs were 3–2446 kW and 3000–91,800 r/min with an efficiency of 63–89.1%. AT is suitable for large-scale ORCs with a power output of greater than hundreds of kW. However, AT with impulse stages is feasible for small-scale ORCs when the pressure ratio is high, and the mass flow rate is small. The power outputs of the designed ROTs were relatively small, at 10–400 kW with a speed of 7200–42,700 r/min and an efficiency of 68.7–85%. For organic working fluids with a large expansion pressure ratio, ROT might be employed. Conventional mean-line models may neglect the effects of supersonic flow, which will be encountered in many ORC turbines. Therefore, adequate models for supersonic expansion loss and shock loss need to be added. Meanwhile, a proper multivariable optimization algorithm such as a gradient-based or stochastic search method should be selected. Finally, the challenges and potential research directions are discussed. The outcomes can provide some insights for the development of ORC turbines and the optimization of ORC systems.

Offline and online parameter estimation of nonlinear systems: Application to a solid oxide fuel cell system

In this paper, an offline tuning strategy and an online parameter estimation method are exploited to calibrate the solid oxide fuel cell mathematical model. Different to existing offline tuning strategy, the developed strategy is designed in order to tune the model under various operation conditions. First, the particle swarm optimization method combined with the gradient-based search method is applied to tune unknown parameters in the state-space model and the steady-state model for each operation condition. Then, the sensitive parameters are expanded to the polynomial equations. Moreover, the reconstructed model including coefficients in the polynomial equations are determined by using the particle swarm optimization method with gradient-based search method for whole operation conditions. To show the slowly time-varying performance of a solid oxide fuel cell, an adaptive optimal learning law based on the optimization technology is proposed to online minimize a cost function with the information of the estimation error. The estimation error is extracted through several low-pass filters and simple algebraic calculation. Finally, the proposed offline tuning strategy and the developed online adaptive estimation method are verified by conducting experiments on a practical solid oxide fuel cell test bench.

Local and Global Linear Convergence of General Low-Rank Matrix Recovery Problems

We study the convergence rate of gradient-based local search methods for solving low-rank matrix recovery problems with general objectives in both symmetric and asymmetric cases, under the assumption of the restricted isometry property. First, we develop a new technique to verify the Polyak-Lojasiewicz inequality in a neighborhood of the global minimizers, which leads to a local linear convergence region for the gradient descent method. Second, based on the local convergence result and a sharp strict saddle property proven in this paper, we present two new conditions that guarantee the global linear convergence of the perturbed gradient descent method. The developed local and global convergence results provide much stronger theoretical guarantees than the existing results. As a by-product, this work significantly improves the existing bounds on the RIP constant required to guarantee the non-existence of spurious solutions.

Active Learning for Well Control Optimization with Surrogate Models

SummaryOptimization of well controls in waterflooding operations has been established as an effective strategy to increase oil recovery efficiency and to reduce operating costs. Optimization problems, however, entail hundreds of full-scale reservoir simulation runs, which can be computationally expensive and impractical for field-scale applications. Fit-for-purpose surrogate models present more efficient alternatives that can be deployed for faster implementation and to enable real-time applications. In recent years, machine learning (ML)-based surrogate models have gained traction in building efficient proxy models to facilitate reservoir engineering and management workflows. For high-dimensional control variables that are frequently encountered in life cycle optimization, gradient-based local search methods have proven to be the optimization algorithms of choice, especially when adjoint models are available to provide the required gradient information in a computationally efficient manner. The use of local search optimization techniques implies that the surrogate models should be locally accurate and adapted to the optimization path that is traversed during the iterations. The adaptation can be accomplished through active learning, where the training of the surrogate model is performed online (throughout the iterations) and tied to the optimization iterations. This paper presents an active learning framework that is tuned to the local search nature of the optimization problem by updating an initially trained surrogate model based on new samples that are generated during the optimization iterations. Specifically, at each iteration, the full reservoir simulation model is performed to evaluate the true objective function and provide the new simulated data to update the surrogate model, thereby enhancing its local prediction accuracy. As the optimization progresses, older samples that become less relevant to the solution at the current iteration are dropped to preserve the local nature of the surrogate model. Various aspects of designing an active learning-based proxy model for well control optimization are discussed and the framework is applied to waterflooding problems to demonstrate its efficiency and superior performance compared to the traditional offline training approach.

A local search-based many-objective five-element cycle optimization algorithm

The conception of memetic algorithms (MAs) has provided a new perspective in algorithmic design through hybridizing and combining of local search techniques and population-based search. Previous research has proven that the performance of some multi-objective evolutionary algorithms (MOEAs) could be improved by hybridization with local search strategy. However, when designing many-objective evolutionary optimization algorithms, if the ranking scheme relies on the Pareto dominance relation, it is quite challenging to evaluate solutions with several objective values and distinguish the most effective solutions from a population full of “the first rank” solutions. In this study, a new many-objective five-element cycle optimization algorithm based on a gradient-based local search method (termed LSMaOFECO) is proposed. In the evolutionary search part of LSMaOFECO, within the five-element cycle model, the forces exerted on every individual on different objective dimensions are summed up as an evaluating criterion to decide whether a solution should be accepted directly or updated. In the selection phase, this evaluation method is used to choose the next generation from the parents and the offspring when the dominance relation ranking loses effectiveness. In the local search part, three search tactics are proposed, among which the worst-objective-search tactic outperforms the other two in the experiment and is subsequently used in the updating phase of LSMaOFECO. The performance of the proposed algorithm is compared with six state-of-the-art multi- and many-objective algorithms by solving a set of many-objective test problems. The results obtained verify the utility of the LSMaOFECO in solving many-objective optimization problems (MaOPs).

Elastic wave full-waveform inversion in the time domain by the trust region method

In this paper, we investigate the elastic wave full-waveform inversion (FWI) by the trust region method. FWI is an optimization problem of minimizing the misfit between the observed data and simulated data. Usually the gradient-based search methods such as the steepest descent method or quasi-Newton's methods are applied to update the model parameters iteratively. And the L-BFGS method with approximated Hessian information is preferred for its high computational efficiency and accuracy. At each iteration, a line search method computes a search direction and then finds a suitable step length or how far to move along the direction. The trust region method is an effective alternative to solve nonlinear optimization problems. In this paper, we investigate the performance of the trust region method in elastic time-domain FWI. In the trust region method, a trial step length called the trust-region radius is forced within a certain neighborhood of the current iterate point and a trust-region subproblem requires to solve. In order to solve the trust-region subproblem, the dogleg method and the two dimensional subspace method are adopted in this paper. In the trust region method, as long as the trust-region radius is well updated, the model updating can be performed for every iteration with the fixed trial step and there is no more extra computation for forward problem like the line search method. Numerical computations for the benchmark Marmousi model are given. The inversion results and comparisons show that the trust region method is very efficient and behaves better than the line search method such as the well-known L-BFGS method.

Fisher Task Distance and its Application in Neural Architecture Search

We formulate an asymmetric (or non-commutative) distance between tasks based on Fisher Information Matrices, called Fisher task distance. This distance represents the complexity of transferring the knowledge of one task to another. We provide a proof of consistency for our distance through theorems and experiments on various classification tasks from MNIST, CIFAR-10, CIFAR-100, ImageNet, and Taskonomy datasets. Next, we construct an online neural architecture search framework using the Fisher task distance, in which we have access to the past learned tasks. By using the Fisher task distance, we can identify the closest learned tasks to the target task, and utilize the knowledge learned from these related tasks on the target task. Here, we show how the proposed distance between a target task and a set of learned tasks can be used to reduce the neural architecture search space for the target task. The complexity reduction in search space for task-specific architectures is achieved by building on the optimized architectures for similar tasks instead of doing a full search and without using this side information. Experimental results for tasks in MNIST, CIFAR-10, CIFAR-100, ImageNet datasets demonstrate the efficacy of the proposed approach and its improvements, in terms of the performance and the number of parameters, over other gradient-based search methods, such as ENAS, DARTS, PC-DARTS.

A Gradient-Based Search Method for Multi-objective Optimization Problems

A gradient-based search method (GBSM) is developed to solve multi-objective optimization problems. It uses the multi-objective gradient information to construct descent directions, i.e., Pareto descent directions (PDDs), to accelerate the convergence. In addition, a multi-objective evolutionary algorithm based on decomposition is adopted to improve the diversity. The comparisons between GBSM with several selected multi-objective evolutionary algorithms and gradient based algorithms on benchmark functions indicate that the proposed method performs competitively and effectively.

Tuning idiophone bar torsional modes with three-dimensional cutaway geometries

The bars of marimbas, vibraphones and similar idiophones are tuned by shaping their “cutaway” or “undercut” geometries. Makers commonly shape bar cutaways to tune three flexural modes of vibration. The first flexural mode is tuned to the fundamental frequency of the bar’s musical note. Two other flexural modes are tuned such that their frequencies become specific multiples of the fundamental. The remaining flexural modes and all other modes are left untuned. Makers have complained of untuned torsional modes polluting bar timbre over specific sections of the keyboard. In wooden marimba bars this problem has proven difficult to predict, filling reject bins with valuable tonewood. This work investigates tuning these torsional modes by varying bar cutaway geometry in three dimensions. No additional mass or heterogeneous materials are employed. Modal frequencies are determined via finite element analysis. Mode shapes are identified algorithmically, enabling analyses to explore the parameter space unsupervised. Geometries are tuned using a gradient-based search method. The approach, designed to efficiently solve underdetermined systems with multiple objectives, is readily applicable to other problems. A selection of tuned example models will be showcased, including rosewood and aluminum bars with common and uncommon tuning ratios.

Accelerating AutoDock4 with GPUs and Gradient-Based Local Search.

AutoDock4 is a widely used program for docking small molecules to macromolecular targets. It describes ligand-receptor interactions using a physics-inspired scoring function that has been proven useful in a variety of drug discovery projects. However, compared to more modern and recent software, AutoDock4 has longer execution times, limiting its applicability to large scale dockings. To address this problem, we describe an OpenCL implementation of AutoDock4, called AutoDock-GPU, that leverages the highly parallel architecture of GPU hardware to reduce docking runtime by up to 350-fold with respect to a single-threaded process. Moreover, we introduce the gradient-based local search method ADADELTA, as well as an improved version of the Solis-Wets random optimizer from AutoDock4. These efficient local search algorithms significantly reduce the number of calls to the scoring function that are needed to produce good results. The improvements reported here, both in terms of docking throughput and search efficiency, facilitate the use of the AutoDock4 scoring function in large scale virtual screening.

Pyrolysis modeling, sensitivity analysis, and optimization techniques for combustible materials: A review

Over the past years, pyrolysis models have moved from thermal models to comprehensive models with great flexibility including multi-step decomposition reactions. However, the downside is the need for a complete set of input data such as the material properties and the parameters related to the decomposition kinetics. Some of the parameters are not directly measurable or are difficult to determine and they carry a certain degree of uncertainty at high temperatures especially for materials that can melt, shrink, or swell. One can obtain input parameters by searching through the literature; however, certain materials may have the same nomenclature but the material properties may vary depending on the manufacturer, thereby inducing uncertainties in the model. Modelers have resorted to the use of optimization techniques such as gradient-based and direct search methods to estimate input parameters from experimental bench-scale data. As an integral part of the model, a sensitivity study allows to identify the role of each input parameter on the outputs. This work presents an overview of pyrolysis modeling, sensitivity analysis, and optimization techniques used to predict the fire behavior of combustible solids when exposed to an external heat flux.

Analytical modeling of laminated composite plates using Kirchhoff circuit and wave digital filters

A physical-based numerical algorithm using Kirchhoff circuit is detailed for modelling the free vibration of moderate thick symmetrically laminated plates based on the first order shear deformation theory (FSDT). With the help of multidimensional passivity of analog circuit and nonlinear optimization solvers, the philosophy gives rise to a nonlinear programming (NLP) model that can apply further to explore stability characteristics and optimum performance of the resultant multidimensional wave digital filtering network representing the FSDT plate. Various optimization methods exploiting gradient-based and direct search methods are adopted with efficient broad search power to tackle the NLP model. As a result, the necessary Courant-Friedrichs-Levy stability criterion can be fully satisfied at all time with least restriction on the spatially discretized geometry of the scattering problem. With full stability guaranteed, the waveform is analyzed by the power cepstrum for spectra peaks detection, which has led to more accurate estimate of various vibration effects in predicting nature frequencies with different fiber orientations, stacking sequences, stiffness ratios and boundary conditions. These results have shown in excellent agreement with early published works based on the finite element solutions of the high-order shear deformation theory and other well known numerical techniques.

Optimization of a thermoelectric generator for heavy-duty vehicles

A thermoelectric generator used to recover energy from the exhaust gases of a heavy-duty freight vehicle is numerically simulated and optimized using two different algorithms: a gradient-based search method, and a global and local direct search method. The model uses a one-dimensional finite volume method to calculate the temperature and the convective heat transfer of the hot and cold fluids along the streamwise direction, as well as the pressure drop, and employs a thermal resistance network and a global energy balance to account for the heat transfer and the thermoelectric effect. Three different heat exchanger configurations are studied with plain, offset strip or triangular fins. The objective function is the net output power, and the geometrical dimensions of the heat exchanger and thermoelectric units are taken as design variables. The results show that the height of the fins, the spacing between them, the height and side length of the cross section of the thermoelectric units, and the distance between the legs are critical parameters in the optimization process. The thickness of the fins, wall ducts, electrical conductor and ceramic strips should be as small as possible, as well as the height of the cooling wall duct. The two optimization methods yield similar optimal solutions.

Exploiting Direct Links in Multiuser Multirelay SWIPT Cooperative Networks With Opportunistic Scheduling

In this paper, we analyze the downlink outage performance of opportunistic scheduling in dual-hop cooperative networks consisting of one source, multiple radio-frequency energy harvesting relays, and multiple destinations. To this end, two low-complexity, suboptimal, yet efficient, relay-destination selection schemes are proposed, namely direct links plus opportunistic channel state information (CSI)-based selection (DOS) scheme and direct links plus partial CSI-based selection (DPS) scheme. Considering three relaying strategies, i.e., decode-and-forward (DF), variable-gain amplify-and-forward (VG-AF), and fixed-gain amplify-and-forward (FG-AF), the performance analysis in terms of outage probability (OP) is carried out for each selection scheme. For the DF and VG-AF strategies, exact analytical expressions and tight closed-form approximate expressions for the OP are derived. For the FG-AF strategy, an exact closed-form expression for the OP is provided. Additionally, we propose a gradient-based search method to find the optimal values of the power-splitting ratio that minimizes the attained OPs. The developed analysis is corroborated through Monte Carlo simulation. Comparisons with the optimal joint selection scheme are performed and it is shown that the proposed schemes significantly reduce the amount of channel estimations while achieving comparable outage performance. In addition, regardless of relaying strategy used, numerical results show that the DOS scheme achieves full diversity gain, i.e., $M+K$ , and the DPS scheme achieves the diversity gain of $M+1$ , where $M$ and $K$ are the numbers of destinations and relays, respectively.

Operation window constrained strategic energy management of microgrid with electric vehicle and distributed resources

In this study, an operation window constrained strategic energy management (OWCSEM) for microgrid operational scheduling is examined. The scheduling problem is formulated as cost minimisation/benefit maximisation of distribution resources. The scheduling operation of distributed resources is evaluated for various perspectives such as customer driven, micro grid operator (MGO) driven and MGO driven with utility constraints. The MGO driven scheduling operation considers the distribution network loss minimisation in addition to economic benefits of distributed resources. The loss estimation of distributed network/microgrid is performed using forward-backward sweep algorithm, considering the ill-natured radial network. Thereafter, the optimisation is performed using modified gradient-based search method, considering the linear nature of the problem. The simulation results are presented and discussed for the three scheduling scenarios proposed under operation window constraints. The customer driven scheduling of distributed resources resulted in highest profit among all the scheduling alternatives. On the other hand, the MGO driven scheduling horizon resulted in lower line loss cost component at a compromise of distribution resource benefit. The utility constrained MGO driven scheduling objective attained a compromise solution in terms of loss component and customer benefit. Thus, it is identified as best practice for overall welfare maximisation of microgrid/distribution network.

Passive tracking in heavy clutter with sensor location uncertainty

In order to address the problem of passive tracking from multiple asynchronous angle-only sensors with location uncertainty in heavy clutter, a new iterative maximum-likelihood probabilistic data-association algorithm is proposed in this paper. An iterative prediction–update framework is adopted in the algorithm to simultaneously estimate the target state as well as the sensor state. At the prediction stage, a deterministic sampling approach is used to adjust the measurement covariance with sensor location uncertainty. Then a two-step grid-search technique is proposed to optimize the log-likelihood ratio, combined with a gradient-based search method. At the update stage, the operational sensor states are updated with target state estimates and measurements in corresponding validation gates. The updated sensor states are used to establish a more accurate log-likelihood ratio in the next iteration, which leads to better parameter estimation. In addition, the effects of the sensor location uncertainty on the track acceptance test and the posterior Cram´er–Rao lower bound are also analyzed. Simulation results show that the proposed method provides a computationally efficient way to improve track initialization performance in heavy clutter with sensor location uncertainty. The proposed work has applications in sonar tracking, geolocation, electronic support measures, and infrared search and tracking systems.