Gradient-based Optimization Techniques Research Articles

An efficient method for disaster tweets classification using gradient-based optimized convolutional neural networks with BERT embeddings

Event of the disastrous scenarios are actively discussed on microblogging platforms like Twitter which can lead to chaotic situations. In the era of machine learning and deep learning, these chaotic situations can be effectively controlled by developing efficient methods and models that can assist in classifying real and fake tweets. In this research article, an efficient method named BERT Embedding based CNN model with RMSProp Optimizer is proposed to effectively classify the tweets related disastrous scenario. Tweet classification is carried out via some of the popular the machine learning algorithms such as logistic regression and decision tree classifiers. Noting the low accuracy of machine learning models, Convolutional Neural Network (CNN) based deep learning model is selected as the primary classification method. CNNs performance is improved via optimization of the parameters with gradient based optimizers. To further elevate accuracy and to capture contextual semantics from the text data, BERT embeddings are included in the proposed model. The performance of proposed method - BERT Embedding based CNN model with RMSProp Optimizer achieved an F1 score of 0.80 and an Accuracy of 0.83. The methodology presented in this research article is comprised of the following key contributions:•Identification of suitable text classification model that can effectively capture complex patterns when dealing with large vocabularies or nuanced language structures in disaster management scenarios.•The method explores the gradient based optimization techniques such as Adam Optimizer, Stochastic Gradient Descent (SGD) Optimizer, AdaGrad, and RMSprop Optimizer to identify the most appropriate optimizer that meets the characteristics of the dataset and the CNN model architecture.•“BERT Embedding based CNN model with RMSProp Optimizer” – a method to classify the disaster tweets and capture semantic representations by leveraging BERT embeddings with appropriate feature selection is presented and models are validated with appropriate comparative analysis.

A random walk based methodology to calculate the sensitivity coefficients in inverse radiant boundary design problems

Radiative heating furnaces are widely used for heat treatment applications in manufacturing industries. Strict requirements on heat treatment characteristics of the workpieces require precise control of furnace heater temperatures. Numerical techniques for the solution of inverse heat transfer problems can be used for the optimization of the radiant heater temperatures. In gradient-based optimization techniques, the optimization is carried out using the sensitivity coefficients between the heater and workpiece temperatures. In this study, we present a methodology for calculating the sensitivity coefficients for the solution of inverse boundary design problems involving transient heating of solid workpieces in a three-dimensional radiant furnace. In our methodology, the sensitivity coefficients on the selected design points within the workpieces are calculated by analytical differentiation of design point temperature formulations derived utilizing the floating random walk method and radiative heat flux formulation with exchange factors. The methodology for calculating sensitivity coefficients has been verified through applications to both a one-dimensional scenario and the three-dimensional radiant furnace model. The developed methodology was integrated into a gradient-based optimization algorithm in two test cases, to determine the furnace heater temperatures required to achieve the desired temperature histories at design points within the workpieces. First test case focuses reconstruction of an assumed heater temperature and examines the impact of design point configuration and input data noise on convergence behavior and calculated estimation. The findings reveal that increasing the number of design points in the workpieces increases solution accuracy, with transverse positioning of the design points in the workpieces resulting in heater temperatures closest to the assumed values. Second test case focused on estimating required heater temperatures for spatially uniform heating pattern on the workpieces. With 72 transversely positioned design points, we achieved uniform heating at these points with a maximum relative temperature deviation of 0.26%.

Human-like Learning in Temporally Structured Environments

Natural environments have correlations at a wide range of timescales. Human cognition is tuned to this temporal structure, as seen by power laws of learning and memory, and by spacing effects whereby the intervals between repeated training data affect how long knowledge is retained. Machine learning is instead dominated by batch iid training or else relatively simple nonstationarity assumptions such as random walks or discrete task sequences. The main contributions of our work are: (1) We develop a Bayesian model formalizing the brain's inductive bias for temporal structure and show our model accounts for key features of human learning and memory. (2) We translate the model into a new gradient-based optimization technique for neural networks that endows them with human-like temporal inductive bias and improves their performance in realistic nonstationary tasks. Our technical approach is founded on Bayesian inference over 1/f noise, a statistical signature of many natural environments with long-range, power law correlations. We derive a new closed-form solution to this problem by treating the state of the environment as a sum of processes on different timescales and applying an extended Kalman filter to learn all timescales jointly. We then derive a variational approximation of this model for training neural networks, which can be used as a drop-in replacement for standard optimizers in arbitrary architectures. Our optimizer decomposes each weight in the network as a sum of subweights with different learning and decay rates and tracks their joint uncertainty. Thus knowledge becomes distributed across timescales, enabling rapid adaptation to task changes while retaining long-term knowledge and avoiding catastrophic interference. Simulations show improved performance in environments with realistic multiscale nonstationarity. Finally, we present simulations showing our model gives essentially parameter-free fits of learning, forgetting, and spacing effects in human data. We then explore the analogue of human spacing effects in a deep net trained in a structured environment where tasks recur at different rates and compare the model's behavioral properties to those of people.

공액경사법을 이용한 LPG 운반선 화물창 내부구조물 중량최소화 설계

Since the cargo tank of LPG carriers is made of expensive low-temperature steel that can maintain its function even at -50oC, the reduction of cargo tank materials has been a major concern for shipyards to improve price competitiveness. The purpose of this study is to minimize the weight of the web frames and stringers that form the interior of the cargo tank of the LPG carriers. A total of 92 design variables including 6 shape-related design variables were selected for the minimum weight design. In order to reduce the time required for the optimization process, the conjugate gradient method, which is a gradient-based optimization technique, was adopted. In addition, for the structural analysis, the cargo tank located at the second position in the bow was selected as the structural analysis target, and in order to improve the reliability of the analysis, a boundary condition considering the deformation of the entire ship for each loading condition was applied to the partial model. As a result of the study, it was confirmed that the error rate did not exceed the maximum 3% by comparing the maximum stress value generated in all internal structures for the entire model structure analysis and the partial model one. A total of 92 dimensions with the minimum objective function were determined, and the weight was reduced by about 39 tons per cargo tank.

Information theoretic clustering for coarse-grained modeling of non-equilibrium gas dynamics

We present a new framework towards the objective of learning coarse-grained models based on the maximum entropy principle. We show that existing methods for assigning clusters using the maximum entropy approach are heuristic or sub-optimal. We propose a machine learning framework informed by rate-distortion theory to learn optimal cluster assignments, aiming to improve the effectiveness of the coarse-graining process. Our approach involves transforming the discrete optimization problem into a probabilistic and continuous form. Our inverse modeling approach involves the development of a fully differentiable, adjoint-driven dynamics solver for use with gradient-based optimization techniques. The entire framework is end-to-end differentiable, facilitating backward pass gradient computation to flow through the ODE solve and probabilistic coarse-graining to train a classifier. We demonstrate application in the evolution of particle quantum states in non-equilibrium conditions. The high dimensionality of these equations poses a significant challenge for efficient and accurate computations, even for seemingly simple chemical systems. Training is performed using a loss function informed by rate-distortion theory to cluster the rovibrational states of some molecule, effectively forming a reduced order model of the master equations. We also introduce variable transformations along with several ways of improving the tractability of the training process. These allow the framework to process the extremely high-dimensional discrete optimization problem successfully. Our method is general, and to demonstrate the effectiveness in a controlled setting, we apply it to a one-dimensional Gaussian source quantization problem for which an analytical solution is known, followed by the problem of isothermal relaxation of a ▪ system with O(104) degrees of freedom.

Inversion for Thermal Properties with Frequency Domain Thermoreflectance.

3D integration of multiple microelectronic devices improves size, weight, and power while increasing the number of interconnections between components. One integration method involves the use of metal bump bonds to connect devices and components on a common interposer platform. Significant variations in the coefficient of thermal expansion in such systems lead to stresses that can cause thermomechanical and electrical failures. More advanced characterization and failure analysis techniques are necessary to assess the bond quality between components. Frequency domain thermoreflectance (FDTR) is a nondestructive, noncontact testing method used to determine thermal properties in a sample by fitting the phase lag between an applied heat flux and the surface temperature response. The typical use of FDTR data involves fitting for thermal properties in geometries with a high degree of symmetry. In this work, finite element method simulations are performed using high performance computing codes to facilitate the modeling of samples with arbitrary geometric complexity. A gradient-based optimization technique is also presented to determine unknown thermal properties in a discretized domain. Using experimental FDTR data from a GaN-diamond sample, thermal conductivity is then determined in an unknown layer to provide a spatial map of bond quality at various points in the sample.

An evolutionary crow search algorithm equipped with interactive memory mechanism to optimize artificial neural network for disease diagnosis

Artificial neural network (ANN) is an information processing paradigm that loosely models the thinking patterns of the human brain with specifications such as real-time learning, self-adaption, and self-organization. The learning process of ANNs is complex and tackles shortcomings such as a slow convergence rate, learning time-consuming, and local minimum trapping, especially when using gradient-based optimization techniques. Although many metaheuristics have been proposed to arm ANN and solve these weaknesses, ANN learning still needs solution quality. Therefore, this study proposes an evolutionary crow search algorithm (ECSA) to optimize the hyperparameters of ANNs for diagnosing chronic diseases. ECSA introduces an evolutionary search strategy, self-adaptive adaptive flight length, and an interactive memory mechanism to alleviate the canonical crow search algorithm's shortcomings. The evolutionary search strategy and self-adaptive flight length provide a meaningful search strategy in which crows effectively explore and exploit problem spaces by maintaining population diversity. During the search process, the interactive memory mechanism records the best solution obtained during optimization. The performance of ECSA was evaluated and compared with well-known metaheuristic algorithms in terms of local and global search abilities, local optima avoidance, and convergence speed towards the promising area. Then, the results were statistically analyzed. Ultimately, the effectiveness of an adaptation of ECSA named ECSA-MLP for optimizing hyperparameters of the multilayer perceptron network for diagnosing diseases including coronavirus, breast cancer, diabetes, cardiovascular, cervical cancer, Parkinson's, mammography, and acute inflammation was compared with state-of-the-art competitor algorithms. The experimental results indicated the superiority of ECSA over competitor algorithms in optimizing the network.

On identifying the non-linear dynamics of a hovercraft using an end-to-end deep learning approach

We present the identification of the non-linear dynamics of a novel hovercraft design, employing end-to-end deep learning techniques. Our experimental setup consists of a hovercraft propelled by racing drone propellers mounted on a lightweight foam base, allowing it to float and be controlled freely on an air hockey table. We learn parametrized physics-inspired nonlinear models directly from data trajectories, leveraging gradient-based optimization techniques prevalent in machine learning research. The chosen model structure allows us to control the position of the hovercraft precisely on the air hockey table. We then analyze the prediction performance and demonstrate the closed-loop control performance on the real system.

Adjoint-Based Design Optimization of a Volute for a Radial Compressor

Numerical optimization methods are widely used for designing turbomachinery components due to the cost and time savings they can provide. In the available literature, the shape optimization of radial compressors mainly focuses on improving the impeller alone. However, it is well-established knowledge that the volute plays a key role in the overall performance of the compressor. The aim of the present paper is to perform an adjoint-based optimization of a volute that is designed for the SRV2-O compressor. The CAD model was first created by using the parametrization of 33 design parameters. Then, a butterfly topology was applied to mesh the computational domain with a multi-block structured grid, and an elliptic smoothing procedure was used to improve the quality of the fluid grid. A steady-state RANS CFD solver with a Spalart-Allmaras turbulence model was used to solve the Navier–Stokes equations, and then the flow sensitivities were computed with an adjoint solver. The objective function consists of minimizing the loss coefficient of the volute. The optimization is performed to obtain an improved design with a 14% loss reduction. A detailed flow and design analysis is carried out to highlight the loss reduction mechanisms, followed by the optimizer. Finally, the compressor map of the full stage is compared between the baseline and the optimized volute from the CFD simulations using a mixing plane interface. This research demonstrates the successful use of a gradient-based optimization technique to improve the volute of a radial compressor and opens the door towards simultaneously optimizing the wheel and the volute.

Determination of Optimum Parameter Space of a Fluidic Thrust Vectoring System based on Coanda Effect Using Gradient-Based Optimization Technique

In the realm of aviation, jet propulsion systems serve to provide enhanced maneuverability and to make sure that the aircraft thrust is accurately and precisely regulated during take-off and landing operations. The movement of aerodynamic control surfaces (flaps, slats, elevators, ailerons, spoilers, wing attachments) determines the mobility of practically all aircraft types. Recognized as dependable components in the aviation world for take-off and landing tasks, these control surfaces are being replaced by fluidic thrust vectoring (FTV) systems, especially in small unmanned aerial vehicles (UAVs) and short or vertical take-off and landing aircraft. The FTV system is capable of directing thrust in any preferred direction without the need for any movable components. This paper numerically examines the FTV system by utilizing computational fluid dynamics (CFD) and an optimization technique based on gradients of the system components to understand the physics of the Coanda effect in FTV systems. This research employs gradient-based optimization for nozzle design in order to optimize the parameter space for different velocity ratios (VR) by calculating the moment around the upper Coanda surface, which is used to represent the jet deflection angle. In that context, four different Coanda surface-pintle pair designs for four different VRs are produced. The parameter space shows significant improvement in all four configurations, and results reveal that all output parameters successfully delay separation on the thrust vectoring system's upper Coanda surface. Finally, four optimum design suggestions are tested at various VRs, and the most efficient and proper design is recommended based on output parameters.

Variational sensitivity analysis and shape optimisation applied to a non-local, ductile damage model

AbstractShape optimisation is applied to an elasto-plastic material model with non-local, regularised damage. Geometries of the same volume are generated which behave better under the aspect of damage accumulation. The underlying material model is taken from published literature and enhanced to derive the sensitivities w.r.t the initial reference geometry. A variational approach, together with an enhanced kinematic concept to decouple geometrical and physical quantities, is utilised to derive these sensitivities. With the inclusion of plastic and damaging effects, the load history of the problem has to be taken into account as well. This introduces additional terms for the sensitivity analysis and requires certain adjustments within the numerical treatment. These gradient information can finally be applied in gradient-based optimisation techniques to efficiently solve the stated optimisation problem. The two discussed examples highlight the benefit of damage optimisation. In the first example, a geometry is reshaped to directly reduce the damage accumulation under certain load. In a second example, the results of a compliance based optimisation with the ductile damage material behaviour are compared to shapes resulting from optimisation problems considering elastic and elasto-plastic material behaviour.

Metalens enhanced ray optics: an end-to-end wave-ray co-optimization framework.

We present a fully differentiable framework for seamlessly integrating wave optical components with geometrical lenses, offering an approach to enhance the performance of large-scale end-to-end optical systems. In this study, we focus on the integration of a metalens, a geometrical lens, and image data. Through the use of gradient-based optimization techniques, we demonstrate the design of nonparaxial imaging systems and the correction of aberrations inherent in geometrical optics. Our framework enables efficient and effective optimization of the entire optical system, leading to improved overall performance.

Optimized FACTS Devices for Power System Enhancement: Applications and Solving Methods

The use of FACTS devices in power systems has become increasingly popular in recent years, as they offer a number of benefits, including improved voltage profile, reduced power losses, and increased system reliability and safety. However, determining the optimal type, location, and size of FACTS devices can be a challenging optimization problem, as it involves mixed integer, nonlinear, and nonconvex constraints. To address this issue, researchers have applied various optimization techniques to determine the optimal configuration of FACTS devices in power systems. The paper provides an in-depth and comprehensive review of the various optimization techniques that have been used in published works in this field. The review classifies the optimization techniques into four main groups: classical optimization techniques, metaheuristic methods, analytic methods, and mixed or hybrid methods. Classical optimization techniques are conventional optimization approaches that are widely used in optimization problems. Metaheuristic methods are stochastic search algorithms that can be effective for nonconvex constraints. Analytic methods involve sensitivity analysis and gradient-based optimization techniques. Mixed or hybrid methods combine different optimization techniques to improve the solution quality. The paper also provides a performance comparison of these different optimization techniques, which can be useful in selecting an appropriate method for a specific problem. Finally, the paper offers some advice for future research in this field, such as developing new optimization techniques that can handle the complexity of the optimization problem and incorporating uncertainties into the optimization model. Overall, the paper provides a valuable resource for researchers and practitioners in the field of power systems optimization, as it summarizes the various optimization techniques that have been used to solve the FACTS optimization problem and provides insights into their performance and applicability.

Training material models using gradient descent algorithms

High temperature design requires accurate constitutive models to describe material inelastic deformation and failure behavior. Oftentimes, calibrating accurate models devolves into the problem of fitting the model parameters against experimental test data. Here, we present the pyopmat package, an open source framework for calibrating constitutive models against experiment data subjected to various loading conditions using machine learning techniques. The package calculates the exact gradient of the model response with respect to the parameters using a combination of automatic differentiation and the adjoint method. Given this exact gradient, we compare the performance of several gradient-based optimization techniques in fitting realistic constitutive models against data. We demonstrate the efficiency and accuracy of our package through example problems using both synthetic data, generated using known parameter sets, under monotonic and cyclic loading conditions and also with an example applying the techniques developed here to actual high temperature creep-fatigue test data.

Constant Power Psychoacoustic Spectrum Optimization for Loudness and Sharpness with Application to Vehicle Interiors

<div>Sound quality assessments are an integral part of vehicle design. Especially now, as manufacturers move towards electrification, vehicle sounds are fundamentally changing. By improving the quality of the interior sounds of a vehicle, consumers’ subjective evaluation of it can be increased. Therefore, the field of psychoacoustics, which is the study of human perception of sound, is broadly applicable here. In fact, the perceived quality of a sound signal is influenced by several psychoacoustic indicators, including loudness, sharpness, and roughness. Of particular utility is identifying in advance how to distribute audible frequency content in a way that optimizes psychoacoustic metrics as this can help automotive engineers obtain specific design targets that optimize vehicle noise, vibration, and harshness (NVH).</div> <div>In this article, a novel modified gradient-based optimization technique (MGOT) is developed to optimize psychoacoustic loudness and sharpness. The new technique is applied to identify targeted adjustments to a measured vehicle interior sound signal that keep the signal energy constant but reduce loudness and/or sharpness. The MGOT numerically approximates the objective function gradient for small changes in the signal power distribution for which constant overall signal power is maintained. These gradient calculations identify power spectrum one-third octave band trades that minimize a sound signal metric that is a weighted sum of loudness and sharpness while conserving the total signal power. A trade consists of a reduction of power content from a one-third octave band designated as a source together with a simultaneous addition of that power to another receiver one-third octave band. In the MGOT, a one-third octave band that is at any time identified as a source can never later become a receiver of power. The MGOT results and execution times are compared with two widely available general-purpose optimization routines (a standard gradient-based optimizer and a “genetic,” non-gradient optimizer) are used to achieve identical optimization objectives. In comparison to existing optimization techniques, MGOT is found to identify spectrum modifications that produce a superior minimization of the objective function for comparable or even reduced execution times. The resultant sound spectrum modifications can guide vehicle structural or calibration design recommendations that realize a preferred frequency distribution for enhancing the vehicle driving experience.</div>

Predicting the optimal heating function for uniform exit temperature in a pipe flow

<abstract> <p>The primary goal of this study is to achieve isothermal temperature control of the fluid at the outlet of a circular pipe through a gradient-based method. Typically, temperatures are set on the outer wall of the pipe to warm the fluid and achieve the desired uniformly consistent exit temperature. However, maintaining a constant temperature on the outer wall of the pipe is challenging due to the heat exchange between the fluid and the wall, as well as external environmental factors. Thus, in this current investigation, the temperature distribution is designated along the inner pipe wall to heat the fluid, employing an optimization algorithm to deduce the appropriate optimal heating function on the outer wall. This enables the attainment of the specified temperature on the inner wall. To guarantee consistent temperature control during optimal heating procedures across extensive distances or expansive areas, a gradient-based optimization technique is employed. In the context of a smaller pipe diameter scenario, applying the designated floor function temperature distribution produces the most accurate outcomes for exit temperatures at the pipe exit. The assessed heating function exhibits abrupt surges and disjointed behaviors, which can be alleviated by introducing a weight coefficient into the cost function. This adaptation preserves the precision of exit temperature and its uniformity. In the context of a larger pipe diameter scenario, while adhering to the floor function temperature specification, the estimated exit temperature remains precise, and the uniformity reaches a contentedly acceptable level. This implies that a larger pipe diameter could potentially require a lengthened tube to enhance the uniformity of exit temperatures.</p> </abstract>

Optimisation of panel component regions subject to hot stamping constraints using a novel deep-learning-based platform

The latest hot stamping processes can enable efficient production of complex shaped panel components with high stiffness-to-weight ratios. However, structural redesign for these intricate processes can be challenging, because compared to cold forming, the non-isothermal and dynamic nature of these processes introduces complexity and unfamiliarity among industrial designers. In industrial practice, trial-and-error approaches are currently used to update non-feasible designs where complicated forming simulations are needed each time a design change is made. A superior approach to structural redesign for hot stamping processes is demonstrated in this paper which applies a novel deep-learning-based optimisation platform. The platform consists of the interaction between two neural networks: a generator that creates 3D panel component geometries and an evaluator that predicts their post-stamping thinning distributions. Guided by these distributions the geometry is iteratively updated by a gradient-based optimisation technique. In the application presented in this paper, panel component geometries are optimised to meet imposed constraints that are derived from post-stamping thinning distributions. In addition, a new methodology is applied to select arbitrary geometric regions that are to be fixed during the optimisation. Overall, it is demonstrated that the platform is capable of optimising selective regions of panel component subject to imposed post-stamped thinning distribution constraints.

Broadband acoustic lens design using gradient-based optimization and adjusting radii and positions of scatterers

In this talk, we present the semi-analytical gradient-based optimization (GBO) technique for an efficient design of the broadband acoustic lens. This idea differs from earlier inverse designs that use topology optimization, shape optimization, and generic algorithms. We derived a formula for the gradients of the absolute pressure at the focal point with respect to positions and radii of a set of cylindrical scatterers. The derived analytic form of gradients of absolute pressure complements the modeling when combined with optimization and parallel computing. The GBO algorithm maximizes the sound amplification at focal point by evaluating pressure derivative with respect to the cylinder positions and radii, and then perturbatively optimizing the position and radius of each cylinder in the lens while considering acoustic multiple scattering between the cylinders. Computations are performed using the MultiStart Global optimization solver with fmincon while supplying the gradient of pressure at the focus and the gradient of nonlinear geometrical constraints. The GBO of the broadband acoustic lens is illustrated performing several performance measures for the dependency on the wavenumber and the incidence angle. The method is presented giving numerical examples for non-uniform configurations of the cylindrical structures of various radii and material submerged in water.

Optimisation of deep drawn corners subject to hot stamping constraints using a novel deep-learning-based platform

State-of-the-art hot stamping processes offer improved material formability and therefore have potential to successfully form challenging components. The feasibility of components to be formed through these processes is dependent on their geometric design and its complex interactions with the hot stamping environment. In industrial practice, trial-and-error approaches are currently used to update non-feasible designs where simulation runs are needed each time a design change is made. These approaches make the design process resource intensive and require considerable numerical and process expertise. To demonstrate a superior approach, this study presents a novel application of a deep-learning-based optimisation platform which adopts a non-parametric geometric modelling strategy. Here, deep drawn corner geometries from different geometry subclasses were optimised to minimise wasted volume due to radii while avoiding excessive post-stamping thinning. A neural network was trained to generate families of deep drawn corner geometries where each geometry was conditioned on an input latent vector. Another neural network was trained to predict the thinning distributions obtained from forming these geometries through a hot stamping process. Guided by these distributions, the latent vector, and therefore geometry, was iteratively updated by a new gradient-based optimisation technique. Overall, it is demonstrated that the platform is capable of optimising geometries, irrespective of complexity, subject to imposed post-stamped thinning constraints.

Differentiable Visual Computing: Challenges and Opportunities.

Classical algorithms typically contain domain-specific insights. This makes them often more robust, interpretable, and efficient. On the other hand, deep-learning models must learn domain-specific insight from scratch from a large amount of data using gradient-based optimization techniques. To have the best of both worlds, we should make classical visual computing algorithms differentiable to enable gradient-based optimization. Computing derivatives of classical visual computing algorithms is challenging: there can be discontinuities, and the computation pattern is often irregular compared to high-arithmetic intensity neural networks. In this article, we discuss the benefits and challenges of combining classical visual computing algorithms and modern data-driven methods, with particular emphasis to my thesis, which took one of the first steps toward addressing these challenges.