Gradient-based Approach Research Articles

A consistent, volume preserving, and adaptive mesh refinement-based framework for modeling non-isothermal gas–liquid–solid flows with phase change

This work expands on our recently introduced low Mach enthalpy method (Thirumalaisamy and Bhalla 2023) for simulating the melting and solidification of a phase change material (PCM) alongside (or without) an ambient gas phase. The method captures PCM’s volume change (shrinkage or expansion) by accounting for density change-induced flows. We present several improvements to the original work. First, we introduce consistent time integration schemes for the mass, momentum, and enthalpy equations, which enhance the method stability. Demonstrating the effectiveness of this scheme, we show that a system free of external forces and heat sources can conserve its initial mass, momentum, enthalpy, and phase composition. This allows the system to transition from a non-isothermal, non-equilibrium, phase-changing state to an isothermal, equilibrium state without exhibiting unrealistic behavior. Furthermore, we show that the low Mach enthalpy method accurately simulates thermocapillary flows without introducing spurious phase changes. To reduce computational costs, we solve the governing equations on adaptively refined grids. We investigate two cell tagging/untagging criteria and find that a gradient-based approach is more effective. This approach ensures that the moving thin mushy region is always captured at fine grid levels, even when it temporarily falls within a subgrid level. We propose an analytical model to validate advanced computational fluid dynamics (CFD) codes used to simulate metal manufacturing processes (welding, 3D printing). These processes involve a heat source (like a laser) melting metal or its alloy in the presence of an ambient (inert) gas. Traditionally, studies relied on artificially manipulating material properties to match complex experiments for validation purposes. Leveraging the analytical solution to the Stefan problem with a density jump, this model offers a straightforward approach to validating multiphysics simulations involving heat sources and phase change phenomena in three-phase flows. Lastly, we demonstrate the practical utility of the method in modeling porosity defects (gas bubble trapping) during metal solidification. A field extension technique is used to accurately apply surface tension forces in a three-phase flow situation. This is where part of the bubble surface is trapped within the (moving) solidification front.

Advanced Applications and Methodologies in Mathematical Optimization and Operations Research: Insights into Linear Programming, Nonlinear Programming, and Decision-Making Frameworks

Mathematical optimization and operations research are pivotal disciplines in solving complex decision-making problems across industries. This research delves into advanced methodologies within these fields, with a focus on linear programming (LP), nonlinear programming (NLP), and their applications in optimizing processes and resource allocation. Linear programming, with its capacity to model and solve large-scale problems, remains a cornerstone for optimization, particularly in logistics, finance, and manufacturing. Nonlinear programming, characterized by its ability to handle complex, real-world systems with non-convex functions, expands the scope of optimization to include dynamic and intricate challenges such as energy management and machine learning. This study investigates state-of-the-art algorithms, including interior-point methods, dual simplex techniques, and gradient-based approaches, to enhance the efficiency and accuracy of solutions. By addressing theoretical advancements and practical implementations, this research bridges the gap between mathematical rigor and real-world applications. The insights gained are expected to contribute significantly to operational efficiency, strategic planning, and decision-making in diverse fields.

Advancing AutoML: A Deep Dive into Neural Architecture Search (NAS) and Its Optimization Techniques

Abstract: Neural Architecture Search (NAS) is a pivotal technique in the field of automated machine learning (AutoML), enabling the automatic design of optimal neural network architectures. As deep learning models grow in complexity, NAS offers a scalable approach to improving model performance by exploring vast search spaces of potential architectures. In our research, we investigate the mathematical foundations and algorithms underpinning NAS, focusing on reinforcement learning-based, evolutionary, and gradient-based approaches. We provide mathematical proofs of convergence and efficiency for each method and analyze real-world applications, such as image classification and natural language processing (NLP). Through a comprehensive exploration of NAS, we aim to highlight its impact on AutoML and its potential to automate neural network design effectively while addressing challenges in computational cost and generalization.

Differential evolution-based topology optimization using Karhunen-Loève expansion

Topology optimization is a widely adopted structural optimization approach that aims to maximize desired performance based on mathematical and physical principles. Gradient-based methods are common for obtaining optimal configurations; however, they struggle with problems involving numerous local optima and non-differentiable problems, which prevent the derivation of design sensitivity. To address these issues, this study explores the application of the Success History-Based Differential Evolution with Linear Population Size Reduction and Semi-Parameter Adaptation, which is regarded as LSHADE-SPA algorithm, known for its strong search capabilities. Nevertheless, its computational cost makes it impractical for topology optimization, especially as the number of design variables increases. To overcome this challenge, a novel structure representation method utilizing the Karhunen-Loève (KL) expansion is constructed. Applying truncated KL expansion instead of conventional density methods to represent structural configurations reduces the number of design variables. This article introduces a differential evolution-based topology optimization method utilizing truncated KL expansion and validates its effectiveness through applications in various structural and fluid problems. The method is particularly suited to highly nonlinear problems, such as negative Poisson's ratio structures and micro-mixer designs. The findings here indicate that differential evolution with truncated KL expansion can serve as an efficient and robust topology optimization method, proving particularly effective for solving optimization problems that are challenging for conventional gradient-based approaches. Notably, the proposed methodology not only delivers superior results but also alleviates computational burdens, thereby advancing the prospects of topology optimization in various engineering applications.

Machine learning-based intelligent localization technique for channel classification in massive MIMO

Multiple-input multiple-output (MIMO) technology has been widely adopted in wireless communications, which enables the simultaneous transmission of multiple data streams via multiple transmitting and receiving antennas. In a MIMO system with non-line-of-sight (NLOS), transmitted signals are reflected by various obstacles along the path, reaching the antenna at different angles and times. In 5G networks, the NLOS problem is a major challenge for massive MIMO localization, significantly reducing positioning accuracy. In this work, an intelligent localization technique based on NLOS identification and mitigation is proposed to address this problem. In this solution, a Convolutional Neural Network (CNN) based hybrid Archimedes-based Salp Swarm Algorithm (HASSA) technique is proposed to detect NLOS or the line of sight (LOS) and estimate the location. The accuracy can be analyzed by considering the angle of arrival of signals, threshold-based time of arrival, and time difference of arrival from different antennas. A novel reinforcement learning-based optimization approach is used for the mitigation of NLOS in the radio wave propagation path, which in turn reduces the computational complexity. We use the Ensemble Deep Deterministic Policy Gradient-Based Approach (EDDPG)-based Honey Badger algorithm (HBA) for the aforementioned process. The simulation of this approach assesses diverse scenarios and considers different parameters, and the approach is compared with various state-of-the-art works. From the simulation results, our proposed approach can be used for the identification and detection of LOS and NLOS components and can precisely enhance the localization compared with other approaches.

Fast Convex Optimization via Time Scale and Averaging of the Steepest Descent

In a Hilbert setting, we develop a gradient-based dynamic approach for fast solving convex optimization problems. By applying time scaling, averaging, and perturbation techniques to the continuous steepest descent (SD), we obtain high-resolution ordinary differential equations of the Nesterov and Ravine methods. These dynamics involve asymptotically vanishing viscous damping and Hessian-driven damping (either in explicit or implicit form). Mathematical analysis does not require developing a Lyapunov analysis for inertial systems. We simply exploit classical convergence results for SD and its external perturbation version, then use tools of differential and integral calculus, including Jensen’s inequality. The method is flexible, and by way of illustration, we show how it applies starting from other important dynamics in optimization. We consider the case in which the initial dynamic is the regularized Newton method, then the case in which the starting dynamic is the differential inclusion associated with a convex lower semicontinuous potential, and finally we show that the technique can be naturally extended to the case of a monotone cocoercive operator. Our approach leads to parallel algorithmic results, which we study in the case of fast gradient and proximal algorithms. Our averaging technique shows new links between the Nesterov and Ravine methods. Funding: The research of R.I. Boţ and D.-K. Nguyen was supported by the Austrian Science Fund (FWF), projects W 1260 and P 34922-N.

Lagrangian gradient regression for the detection of coherent structures from sparse trajectory data.

Complex flows are often characterized using the theory of Lagrangian coherent structures (LCS), which leverages the motion of flow-embedded tracers to highlight features of interest. LCS are commonly employed to study fluid mechanical systems where flow tracers are readily observed, but they are broadly applicable to dynamical systems in general. A prevailing class of LCS analyses depends on reliable computation of flow gradients. The finite-time Lyapunov exponent (FTLE), for example, is derived from the Jacobian of the flow map, and the Lagrangian-averaged vorticity deviation (LAVD) relies on velocity gradients. Observational tracer data, however, are typically sparse (e.g. drifters in the ocean), making accurate computation of gradients difficult. While a variety of methods have been developed to address tracer sparsity, they do not provide the same information about the flow as gradient-based approaches. This work proposes a purely Lagrangian method, based on the data-driven machinery of regression, for computing instantaneous and finite-time flow gradients from sparse trajectories. The tool is demonstrated on a common analytical benchmark to provide intuition and demonstrate performance. The method is seen to effectively estimate gradients using data with sparsity representative of observable systems.

Deep Convolutional Tables: Deep Learning Without Convolutions.

We propose a novel formulation of deep networks that do not use dot-product neurons and rely on a hierarchy of voting tables instead, denoted as convolutional tables (CTs), to enable accelerated CPU-based inference. Convolutional layers are the most time-consuming bottleneck in contemporary deep learning techniques, severely limiting their use in the Internet of Things and CPU-based devices. The proposed CT performs a fern operation at each image location: it encodes the location environment into a binary index and uses the index to retrieve the desired local output from a table. The results of multiple tables are combined to derive the final output. The computational complexity of a CT transformation is independent of the patch (filter) size and grows gracefully with the number of channels, outperforming comparable convolutional layers. It is shown to have a better capacity:compute ratio than dot-product neurons, and that deep CT networks exhibit a universal approximation property similar to neural networks. As the transformation involves computing discrete indices, we derive a soft relaxation and gradient-based approach for training the CT hierarchy. Deep CT networks have been experimentally shown to have accuracy comparable to that of CNNs of similar architectures. In the low-compute regime, they enable an error:speed tradeoff superior to alternative efficient CNN architectures.

Towards diagnostics of aerospace structural defects using a novel physics-based post-processing scheme employing lock-in thermography

During its service life, an aircraft experiences various loadings that affect its structural performance. Thin sheet riveted joints are ubiquitous in aircraft and require thorough inspections to arrest any defects that may cause catastrophic failure. Subsequently once flown off, loose and/or cracked rivets may get stuck in critical areas such as flight controls or get ingested into the engine and become a source of Internal Object Damage (IoD). While several inspection schemes are employed in the aviation industry for rivets-related defect detection, most of them are subjective and skill-dependent. Towards the aim of developing an efficient rivet defects diagnostic technique, in this research work, a novel physics-based post-processing scheme employing lock-in thermography (LIT) is proposed for the detection of such defects. For this purpose, a hierarchical experimental thrust has been adopted, where in the beginning the scheme was developed on the lab experiments which was further extended/refined to detect defects on actual aircraft. Two different post-processing techniques have been used for the interpretation of LIT, namely the Fast Fourier Transform (FFT) technique and an indigenously developed temporal temperature gradient-based approach to capture defects whereby localized thermal emissivity variance at the defect location affects the local thermal conductivity and the surface temperature profile. The study considers two common defect i.e., looseness and sub-surface cracks in rivets. The results obtained from the novel post-processing technique provide empirical understanding, better quantification, and clearer comprehension of the two types of rivets-related defects by making it more objective as compared to the FFT technique. The proposed novel methodology in this work exhibits higher fidelity than conventional FFT by accurately identifying the rivet defects thereby providing a promising alternative to in-vogue visual inspections.

Comparative study of kernel- and deep learning-based proxy models for nonlinearly constrained life-cycle production optimization

This study presents a comparative study for the application of deep learning–based and kernel-based proxy models in hydrocarbon production optimization with nonlinear constraints, comparing their performance against high-fidelity simulators (HFS) in terms of computational efficiency and the quality of the results achieved. One of the proxy models employed is referred to as embed to control and observe (E2CO), a deep learning–based model. The other model is a kernel-based proxy model known as least-squares support-vector regression (LS-SVR). Both proxy models can predict production/injection outputs for each well. We use the sequential quadratic programming (SQP) method for nonlinearly constrained production optimization. Our optimization objective revolves around the net present value (NPV), while the nonlinear state constraints involve field liquid production rate (FLPR) and field water production rate (FWPR). NPV, FLPR, and FWPR are estimated using two different types of proxy models. The gradient of the objective function and the Jacobian matrix of constraints are computed analytically for LS-SVR. In contrast, the method of stochastic simplex approximated gradient (StoSAG) is employed for optimization with E2CO and HFS. The reservoir model considered in this study is a two-phase, three-dimensional reservoir characterized by channelized and heterogeneous permeability, sourced from the SPE10 benchmark case. We optimize well controls to maximize NPV in an oil-water waterflooding scenario. All proxy models can achieve optimal NPV results like those obtained through HFS but with significantly lower computational resources. Both proxy models are found to be approximately 15-fold more efficient than using HFS when the number of simulation runs is compared. Forward prediction results show that both proxy models outperform numerical simulator, and the E2CO can predict up to 2400 times faster than HFS whereas speedups gained over HFS go up to 5000-fold with LS-SVR. The training time required for E2CO is at least an order of magnitude longer than that for LS-SVR. The main contributions of the paper are (i) development of a novel methodology that uses an E2CO proxy model within a gradient-based iterative retraining proxy approach for life-cycle production optimization with nonlinear state constraints, (ii) comparison of this methodology in terms of computational cost and accuracy with an LS-SVR proxy model, and (iii) new insights into the accuracy and prediction performances of these machine learning-based proxy models for 3D oil-water systems as well as their efficiency in nonlinearly constrained production optimization for waterflooding applications.

Propagating Transparency: A Deep Dive into the Interpretability of Neural Networks

In the rapidly evolving landscape of deep learning (DL), understanding the inner workings of neural networks remains a significant challenge. This need for transparency and accountability from DL models assumes particular importance as DL models become increasingly prevalent in decision-making processes. Interpreting these models is key to addressing this challenge. This paper offers a comprehensive overview of interpretable deep learning methods. It emphasizes gradient-based propagation techniques that shed light on the complex mechanisms driving neural network predictions. Through a systematic review, we categorize gradient-based interpretability approaches, delve into the theory of notable methods, and compare their strengths and weaknesses. Furthermore, we investigate various evaluation metrics for interpretable systems, often generalized under the term eXplainable Artificial Intelligence (XAI). We highlight their significance in assessing the faithfulness, robustness, localization, complexity, randomization, and adherence to the axiomatic principles of XAI methods. We aim to help researchers and practitioners work towards a more transparent future for artificial intelligence by providing an overview of the most recent developments in the field.

Multiphase flow simulation of fractured karst oil reservoirs applying three-dimensional network models

The characteristics of karst reservoirs are extremely varied and anisotropic, exhibiting notable differences in porosity, permeability, and corresponding fluid flow pathways. Fractured karst petroleum reservoirs, such as distinct caverns and fractures, are an example of a typical discrete media type. The traditional reservoir modeling approach and discrete fracture-like local refinement models are unsuitable for field application of fractured karst oil reservoirs due to the needs of high fidelity geological description and huge computing efforts. Based directly on the spatial characteristics of seismic surveys, a numerical simulation model in three dimensions, akin to a node-like network, is presented here for cracked karst oil reserves. First, the watershed image processing technique and the automatic connection identification procedure are used to extract the three-dimensional node-network model. After that, automatic differentiation is used to build the numerical finite volume scheme, and the proper gradient-based adjoint approach is used to conduct the related historical matching rapidly. After validation by a synthetic model in a commercial simulator, this proposed three-dimensional network numerical model is used for a field reservoir block of deep formation in the Tarim basin to demonstrate its computational efficiency and viability for enormously comparable karst oil reservoirs.

Image interpolation with spiking neural network based pixel similarity

Image interpolation is an important topic in the field of image processing. It is defined as the process of transforming low-resolution images into high-resolution ones using image processing methods. Recent studies on interpolation have shown that researchers are focusing on successful interpolation techniques that preserve edge information. Therefore, the edge detection phase plays a vital role in interpolation studies. However, these approaches typically rely on gradient-based linear computations for edge detection. On the other hand, non-linear structures that effectively simulate the human visual system have gained attention. In this study, a non-linear method was developed to detect edge information using a pixel similarity approach. Pixel similarity-based edge detection approach offers both lower computational complexity and more successful interpolation results compared to gradient-based approaches. 1D cubic interpolation was applied to the pixels identified as edges based on pixel similarity, while bicubic interpolation was applied to the remaining pixels. The algorithm was tested on 12 commonly used images and compared with various interpolation techniques. The results were evaluated using metrics such as SSIM and PSNR, as well as visual assessment. The experimental findings clearly demonstrated that the proposed method outperformed other approaches. Additionally, the method offers significant advantages, such as not requiring any parameters and having competitive computational cost.

Multi-material and thickness optimization of a wind turbine blade root section

Structural optimization has been shown to be an invaluable tool for solving large-scale challenging design problems, and this work concerns such optimization of a state-of-the-art laminated composite wind turbine blade root section. For laminated composites structures, the key design parameters are material choice, fiber orientation, stacking sequence, and layer thickness, however a framework for treating these simultaneously in optimization, on the current wind turbine blade scale, has not been demonstrated. Thus, the motivation and novelty of the present work is providing and demonstrating a general gradient-based approach applicable to wind turbine blades, where the key design parameters and structural criteria, i.e., buckling, static strength, and fatigue damage, are considered for multiple design load cases. The optimization framework is based on a variation of the Discrete Material and Thickness Optimization approach, where the thickness is directly parametrized, allowing for appropriately treating the sandwich parts of the blade. It is demonstrated how optimization leads to a design consisting of complex variable-thickness laminates, a good overall distribution of the structural criteria in the model, and a significant reduction in mass compared to the initial design.

Analytically differentiable metrics for phase stability

In this work, a long-established but sparsely documented method of obtaining semi-analytic derivatives of thermodynamic properties with respect to equilibrium conditions is briefly reviewed and rigorously derived. This procedure is then leveraged to construct general forms of derivatives of the residual driving force, a metric for measuring phase stability used in CALPHAD model optimization, with respect to overall system and individual phase compositions. Applied examples – calculating heat capacity in the Al-Fe system, thermodynamic factors in the Nb-V-W system, and residual driving force derivatives in the Ni-Ti system – demonstrate the versatility, accuracy, and extensibility of this method. Using the developed method, residual driving force gradients can be applied directly in CALPHAD model optimizers, as well as in materials design frameworks, to identify regions of phase stability with an efficient, gradient-based approach.

Robust wind farm layout optimization

Wake interactions in wind farms cause losses in annual energy production (AEP) on the order of 10%. Wind farm designers optimize the layout of the farm to mitigate wake losses, especially in the dominant site-specific wind directions. As wind turbines and wind farms grow in scale, optimization becomes more complex. Offshore wind farms regularly comprise more than 100 wind turbines and are characterized by complex boundaries due to shipping lanes, neighboring wind farms, and other constraints.Layout optimization methods are broadly split between gradient-based and gradient-free approaches. Gradient-based approaches can converge quickly and perform well for smaller, academic problems but are often sensitive to initial conditions and tuning parameters and require expert knowledge to use. On the other hand, gradient-free approaches can be more robust to problem complexities. We present a robust layout optimization approach based on a random search algorithm. The algorithm is intended for those who are not optimization experts and has few tuning parameters that need specification to achieve satisfactory results. Unlike off-the-shelf methods, which use generally available, non-domain-specific optimization routines that accept as inputs an optimization function and constraint definitions, this approach takes advantage of the relative computational costs of the different evaluations by evaluating cheaper computations first (boundary and minimum distance constraints) and running expensive AEP evaluations only if all other checks pass. Moreover, an outer genetic algorithm allows multiple solutions to evolve in parallel, enabling rapid solution development on high-performance computers. We discuss the relative ease of selecting necessary tuning parameters and demonstrate the efficacy of the genetic random search on a complex layout problem consisting of placing 70 turbines in a nonconvex and unconnected boundary region.

Quantum optimization for training quantum neural networks

Training quantum neural networks (QNNs) using gradient-based or gradient-free classical optimization approaches is severely impacted by the presence of barren plateaus in the cost landscapes. In this paper, we devise a framework for leveraging quantum optimization algorithms to find optimal parameters of QNNs for certain tasks. To cast the optimization problem of training QNN into the context of quantum optimization, the parameters in QNN are quantized—moved from being classical to being stored in quantum registers which are in addition to those upon which the QNN is performing its computation. We then coherently encode the cost function of QNNs onto relative phases of a superposition state in the Hilbert space of the QNN parameters. The parameters are tuned with an iterative quantum optimization structure using adaptively selected Hamiltonians. The quantum mechanism of this framework exploits hidden structure in the QNN optimization problem and hence is expected to provide beyond-Grover speed up, mitigating the barren plateau issue.

A three-dimensional spring-loaded inverted pendulum walking model considering human movement speed and frequency

The spring-loaded inverted pendulum (SLIP) model is an effective model to capture the essential dynamics during human walking and/or running. However, most of the existing three-dimensional (3D) SLIP model does not explicitly account for human movement speed and frequency. To address this knowledge gap, this paper develops a new SLIP model, which includes a roller foot, massless spring, and concentrated mass. The governing equations-of-motion for the SLIP model during its double support phase are derived. It is noted that in the current formulation, the motion of the roller foot is prescribed; therefore, only the equations for the concentrated mass need to be solved. To yield model parameters leading to a periodic walking gait, a constrained optimization problem is formulated and solved using a gradient-based approach with a global search strategy. The optimization results show that when the attack angle ranges from 68° to 74°, the 3D SLIP model can yield a periodic walking gait with walking speeds varying from 0.5 to 2.0 m s−1. The predicted human walking data are also compared with published experimental data, showing reasonable accuracy.

Random-Effects Substitution Models for Phylogenetics via Scalable Gradient Approximations.

Phylogenetic and discrete-trait evolutionary inference depend heavily on an appropriate characterization of the underlying character substitution process. In this paper, we present random-effects substitution models that extend common continuous-time Markov chain models into a richer class of processes capable of capturing a wider variety of substitution dynamics. As these random-effects substitution models often require many more parameters than their usual counterparts, inference can be both statistically and computationally challenging. Thus, we also propose an efficient approach to compute an approximation to the gradient of the data likelihood with respect to all unknown substitution model parameters. We demonstrate that this approximate gradient enables scaling of sampling-based inference, namely Bayesian inference via Hamiltonian Monte Carlo, under random-effects substitution models across large trees and state-spaces. Applied to a dataset of 583 SARS-CoV-2 sequences, an HKY model with random-effects shows strong signals of nonreversibility in the substitution process, and posterior predictive model checks clearly show that it is a more adequate model than a reversible model. When analyzing the pattern of phylogeographic spread of 1441 influenza A virus (H3N2) sequences between 14 regions, a random-effects phylogeographic substitution model infers that air travel volume adequately predicts almost all dispersal rates. A random-effects state-dependent substitution model reveals no evidence for an effect of arboreality on the swimming mode in the tree frog subfamily Hylinae. Simulations reveal that random-effects substitution models can accommodate both negligible and radical departures from the underlying base substitution model. We show that our gradient-based inference approach is over an order of magnitude more time efficient than conventional approaches.

A Self-Optimized Machine Learning Approach for Constitutive Parameters Identification of Aortic Walls

The accurate identification of constitutive parameters is considered the key challenge for the study of mechanical properties of biological soft tissues. The popular machine learning (ML)-based inverse identification frameworks always require large amounts of training datasets. This work proposes an ML framework called self-optimized ML for accurate identifying the constitutive parameters of aortic walls under few training datasets conditions. The self-optimized ML includes three steps: Step 1: the forward physical FEM models first simulate the nonlinear deformation of aortic walls subject to uniaxial tension tests and are used to establish the nonlinear relationship datasets, Step 2: the carefully designed inverse random forest (RF) of the ML model can offer rapid identification by learning the established nonlinear relationship datasets, and Step 3: forward physical FEM models are recalled to evaluate the error between the identification results in Step 2 and real values, and then the accuracy are embedded into RF for guiding optimization directions to ensure that the final identification results are accurately and physically reasonable. The accuracy and robustness validation of proposed framework was conducted by constitutive parameters identification of uniaxial tension experiment samples of bovine aortic walls. The approach proposed achieves the R-squared exceeding 96.90% in longitudinal direction and 98.30% in circumferential direction, which is better than directly ML approach and gradient-based approach under the same amounts of datasets, respectively. The comparison results show that self-optimized ML can not only achieve accurate identification of the constitutive parameters of aortic walls, but also can decrease the identification results dependency of the initial number of sampling data effectively. The identification approach developed herein provides a common and convenient framework for constitutive parameters identification of biological soft tissues.