Gradient-based Method Research Articles

Modeling and multi-objective optimization of forward osmosis process

In order to ensure efficient wastewater treatment and seawater desalination, adequate modeling and optimization of the forward osmosis (FO) process has the potential to be very helpful. This paper deals with the FO model parameters calibration and FO process optimization by a gradient-based optimization method. For this purpose, an upgraded FO model, which involves temperature- and agent-dependent parameters, was developed. The FO model calibration was done using NaCl as agents in draw solution, while MgCl2 was used for model validation. The agreements between simulated and measured FO performance were satisfactory; relative index of agreement are higher than 0.99. By using the proposed FO model, the optimization of FO process conditions was performed with various definitions of the objective and constraint functions. In case of maximizing the water flux, minimizing reverse solute flux, and fulfilling the required constraints, the ratio of water flux and reverse solute flux increased up to 40 % for NaCl and up to 20 % for MgCl2; meanwhile the effective osmotic pressure difference was improved 2-times for NaCl and up to 3.8-times for MgCl2. The optimization process proved to be stable and efficient and can easily be adapted or upgraded for more complex dynamic FO modeling.

Maximum likelihood localization of a network of moving agents from ranges, bearings and velocity measurements

Localization is a fundamental enabler technology for many applications, like vehicular networks, IoT, and even medicine. While Global Navigation Satellite Systems solutions offer great performance, they are unavailable in scenarios like indoor or underwater environments, and, for large networks, the instrumentation cost is prohibitive. We develop a localization algorithm from ranges and bearings, suitable for generic mobile networks. Our algorithm is built on a tight convex relaxation of the Maximum Likelihood position estimator. To serve positioning to mobile agents, a horizon-based version is developed accounting for velocity measurements at each agent. To solve the convex problem, a distributed gradient-based method is provided. This constitutes an advantage over centralized approaches, which usually exhibit high latency for large networks and present a single point of failure. Additionally, the algorithm estimates all required parameters and effectively becomes parameter-free. Our solution to the dynamic network localization problem is theoretically well-founded and still easy to understand. We obtain a parameter-free, outlier-robust and trajectory-agnostic algorithm, with nearly constant positioning error regardless of the trajectories of agents and anchors, achieving better or comparable performance to state-of-the-art methods, as our simulations show. Furthermore, the method is distributed, convex and does not require any particular anchor configuration.

Robust and accurate corneal interfaces segmentation in 2D and 3D OCT images.

Segmentation of corneal layer interfaces in optical coherence tomography (OCT) images is important for diagnostic and surgical purposes, while manual segmentation is a time-consuming and tedious process. This paper presents a novel technique for the automatic segmentation of corneal layer interfaces using customized initial layer estimation and a gradient-based segmentation method. The proposed method was also extended to three-dimensional OCT images. Validation was performed on two corneal datasets, one with 37 B-scan images of healthy human eyes and the other with a 3D volume scan of a porcine eye. The approach showed robustness in extracting different layer boundaries in the low-SNR region with lower computational cost but higher accuracy compared to existing techniques. It achieved segmentation errors below 2.1 pixels for both the anterior and posterior layer boundaries in terms of mean unsigned surface positioning error for the first dataset and 2.6 pixels (5.2 μm) for segmenting all three layers that can be resolved in the second dataset. On average, it takes 0.7 and 0.4 seconds to process a cross-sectional B-scan image for datasets one and two, respectively. Our comparative study also showed that it outperforms state-of-the-art methods for quantifying layer interfaces in terms of accuracy and time efficiency.

Reusing Convolutional Neural Network Models through Modularization and Composition

With the widespread success of deep learning technologies, many trained deep neural network (DNN) models are now publicly available. However, directly reusing the public DNN models for new tasks often fails due to mismatching functionality or performance. Inspired by the notion of modularization and composition in software reuse, we investigate the possibility of improving the reusability of DNN models in a more fine-grained manner. Specifically, we propose two modularization approaches named CNNSplitter and GradSplitter, which can decompose a trained convolutional neural network (CNN) model for N -class classification into N small reusable modules. Each module recognizes one of the N classes and contains a part of the convolution kernels of the trained CNN model. Then, the resulting modules can be reused to patch existing CNN models or build new CNN models through composition. The main difference between CNNSplitter and GradSplitter lies in their search methods: the former relies on a genetic algorithm to explore search space, while the latter utilizes a gradient-based search method. Our experiments with three representative CNNs on three widely used public datasets demonstrate the effectiveness of the proposed approaches. Compared with CNNSplitter, GradSplitter incurs less accuracy loss, produces much smaller modules (19.88% fewer kernels), and achieves better results on patching weak models. In particular, experiments on GradSplitter show that (1) by patching weak models, the average improvement in terms of precision, recall, and F1-score is 17.13%, 4.95%, and 11.47%, respectively, and (2) for a new task, compared with the models trained from scratch, reusing modules achieves similar accuracy (the average loss of accuracy is only 2.46%) without a costly training process. Our approaches provide a viable solution to the rapid development and improvement of CNN models.

Local physical gradient-based coupling information interpolate method and application on double-wall turbine blade multidisciplinary analysis

A novel generalization interpolate modification method based on local gradient is presented. The main idea was to remove the sample points located in the gradient direction away from the unknown points compared to the sample points located in the perpendicular direction. This method first uses the Taylor expansion to construct a set of equations and solves the local physical gradient. Subsequently, a scale vector is derived to modify the interpolate space. Three classical interpolation methods were modified: Inverse Distance Weighted (IDW), Radial Basis Function (RBF), and Kriging method. The results of the test functions prove that the proposed method can effectively decrease the interpolate error regardless of low or high dimensional interpolate problem and irrespective of the sample point size. Surface and volume interpolate validations demonstrate that the local gradient generates a large interpolate error, and the modified method can essentially diminish this error with a maximum reduction of 9.94 %. Subsequently, this method is applied to implement a multidisciplinary analysis of a practical double-wall cooling blade with a large physical gradient. The results showed that the pressure and temperature interpolate errors could be decreased by 5.18 % and 14.5 %, respectively, with only an additional 115 s, at the fillet and film hole area where the local gradient was high. When the strength analysis was implemented, the maximum stress error reduction was 25.04 MPa at the stress concentration region.

Manifold-guided multi-objective gradient algorithm combined with adjoint method for supersonic aircraft shape design

The aerodynamic design of supersonic aircraft necessitates the comprehensive consideration of multiple conflicting design objectives and multi-disciplinary characteristics. However, the efficiency of heuristic algorithms in solving multi-objective aerodynamic problems significantly decreases with the increase in the number of design variables and objectives. Although gradient-based adjoint optimization methods can eliminate the constraints of design variables on aerodynamic optimization and have high optimization efficiency, they are prone to getting stuck in local optima and are difficult to apply in multi-objective optimization. To overcome these shortcomings, a manifold-guided multi-objective gradient-based algorithm (MG-MOGBA) is proposed. This algorithm enhances the optimization efficiency of the algorithm by coupling the multi-objective gradient operator with the predicted Pareto front (PF) manifold structure to guide the population search direction and reduce the search space. A multi-objective aerodynamic optimization framework based on the discrete adjoint method is developed to automate the aerodynamic optimization process. Test functions and airfoil optimization results confirm that the proposed algorithm achieves superior results with fewer iterations and is more efficient than heuristic algorithms. The four-objective supersonic civil aircraft example demonstrates that the algorithm can effectively handle high-dimensional multi-objective aerodynamic optimization problems. A potential equilibrium solution increased the cruise factor by 17.5% and 12.8% under supersonic and transonic conditions respectively, reduced the maximum overpressure of the sonic boom by 25.1%, and decreased the root bending moment by 20.3%. Overall, MG-MOGBA can efficiently solve high-dimensional multi-objective aerodynamic optimization problems and has good scalability for the dimensions of design objectives and variables.

Weighted Similarity-Confidence Laplacian Synthesis for High-Resolution Art Painting Completion

Artistic image completion assumes a significant role in the preservation and restoration of invaluable art paintings, marking notable advancements through the adoption of deep learning methodologies. Despite progress, challenges persist, particularly in achieving optimal results for high-resolution paintings. The intricacies of complex structures and textures in art paintings pose difficulties for sophisticated approaches like Generative Adversarial Networks (GANs), leading to issues such as small-scale texture synthesis and the inference of missing information, resulting in distortions in lines and unnatural colors. Simultaneously, patch-based image synthesis, augmented with global optimization on the image pyramid, has evolved to enhance structural coherence and details. However, gradient-based synthesis methods face obstacles related to directionality, inconsistency, and the computational burdens associated with solving the Poisson equation in non-integrable gradient fields. This paper introduces a pioneering approach, integrating Weighted Similarity-Confidence Laplacian Synthesis to comprehensively address these challenges and advance the field of artistic image completion. Experimental results affirm the effectiveness of our approach, offering promising outcomes for the preservation and restoration of art paintings with intricate details and irregular missing regions. The integration of weighted Laplacian synthesis and patch-based completion across multi-regions ensures precise and targeted completion, outperforming existing methods. A comparative analysis underscores our method’s superiority in artifact reduction and minimizing blurriness, particularly addressing challenges related to color discrepancies in texture areas. Additionally, the incorporation of pyramid blending proves advantageous, ensuring smoother transitions and preventing noticeable seams or artifacts in blended results. Based on empirical results, our method consistently outperforms previous methods across both high and low resolutions. Responding to these insights, our approach emerges as an invaluable guide for both curators and artists. The algorithm’s performance yields insights that underscore the central role of thoughtful decision making in the creation of art paintings. This guidance extends to informing choices related to color selection, brushstrokes, and various other elements integral to the artistic process. During the creation phase, employing these insights enables artists and curators to optimize not only the digitization but also the subsequent restoration process. This proves especially vital when dealing with the intricacies involved in physically restoring damaged original art paintings. Importantly, our approach not only streamlines the restoration process but also contributes significantly to the preservation and enhancement of the digital representations of these distinctive and often intricate works of art.

Distributed load identification for hyperelastic plates using gradient-based and machine learning methods

Distributed load identification for hyperelastic plates using gradient-based and machine learning methods

Improvement of Biogas Production Utilizing a Complex Anaerobic Digestion Model and Gradient-Based Optimization

Anaerobic digestion (AD) is a promising technology for renewable energy production from organic waste. In order to maximize the produced biogas quantity and quality, this paper deals with the optimization of the AD process in a CSTR bioreactor of a full-scale biogas plant. For this purpose, a novel approach was adopted coupling, a highly complex BioModel for AD simulation, and a gradient-based optimization method. In order to improve AD performance, the dosages of various types of biological additives, the dosages of inorganic additives, and the temperature in the bioreactor were optimized in three different scenarios. The best biogas quality was obtained using multi-objective optimization, where the objective function involves the following two conflicting objectives: the maximization of biogas production and minimization of the needed heating energy. The obtained results show that, potentially, the content of CH4 can be increased by 11%, while the contents of H2, H2S, and NH3 can be reduced by 30%, 20%, and 81% when comparing the simulation results with the experimental data. The obtained results confirm the usefulness of the proposed approach, which can easily be adapted or upgraded for other bioreactor types.

Automated Equations of State Tuning Workflow Using Global Optimization and Physical Constraints

A computational model that can accurately describe the thermodynamics of a hydrocarbon system and its properties under various conditions is a prerequisite for running reservoir and pipeline simulations. Cubic Equations of State (EoS) are mathematical tools used to model the phase and volumetric behavior of reservoir fluids when compositional effects need to be considered. To anticipate uncertainty and enhance the quality of their predictions, EoS models must be adjusted to adequately match the available lab-measured PVT values. This task is challenging given that there are many potential tuning parameters, thus leading to various tuning results of questionable validity. In this paper, we present an automated EoS tuning workflow that employs a Generalized Pattern Search (GPS) optimizer for efficient tuning of a cubic EoS model. Specifically, we focus on the Peng–Robinson (PR) model, which is the oil and gas industry standard, to accurately capture the behavior of diverse multicomponent, complex hydrocarbon mixtures encountered in subsurface reservoirs. This approach surpasses the limitations of conventional gradient-based (GB) methods, which are susceptible to getting trapped in local optima. The proposed technique also allows physical constraints to be imposed on the optimization procedure. A gas condensate and an H2S-rich oil were used to demonstrate the effectiveness of the GPS algorithm in finding an optimized solution for high-dimensional search spaces, and its superiority over conventional gradient-based optimization was confirmed by automatically tracking globally optimal and physically sound solutions.

Improving adversarial transferability through frequency enhanced momentum

The emergence of adversarial examples seriously affects the practical security deployment of convolutional neural networks. The existing attack algorithms perform brilliantly under white-box scenarios, but they show weak transferability when faced with unknown black-box models. Recent studies have revealed that models have different interests in different frequency components of images, and the low-frequency characteristics play a non-negligible role in the decision-making of models. In this article, we present the frequency enhanced momentum iterative attack, called FE-MI-FGSM. Specifically, we use multiple convolution kernels for Gaussian filtering of the image before each gradient update to push the processed image closer to the common decision boundaries of multiple models. Then, the average gradient of the white-box model to these processed images is obtained and used as the perturbation direction to generate adversarial examples with a high success rate of white-box attack and high transferability. The empirical results show that compared with the current mainstream gradient-based methods, our method performs better on both normally trained and adversarially trained models. Besides, our method can combine with gradient-based methods which integrate convergence algorithms or input transformations for the sake of reaching satisfactory improvement of the transferability.

Slow Down to Go Better: A Survey on Slow Feature Analysis.

Temporal data contain a wealth of valuable information, playing an essential role in various machine-learning tasks. Slow feature analysis (SFA), one of the most classic temporal feature extraction models, has been deeply explored in two decades of development. SFA extracts slowly varying features as high-level representations of temporal data. Its core idea of "slow" has been proven to be consistent with the nature of biological vision and beneficial in capturing significant temporal information for various tasks. So far, SFA has evolved into numerous improved versions and is widely applied in many fields such as computer vision, industrial control, remote sensing, signal processing, and computational biology. However, there currently lacks an insightful review of SFA. In this article, a comprehensive overview of SFA and its extensions is provided for the first time. The formulation and optimization of SFA are introduced. Two mainstream solutions, geometric interpretation, and a gradient-based training method of SFA are presented and discussed. Following that, a taxonomy of the current progress of SFA is proposed. We classify improved versions of SFA into six categories, including dual-input SFA (DISFA), online slow feature analysis (OSFA), probabilistic SFA (PSFA), multimode SFA, nonlinear SFA, and discrete labeled SFA. For each category, we illustrate its main ideas, mathematical principles, and applicable scenarios. In addition, the practical applications of SFA are summarized and presented. Finally, we bring new insights into SFA according to its research status and provide potential research directions, which may serve as a good reference for promoting future work.

A Keyframe Extraction Approach for 3D Videogrammetry Based on Baseline Constraints

In this paper, we propose a novel approach for the extraction of high-quality frames to enhance the fidelity of videogrammetry by combining fuzzy frames removal and baseline constraints. We first implement a gradient-based mutual information method to filter out low-quality frames while preserving the integrity of the videos. After frame pose estimation, the geometric properties of the baseline are constrained by three aspects to extract the keyframes: quality of relative orientation, baseline direction, and base to distance ratio. The three-dimensional model is then reconstructed based on these extracted keyframes. Experimental results demonstrate that our approach maintains a strong robustness throughout the aerial triangulation, leading to high levels of reconstruction precision across diverse video scenarios. Compared to other methods, this paper improves the reconstruction accuracy by more than 0.2 mm while simultaneously maintaining the completeness.

Attacking Click-through Rate Predictors via Generating Realistic Fake Samples

How to construct imperceptible (realistic) fake samples is critical in adversarial attacks. Due to the sample feature diversity of a recommender system (containing both discrete and continuous features), traditional gradient-based adversarial attack methods may fail to construct realistic fake samples. Meanwhile, most recommendation models adopt click-through rate (CTR) predictors, which usually utilize black-box deep models with discrete features as input. Thus, how to efficiently construct realistic fake samples for black-box recommender systems is still full of challenges. In this article, we propose a hierarchical adversarial attack method against black-box CTR models via generating realistic fake samples, named CTRAttack. To better train the generation network, the weights of its embedding layer are shared with those of the substitute model, with both the similarity loss and classification loss used to update the generation network. To ensure that the discrete features of the generated fake samples are all real, we first adopt the similarity loss to ensure that the distribution of the generated perturbed samples is sufficiently close to the distribution of the real features, and then the nearest neighbor algorithm is used to retrieve the most appropriate features for non-existent discrete features from the candidate instance set. Extensive experiments demonstrate that CTRAttack can not only effectively attack the black-box recommender systems but also improve the robustness of these models while maintaining prediction accuracy.

Modern software capabilities for shape optimization of shells

The article is devoted to the shape optimization of shell structures in Comsol Multiphysics using three gradient-based methods: IPOPT (Interior Point OPTimizer), SNOPT (Sparse Nonlinear OPTimizer) and MMA or GCMMA (the Method of Moving Asymptotes). Two types of complex shapes, such as right helicoid and developable helicoid are taken for the computational experiment. The task is to investigate the initial design and optimization process of two helicoids. To obtain a more accurate result and an interesting design solution, the calculation is carried out using three physics-controlled mesh sizes: extra coarse, fine and extra fine with varying values of special optimization settings, such as maximum displacement (dmax) and filter radius (Rmin). The results obtained using the three methods allow to conclude that the mesh element size and studied parameters dmax and Rmin have a significant impact on the final optimization result.

Dynamic vehicle–track interaction and structural loading in a crossing panel – calibration and assessment of a model with a 3D representation of the crossing rail

A three-dimensional (3D) finite element model of a railway crossing panel for use in multibody simulation (MBS) of dynamic vehicle–track interaction is presented. It is a two-layer track model with stock rails and sleepers represented by beam elements and a crossing rail represented by 3D solid elements. The track model uses linear bushings for the rail fastenings and bi-linear bushings for the ballast to allow for potentially voided sleepers. Based on the output from the MBS, the structural loading of the crossing in terms of strains, stresses and sleeper-ballast contact pressures is extracted in a post-processing step. The model is calibrated and validated to measurement data from a comprehensively instrumented switch & crossing (S&C) demonstrator installed in the Austrian railway network as a part of the European research programme Shift2Rail. The applied procedure for the calibration and critical assessment of the crossing model is described in detail. It is based on a model parameterisation with eight parameters relating to the rail fastening and foundation stiffnesses and to a distribution of the ballast voids. The calibration method uses Latin hypercube samples to explore the parameter space in a sensitivity analysis before a parameter optimisation is performed using a gradient-based method on a response surface built from a polyharmonic spline. In a comparative study it is shown that the 3D model and a more conventional beam model of the crossing rail show similar calibration results and good agreement with the measured data. The 3D model allows for the extraction of stress concentrations in the crossing rail but has an increased computational time of about 30% compared to the beam model.

MT2DInv-Unet: A 2D magnetotelluric inversion method based on deep-learning technology

Traditional gradient-based inversion methods usually suffer from the problems of falling into local minima and relying heavily on initial guesses. Deep-learning methods have received increasing attention due to their excellent nonlinear fitting ability. However, given the recent application of deep-learning methods in the field of magnetotelluric (MT) inversion, there are currently challenges associated with achieving high inversion resolution and extracting sufficient features. We develop a neural network model (called MT2DInv-Unet) based on the deformable convolution for 2D MT inversion to approximate the nonlinear mapping from the MT response data to the resistivity model. The deformable convolution is achieved by adding an offset to each sample point of the conventional convolution operation, which extracts hidden relationships and allows the flexible adjustment of the size and shape of the feature region. Meanwhile, we design the network structure with multiscale residual blocks, which effectively extract the multiscale features of the MT response data. This design not only enhances the network performance but also alleviates issues such as vanishing gradients and network degradation. The results of synthetic models indicate that our network inversion method has stable convergence, good robustness, and generalization performance, and it performs better than the fully convolutional neural network and U-Net network. Finally, the inversion results of field data show that MT2DInv-Unet can effectively obtain a reliable underground resistivity structure and has a good application prospect in MT inversion.

Investigating Hessian-based inversion velocity analysis

Inversion velocity analysis (IVA) is an image-domain method built upon the spatial scale separation of the model. Accordingly, the IVA method is performed with an iterative process composed of two minimization steps consisting of migration (inner loop) and tomography (outer loop), respectively, with each step accounting for its Hessian or not. The migration part provides the common-image gathers (CIGs) with extension in the horizontal subsurface offset. Then, the differential semblance optimization (DSO) misfit measures the focusing of the events in the CIGs, which indicates the quality of the velocity model. Commonly, the velocity updates are obtained from the DSO gradient. IVA is a modified version in which the approximate inverse replaces the adjoint of the inner loop process; in that case, the migration Hessian is approximately diagonal in the high-frequency regime. In this work, we report on implementation of the tomographic Hessian (i.e., the second derivative of the DSO misfit with respect to the background model) for the estimation of the background velocity model. We apply the second-order adjoint-state method to obtain the application of the tomographic Hessian on a vector. Then, we use the truncated-Newton (TN) method to obtain the update directions by computing approximately the application of the inverse of the tomographic Hessian on the descent direction. We also make a theoretical comparison between tomography in the IVA and full-waveform inversion contexts. Two numerical examples are used to compare, in terms of geophysical results and computational costs, the TN method with different gradient-based optimization methods applied to the IVA. A small model allows us to evaluate the eigenvalues of the tomographic Hessian, which explains the large damping needed in the TN case.

Qualitative and quantitative enhancement of parameter estimation for model-based diagnostics using automatic differentiation with an application to inertial fusion

Parameter estimation using observables is a fundamental concept in the experimental sciences. Mathematical models that represent the physical processes can enable reconstructions of the experimental observables and greatly assist in parameter estimation by turning it into an optimization problem which can be solved by gradient-free or gradient-based methods. In this work, the recent rise in flexible frameworks for developing differentiable scientific computing programs is leveraged in order to dramatically accelerate data analysis of a common experimental diagnostic relevant to laser–plasma and inertial fusion experiments, Thomson scattering. A differentiable Thomson-scattering data analysis tool is developed that uses reverse-mode automatic differentiation (AD) to calculate gradients. By switching from finite differencing to reverse-mode AD, three distinct outcomes are achieved. First, gradient descent is accelerated dramatically to the extent that it enables near real-time usage in laser–plasma experiments. Second, qualitatively novel quantities which require O(103) parameters can now be included in the analysis of data which enables unprecedented measurements of small-scale laser–plasma phenomena. Third, uncertainty estimation approaches that leverage the value of the Hessian become accurate and efficient because reverse-mode AD can be used for calculating the Hessian.

Exploring the performance and explainability of fine-tuned BERT models for neuroradiology protocol assignment

BackgroundDeep learning has demonstrated significant advancements across various domains. However, its implementation in specialized areas, such as medical settings, remains approached with caution. In these high-stake environments, understanding the model's decision-making process is critical. This study assesses the performance of different pretrained Bidirectional Encoder Representations from Transformers (BERT) models and delves into understanding its decision-making within the context of medical image protocol assignment.MethodsFour different pre-trained BERT models (BERT, BioBERT, ClinicalBERT, RoBERTa) were fine-tuned for the medical image protocol classification task. Word importance was measured by attributing the classification output to every word using a gradient-based method. Subsequently, a trained radiologist reviewed the resulting word importance scores to assess the model’s decision-making process relative to human reasoning.ResultsThe BERT model came close to human performance on our test set. The BERT model successfully identified relevant words indicative of the target protocol. Analysis of important words in misclassifications revealed potential systematic errors in the model.ConclusionsThe BERT model shows promise in medical image protocol assignment by reaching near human level performance and identifying key words effectively. The detection of systematic errors paves the way for further refinements to enhance its safety and utility in clinical settings.