Gradient-based Procedure Research Articles

On Unsupervised Artificial Intelligence-Assisted Design of Antennas for High-Performance Planar Devices

Designing modern antenna structures is a challenging endeavor. It is laborious and heavily reliant on engineering insight and experience, especially at the initial stages oriented towards the development of a suitable antenna architecture. Due to its interactive nature and hands-on procedures (mainly parametric studies) for validating the suitability of particular geometric setups, typical antenna development requires many weeks and significant involvement of a human expert. The same reasons only allow the designer to try out a very limited number of options in terms of antenna geometry arrangements. Automated topology development and dimension sizing is therefore of high interest, especially from an industry perspective where time-to-market and expert-related expenses are of paramount importance. This paper discusses a novel approach to unsupervised specification-driven design of planar antennas. The presented methodology capitalizes on a flexible and scalable antenna parameterization, which enables the realization of complex geometries while maintaining reasonably small parameter space dimensionality. A customized nature-inspired algorithm is employed to carry out space exploration and identification of a quasi-optimum antenna topology in a global sense. A fast gradient-based procedure is then incorporated to fine-tune antenna dimensions. The design framework works entirely in a black-box fashion with the only input being design specifications, and optional constraints, e.g., concerning the structure size. Numerous illustration case studies demonstrate the capability of the presented technique to generate unconventional antenna topologies of satisfactory performance using reasonable computational budgets, and with no human expert interaction necessary whatsoever.

Umbilical Cord Blood-Derived Monocytes as A Reliable Source of Functional Macrophages for Biomedical Research.

Macrophages are multifunctional immune cells widely used in immunological research. While autologous macrophages have been widely used in several biomedical applications, allogeneic macrophages have also demonstrated similar or even superior therapeutic potential. The umbilical cord blood (UCB) is a well-described source of abundant allogenic monocytes and macrophages that is easy to collect and can be processed without invasive methods. Current monocyte isolation procedures frequently result in heterogenous cell products, with limited yields, activated cells, and high cost. This study outlines a simple isolation method that results in high yields and pure monocytes with the potential to differentiate into functional macrophages. In the experimental study, we describe a simple and efficient protocol to isolate highpurity monocytes. After collection of human UCB samples, we used a gradient-based procedure composed of three consecutive gradient steps: i. Hydroxyethyl starch-based erythrocytes sedimentation, followed by ii. Mononuclear cells (MNCs) isolation by Ficoll-Hypaque gradient, and iii. Separation of monocytes from lymphocytes by a slight hyperosmolar Percoll gradient (0.573 g/ml). Then the differentiation potential of isolated monocytes to pro- and antiinflammatory macrophages were evaluated in the presence of granulocyte colony-stimulating factor (GM-CSF) and macrophage CSF (M-CSF), respectively. The macrophages were functionally characterized as well. A high yield of monocytes after isolation (25 to 50 million) with a high purity (>95%) could be obtained from every 100-150 ml UCB. Isolated monocytes were defined based on their phenotype and surface markers expression pattern. Moreover, they possess the ability to differentiate into pro- or anti-inflammatory macrophages with specific phenotypes, gene/surface protein markers, cytokine secretion patterns, T-cell interactions, and phagocytosis activity. Here we describe a simple and reproducible procedure for isolation of pure monocytes from UCB, which could be utilized to provide functional macrophages as a reliable and feasible source of allogenic macrophages for biomedical research.

Accelerated Gradient-Based Optimization of Antenna Structures Using Multifidelity Simulations and Convergence-Based Model Management Scheme

The importance of numerical optimization has been steadily growing in the design of contemporary antenna structures. The primary reason is the increasing complexity of antenna topologies, a typically large number of adjustable parameters that have to be simultaneously tuned. Design closure is no longer possible using traditional methods, including theoretical models or supervised parameter sweeping. To ensure reliability, optimization is normally carried out at the level of full-wave electromagnetic (EM) simulations, which incurs major computational expenses. The issue can be alleviated using a variety of methods such as the incorporation of adjoint sensitivities, sparse sensitivity updates (for local optimization), or the employment of surrogate modeling methods (in the context of global search). Another possibility is utilization of variable-fidelity simulation models, which, in practice, is most often restricted to two levels (coarse/fine or low-/high-fidelity models), and accompanied by appropriate low-fidelity model correction (e.g., space mapping). This article proposes an accelerated version of a trust-region (TR) gradient-based procedure, which involves simulation model management by continuous adjustment of EM analysis fidelity throughout the optimization process. Decision-making process is based on the convergence status of the algorithm. The initial stages of the optimization run are executed using the coarsest discretization of the structure at hand with the model being gradually refined toward the end of the process. This enables considerable computational savings without degrading the quality of the final design. The presented approach has been comprehensively validated using a benchmark set of four broadband antennas and compared with the reference TR procedure and two state-of-the-art accelerated algorithms. The average computational savings are almost 60% as compared with the reference.

Infinite-dimensional gradient-based descent for alpha-divergence minimisation

This paper introduces the $(\alpha, \Gamma)$-descent, an iterative algorithm which operates on measures and performs $\alpha$-divergence minimisation in a Bayesian framework. This gradient-based procedure extends the commonly-used variational approximation by adding a prior on the variational parameters in the form of a measure. We prove that for a rich family of functions $\Gamma$, this algorithm leads at each step to a systematic decrease in the $\alpha$-divergence and derive convergence results. Our framework recovers the Entropic Mirror Descent algorithm and provides an alternative algorithm that we call the Power Descent. Moreover, in its stochastic formulation, the $(\alpha, \Gamma)$-descent allows to optimise the mixture weights of any given mixture model without any information on the underlying distribution of the variational parameters. This renders our method compatible with many choices of parameters updates and applicable to a wide range of Machine Learning tasks. We demonstrate empirically on both toy and real-world examples the benefit of using the Power descent and going beyond the Entropic Mirror Descent framework, which fails as the dimension grows.

Asymptotic optimality in stochastic optimization

We study local complexity measures for stochastic convex optimization problems, providing a local minimax theory analogous to that of Hájek and Le Cam for classical statistical problems. We give complementary optimality results, developing fully online methods that adaptively achieve optimal convergence guarantees. Our results provide function-specific lower bounds and convergence results that make precise a correspondence between statistical difficulty and the geometric notion of tilt-stability from optimization. As part of this development, we show how variants of Nesterov’s dual averaging—a stochastic gradient-based procedure—guarantee finite time identification of constraints in optimization problems, while stochastic gradient procedures fail. Additionally, we highlight a gap between problems with linear and nonlinear constraints: standard stochastic-gradient-based procedures are suboptimal even for the simplest nonlinear constraints, necessitating the development of asymptotically optimal Riemannian stochastic gradient methods.

Bimodal dynamics of granular organelles in primary renin-expressing cells revealed using TIRF microscopy.

Renin is the initiator and rate-limiting factor in the renin-angiotensin blood pressure regulation system. Although renin is not exclusively produced in the kidney, in nonmurine species the synthesis and secretion of the active circulatory enzyme is confined almost exclusively to the dense core granules of juxtaglomerular (JG) cells, where prorenin is processed and stored for release via a regulated pathway. Despite its importance, the structural organization and regulation of granules within these cells is not well understood, in part due to the difficulty in culturing primary JG cells in vitro and the lack of appropriate cell lines. We have streamlined the isolation and culture of primary renin-expressing cells suitable for high-speed, high-resolution live imaging using a Percoll gradient-based procedure to purify cells from RenGFP+ transgenic mice. Fibronectin-coated glass coverslips proved optimal for the adhesion of renin-expressing cells and facilitated live cell imaging at the plasma membrane of primary renin cells using total internal reflection fluorescence microscopy (TIRFM). To obtain quantitative data on intracellular function, we stained mixed granule and lysosome populations with Lysotracker Red and stimulated cells using 100 nM isoproterenol. Analysis of membrane-proximal acidic granular organelle dynamics and behavior within renin-expressing cells revealed the existence of two populations of granular organelles with distinct functional responses following isoproterenol stimulation. The application of high-resolution techniques for imaging JG and other specialized kidney cells provides new opportunities for investigating renal cell biology.

Determination of lower and upper bounds of predicted production from history-matched models

We present a method to determine lower and upper bounds to the predicted production or any other economic objective from history-matched reservoir models. The method consists of two steps: 1) performing a traditional computer-assisted history match of a reservoir model with the objective to minimize the mismatch between predicted and observed production data through adjusting the grid block permeability values of the model. 2) performing two optimization exercises to minimize and maximize an economic objective over the remaining field life, for a fixed production strategy, by manipulating the same grid block permeabilities, however without significantly changing the mismatch obtained under step 1. This is accomplished through a hierarchical optimization procedure that limits the solution space of a secondary optimization problem to the (approximate) null space of the primary optimization problem. We applied this procedure to two different reservoir models. We performed a history match based on synthetic data, starting from a uniform prior and using a gradient-based minimization procedure. After history matching, minimization and maximization of the net present value (NPV), using a fixed control strategy, were executed as secondary optimization problems by changing the model parameters while staying close to the null space of the primary optimization problem. In other words, we optimized the secondary objective functions, while requiring that optimality of the primary objective (a good history match) was preserved. This method therefore provides a way to quantify the economic consequences of the well-known problem that history matching is a strongly ill-posed problem. We also investigated how this method can be used as a means to assess the cost-effectiveness of acquiring different data types to reduce the uncertainty in the expected NPV.

AUC-Maximized Deep Convolutional Neural Fields for Protein Sequence Labeling.

Deep Convolutional Neural Networks (DCNN) has shown excellent performance in a variety of machine learning tasks. This paper presents Deep Convolutional Neural Fields (DeepCNF), an integration of DCNN with Conditional Random Field (CRF), for sequence labeling with an imbalanced label distribution. The widely-used training methods, such as maximum-likelihood and maximum labelwise accuracy, do not work well on imbalanced data. To handle this, we present a new training algorithm called maximum-AUC for DeepCNF. That is, we train DeepCNF by directly maximizing the empirical Area Under the ROC Curve (AUC), which is an unbiased measurement for imbalanced data. To fulfill this, we formulate AUC in a pairwise ranking framework, approximate it by a polynomial function and then apply a gradient-based procedure to optimize it. Our experimental results confirm that maximum-AUC greatly outperforms the other two training methods on 8-state secondary structure prediction and disorder prediction since their label distributions are highly imbalanced and also has similar performance as the other two training methods on solvent accessibility prediction, which has three equally-distributed labels. Furthermore, our experimental results show that our AUC-trained DeepCNF models greatly outperform existing popular predictors of these three tasks. The data and software related to this paper are available at https://github.com/realbigws/DeepCNF_AUC.

A method of identifying parameters of a time-varying time-delay chaotic system

Unknown time-varying parameters, including time-delay and system parameters, commonly exist in chaotic systems. These unknown parameters increase the difficulties in controlling the chaotic systems, and make most of the existing control methods fail to be applied. However, if these parameters can be estimated, they will facilitate the controller design. Therefore, in this paper, a parameter identification problem for a general time-delay chaotic system with unknown and time-varying parameters is considered, where these unknown time-delay and parameters are slow time-varying. It is very difficult to solve this problem analytically. Thus, a unified identification method is proposed to solve the identification problem numerically. To solve this identification problem, firstly, the time horizon is divided into several subintervals evenly. Then the time-varying parameters are approximated by piecewise constant functions. The height vectors of the piecewise constant functions are unknown and to be determined. Furthermore, the heights of the piecewise constant functions keep constant between each pair of the successive partition time points but switch values at the partition time points. After the approximation, the original identification problem for finding the nonlinear functions of the unknown parameters is transformed into a problem of selecting approximate parameter vectors, where the heights of the piecewise constan functions are unknown parameter vectors to be determined. Secondly, to solve the problem of selecting approximate parameter vectors quickly, the partial gradients of the objective function with respect to the parameter vectors are derived; and they are then integrated with a gradient-based procedure to obtain the unknown heights. As the number of partitions for the piecewise function increases, the optimal results of the approximate problem will approach to the optimal results of the original parameter identification problem. Hence, the optimal piecewise functions will approach to the real nonlinear functions for the unknown parameters. Finally, parameter identification experiments on time-delayed Mackey-Class and time-delayed logistic chaotic systems are carried out. The effects of the partition number on the estimated results are discussed. Numerical results demonstrate that when some switching times of the unknown parameters do not coincide with any partition time points, small error between the estimated results and the real values are present. However, these errors can be filtered and the estimated results are consistent well with the real values. Hence, the proposed method is reasonable and effective.

Global Convergent Sequence for Solving Optimization Problems

The paper presents an effective, gradient-based procedure for locating the optimum for either constrained or unconstrained response surfaces. The procedure is widely applicable covering linear and curvilinear constraints as well as constraints with redundancy. Each move of the sequence is made by pooling information from all the support points in the feasible region, making the search a global one and convergence to the global optimum almost certain.

Utilization of efficient gradient and Hessian computations in the force field optimization process of molecular simulations

Computer simulations of chemical systems, especially systems of condensed matter, are highly important for both scientific and industrial applications. Thereby, molecular interactions are modeled on a microscopic level in order to study their impact on macroscopic phenomena. To be capable of predicting physical properties quantitatively, accurate molecular models are indispensable. Molecular interactions are described mathematically by force fields, which have to be parameterized. Recently, an automated gradient-based optimization procedure was published by the authors based on the minimization of a loss function between simulated and experimental physical properties. The applicability of gradient-based procedures is not trivial at all because of two reasons: firstly, simulation data are affected by statistical noise, and secondly, the molecular simulations required for the loss function evaluations are extremely time-consuming. Within the optimization process, gradients and Hessians were approximated by finite differences so that additional simulations for the respective modified parameter sets were required. Hence, a more efficient approach to computing gradients and Hessians is presented in this work. The method developed here is based on directional instead of partial derivatives. It is compared with the classical computations with respect to computation time. Firstly, molecular simulations are replaced by fit functions that define a functional dependence between specific physical observables and force field parameters. The goal of these simulated simulations is to assess the new methodology without much computational effort. Secondly, it is applied to real molecular simulations of the three chemical substances phosgene, methanol and ethylene oxide. It is shown that up to 75% of the simulations can be avoided using the new algorithm.

Planning identification experiments for cell signaling pathways: An NFκB case study

Planning identification experiments for cell signaling pathways: An NFκB case studyMathematical modeling of cell signaling pathways has become a very important and challenging problem in recent years. The importance comes from possible applications of obtained models. It may help us to understand phenomena appearing in single cells and cell populations on a molecular level. Furthermore, it may help us with the discovery of new drug therapies. Mathematical models of cell signaling pathways take different forms. The most popular way of mathematical modeling is to use a set of nonlinear ordinary differential equations (ODEs). It is very difficult to obtain a proper model. There are many hypotheses about the structure of the model (sets of variables and phenomena) that should be verified. The next step, fitting the parameters of the model, is also very complicated because of the nature of measurements. The blotting technique usually gives only semi-quantitative observations, which are very noisy and collected only at a limited number of time moments. The accuracy of parameter estimation may be significantly improved by a proper experiment design. Recently, we have proposed a gradient-based algorithm for the optimization of a sampling schedule. In this paper we use the algorithm in order to optimize a sampling schedule for the identification of the mathematical model of the NFκB regulatory module, known from the literature. We propose a two-stage optimization approach: a gradient-based procedure to find all stationary points and then pair-wise replacement for finding optimal numbers of replicates of measurements. Convergence properties of the presented algorithm are examined.

RECM: Relational evidential c-means algorithm

A new clustering algorithm for proximity data, called RECM (relational evidential c-means) is presented. This algorithm generates a credal partition, a new clustering structure based on the theory of belief functions, which extends the existing concepts of hard, fuzzy and possibilistic partitions. Two algorithms, EVCLUS (Evidential Clustering) and ECM (evidential c-means) were previously available to derive credal partitions from data. EVCLUS was designed to handle proximity data, whereas ECM is a direct extension of fuzzy clustering algorithms for vectorial data. In this article, the relational version of ECM is introduced. It is compared to EVCLUS using various datasets. It is shown that RECM provides similar results to those given by EVCLUS. However, the optimization procedure of RECM, based on an alternate minimization scheme, is computationally much more efficient than the gradient-based procedure used in EVCLUS.

Hybrid approach for sub-arrayed monopulse antenna synthesis

A computationally effective hybrid approach to define the 'optimal' compromise between sum and difference patterns in monopulse arrays is presented. First, the partitioning into sub-arrays is performed by exploiting the knowledge of independently optimal sum and difference excitations. Then, the sub-array gains are computed by means of a gradient-based procedure, which takes advantage from the convexity of the problem at hand. Selected results are shown and compared with those from state-of-the-art methods in dealing with representative test cases.

Rheological Parameter Estimation for a Ferrous Nanoparticle-based Magnetorheological Fluid using Genetic Algorithms

This study examines identification of rheological parameters for a constitutive model characterizing the rheological behavior of a ferrous nanoparticle-based magnetorheological (MR) fluid. Particle size is nominally 28 nm and the MR fluid has a weight fraction of 27.5% Fe. A constant shear rate rheometer is used to measure flow curves (shear stress vs. shear rate), as a function of applied magnetic field, of an MR suspension of nanometer-sized iron particles in hydraulic oil. The MR fluid is characterized using both Bingham-plastic (BP) and Herschel-Bulkley (HB) constitutive models. These models have two regimes that can be characterized by a field-dependent yield stress: pre-yield implies that the local shear stress is less than the yield stress, and post-yield implies that the local shear stress is greater than the yield stress. Both models of MR fluid behavior assume that the MR fluid is rigid in the pre-yield regime. However, the post-yield behavior is different. The BP model assumes that the post-yield increase in shear stress is proportional to shear rate. However, the HB model assumes that the post-yield increase in shear stress is proportional to a power law of shear rate. Identification of the model parameters is complicated by model non-linearities, as well as variance in experimental data. The rheological parameters of the BP and HB models are identified using both a gradient-based least mean square minimization procedure, as well as a genetic algorithm (GA). The HB model is shown to represent better, the rheological behavior of the ferrous nanoparticle-based MR fluid. Also, the GA performs better than the gradient-based procedure in minimizing modeling error.

Model-Based Global and Local Motion Estimation for Videoconference Sequences

In this work, we present an algorithm for face 3-D motion estimation in videoconference sequences. The algorithm is able to estimate both the position of the face as an object in 3-D space (global motion) and the movements of portions of the face, like the mouth or the eyebrows ( local motion). The algorithm uses a modified version of the standard 3-D face model CANDIDE. We present various techniques to increase robustness of the global motion estimation which is based on feature tracking and an extended Kalman filter. Global motion estimation is used as a starting point for local motion detection in the mouth and eyebrow areas. To this purpose, synthetic images of these areas (templates) are generated with texture mapping techniques, and then compared to the corresponding regions in the current frame. A set of parameters, called action unit vectors (AUVs) influences the shape of the synthetic mouth and eyebrows. The optimal AUV values are determined via a gradient-based minimization procedure of the error energy between the templates and the actual face areas. The proposed scheme is robust and was tested with success on sequences of many hundreds of frames.

A gradient‐based adaptation procedure and its implementation in the element‐free Galerkin method

AbstractA gradient‐based adaptation procedure is proposed in this paper. The relative error in the total strain energy from two adjacent adaptation stages is used as a stop‐criterion. The refinement–coarsening process is guided by the gradient of strain energy density, based on the assumption: a larger gradient needs a richer mesh and vice versa. The procedure is then implemented in the element‐free Galerkin method for linear elasto‐static problems. Numerical examples are presented to show the performance of the proposed procedure. Copyright © 2003 John Wiley & Sons, Ltd.

On-line learning algorithms for locally recurrent neural networks

This paper focuses on on-line learning procedures for locally recurrent neural networks with emphasis on multilayer perceptron (MLP) with infinite impulse response (IIR) synapses and its variations which include generalized output and activation feedback multilayer networks (MLN's). We propose a new gradient-based procedure called recursive backpropagation (RBP) whose on-line version, causal recursive backpropagation (CRBP), presents some advantages with respect to the other on-line training methods. The new CRBP algorithm includes as particular cases backpropagation (BP), temporal backpropagation (TBP), backpropagation for sequences (BPS), Back-Tsoi algorithm among others, thereby providing a unifying view on gradient calculation techniques for recurrent networks with local feedback. The only learning method that has been proposed for locally recurrent networks with no architectural restriction is the one by Back and Tsoi. The proposed algorithm has better stability and higher speed of convergence with respect to the Back-Tsoi algorithm, which is supported by the theoretical development and confirmed by simulations. The computational complexity of the CRBP is comparable with that of the Back-Tsoi algorithm, e.g., less that a factor of 1.5 for usual architectures and parameter settings. The superior performance of the new algorithm, however, easily justifies this small increase in computational burden. In addition, the general paradigms of truncated BPTT and RTRL are applied to networks with local feedback and compared with the new CRBP method. The simulations show that CRBP exhibits similar performances and the detailed analysis of complexity reveals that CRBP is much simpler and easier to implement, e.g., CRBP is local in space and in time while RTRL is not local in space.

Optimal weighted L2 model reduction of delay systems

A model reduction method for stable delay systems under L2 optimality is introduced in this paper. The reduced models may take the form of either a stable finite dimensional system or a delay system with a reduced-order finite dimensional part. The two cases are studied under a unified framework. Contrary to many existing algorithms, the formulation is entirely given in terms of state-space matrices. The Routh canonical realization is used to parametrize the stable reduced models. The parameters which form the reduced order system are obtained by minimizing the weighted L2 norm of the approximation error through an unconstrained gradient-based minimization procedure. Numerical examples are used to illustrate the effectiveness of the proposed method.

Geometric Elimination of Constraint Violations in Numerical Simulation of Lagrangian Equations

Conventional holonomic or nonholonomic constraints are defined as geometric constraints. The total enregy of a dynamic system can be treated as a constrained quantity for the purpose of accurate numerical simulation. In the simulation of Lagrangian equations of motion with constraint equations, the Geometric Elimination Method turns out to be more effective in controlling constraint violations than any conventional methods, including Baumgarte’s Constraint Violation Stabilization Method (CVSM). At each step, this method first goes through the numerical integration process without correction to obtain updated values of the state variables. These values are then used in a gradient-based procedure to eliminate the geometric and energy errors simultaneously before processing to the next step. For small step size, this procedure is stable and very accurate.