Gradient Calculations Research Articles

Coresets over multiple tables for feature-rich and data-efficient machine learning

Successful machine learning (ML) needs to learn from good data. However, one common issue about train data for ML practitioners is the lack of good features. To mitigate this problem, feature augmentation is often employed by joining with (or enriching features from) multiple tables, so as to become feature-rich ML. A consequent problem is that the enriched train data may contain too many tuples, especially if the feature augmentation is obtained through 1 (or many)-to-many or fuzzy joins. Training an ML model with a very large train dataset is data-inefficient. Coreset is often used to achieve data-efficient ML training, which selects a small subset of train data that can theoretically and practically perform similarly as using the full dataset. However, coreset selection over a large train dataset is also known to be time-consuming. In this paper, we aim at achieving both feature-rich ML through feature augmentation and data-efficient ML through coreset selection. In order to avoid time-consuming coreset selection over a feature augmented (or fully materialized) table, we propose to efficiently select the coreset without materializing the augmented table. Note that coreset selection typically uses weighted gradients of the subset to approximate the full gradient of the entire train dataset. Our key idea is that the gradient computation for coreset selection of the augmented table can be pushed down to partial feature similarity of tuples within each individual table, without join materialization. These partial feature similarity values can be aggregated to estimate the gradient of the augmented table, which is upper bounded with provable theoretical guarantees. Extensive experiments show that our method can improve the efficiency by nearly 2 orders of magnitudes, while keeping almost the same accuracy as training with the fully augmented train data.

Constrained non-linear AVO inversion based on the adjoint-state optimization

Pre-stack AVO inversion of seismic data is a modeling tool for estimating subsurface elastic properties. Our focus is on the model-based inversion method where then unknown variables are estimated by minimizing the misfit to the observed data. Standard approaches for non-linear AVO inversion are based on the gradient descent optimization algorithms that require the calculation of the gradient equations of the objective function. To improve the accuracy and efficiency of these methods, we developed a technique that uses an implementation of the adjoint-state-based gradient computation. The inversion algorithm relies on three basic modeling components consisting of a convolution-based forward model using a linearized approximation of the Zoeppritz equation, the definition of the objective function, and the adjoint-computed gradient. To achieve an accurate solution, we choose a second-order optimization algorithm known as the Limited memory-BFGS (L-BFGS) that implicitly approximates the inverse Hessian matrix. This approach is more efficient than traditional optimization methods. The main novelty of the proposed approach is the derivation of the adjoint-state equations for the gradient of the objective function. The application of the proposed method is demonstrated using 1D and 2D synthetic datasets based on data from the Edvard Grieg oil field. The seismic data for these applications is generated by using both convolutional modeling and finite difference methods. The results of the proposed method are accurate and the computational approach is efficient. The results show that the algorithm reliably retrieves the elastic variables, P- and S-wave velocities and density for both convolutional and finite difference models.

A general deep hybrid model for bioreactor systems: Combining first principles with deep neural networks

Numerous studies have reported the use of hybrid semiparametric systems that combine shallow neural networks with First Principles for bioprocess modeling. Here we revisit the general bioreactor hybrid model and introduce some deep learning techniques. Multi-layer networks with varying depths were combined with First Principles equations in the form of deep hybrid models. Deep learning techniques, namely the adaptive moment estimation method (ADAM), stochastic regularization and depth-dependent weights initialization were evaluated in a hybrid modeling context. Modified sensitivity equations are proposed for the computation of gradients in order to reduce CPU time for the training of deep hybrid models. The methods are illustrated with applications to a synthetic dataset and a pilot 50 L MUT+ Pichia pastoris process expressing a single chain antibody fragment. All in all, the results point to a systematic generalization improvement of deep hybrid models over its shallow counterpart. Moreover, the CPU cost to train the deep hybrid models is shown to be lower than for the shallow counterpart. In the pilot 50L MUT+ Pichia pastoris data set, the prediction accuracy was increased by 18.4% and the CPU decreased by 43.4%.

Deep learning-based alzheimer disease detection techniques

Alzheimer disease (AD) is one of the most common degenerative illnesses of the elderly worldwide. It is a progressive neurological condition that impairs cognitive memory. As a result, Alzheimer's sufferers struggle to recall daily activities, recollect family members, and solve logical problems. Medication which reduces the creation of proteins, block data communication between brain neurons and it can also delay the course of Alzheimer's disease. Mild Cognitive Impairment (MCI) seems to be a common disorder that does not usually progress to Alzheimer's. It is difficult to find patients with modest cognitive decline who may acquire Alzheimer's. As a result, creating deep learning-based disease detection techniques to assist clinicians in detecting prospective Alzheimer's patients is crucial. The performance comparison of the Imaging, Electronic Health Record (EHR), and Single nucleotide polymorphisms (SNP) datasets is evaluated using the metrics Accuracy, Sensitivity, Specificity, and Multi Area. Different mistakes are added under the curves for gradient calculation.

Training Physics‐Based Machine‐Learning Parameterizations With Gradient‐Free Ensemble Kalman Methods

AbstractMost machine learning applications in Earth system modeling currently rely on gradient‐based supervised learning. This imposes stringent constraints on the nature of the data used for training (typically, residual time tendencies are needed), and it complicates learning about the interactions between machine‐learned parameterizations and other components of an Earth system model. Approaching learning about process‐based parameterizations as an inverse problem resolves many of these issues, since it allows parameterizations to be trained with partial observations or statistics that directly relate to quantities of interest in long‐term climate projections. Here, we demonstrate the effectiveness of Kalman inversion methods in treating learning about parameterizations as an inverse problem. We consider two different algorithms: unscented and ensemble Kalman inversion. Both methods involve highly parallelizable forward model evaluations, converge exponentially fast, and do not require gradient computations. In addition, unscented Kalman inversion provides a measure of parameter uncertainty. We illustrate how training parameterizations can be posed as a regularized inverse problem and solved by ensemble Kalman methods through the calibration of an eddy‐diffusivity mass‐flux scheme for subgrid‐scale turbulence and convection, using data generated by large‐eddy simulations. We find the algorithms amenable to batching strategies, robust to noise and model failures, and efficient in the calibration of hybrid parameterizations that can include empirical closures and neural networks.

Hybrid domain frequency controllable envelope inversion with lowrank approximation for long-wavelength velocity model building

Full waveform inversion (FWI) needs an accurate initial model to guarantee convergence. Time domain frequency controllable envelope inversion (FCEI) is capable to build a long-wavelength velocity model that can be used as an initial model for conventional FWI. However, the implementation of time domain FCEI introduces tremendous burden in storage or computation, which hinders its efficient implementation especially on a memory-limited device. We propose a hybrid domain FCEI (HFCEI) method, in which the wave propagation is performed in the time domain while the computation of gradient is performed in the frequency domain. We demonstrate that the frequency control property of FCEI makes it suitable to be efficiently implemented in the hybrid domain. Only a few frequency domain wavefields need to be saved and no extra wave propagation is needed in calculating gradient. We also demonstrate that the frequency domain wavefields in HFCEI satisfy the lowrank assumption, which enables us to use the lowrank approximation to accurately represent the frequency domain wavefields to further reduce the storage requirement of the method. As a result, HFCEI has significant advantages than time domain FCEI in implementation especially for large-scale problems. Synthetic examples are given to verify the effectiveness of HFCEI and demonstrate its advantages over time domain FCEI.

Nonstationary phase-corrected full-waveform inversion with attenuation compensation in viscoacoustic medium

Abstract Full-waveform inversion (FWI) acts as an effective technique to estimate subsurface parameter by iteratively reducing the difference between the predictions and the observations. The classic FWI suffers from the problem of converging to the local minimum when the starting model is poor, which is known as the notorious cycle skipping phenomenon. Moreover, due to the anelasticity of the earth, seismic waves always suffer from energy dissipation and phase distortion while their propagation, which leads to an attenuated gradient for FWI, decelerates the convergence rate of the inversion processing. We have proposed a new method referred to as Q-compensated nonstationary phase-corrected FWI (QNPCFWI) to compensate for the attenuation-induced gradient energy loss and the phase mismatch caused by the less-accurate initial velocity model and phase dispersion simultaneously in viscoacoustic medium. We incorporated attenuation compensation mechanism and nonstationary phase correction method for improved inversion efficiency in the case that a poor initial model is used. The main points of this paper can be concluded as follows: (i) we compensate the lost energy for gradient calculation during wave propagation for improved inversion efficiency. (ii) As we know, Q model estimation for real data is challenging and an accurate Q model is hard to get. The proposed QNPCFWI can also work using an approximate Q model. (iii) The proposed method has the ability to mitigate cycle skipping even if the low-frequency components of seismic data are absent. Numerical examples validate the effectiveness and efficiency of our proposed method.

Thermodynamic basis for the demarcation of Arctic and alpine treelines

At the edge of alpine and Arctic ecosystems all over the world, a transition zone exists beyond which it is either infeasible or unfavorable for trees to exist, colloquially identified as the treeline. We explore the possibility of a thermodynamic basis behind this demarcation in vegetation by considering ecosystems as open systems driven by thermodynamic advantage—defined by vegetation’s ability to dissipate heat from the earth’s surface to the air above the canopy. To deduce whether forests would be more thermodynamically advantageous than existing ecosystems beyond treelines, we construct and examine counterfactual scenarios in which trees exist beyond a treeline instead of the existing alpine meadow or Arctic tundra. Meteorological data from the Italian Alps, United States Rocky Mountains, and Western Canadian Taiga-Tundra are used as forcing for model computation of ecosystem work and temperature gradients at sites on both sides of each treeline with and without trees. Model results indicate that the alpine sites do not support trees beyond the treeline, as their presence would result in excessive CO_2 loss and extended periods of snowpack due to temperature inversions (i.e., positive temperature gradient from the earth surface to the atmosphere). Further, both Arctic and alpine sites exhibit negative work resulting in positive feedback between vegetation heat dissipation and temperature gradient, thereby extending the duration of temperature inversions. These conditions demonstrate thermodynamic infeasibility associated with the counterfactual scenario of trees existing beyond a treeline. Thus, we conclude that, in addition to resource constraints, a treeline is an outcome of an ecosystem’s ability to self-organize towards the most advantageous vegetation structure facilitated by thermodynamic feasibility.

Image Interpolation with Regional Gradient Estimation

This paper proposes an image interpolation method with regional gradient estimation (GEI) to solve the problem of the nonlinear interpolation method not sufficiently considering non-edge pixels. First, the approach presented in this paper expanded on the edge diffusion idea used in CGI and proposed a regional gradient estimation strategy to improve the problem of gradient calculation in the CGI method. Next, the gradient value was used to determine whether a pixel was an edge pixel. Then, a 1D directional filter was employed to process edge pixels while interpolating non-edge pixels using a 2D directionless filter. Finally, we experimented with various representative interpolation methods for grayscale and color images, including the one presented in this paper, and compared them in terms of subjective results, objective criteria, and computational complexity. The experimental results showed that GEI performed better than the other methods in an experiment concerning the visual effect, objective criteria, and computational complexity.

Adaptive stochastic resonance based convolutional neural network for image classification

In this paper, we exploit the adaptive stochastic resonance effect in the convolutional neural network with threshold activation functions for enabling the back-propagation gradient computation. During training, the injection of noise into the threshold activation function makes it to be differentiable at forward pass, thus exact gradients of the loss function with respect to network parameters including the injected noise level can smoothly propagate at backforward pass. This training method effectively uses the non-zero noise to improve the performance of the designed convolutional neural network. In the testing phase, the differentiable activation function is decoupled into a set of threshold functions driven by mutually independent noise components, which endows a hardware-friendly feature of the designed neural network. Experiments show that the designed low-precision convolutional neural network has a comparable performance with the full-precision one with ReLU activation functions on two benchmark data sets. These results also extend the potential application of adaptive stochastic resonance into the convolutional neural network for image classification.

A Real-Time Energy Monitoring System for an MRI Hybrid Ablation System.

The MRI hybrid ablation system is an approach to use the MR (magnetic resonance) scanner's radiofrequency amplifier itself as power source for ablation. Hereby, an electrode is connected to the MR internal radiofrequency amplifier. An average RF power is provided through a train of short RF pulses, which is sufficient to thermally destroy tissue. However, ablation with too high power values can cause tissue carbonizations, thus impeding the ablation procedure. Therefore, monitoring of the energy and the power absorbed inside the tissue is necessary. For this purpose, a measurement system was designed to measure the energy applied to the tissue when using the concept of an MRI hybrid ablation system. The system consists of a dual-directional coupler, RF-to-RMS sensors, and a microcontroller. The gradient calculation of the measured energy curve provides information about the absorbed RF power in the tissue. Validation measurements of the system were performed and compared with measurements from the MR-internal power measurement system. The energy monitoring system was also tested in an ablation experiment with ex-vivo animal tissue using the MRI hybrid ablation system. The measurements showed that the applied RF power can be monitored in real-time. It has been shown that the mean RF power absorbed in the patient decreased during an ablation procedure due to an occurring impedance mismatch and tissue changes. In further work, the influence of the monitoring system on the quality of the MR images should be investigated. Clinical relevance- This paper demonstrates an energy monitoring system for an RF ablation system, which can be used inside an MR environment.

Information geometry of physics-informed statistical manifolds and its use in data assimilation

The data-aware method of distributions (DAMD) is a low-dimensional data assimilation procedure to forecast the behavior of dynamical systems described by differential equations. The core of DAMD is the minimization of a distance between an observation and a prediction in distributional terms, with prior and posterior distributions constrained to a statistical manifold defined by the method of distributions (MD). We leverage the information-geometric properties of the statistical manifold to reduce predictive uncertainty via data assimilation. Specifically, we exploit the information-geometric structures induced by two discrepancy metrics, the Kullback-Leibler divergence and the Wasserstein distance, which explicitly yield natural gradient descent. The use of a deep neural network as a surrogate model for MD enables automatic differentiation, further accelerating optimization. The manifold's geometry is quantified without sampling, yielding an accurate approximation of the gradient descent direction. Our numerical experiments demonstrate that accounting for the manifold's geometry significantly reduces the computational cost of data assimilation by both facilitating the calculation of gradients and reducing the number of required iterations.

Robust binary fringe generation method with defocus adaptability

Current binary defocus technology mainly focuses on fringe generation suitable for specific frequencies without considering fringe adaptability for defocus-degree variation, which decreases the measuring accuracy for this scenario. To achieve a high-quality measurement, we propose a robust binary fringe generation method to minimize the phase error caused by changes in defocus and the random error in measurement. In the method, we establish a complete phase error model and construct a novel objective function, to the best of our knowledge, to optimize the binarization threshold of each pixel. Through derivation of the threshold gradient calculation formula, we quickly obtain optimal binary fringes that can adapt to different fringe pitches and various degrees of defocus. The experimental results verify that the proposed method can generate robust binary fringes adaptive to different fringe pitches and defocus degree variation, and thus achieve high-quality 3D measurements.

Exploring the Effects of Caputo Fractional Derivative in Spiking Neural Network Training

Fractional calculus is an emerging topic in artificial neural network training, especially when using gradient-based methods. This paper brings the idea of fractional derivatives to spiking neural network training using Caputo derivative-based gradient calculation. We focus on conducting an extensive investigation of performance improvements via a case study of small-scale networks using derivative orders in the unit interval. With particle swarm optimization we provide an example of handling the derivative order as an optimizable hyperparameter to find viable values for it. Using multiple benchmark datasets we empirically show that there is no single generally optimal derivative order, rather this value is data-dependent. However, statistics show that a range of derivative orders can be determined where the Caputo derivative outperforms first-order gradient descent with high confidence. Improvements in convergence speed and training time are also examined and explained by the reformulation of the Caputo derivative-based training as an adaptive weight normalization technique.

Novel methods for measuring the thermal diffusivity and the thermal conductivity of a lithium-ion battery

Thermal conductivity is a fundamental parameter in every battery pack model. It allows for the calculation of internal temperature gradients which affect cell safety and cell degradation. The accuracy of the measurement for thermal conductivity is directly proportional to the accuracy of any thermal calculation. Currently the battery industry uses archaic methods for measuring this property which have errors up to 50 %. This includes the constituent material approach, the Searle’s bar method, laser/Xeon flash and the transient plane source method. In this paper we detail three novel methods for measuring both the thermal conductivity and the thermal diffusivity to within 5.6 %. These have been specifically designed for bodies like lithium-ion batteries which are encased in a thermally conductive material. The novelty in these methods comes from maintaining a symmetrical thermal boundary condition about the middle of the cell. By using symmetric boundary conditions, the thermal pathway around the cell casing can be significantly reduced, leading to improved measurement accuracy. These novel methods represent the future for thermal characterisation of lithium-ion batteries. Continuing to use flawed measurement methods will only diminish the performance of battery packs and slow the rate of decarbonisation in the transport sector.

Spatially coherent modeling of 3D FDG-PET data for assessment of intratumoral heterogeneity and uptake gradients.

Purpose: Radiomics have become invaluable for non-invasive cancer patient risk prediction, and the community now turns to exogenous assessment, e.g., from genomics, for interpretability of these agnostic analyses. Yet, some opportunities for clinically interpretable modeling of positron emission tomography (PET) imaging data remain unexplored, that could facilitate insightful characterization at voxel level. Approach: Here, we present a novel deformable tubular representation of the distribution of tracer uptake within a volume of interest, and derive interpretable prognostic summaries from it. This data-adaptive strategy yields a 3D-coherent and smooth model fit, and a profile curve describing tracer uptake as a function of voxel location within the volume. Local trends in uptake rates are assessed at each voxel via the calculation of gradients derived from this curve. Intratumoral heterogeneity can also be assessed directly from it. Results: We illustrate the added value of this approach over previous strategies, in terms of volume rendering and coherence of the structural representation of the data. We further demonstrate consistency of the implementation via simulations, and prognostic potential of heterogeneity and statistical summaries of the uptake gradients derived from the model on a clinical cohort of 158 sarcoma patients imaged with -fluorodeoxyglucose-PET, in multivariate prognostic models of patient survival. Conclusions: The proposed approach captures uptake characteristics consistently at any location, and yields a description of variations in uptake that holds prognostic value complementarily to structural heterogeneity. This creates opportunities for monitoring of local areas of greater interest within a tumor, e.g., to assess therapeutic response in avid locations.

Strategic Generation Investment in Energy Markets: A Multiparametric Programming Approach

An investor has to carefully select the location and size of new generation units it intends to build, since adding capacity in a market affects the profit from units this investor may already own. To capture this closed-loop characteristic, strategic investment (SI) of generation can be posed as a bilevel optimization. By analytically studying a small market, we first show that its objective function can be non-convex and discontinuous. Realizing that existing mixed-integer problem formulations become impractical for larger markets and number of instances, this work put forth two SI solvers: a grid search to handle setups where the candidate investment locations are few, and a stochastic gradient descent approach for otherwise. Both solvers leverage powerful results of multiparametric programming (MPP), each in a unique way. The grid search entails finding the primal/dual solutions for a large number of optimal power flow (OPF) problems, which nonetheless can be efficiently computed several at once thanks to the properties of MPP. The same properties facilitate the rapid calculation of gradients in a mini-batch fashion, thus accelerating the implementation of a stochastic (sub)-gradient descent search. Tests on the IEEE 30- and 118-bus systems using real-world data corroborate the advantages of the novel solvers.

Shrub Ensembles for Online Classification

Online learning algorithms have become a ubiquitous tool in the machine learning toolbox and are frequently used in small, resource-constraint environments. Among the most successful online learning methods are Decision Tree (DT) ensembles. DT ensembles provide excellent performance while adapting to changes in the data, but they are not resource efficient. Incremental tree learners keep adding new nodes to the tree but never remove old ones increasing the memory consumption over time. Gradient-based tree learning, on the other hand, requires the computation of gradients over the entire tree which is costly for even moderately sized trees. In this paper, we propose a novel memory-efficient online classification ensemble called shrub ensembles for resource-constraint systems. Our algorithm trains small to medium-sized decision trees on small windows and uses stochastic proximal gradient descent to learn the ensemble weights of these `shrubs'. We provide a theoretical analysis of our algorithm and include an extensive discussion on the behavior of our approach in the online setting. In a series of 2~959 experiments on 12 different datasets, we compare our method against 8 state-of-the-art methods. Our Shrub Ensembles retain an excellent performance even when only little memory is available. We show that SE offers a better accuracy-memory trade-off in 7 of 12 cases, while having a statistically significant better performance than most other methods. Our implementation is available under https://github.com/sbuschjaeger/se-online .

Differentially Private Normalizing Flows for Synthetic Tabular Data Generation

Normalizing flows have shown to be a promising approach to deep generative modeling due to their ability to exactly evaluate density --- other alternatives either implicitly model the density or use approximate surrogate density. In this work, we present a differentially private normalizing flow model for heterogeneous tabular data. Normalizing flows are in general not amenable to differentially private training because they require complex neural networks with larger depth (compared to other generative models) and use specialized architectures for which per-example gradient computation is difficult (or unknown). To reduce the parameter complexity, the proposed model introduces a conditional spline flow which simulates transformations at different stages depending on additional input and is shared among sub-flows. For privacy, we introduce two fine-grained gradient clipping strategies that provide a better signal-to-noise ratio and derive fast gradient clipping methods for layers with custom parameterization. Our empirical evaluations show that the proposed model preserves statistical properties of original dataset better than other baselines.

Flexible and efficient Bayesian pharmacometrics modeling using Stan and Torsten, Part I.

Stan is an open‐source probabilistic programing language, primarily designed to do Bayesian data analysis. Its main inference algorithm is an adaptive Hamiltonian Monte Carlo sampler, supported by state‐of‐the‐art gradient computation. Stan's strengths include efficient computation, an expressive language that offers a great deal of flexibility, and numerous diagnostics that allow modelers to check whether the inference is reliable. Torsten extends Stan with a suite of functions that facilitate the specification of pharmacokinetic and pharmacodynamic models and makes it straightforward to specify a clinical event schedule. Part I of this tutorial demonstrates how to build, fit, and criticize standard pharmacokinetic and pharmacodynamic models using Stan and Torsten.