Invertible Neural Networks Research Articles

Variational inference of fractional Brownian motion with linear computational complexity.

We introduce a simulation-based, amortized Bayesian inference scheme to infer the parameters of random walks. Our approach learns the posterior distribution of the walks' parameters with a likelihood-free method. In the first step a graph neural network is trained on simulated data to learn optimized low-dimensional summary statistics of the random walk. In the second step an invertible neural network generates the posterior distribution of the parameters from the learned summary statistics using variational inference. We apply our method to infer the parameters of the fractional Brownian motion model from single trajectories. The computational complexity of the amortized inference procedure scales linearly with trajectory length, and its precision scales similarly to the Cramér-Rao bound over a wide range of lengths. The approach is robust to positional noise, and generalizes to trajectories longer than those seen during training. Finally, we adapt this scheme to show that a finite decorrelation time in the environment can furthermore be inferred from individual trajectories.

ERGO-ML I: inferring the assembly histories of IllustrisTNG galaxies from integral observable properties via invertible neural networks

ABSTRACT A fundamental prediction of the ΛCDM cosmology is the hierarchical build-up of structure and therefore the successive merging of galaxies into more massive ones. As one can only observe galaxies at one specific time in the cosmic history, this merger history remains, in principle, unobservable. By using the TNG100 simulation of the IllustrisTNG project, we show that it is possible to infer the unobservable stellar assembly and merger history of central galaxies from their observable properties by using machine learning techniques. In particular, in this first paper of ERGO-ML (Extracting Reality from Galaxy Observables with Machine Learning), we choose a set of seven observable integral properties of galaxies to infer the stellar ex-situ fraction, the average merger lookback times and mass ratios, and the lookback time and stellar mass of the last major merger. To infer the posterior distribution for these parameters and hence estimate the uncertainties in the predictions, we use a conditional Invertible Neural Network (cINN). We find that the stellar ex-situ fraction and the time of the last major merger are well-determined by the selected set of observables, that the mass-weighted merger mass ratio is unconstrained, and that, beyond stellar mass, stellar morphology and stellar age are the most informative properties. Finally, we show that the cINN recovers the remaining unexplained scatter and secondary cross-correlations. Overall, this is a first step towards a tool that can be applied to large galaxy surveys in order to infer unobservable properties of the galaxies’ past, enabling empirical studies of galaxy evolution enriched by cosmological simulations.

LINES: Log-Probability Estimation via Invertible Neural Networks for Enhanced Sampling.

It is very challenging to sample a molecular process with large activation energies using molecular dynamics simulations. Current enhanced sampling methodologies, such as umbrella sampling and metadynamics, rely on the identification of appropriate reaction coordinates for a system. In this paper, we developed a method for log-probability estimation via invertible neural networks for enhanced sampling (LINES). This iterative scheme utilizes a normalizing flow machine learning model to learn the underlying free energy surface (FES) of a system as a function of molecular coordinates and then applies a gradient-based optimization method to the learned normalizing flow to identify reaction coordinates. A biasing potential is then evaluated over a tabulated grid of the reaction coordinate values, which can be applied to the next round of simulations for enhanced sampling, resulting in more efficient sampling. We tested the accuracy and efficiency of the LINES method in sampling the FES using the alanine dipeptide system. We also demonstrated the effectiveness of identification of reaction coordinates through simulation of cyclobutanol unbinding from β-cyclodextrin and the folding/unfolding of CLN025─a variant of the peptide Chignolin. The LINES method can be extended to the study of large-scale protein systems with complex nonlinear reaction pathways.

On Sampling Minimum Energy Path

Sampling the minimum energy path (MEP) between two minima of a system is often hindered by the presence of an energy barrier separating the two metastable states. As a consequence, direct sampling based on molecular dynamics or Markov Chain Monte Carlo methods becomes inefficient, the crossing of the energy barrier being associated to a rare event. Augmented sampling methods based on the definition of collective variables or reaction coordinates allow us to circumvent this limitation at the price of an arbitrary choice of the dimensionality reduction algorithm. We couple the statistical sampling techniques, namely, metadynamics and invertible neural networks, with autoencoders so as to gradually learn the MEP and the collective variable at the same time. Learning is achieved through a succession of two steps: statistical sampling of the most probable path between the two minima and redefinition of the collective variable from the updated data points. The prototypical Mueller potential with nearly orthogonal minima is employed to demonstrate the ability of such coupling to unravel a complex MEP.

Generative Inverse Design of Aerodynamic Shapes Using Conditional Invertible Neural Networks

Abstract Conditional invertible neural networks (cINNs) were used for generative inverse design of aerodynamic shapes for a given aerodynamic performance target. The methodology was used to generate two-dimensional (2D) airfoil shapes for a target lift coefficient and three-dimensional (3D) vehicle shapes for a low drag vehicle given an aerodynamic drag coefficient target. Training data for both cases were generated for the forward process i.e., aerodynamic performance as a function of design variables that define the airfoil or vehicle shape, using design of experiments (DOE) and computational fluid dynamics (CFD) simulations. Due to the structure of the cINNs, the inverse process was learned implicitly, i.e., samples from latent space were transformed back to the design variables. The designs generated by the trained cINN model were simulated under identical conditions to check if they met the desired aerodynamic performance target. The distribution of design variables conditioned on a performance target learned by the cINN model was compared to the distribution in the training data. cINNs provide an easy-to-use tool to generate new designs that meet the desired aerodynamic performance, thereby, reducing the iteration time between aerodynamicists and stylists. In case of vehicle shape generation, since all generated vehicle shapes meet the aerodynamic performance target, the designer can select the shapes that do not conflict with other design constraints such as the interior volume, comfort, styling, and various safety requirements.

Controlling non-stationarity and periodicities in time series generation using conditional invertible neural networks

Generated synthetic time series aim to be both realistic by mirroring the characteristics of real-world time series and useful by including characteristics that are useful for subsequent applications, such as forecasting and missing value imputation. To generate such realistic and useful time series, we require generation methods capable of controlling the non-stationarity and periodicities of the generated time series. However, existing approaches do not consider such explicit control. Therefore, in the present paper, we present a novel approach to control non-stationarity and periodicities with calendar and statistical information when generating time series. We first define the requirements for methods to generate time series with non-stationarity and periodicities, which we show are not fulfilled by existing generation methods. Second, we formally describe the novel approach for controlling non-stationarity and periodicities in generated time series. Thirdly, we introduce an exemplary implementation of this approach using a conditional Invertible Neural Network (cINN). We evaluate this cINN empirically in experiments with real-world data sets and compare it to state-of-the-art time series generation methods. Our experiments show that the evaluated cINN can generate time series with controlled periodicities and non-stationarity, and it also generally outperforms the selected benchmarks.

Pan-Sharpening with Customized Transformer and Invertible Neural Network

In remote sensing imaging systems, pan-sharpening is an important technique to obtain high-resolution multispectral images from a high-resolution panchromatic image and its corresponding low-resolution multispectral image. Owing to the powerful learning capability of convolution neural network (CNN), CNN-based methods have dominated this field. However, due to the limitation of the convolution operator, long-range spatial features are often not accurately obtained, thus limiting the overall performance. To this end, we propose a novel and effective method by exploiting a customized transformer architecture and information-lossless invertible neural module for long-range dependencies modeling and effective feature fusion in this paper. Specifically, the customized transformer formulates the PAN and MS features as queries and keys to encourage joint feature learning across two modalities while the designed invertible neural module enables effective feature fusion to generate the expected pan-sharpened results. To the best of our knowledge, this is the first attempt to introduce transformer and invertible neural network into pan-sharpening field. Extensive experiments over different kinds of satellite datasets demonstrate that our method outperforms state-of-the-art algorithms both visually and quantitatively with fewer parameters and flops. Further, the ablation experiments also prove the effectiveness of the proposed customized long-range transformer and effective invertible neural feature fusion module for pan-sharpening.

RID-Noise: Towards Robust Inverse Design under Noisy Environments

From an engineering perspective, a design should not only perform well in an ideal condition, but should also resist noises. Such a design methodology, namely robust design, has been widely implemented in the industry for product quality control. However, classic robust design requires a lot of evaluations for a single design target, while the results of these evaluations could not be reused for a new target. To achieve data-efficient robust design, we propose Robust Inverse Design under Noise (RID-Noise), which can utilize existing data to train a conditional invertible neural network. Specifically, we estimate the robustness of a design parameter by its predictability, measured by the prediction error of a forward neural network. We also define a sample-wise weight, which can be used in the maximum weighted likelihood estimation of an inverse model based on a conditional invertible neural network. With the visual results from experiments, we clearly justify how RID-Noise works by learning the distribution and robustness from data. Further experiments on several real-world benchmark tasks with noises confirm that our method is more effective than other state-of-the-art inverse design methods. Code and supplementary is publicly available at https://github.com/ThyrixYang/rid-noise-aaai22

Manifold embedding data-driven mechanics

This article introduces a manifold embedding data-driven paradigm to solve small- and finite-strain elasticity problems without a conventional constitutive law. This formulation follows the classical data-driven paradigm by seeking the solution that obeys the balance of linear momentum and compatibility conditions while remaining consistent with the material data by minimizing a distance measure. Our key point of departure is the introduction of a global manifold embedding as a means to learn the geometrical trend of the constitutive data mathematically represented by a smooth manifold. By training an invertible neural network to embed the data of an underlying constitutive manifold onto a Euclidean space, we reformulate the local distance-minimization problem that requires a computationally intensive combinatorial search to identify the optimal data points closest to the conservation law with a cost-efficient projection step. Meanwhile, numerical experiments performed on path-independent elastic materials of different material symmetries suggest that the geometrical inductive bias learned by the neural network is helpful to ensure more consistent predictions when dealing with data sets of limited sizes or those with missing data.

Machine Learning Hidden Symmetries.

We present an automated method for finding hidden symmetries, defined as symmetries that become manifest only in a new coordinate system that must be discovered. Its core idea is to quantify asymmetry as violation of certain partial differential equations, and to numerically minimize such violation over the space of all invertible transformations, parametrized as invertible neural networks. For example, our method rediscovers the famous Gullstrand-Painlevé metric that manifests hidden translational symmetry in the Schwarzschild metric of nonrotating black holes, as well as Hamiltonicity, modularity, and other simplifying traits not traditionally viewed as symmetries.

Wind turbine blade design with airfoil shape control using invertible neural networks

Wind turbine blade design is a highly multidisciplinary process that involves aerodynamics, structures, controls, manufacturing, costs, and other considerations. More efficient blade designs can be found by controlling the airfoil cross-sectional shapes simultaneously with the bulk blade twist and chord distributions. Prior work has focused on incorporating panel-based aerodynamic solvers with a blade design framework to allow for airfoil shape control within the design loop in a tractable manner. Including higher fidelity aerodynamic solvers, such as computational fluid dynamics, makes the design problem computationally intractable. In this work, we couple an invertible neural network trained on high-fidelity airfoil aerodynamic data to a turbine design framework to enable the design of airfoil cross sections within a larger blade design problem. We detail the methodology of this coupled framework and showcase its efficacy by aerostructurally redesigning the IEA 15-MW reference wind turbine blade. The coupled approach reduces the cost of energy by 0.9% compared to a more conventional design approach. This work enables the inclusion of high-fidelity aerodynamic data earlier in the design process, reducing cycle time and increasing certainty in the performance of the optimal design.

Invertible neural networks for reconstructing acoustic fields

Sound field reconstruction from finite measurement arrays provides a means to interpolate and extrapolate acoustic quantities that describe the field. By assuming a linear projection on a basis that follows a principled source propagation, one can recover accurate estimates of the aforementioned sound fields. However, the recovery of the basis coefficients relies on explicit models of which measurement noise and data incompleteness can profoundly affect the uncertainty of the solution. This work aims to estimate the distribution of the underlying pressure conditioned on the observations of the measured pressure in a room. A framework for approximate inference is adapted for sound field reconstruction by applying generative flow-based models and invertible neural network architectures. In particular, we use conditional normalising flows for fast conditional posterior estimation and uncertainty quantification. The model's evaluation is carried out using experimental data measured with a spherical array and compared to hierarchical Bayes with Markov Chain Monte-Carlo sampling.

Fast location and segmentation of high-throughput damaged soybean seeds with invertible neural networks.

Fast identification of damaged soybean seeds has undeniable importance in seed sorting and food quality. Mechanical vibration is generally used in soybean seed sorting, but this can seriously damage soybean seeds. The convolutional neural network (CNN) is considered an effective method for location and segmentation tasks. However, a CNN requires a large amount of ground truth data and has high computational cost. First, we propose a self-supervision manner to automatically generate ground truths, which can theoretically create an almost unlimited number of labeled images. Second, instead of using popular CNNs, a novel invertible convolution (involution)-enabled scheme is proposed by using the bottleneck block of the residual networks. Third, a feature selection feature pyramid network (FS-FPN) based on involution is designed, which selects features more flexibly and adaptively. We further merge involution-based backbones and FS-FPN into a unified network, achieving an end-to-end seed location and segmentation model; the best mean average precision of location and segmentation achieved was 85.1% and 81% respectively. The experimental results demonstrate that the proposed method greatly improves the performance of the baseline network with faster speed and fewer parameters, enabling it to detect soybean seeds more effectively. © 2022 Society of Chemical Industry.

Event Detection and Identification in Distribution Networks Based on Invertible Neural Networks and Pseudo Labels

Anomalous event detection and identification are important to support situational awareness and security analysis in power grids. Particularly, the distribution network is with complicated topology, variable load behaviors, and integration of nonlinear distributed generators (DGs), which is difficult to implement complete modeling mathematically. With the deployment of advanced measurement devices such as μPMUs in distribution networks, massive data containing rich system status information becomes available. In this paper, a framework for event detection, localization, and classification is studied to extract event features from measurements in distribution networks. Specifically, a method based on an invertible neural network (INN) is employed to model the complex distributions of normal-state measurements offline in a flexible way. It then establishes explicit likelihoods as the indicator to enable real-time event detection. Furthermore, a Jacobian-based method is utilized for spatial localization. Finally, as the events in practical power grids are mostly recorded unlabeled, the pseudo label (PL) based approach, superior in the separating ability for events under a low labeling rate, and is used to implement event classification. Several typical types of events simulated in the IEEE 34-bus system and real-world cases in a low-voltage system verify the effectiveness and superiorities of the framework.

Model updating of a wind turbine blade finite element Timoshenko beam model with invertible neural networks

Abstract. Digitalization, especially in the form of a digital twin, is fast becoming a key instrument for the monitoring of a product's life cycle from manufacturing to operation and maintenance and has recently been applied to wind turbine blades. Here, model updating plays an important role for digital twins, in the form of adjusting the model to best replicate the corresponding real-world counterpart. However, classical updating methods are generally limited to a reduced parameter space due to low computational efficiency. Moreover, these approaches most likely lack a probabilistic evaluation of the result. The purpose of this paper is to extend a previous feasibility study to a finite element Timoshenko beam model of a full blade for which the model updating process is conducted through the novel approach with invertible neural networks (INNs). This type of artificial neural network is trained to represent an inversion of the physical model, which in general is complex and non-linear. During the updating process, the inverse model is evaluated based on the target model's modal responses. It then returns the posterior prediction for the input parameters. In advance, a global sensitivity study will reduce the parameter space to a significant subset on which the updating process will focus. The finally trained INN excellently predicts the input parameters' posterior distributions of the proposed generic updating problem. Moreover, intrinsic model ambiguities, such as material densities of two closely located laminates, are correctly captured. A robustness analysis with noisy response reveals a few sensitive parameters, though most can still be recovered with equal accuracy. And, finally, after the resimulation analysis with the updated model, the modal response perfectly matches the target values. Thus, we successfully confirmed that INNs offer an extraordinary capability for structural model updating of even more complex and larger models of wind turbine blades.

Retrieval of aerosol properties from in situ, multi-angle light scattering measurements using invertible neural networks

Atmospheric aerosols have a major influence on the earth’s climate and public health. Hence, studying their properties and recovering them from light scattering measurements is of great importance. State of the art retrieval methods such as pre-computed look-up tables and iterative, physics-based algorithms can suffer from either accuracy or speed limitations. These limitations are becoming increasingly restrictive as instrumentation technology advances and measurement complexity increases. Machine learning algorithms offer new opportunities to overcome these problems, by being quick and precise. In this work we present a method, using invertible neural networks to retrieve aerosol properties from in situ light scattering measurements. In addition, the algorithm is capable of simulating the forward direction, from aerosol properties to measurement data. The applicability and performance of the algorithm are demonstrated with simulated measurement data, mimicking in situ laboratory and field measurements. With a retrieval time in the millisecond range and a weighted mean absolute percentage error of less than 1.5%, the algorithm turned out to be fast and accurate. By introducing Gaussian noise to the data, we further demonstrate that the method is robust with respect to measurement errors. In addition, realistic case studies are performed to demonstrate that the algorithm performs well even with missing measurement data.

Inversion of low- to medium-frequency velocities and densities from AVO data using invertible neural networks

Amplitude-variation-with-offset (AVO) inversion and neural networks (NN) are widely used to invert elastic parameters. With more constraints from well-log data, neural network-based inversion may estimate elastic parameters with greater precision and resolution than traditional AVO inversion; however, neural network approaches necessitate a massive number of reliable training samples. Furthermore, because the lack of low-frequency information in seismic gathers leads to multiple solutions of the inverse problem, both inversions rely heavily on proper low-frequency initial models. To mitigate the dependence of inversions on accurate training samples and initial models, we have adopted solving inverse problems with the recently developed invertible neural networks (INNs). Unlike conventional neural networks, which address the ambiguous inverse issues directly, INNs learn definite forward modeling and use additional latent variables to increase the uniqueness of solutions. Motivated by the newly developed neural networks, we adopt an INN-based AVO inversion method, which can reliably invert low- to medium-frequency velocities and densities with randomly generated easy-to-access data sets rather than trustworthy training samples or well-prepared initial models. Tests on synthetic and field data indicate that our method is feasible, antinoise capable, and practicable.

Invertible Neural Networks for Airfoil Design

The airfoil design problem, in which an engineer seeks a shape with desired performance characteristics, is fundamental to aerodynamics. Design workflows traditionally rely on iterative optimization methods using low-fidelity integral boundary-layer methods as higher-fidelity adjoint-based computational fluid dynamics methods are computationally expensive. Surrogate-based approaches can accelerate the design process but still rely on some iterative inverse design procedure. In this work, we leverage emerging invertible neural network (INN) tools to enable the rapid inverse design of airfoil shapes for wind turbines. INNs are specialized deep-learning models with well-defined inverse mappings. When trained appropriately, INN surrogate models are capable of forward prediction of aerodynamic and structural quantities for a given airfoil shape as well as inverse recovery of airfoil shapes with specified aerodynamic and structural characteristics. The INN approach offers a roughly 100 times speed-up compared to adjoint-based methods for inverse design. We demonstrate the INN tool for inverse design on three test cases of 100 airfoils each that satisfy the performance characteristics close to those of airfoils used in wind-turbine blades. All generated shapes satisfy the desired aerodynamic characteristics, demonstrating the success of the INN approach for inverse design of airfoils.

Inference of cosmic-ray source properties by conditional invertible neural networks

The inference of physical parameters from measured distributions constitutes a core task in physics data analyses. Among recent deep learning methods, so-called conditional invertible neural networks provide an elegant approach owing to their probability-preserving bijective mapping properties. They enable training the parameter-observation correspondence in one mapping direction and evaluating the parameter posterior distributions in the reverse direction. Here, we study the inference of cosmic-ray source properties from cosmic-ray observations on Earth using extensive astrophysical simulations. We compare the performance of conditional invertible neural networks (cINNs) with the frequently used Markov Chain Monte Carlo (MCMC) method. While cINNs are trained to directly predict the parameters’ posterior distributions, the MCMC method extracts the posterior distributions through a likelihood function that matches simulations with observations. Overall, we find good agreement between the physics parameters derived by the two different methods. As a result of its computational efficiency, the cINN method allows for a swift assessment of inference quality.

Emission-line diagnostics of H ii regions using conditional invertible neural networks

ABSTRACT Young massive stars play an important role in the evolution of the interstellar medium (ISM) and the self-regulation of star formation in giant molecular clouds (GMCs) by injecting energy, momentum, and radiation (stellar feedback) into surrounding environments, disrupting the parental clouds, and regulating further star formation. Information of the stellar feedback inheres in the emission we observe, however inferring the physical properties from photometric and spectroscopic measurements is difficult, because stellar feedback is a highly complex and non-linear process, so that the observational data are highly degenerate. On this account, we introduce a novel method that couples a conditional invertible neural network (cINN) with the WARPFIELD-emission predictor (WARPFIELD-EMP) to estimate the physical properties of star-forming regions from spectral observations. We present a cINN that predicts the posterior distribution of seven physical parameters (cloud mass, star formation efficiency, cloud density, cloud age which means age of the first generation stars, age of the youngest cluster, the number of clusters, and the evolutionary phase of the cloud) from the luminosity of 12 optical emission lines, and test our network with synthetic models that are not used during training. Our network is a powerful and time-efficient tool that can accurately predict each parameter, although degeneracy sometimes remains in the posterior estimates of the number of clusters. We validate the posteriors estimated by the network and confirm that they are consistent with the input observations. We also evaluate the influence of observational uncertainties on the network performance.