Invertible Neural Networks Research Articles

Missing data in amortized simulation-based neural posterior estimation.

Amortized simulation-based neural posterior estimation provides a novel machine learning based approach for solving parameter estimation problems. It has been shown to be computationally efficient and able to handle complex models and data sets. Yet, the available approach cannot handle the in experimental studies ubiquitous case of missing data, and might provide incorrect posterior estimates. In this work, we discuss various ways of encoding missing data and integrate them into the training and inference process. We implement the approaches in the BayesFlow methodology, an amortized estimation framework based on invertible neural networks, and evaluate their performance on multiple test problems. We find that an approach in which the data vector is augmented with binary indicators of presence or absence of values performs the most robustly. Indeed, it improved the performance also for the simpler problem of data sets with variable length. Accordingly, we demonstrate that amortized simulation-based inference approaches are applicable even with missing data, and we provide a guideline for their handling, which is relevant for a broad spectrum of applications.

SIHNet: A safe image hiding method with less information leaking

AbstractImage hiding is a task that hides secret images into cover images. The purposes of image hiding are to ensure the secret images are invisible to the human and the secret images can be recovered. The current state‐of‐the‐art steganography methods run the risk of secret information leakage. A safe image hiding network (SIHNet) is presented to reduce the leakage of secret information. Based on some phenomena of image hiding methods which use invertible neural network, a reversible secret image processing (SIP) module is proposed to make the secret images suitable for hiding and make the stego images leak less secret information. Besides, a reversible lost information hiding (LIH) module is used to hide the lost information into the cover images, thus the method can recover the secret images better than the method that uses random noise to replace the lost information. Experimental results show that SIHNet outperforms other state‐of‐the‐art methods on the PSNR and SSIM values of the recovered secret images and the stego images. Besides, residual images of other state‐of‐the‐art methods all contain information about secret images while residual images of SIHNet leak almost no secret information. Thus the method can prevent the listener of transmission channel from obtaining the information of the secret image through the residual image, which means SIHNet performs better in security than other state‐of‐the‐art methods.

Generative invertible quantum neural networks

Invertible Neural Networks (INN) have become established tools for the simulation and generation of highly complex data. We propose a quantum-gate algorithm for a Quantum Invertible Neural Network (QINN) and apply it to the LHC data of jet-associated production of a Z-boson that decays into leptons, a standard candle process for particle collider precision measurements. We compare the QINN’s performance for different loss functions and training scenarios. For this task, we find that a hybrid QINN matches the performance of a significantly larger purely classical INN in learning and generating complex data.

Invertible neural networks for uncertainty quantification in refraction tomography

Uncertainty quantification (UQ) should be an essential ingredient of geophysical inversion because it measures the confidence in the results and enables the assessment of the value of information in the data. However, UQ using established methods ranges from very expensive to prohibitively costly, and estimating noise levels and integrating prior information is challenging, so it is not yet widely undertaken. In this paper, we explore the capabilities of a machine learning-based UQ tool known as the invertible neural network (INN) and focus on its application to a 2D tomography problem within a complex foothills environment. We propose a novel approach to handle realistic problem dimensions that uses variational autoencoders to compress the velocity model and data. The INN relates the respective latent spaces, significantly reducing memory requirements. Our findings reveal that this INN-based workflow can perform tomographic inversion while integrating an implicit prior in the form of a set of velocity models with pertinent features. Furthermore, we can address both epistemic and aleatoric uncertainties by adopting a deep ensemble strategy. This integrated approach yields plausible estimates of relative confidence in the inverted velocities, showcasing the potential of INN as a tool for UQ in geophysical inversion.

Learning to solve Bayesian inverse problems: An amortized variational inference approach using Gaussian and Flow guides

Inverse problems, i.e., estimating parameters of physical models from experimental data, are ubiquitous in science and engineering. The Bayesian formulation is the gold standard because it alleviates ill-posedness issues and quantifies epistemic uncertainty. Since analytical posteriors are not typically available, one resorts to Markov chain Monte Carlo sampling or approximate variational inference. However, inference needs to be rerun from scratch for each new set of data. This drawback limits the applicability of the Bayesian formulation to real-time settings, e.g., health monitoring of engineered systems, and medical diagnosis. The objective of this paper is to develop a methodology that enables real-time inference by learning the Bayesian inverse map, i.e., the map from data to posteriors. Our approach is as follows. We parameterize the posterior distribution as a function of data. This work outlines two distinct approaches to do this. The first method involves parameterizing the posterior using an amortized full-rank Gaussian guide, implemented through neural networks. The second method utilizes a Conditional Normalizing Flow guide, employing conditional invertible neural networks for cases where the target posterior is arbitrarily complex. In both approaches, we learn the network parameters by amortized variational inference which involves maximizing the expectation of evidence lower bound over all possible datasets compatible with the model. We demonstrate our approach by solving a set of benchmark problems from science and engineering. Our results show that the posterior estimates of our approach are in agreement with the corresponding ground truth obtained by Markov chain Monte Carlo. Once trained, our approach provides the posterior distribution for a given observation just at the cost of a forward pass of the neural network.

Explainable Intelligent Fault Diagnosis for Nonlinear Dynamic Systems: From Unsupervised to Supervised Learning.

The increased complexity and intelligence of automation systems require the development of intelligent fault diagnosis (IFD) methodologies. By relying on the concept of a suspected space, this study develops explainable data-driven IFD approaches for nonlinear dynamic systems. More specifically, we parameterize nonlinear systems through a generalized kernel representation for system modeling and the associated fault diagnosis. An important result obtained is a unified form of kernel representations, applicable to both unsupervised and supervised learning. More importantly, through a rigorous theoretical analysis, we discover the existence of a bridge (i.e., a bijective mapping) between some supervised and unsupervised learning-based entities. Notably, the designed IFD approaches achieve the same performance with the use of this bridge. In order to have a better understanding of the results obtained, both unsupervised and supervised neural networks are chosen as the learning tools to identify the generalized kernel representations and design the IFD schemes; an invertible neural network is then employed to build the bridge between them. This article is a perspective article, whose contribution lies in proposing and formalizing the fundamental concepts for explainable intelligent learning methods, contributing to system modeling and data-driven IFD designs for nonlinear dynamic systems.

Using conditional Invertible Neural Networks to perform mid‐term peak load forecasting

AbstractMeasures for balancing the electrical grid, such as peak shaving, require accurate peak forecasts for lower aggregation levels of electrical loads. Thus, the Big Data Energy Analytics Laboratory (BigDEAL) challenge—organised by the BigDEAL—focused on forecasting three different daily peak characteristics in low aggregated load time series. In particular, participants of the challenge were asked to provide long‐term forecasts with horizons of up to 1 year in the qualification. The authors present the approach of the KIT‐IAI team from the Institute for Automation and Applied Informatics at the Karlsruhe Institute of Technology. The approach to the challenge is based on a hybrid generative model. In particular, the authors use a conditional Invertible Neural Network (cINN). The cINN gets the forecast of a sliding mean as representative of the trend, different weather features, and calendar information as conditioning input. By this, the proposed hybrid method achieved second place overall and won two out of three tracks of the BigDEAL challenge.

StegEraser: Defending cybersecurity against malicious covert communications

Traditionally, the mission of intercepting malicious traffic between the Internet and the internal network of entities like organizations and corporations, is largely fulfilled by techniques such as deep packet inspection (DPI). However, steganography, the methodology of hiding secret data in seemingly benign public mediums (e.g., images), has been leveraged by advanced persistent threat (APT) groups in recent years, and is almost impossible to be detected and intercepted by traditional techniques, posing a pervasive and realistic threat to cybersecurity. Additionally, internal networks’ vulnerability to steganography is further exacerbated by the connectivity and large attack surface of the Internet of Things (IoT), whose adoption and deployment are quickly expanding. To protect computer systems against malicious communications that apply steganographic methods potentially unknown to cybersecurity stakeholders, we propose StegEraser, an approach to removing the secret information embedded in public mediums by adversaries, that is fundamentally distinct from existing research which is primarily designed for known steganographic methods. Implemented for images, StegEraser injects an excessively huge amount of random binary data with a novel steganographic method into the images, by utilizing the information-merging capabilities of invertible neural networks (INNs), in order to “overload” adversaries’ steganographic hiding capacity of images transmitted through the firewall performing DPI. In the meantime, StegEraser preserves the perceptual quality of the images. In other words, StegEraser “defeats unknown steganography with steganography”. Extensive evaluation verifies that StegEraser significantly outperforms state-of-the-art (SOTA) methods in terms of removing secret information embedded with both traditional and neural network-based steganographic methods, while visually maintaining the image quality.

Learning the long-tail distribution in latent space for Weighted Link Prediction via conditional Invertible Neural Networks

Link prediction is an important problem in dynamic systems, where the goal is to predict future unweighted or weighted link topologies using historical context. Compared with unweighted links, weighted links can preferably reveal the nature and strength of the interactions among entities. However, weighted links also bring greater challenges because they require subtle structural adjustments and numerical variations to be captured. Existing methods are primarily tailored for unweighted links and most generally suffer from low-quality performance when applied to Weighted Link Prediction (WLP) task. In this study, we propose a novel generative framework that adopts conditional Invertible Neural Networks (INNs) to achieve WLP. The proposed framework leverages the benefits of conditional INNs to exactly optimize the log-likelihood in the latent space conditioned on the historical context, which can be sensitive to minor replacements in real-world systems and derive accurate WLPs. Furthermore, to deal with the long-tail statistical phenomenon of edge weights observed in real life, a tail-adaptive distribution is learned in latent space to capture the tail properties and enhance the model’s ability. To verify the effectiveness of the proposed method, we conduct extensive experiments on four datasets from different systems. The experimental results demonstrate that our model achieves impressive results compared to state-of-the-art competitors.

Generating probabilistic forecasts from arbitrary point forecasts using a conditional invertible neural network

In various applications, probabilistic forecasts are required to quantify the inherent uncertainty associated with the forecast. However, many existing forecasting methods still only generate point forecasts. Although methods exist to generate probabilistic forecasts from these point forecasts, these are often limited to prediction intervals or must be trained together with a specific point forecast. Therefore, the present article proposes a novel approach for generating probabilistic forecasts from arbitrary point forecasts. In order to implement this approach, we apply a conditional Invertible Neural Network (cINN) to learn the underlying distribution of the data and then combine the uncertainty from this distribution with an arbitrary point forecast to generate probabilistic forecasts. We evaluate our approach by generating probabilistic forecasts from multiple point forecasts and comparing these forecasts to six probabilistic benchmarks on four data sets. We show that our approach generally outperforms all benchmarks with regard to CRPS and Winkler scores and generates probabilistic forecasts with the narrowest prediction intervals whilst remaining reasonably calibrated. Furthermore, our approach enables simple point forecasting methods to rank highly in the Global Energy Forecasting Competition 2014.

Tandem neural network-assisted inverse design of highly efficient diffractive slanted waveguide grating.

Virtual reality devices featuring diffractive grating components have emerged as hotspots in the field of near-to-eye displays. The core aim of our work is to streamline the intricacies involved in devising the highly efficient slanted waveguide grating using the deep-learning-driven inverse design technique. We propose and establish a tandem neural network (TNN) comprising a generative flow-based invertible neural network and a fully connected neural network. The proposed TNN can automatically optimize the coupling efficiencies of the proposed grating at multi-wavelengths, including red, green, and blue beams at incident angles in the range of 0°-15°. The efficiency indicators manifest in the peak transmittance, average transmittance, and illuminance uniformity, reaching approximately 100%, 92%, and 98%, respectively. Additionally, the structural parameters of the grating can be deduced inversely based on the indicators within a short duration of hundreds of milliseconds to seconds using the TNN. The implementation of the inverse-engineered grating is anticipated to serve as a paradigm for simplifying and expediting the development of diverse types of waveguide gratings.

Lossless image steganography: Regard steganography as super-resolution

Image steganography attempts to imperceptibly hide the secret image within the cover image. Most of the existing deep learning-based steganography approaches have excelled in payload capacity, visual quality, and steganographic security. However, they are difficult to losslessly reconstruct secret images from stego images with relatively large payload capacity. Recently, although some studies have introduced invertible neural networks (INNs) to achieve large-capacity image steganography, these methods still cannot reconstruct the secret image losslessly due to the existence of lost information on the output side of the concealing network. We present an INN-based framework in this paper for lossless image steganography. Specifically, we regard image steganography as an image super-resolution task that converts low-resolution cover images to high-resolution stego images while hiding secret images. The feature dimension of the generated stego image matches the total dimension of the input secret and cover images, thereby eliminating the lost information. Besides, a bijective secret projection module is designed to transform various secret images into a latent variable that follows a simple distribution, improving the imperceptibility of the secret image. Comprehensive experiments indicate that the proposed framework achieves secure hiding and lossless extraction of the secret image.

A deep-learning approach to the 3D reconstruction of dust density and temperature in star-forming regions

Aims. We introduce a new deep-learning approach for the reconstruction of 3D dust density and temperature distributions from multi-wavelength dust emission observations on the scale of individual star-forming cloud cores (<0.2 pc). Methods. We constructed a training data set by processing cloud cores from the Cloud Factory simulations with the POLARIS radiative transfer code to produce synthetic dust emission observations at 23 wavelengths between 12 and 1300 µm. We simplified the task by reconstructing the cloud structure along individual lines of sight (LoSs) and trained a conditional invertible neural network (cINN) for this purpose. The cINN belongs to the group of normalising flow methods and it is able to predict full posterior distributions for the target dust properties. We tested different cINN setups, ranging from a scenario that includes all 23 wavelengths down to a more realistically limited case with observations at only seven wavelengths. We evaluated the predictive performance of these models on synthetic test data. Results. We report an excellent reconstruction performance for the 23-wavelength cINN model, achieving median absolute relative errors of about 1.8% in log(n/m−3) and 1% in log(Tdust/K), respectively. We identify trends towards an overestimation at the low end of the density range and towards an underestimation at the high end of both the density and temperature values, which may be related to a bias in the training data. After limiting our coverage to a combination of only seven wavelengths, we still find a satisfactory performance with average absolute relative errors of about 2.8% and 1.7% in log(n/m−3) and log(Tdust/K). Conclusions. This proof-of-concept study shows that the cINN-based approach for 3D reconstruction of dust density and temperature is very promising and it is even compatible with a more realistically constrained wavelength coverage.

Reconstructing axion-like particles from beam dumps with simulation-based inference

Axion-like particles (ALPs) that decay into photon pairs pose a challenge for experiments that rely on the construction of a decay vertex in order to search for long-lived particles. This is particularly true for beam-dump experiments, where the distance between the unknown decay position and the calorimeter can be very large. In this work we use machine learning to explore the possibility to reconstruct the ALP properties, in particular its mass and lifetime, from such inaccurate observations. We use a simulation-based inference approach based on conditional invertible neural networks to reconstruct the posterior probability of the ALP parameters for a given set of events. We find that for realistic angular and energy resolution, such a neural network significantly outperforms parameter reconstruction from conventional high-level variables while at the same time providing reliable uncertainty estimates. Moreover, the neural network can quickly be re-trained for different detector properties, making it an ideal framework for optimizing experimental design.

On the approximation of bi-Lipschitz maps by invertible neural networks

Invertible neural networks (INNs) represent an important class of deep neural network architectures that have been widely used in applications. The universal approximation properties of INNs have been established recently. However, the approximation rate of INNs is largely missing. In this work, we provide an analysis of the capacity of a class of coupling-based INNs to approximate bi-Lipschitz continuous mappings on a compact domain, and the result shows that it can well approximate both forward and inverse maps simultaneously. Furthermore, we develop an approach for approximating bi-Lipschitz maps on infinite-dimensional spaces that simultaneously approximate the forward and inverse maps, by combining model reduction with principal component analysis and INNs for approximating the reduced map, and we analyze the overall approximation error of the approach. Preliminary numerical results show the feasibility of the approach for approximating the solution operator for parameterized second-order elliptic problems.

An unfolding method based on conditional invertible neural networks (cINN) using iterative training

The unfolding of detector effects is crucial for the comparison of data to theory predictions. While traditional methods are limited to representing the data in a low number of dimensions, machine learning has enabled new unfolding techniques while retaining the full dimensionality. Generative networks like invertible neural networks~(INN) enable a probabilistic unfolding, which map individual data events to their corresponding unfolded probability distribution. The accuracy of such methods is however limited by how well simulated training samples model the actual data that is unfolded. We introduce the iterative conditional INN (IcINN) for unfolding that adjusts for deviations between simulated training samples and data. The IcINN unfolding is first validated on toy data and then applied to pseudo-data for the pp \to Z \gamma \gammapp→Zγγ process.

Extended Dynamic Mode Decomposition with Invertible Dictionary Learning

The Koopman operator has received attention for providing a potentially global linearization representation of the nonlinear dynamical system. To estimate or control the original system, the invertibility problem is introduced into the data-driven modeling, i.e., the observables are required to be reconstructed the original system’s states. Existing methods cannot solve this problem perfectly. Only linear or nonlinear but lossy reconstruction can be achieved. This paper proposed a novel data-driven modeling approach, denoted as the Extended Dynamic Mode Decomposition with Invertible Dictionary Learning (EDMD-IDL) to address this issue, which can be interpreted as a further extension of the classical Extended Dynamic Mode Decomposition (EDMD). The Invertible Neural Network (INN) is introduced in the proposed method, where its inverse process provides the explicit inverse on the dictionary functions, thus allowing the nonlinear and lossless reconstruction. An iterative algorithm is designed to solve the extended optimization problem defined by the Koopman operator and INN by combining the optimization algorithm based on the gradient descent and the classical EDMD method, making the method successfully obtain the finite-dimensional approximation of the Koopman operator. The method is tested on various canonical nonlinear dynamical systems and is shown that the predictions obtained in a linear fashion and the ground truth match well over the long-term, where only the initial status is provided. Comparison experiments highlight the superiority of the proposed method over the other EDMD-based methods. Notably, a typical example in fluid dynamics, cylinder wake, illustrates the potential of the method to be further extended to the high-dimensional system with tens of thousands of states. By combining the Proper Orthogonal Decomposition technique, nontrivial Kármán vortex sheet phenomenon is perfectly reconstructed. Our proposed method provides a new paradigm for solving the finite-dimensional approximation of the Koopman operator and applying it to data-driven modeling.

Robust Reconstruction of the Void Fraction from Noisy Magnetic Flux Density Using Invertible Neural Networks.

Electrolysis stands as a pivotal method for environmentally sustainable hydrogen production. However, the formation of gas bubbles during the electrolysis process poses significant challenges by impeding the electrochemical reactions, diminishing cell efficiency, and dramatically increasing energy consumption. Furthermore, the inherent difficulty in detecting these bubbles arises from the non-transparency of the wall of electrolysis cells. Additionally, these gas bubbles induce alterations in the conductivity of the electrolyte, leading to corresponding fluctuations in the magnetic flux density outside of the electrolysis cell, which can be measured by externally placed magnetic sensors. By solving the inverse problem of the Biot-Savart Law, we can estimate the conductivity distribution as well as the void fraction within the cell. In this work, we study different approaches to solve the inverse problem including Invertible Neural Networks (INNs) and Tikhonov regularization. Our experiments demonstrate that INNs are much more robust to solving the inverse problem than Tikhonov regularization when the level of noise in the magnetic flux density measurements is not known or changes over space and time.

Transformer-based dual path cross fusion for pansharpening remote sensing images

ABSTRACT The purpose of pan-sharpening is to generate high-resolution multispectral (HRMS) images by combining multispectral images (MS) and panchromatic images (PAN), which have become an important part of remote sensing image processing. Therefore, how to better extract complete feature information from MS images and PAN images has become the focus of our attention. In this paper, we propose a new pansharpening method, called Transformer-based Dual-path cross fusion network (TDF) for Pan-sharpening remote sensing images, which aims to extract the spatial details of PAN images while maintaining the spectral fidelity of MS images. The whole network structure can be divided into two parts: in the encoder part, we adopt the Swin-Transformer module for the downsampling operation, which expands the sensory field of the network to the feature map, and then extracts the global information. However, since the Swin-Transformer module is not good at extracting pixel-level details, we introduce the Base Feature Extraction (BFE) and the Invertible Neural Network Block (INNB) modules for the interaction between the local and the global feature information. we also introduce the Edge-Enhancement Block (EEB) to further enhance the feature extraction at multi-scales during the image fusion process. In the decoder section, we once again employ the Swin Transformer module for downsampling. After the convolution operation and activation function, we utilize the Sub-Pixel Convolutional Neural Network for upsampling to generate the ultimate high-resolution multispectral images. Simulation experiments and real experiments are conducted on QuickBird (QB) and WorldView2 (WV2) datasets, which demonstrated our method are superior to the current methods.

Koopman operator learning using invertible neural networks

In Koopman operator theory, a finite-dimensional nonlinear system is transformed into an infinite but linear system using a set of observable functions. However, manually selecting observable functions that span the invariant subspace of the Koopman operator based on prior knowledge is inefficient and challenging, particularly when little or no information is available about the underlying systems. Furthermore, current methodologies tend to disregard the importance of the invertibility of observable functions, which leads to inaccurate results. To address these challenges, we propose the so-called FlowDMD, aka Flow-based Dynamic Mode Decomposition, that utilizes the Coupling Flow Invertible Neural Network (CF-INN) framework. FlowDMD leverages the intrinsically invertible characteristics of the CF-INN to learn the invariant subspaces of the Koopman operator and accurately reconstruct state variables. Numerical experiments demonstrate the superior performance of our algorithm compared to state-of-the-art methodologies.