Input-output Pairs Research Articles

Learning Elliptic Partial Differential Equations with Randomized Linear Algebra

Given input–output pairs of an elliptic partial differential equation (PDE) in three dimensions, we derive the first theoretically rigorous scheme for learning the associated Green’s function G. By exploiting the hierarchical low-rank structure of G, we show that one can construct an approximant to G that converges almost surely and achieves a relative error of mathcal {O}(varGamma _epsilon ^{-1/2}log ^3(1/epsilon )epsilon ) using at most mathcal {O}(epsilon ^{-6}log ^4(1/epsilon )) input–output training pairs with high probability, for any 0<epsilon <1. The quantity 0<varGamma _epsilon le 1 characterizes the quality of the training dataset. Along the way, we extend the randomized singular value decomposition algorithm for learning matrices to Hilbert–Schmidt operators and characterize the quality of covariance kernels for PDE learning.

From MPC-Based to End-to-End (E2E) Learning-Based Control Policy for Grid-Tied 3L-NPC Transformerless Inverter

This paper proposes an end-to-end (E2E) learning-based control policy to directly control a transformerless grid-tied three-level neutral-point-clamped (3L-NPC) inverter powered by a photovoltaic (PV) array. This E2E control policy is represented by an artificial neural network (ANN) and a time-delay neural network (TDNN), namely, ANN- and TDNN-based control policies, to properly estimate the optimal switching vector of the 3L-NPC. With such learning-based control policy, there exists no need for deriving or understanding deeply the complex mathematical model of the 3L-NPC, as the dynamics of both the system and the control scheme, as well as the cost function to be minimized, are learned via an end-to-end learning fashion that maps directly from the raw observations to the optimal switching states. This definitely eliminates the major barriers of the model-based control strategies (i.e., model predictive control (MPC)) such as (i) the need for an accurate system model, and (ii) the exponential increase in computational complexity. In order to train the two control policies, the conventional MPC is employed, as an expert, for acquiring a set of training data (i.e., input-output pairs) and, thereafter, for assessing our proposed control schemes. The proposed E2E control strategies are validated using MATLAB/Simulink software, where the impact of having different input features and training data are studied. With the proposed control policies, especially the TDNN-based control policy that has only one time-delay window, a high-quality sinusoidal grid current is achieved with low total harmonic distortion (THD), resulting in enhancing the power quality of the utility grid. In addition, the leakage current is minimized compared to the conventional MPC by more than 25%. However, the same dynamic behavior is almost obtained during the irradiation changes compared to the MPC strategy. Moreover, the experimental verification of the proposed E2E control strategy is implemented on the basis of the Hardware-in-the-Loop (HIL) real-time simulator using the C2000TM-microcontroller-LaunchPadXL TMS320F28379D kit, demonstrating the applicability and good performance of our proposed control strategy under realistic conditions.

Physics-informed learning of aerosol microphysics

Abstract Aerosol particles play an important role in the climate system by absorbing and scattering radiation and influencing cloud properties. They are also one of the biggest sources of uncertainty for climate modeling. Many climate models do not include aerosols in sufficient detail due to computational constraints. To represent key processes, aerosol microphysical properties and processes have to be accounted for. This is done in the ECHAM-HAM (European Center for Medium-Range Weather Forecast-Hamburg-Hamburg) global climate aerosol model using the M7 microphysics, but high computational costs make it very expensive to run with finer resolution or for a longer time. We aim to use machine learning to emulate the microphysics model at sufficient accuracy and reduce the computational cost by being fast at inference time. The original M7 model is used to generate data of input–output pairs to train a neural network (NN) on it. We are able to learn the variables’ tendencies achieving an average $ {R}^2 $ score of 77.1%. We further explore methods to inform and constrain the NN with physical knowledge to reduce mass violation and enforce mass positivity. On a Graphics processing unit (GPU), we achieve a speed-up of up to over 64 times faster when compared to the original model.

Nonparametric, Nonasymptotic Confidence Bands With Paley-Wiener Kernels for Band-Limited Functions

The paper introduces a method to construct confidence bands for bounded, band-limited functions based on a finite sample of input-output pairs. The approach is distribution-free w.r.t. the observation noises and only the knowledge of the input distribution is assumed. It is nonparametric, that is, it does not require a parametric model of the regression function and the regions have non-asymptotic guarantees. The algorithm is based on the theory of Paley-Wiener reproducing kernel Hilbert spaces. The paper first studies the fully observable variant, when there are no noises on the observations and only the inputs are random; then it generalizes the ideas to the noisy case using gradient-perturbation methods. Finally, numerical experiments demonstrating both cases are presented.

Probabilistic, high-resolution tsunami predictions in northern Cascadia by exploiting sequential design for efficient emulation

Abstract. The potential of a full-margin rupture along the Cascadia subduction zone poses a significant threat over a populous region of North America. Previous probabilistic tsunami hazard assessment studies produced hazard curves based on simulated predictions of tsunami waves, either at low resolution or at high resolution for a local area or under limited ranges of scenarios or at a high computational cost to generate hundreds of scenarios at high resolution. We use the graphics processing unit (GPU)-accelerated tsunami simulator VOLNA-OP2 with a detailed representation of topographic and bathymetric features. We replace the simulator by a Gaussian process emulator at each output location to overcome the large computational burden. The emulators are statistical approximations of the simulator's behaviour. We train the emulators on a set of input–output pairs and use them to generate approximate output values over a six-dimensional scenario parameter space, e.g. uplift/subsidence ratio and maximum uplift, that represent the seabed deformation. We implement an advanced sequential design algorithm for the optimal selection of only 60 simulations. The low cost of emulation provides for additional flexibility in the shape of the deformation, which we illustrate here considering two families – buried rupture and splay-faulting – of 2000 potential scenarios. This approach allows for the first emulation-accelerated computation of probabilistic tsunami hazard in the region of the city of Victoria, British Columbia.

Brain Computer Interface for Speech Synthesis Based on Multilayer Differential Neural Networks

This manuscript proposes the design of a speech synthesis algorithm based on measured electroencephalographic (EEG) signals previously classified by a class of neural network with continuous dynamics. A novel multilayer differential neural network (MDNN) classifies a database with the EEG studies of 20 volunteers. The database contains information described by input-output pairs corresponding to EEG signals and a corresponding word imagined by the volunteer. The suggested MDNN estimates the unknown relationship between the information instances and suggests the most-likely word that the user wants mentioning. The proposed MDNN satisfactory classifies over 95% a set of words obtained from the suggested EEG study in which, the users have to watch four different geometric figures on a screen.

An N-Port Universal Multimode Optical Router Supporting Mode-Division Multiplexing.

Optical network-on-chip (ONoC) is based on optical interconnects and optical routers (ORs), which have obvious advantages in bandwidth and power consumption. Transmission capacity is a significant performance in ONoC architecture, which has to be fully considered during the design process. Relying on mode-division multiplexing (MDM) technology, the system capacity of optical interconnection is greatly improved compared to the traditional multiplexing technology. With the explosion in MDM technology, the optical router supporting MDM came into being. In this paper, we design a multimode optical router (MDM-OR) model and analyze its indicators. Above all, we propose a novel multimode switching element and design an N-port universal multimode optical router (MDM-OR) model. Secondly, we analyze the insertion loss model of different optical devices and the crosstalk noise model of N-port MDM-OR. On this basis, a multimode router structure of a single-mode five-port optical router is proposed. At the same time, we analyze the transmission loss, crosstalk noise, signal-to-noise radio (OSNR), and bit error rate (BER) of different input–output pairs by inputting the 1550 nm TE, TE, and TE modes to the router.

A Data-Centric Approach to Improve Machine Learning Model’s Performance in Production

Machine learning teaches computers to think in a similar way to how humans do. An ML models work by exploring data and identifying patterns with minimal human intervention. A supervised ML model learns by mapping an input to an output based on labeled examples of input-output (X, y) pairs. Moreover, an unsupervised ML model works by discovering patterns and information that was previously undetected from unlabelled data. As an ML project is an extensively iterative process, there is always a need to change the ML code/model and datasets. However, when an ML model achieves 70-75% of accuracy, then the code or algorithm most probably works fine. Nevertheless, in many cases, e.g., medical or spam detection models, 75% accuracy is too low to deploy in production. A medical model used in susceptible tasks such as detecting certain diseases must have an accuracy label of 98-99%. Furthermore, that is a big challenge to achieve. In that scenario, we may have a good working model, so a model-centric approach may not help much achieve the desired accuracy threshold. However, improving the dataset will improve the overall performance of the model. Improving the dataset does not always require bringing more and more data into the dataset. Improving the quality of the data by establishing a reasonable baseline level of performance, labeler consistency, error analysis, and performance auditing will thoroughly improve the model's accuracy. This review paper focuses on the data-centric approach to improve the performance of a production machine learning model.

A new method to build an associative memory model

An associative memory is an artificial neural network model designed to store and recall input-output patterns pairs by association. A new method for obtaining two associative memories +M and -M is presented in this article, which uses a framework focused on two new binary operations boxplus and boxminus, and a unary operation called projection. The conditions to obtaining perfect recovery and the noise boundaries that the memories can tolerate are studied. Memories +M and -M are robust to additive and subtractive noise, respectively, both types converge in one-step and operate in heteroassociative and autoassociative modes. The performance of the proposed memories is tested against other memory models under identical conditions using the gamma binary distance for measuring similarity between binary patterns. The computer simulation results based on the average of 500 trials with the binary set of 26 lowercase letters of 7x7 pixel size, showed that the proposed memories in autoassociative mode exhibited a gamma binary distance of above .8, .75 and .7 by distorting up to 10 pixels by additive, subtractive, and mixed noise, respectively, which implied that the recovered images had a high similarity. In absence of noise the performance was excellent, i.e., the 100% of the recovered images were identical.

Generalized correntropy induced metric based total least squares for sparse system identification

The total least squares (TLS) method has been successfully applied to system identification in the errors-in-variables (EIV) model, which can efficiently describe systems where input–output pairs are contaminated by noise. In this paper, we propose a new gradient-descent TLS filtering algorithm based on the generalized correntropy induced metric (GCIM), called as GCIM-TLS, for sparse system identification. By introducing GCIM as a penalty term to the TLS problem, we can achieve improved accuracy of sparse system identification. We also characterize the convergence behaviour analytically for GCIM-TLS. To reduce computational complexity, we use the first-order Taylor series expansion and further derive a simplified version of GCIM-TLS. Simulation results verify the effectiveness of our proposed algorithms in sparse system identification.

Learning viscoelasticity models from indirect data using deep neural networks

We propose a novel approach to model viscoelasticity materials, where rate-dependent and non-linear constitutive relationships are approximated with deep neural networks. We assume that inputs and outputs of the neural networks are not directly observable, and therefore common training techniques with input–output pairs for the neural networks are inapplicable. To that end, we develop a novel computational approach to both calibrate parametric and learn neural-network-based constitutive relations of viscoelasticity materials from indirect displacement data in the context of multiple-physics systems. We show that limited displacement data holds sufficient information to quantify the viscoelasticity behavior. We formulate the inverse computation – modeling viscoelasticity properties from observed displacement data – as a PDE-constrained optimization problem and minimize the error functional using a gradient-based optimization method. The gradients are computed by a combination of automatic differentiation and implicit function differentiation rules. The effectiveness of our method is demonstrated through numerous benchmark problems in geomechanics and porous media transport.

Operation Planning Method Using Convolutional Neural Network for Combined Heat and Power System

More and more factories and commercial buildings are installing combined heat and power (CHP) systems that include various energy storage devices. To reduce the energy cost of CHPs, optimal operation plans to satisfy time‐varying energy demands with minimum energy cost are required. Conventional operation planning methods using optimized calculation have an issue with long computing time. To address this, we adapted a convolutional neural network (CNN) to learn the time‐series features of input–output pairs of optimized calculations. The architecture of the proposed CNN has the following key features: (i) the application of one‐dimensional convolutional filters to extract the time‐series features of input data and (ii) the elimination of pooling layers that compress the input data. We also designed rule‐based flows to remove constraint violations from the generated operation plans. Our evaluations showed that the proposed method could generate operation plans with the average error of 3.3%, which is much lower than the 8.0% average error when using the conventional CNN with pooling calculation. © 2021 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

Learning Convex Optimization Models

A convex optimization model predicts an output from an input by solving a convex optimization problem. The class of convex optimization models is large, and includes as special cases many well-known models like linear and logistic regression. We propose a heuristic for learning the parameters in a convex optimization model given a dataset of input-output pairs, using recently developed methods for differentiating the solution of a convex optimization problem with respect to its parameters. We describe three general classes of convex optimization models, maximum a posteriori (MAP) models, utility maximization models, and agent models, and present a numerical experiment for each.

Dynamical system parameter identification using deep recurrent cell networks

In this paper, we investigate the parameter identification problem in dynamical systems through a deep learning approach. Focusing mainly on second-order, linear time-invariant dynamical systems, the topic of damping factor identification is studied. By utilizing a six-layer deep neural network with different recurrent cells, namely GRUs, LSTMs or BiLSTMs; and by feeding input-output sequence pairs captured from a dynamical system simulator, we search for an effective deep recurrent architecture in order to resolve damping factor identification problem. Our study results show that, although previously not utilized for this task in the literature, bidirectional gated recurrent cells (BiLSTMs) provide better parameter identification results when compared to unidirectional gated recurrent memory cells such as GRUs and LSTM. Thus, indicating that an input-output sequence pair of finite length, collected from a dynamical system and when observed anachronistically, may carry information in both time directions for prediction of a dynamical systems parameter.

Quadratic Support Vector Machine and K-Nearest Neighbor Based Robust Sensor Fault Detection and Isolation

Fault detection plays a serious role in high-cost and safety-critical processes. There are two main drivers for continuous improvement in the area of early detection of process faults safety and reliability of technical plants. Detect fault in Geophone string sensors (SG-10) are very important in oil exploration to avoid loss economy. Methods are developed to enable earlier detection of process faults than the traditional limit and trend checking based on a single process variable and the development of these methods is a key matter. Classification methods will be used for pattern recognition and as such is appropriate for fault detection. In supervised training input-output pairs, both for normal and fault conditions, are presented to the network. The models were trained on the free fault and fault sensors. Then the Quadratic Support Vector Machine (QSVM) and k-Nearest Neighbor (KNN) as the classifiers are used. The test results for measuring the performance of 1232 sample classifiers from data show that the accuracy of fault-free sensor recognition is 97.4 % and 100% consecutively for these classifiers.

Towards Extracting Graph Neural Network Models via Prediction Queries (Student Abstract)

Graph data has been widely used to represent data from various domain, e.g., social networks, recommendation system. With great power, the GNN models, usually as valuable properties of their owners, also become attractive targets of the adversary who covets to steal them. While existing works show that simple deep neural networks can be reproduced by so-called Model Extraction Attacks, how to extract a GNN model has not been explored. In this paper, we exploit the threat of model extraction attacks against GNN models. Unlike ordinary attacks which obtain model information via only the input-output query pairs, we utilize both the node queries and the graph structure to extract the GNNs. Furthermore, we consider the stealthiness of the attack and propose to generate legitimate queries so the extraction can be applied discreetly. We implement our attack by leveraging the responses of these queries, as well as other accessible knowledge, e.g., neighbor connectives of the queried nodes. By evaluating over three real-world datasets, our attack is shown to effectively produce a surrogate model with more than 80% equivalent predictions as the target model.

Broad learning robust semi-active structural control: A nonparametric approach

We propose a novel algorithm for dynamic response suppression via semi-active control devices, which we refer to as broad learning, robust, semi-active control (BLRSAC). To configure the semi-active controller, a nonparametric reliability-based output feedback control strategy is introduced. In particular, an adaptive broad learning network is developed to formulate the control strategy using the clipped-optimal control technique. The learning network is augmented incrementally to adopt additional training data based on the inherited information of the trained learning network. By utilizing a robust failure probability, the training dataset is obtained adaptively to include the training input–output pairs with optimal structural control performance. The robust failure probability we propose incorporates both predicted failure probability and the uncertainty of the underlying structure. Therefore, the resultant control strategy can handle the inevitable uncertainty of the actual control situation to achieve optimal structural control. To examine the efficacy of the proposed BLRSAC algorithm, illustrative examples of a shear building and a three-dimensional braced frame under various external excitation and structural damaging conditions are presented.

Application of physics-informed neural networks for self-similar and transient solutions of spontaneous imbibition

Spontaneous imbibition is a natural process where the wetting phase spontaneously replaces the non-wetting phase due to the capillary force in porous media. This multiphase flow mechanism happens frequently in many natural processes such as during nonaqueous phase liquids transport, CO2 storage, soil infiltration, improved oil recovery, etc. Due to the nature of the capillary dominated flow, the imbibition rate is a natural consequence of the rock-fluid interaction and remains unknown from the problem configuration. This has made the solution to spontaneous imbibition unique and differentiates it from the traditional Buckley-Leverett type of displacement problems. While many attempts have been made in the literature to solve the spontaneous imbibition governing equations, most of them are quite mathematically involved even for the self-similar solution governed by an ordinary differential equation (ODE). Only a few more complex solutions exist for the transient spontaneous imbibition problems. In this research, we have adopted the concept and workflow of the deep physics-informed neural networks (PINN) to solve the spontaneous imbibition problems, both self-similar and transient. It focuses on learning the prior knowledge embedded within the governing differential equations rather than depending on input-output data pairs. The one-dimensional unsteady state immiscible, incompressible horizontal flow equations with a saturation-dependent dispersion coefficient under Lagrangian formulation is included in the workflow as the main loss function, along with the physics governed boundary conditions such as capillary end effect. The methodology represents a robust and straightforward function approximator for the solutions of this class, and the effectiveness of the proposed framework is demonstrated through two example calculations.

Fundamental As Fewer Bits

A physics theory predicts precise experimental results for some set of naturally occurring phenomena. Consequently, every well-formed physics theory is equivalent to a computer program that uses input descriptions of specific experimental setups to generate the outputs expected from those setups. The Kolmogorov complexity (or Kolmogorov minimum) of such a computer program is the program that uses the smallest number of bits to represent the largest possible of such input-output data pairs accurately. The principle of concise prediction asserts that the theory whose program length is shortest for a given set of experimental inputs and results is the one most likely to lead to deeper insights and new physics.

Deep Model Intellectual Property Protection via Deep Watermarking

Despite the tremendous success, deep neural networks are exposed to serious IP infringement risks. Given a target deep model, if the attacker knows its full information, it can be easily stolen by fine-tuning. Even if only its output is accessible, a surrogate model can be trained through student-teacher learning by generating many input-output training pairs. Therefore, deep model IP protection is important and necessary. However, it is still seriously under-researched. In this work, we propose a new model watermarking framework for protecting deep networks trained for low-level computer vision or image processing tasks. Specifically, a special task-agnostic barrier is added after the target model, which embeds a unified and invisible watermark into its outputs. When the attacker trains one surrogate model by using the input-output pairs of the barrier target model, the hidden watermark will be learned and extracted afterwards. To enable watermarks from binary bits to high-resolution images, a deep invisible watermarking mechanism is designed. By jointly training the target model and watermark embedding, the extra barrier can even be absorbed into the target model. Through extensive experiments, we demonstrate the robustness of the proposed framework, which can resist attacks with different network structures and objective functions.