Input Space Research Articles

Robust and provably monotonic networks

The Lipschitz constant of the map between the input and output space represented by a neural network is a natural metric for assessing the robustness of the model. We present a new method to constrain the Lipschitz constant of dense deep learning models that can also be generalized to other architectures. The method relies on a simple weight normalization scheme during training that ensures the Lipschitz constant of every layer is below an upper limit specified by the analyst. A simple monotonic residual connection can then be used to make the model monotonic in any subset of its inputs, which is useful in scenarios where domain knowledge dictates such dependence. Examples can be found in algorithmic fairness requirements or, as presented here, in the classification of the decays of subatomic particles produced at the CERN Large Hadron Collider. Our normalization is minimally constraining and allows the underlying architecture to maintain higher expressiveness compared to other techniques which aim to either control the Lipschitz constant of the model or ensure its monotonicity. We show how the algorithm was used to train a powerful, robust, and interpretable discriminator for heavy-flavor-quark decays, which has been adopted for use as the primary data-selection algorithm in the LHCb real-time data-processing system in the current LHC data-taking period known as Run 3. In addition, our algorithm has also achieved state-of-the-art performance on benchmarks in medicine, finance, and other applications.

Deep residual error and bag-of-tricks learning for gravitational wave surrogate modeling

Deep learning methods have been employed in gravitational-wave astronomy to accelerate the construction of surrogate waveforms for the inspiral of spin-aligned black hole binaries, among other applications. We face the challenge of modeling the residual error of an artificial neural network that models the coefficients of the surrogate waveform expansion (especially those of the phase of the waveform) which we demonstrate has sufficient structure to be learnable by a second network. Adding this second network, we were able to reduce the maximum mismatch for waveforms in a validation set by 13.4 times. We also explored several other ideas for improving the accuracy of the surrogate model, such as the exploitation of similarities between waveforms, the augmentation of the training set, the dissection of the input space, using dedicated networks per output coefficient and output augmentation. In several cases, small improvements can be observed, but the most significant improvement still comes from the addition of a second network that models the residual error. Since the residual error for more general surrogate waveform models (when e.g., eccentricity is included) may also have a specific structure, one can expect our method to be applicable to cases where the gain in accuracy could lead to significant gains in computational time.

Multi-kernel learning for multi-label classification with local Rademacher complexity

Multi-label classification aims to construct prediction models from input space to output space for multi-label datesets. However, the feature space of multi-label dataset and the hypothesis space of classifier are always complex. In order to obtain high performance multi-kernel learning algorithms, the core problem is to design algorithms that compress both hypothesis space and feature space. In this paper, by combining the local Rademacher complexity and Hilbert-Schmidt independence criterion (HSIC), an effective multi-kernel learning algorithm for multi-label classification is proposed to compress the feature space and hypothesis space simultaneously. Based on the tail sum of eigenvalues of the integral operator corresponding to kernels, the upper bound of local Rademacher complexity of linear functions class in the feature space is determined. The Hilbert-Schmidt independence criterion is applied to maximize the correlation between the convex combination of the input kernel matrices and the ideal kernel matrix in the label space. Finally, the Laplace regularization model is built by controlling the tail sum of eigenvalues of the integral operator corresponding to kernels of this criterion. Therefore, the above process obtains a combined kernel, simultaneously by sorting the coefficients of the combined kernel, a feature selection algorithm can be constructed. On the basis, the combined kernel can also be used to design a new binary classifier for multi-label classification. The experimental results verify that our proposed multi-kernel learning algorithms can effectively compress the hypothesis space, while the feature selection algorithms can effectively compress the feature space.

A novel semi-supervised fault detection and isolation method for battery system of electric vehicles

The detection and isolation of early and minor faults in vehicle battery systems is vital to safe driving and improving power utilization. This paper proposes a data-driven model to achieve accurate, early, and economical fault detection and isolation. The model is based on kernel principal component analysis (KPCA), which maps complex nonlinear data from the input space into a high-dimensional feature space to gain a detection model with good performance. To overcome the difficulty of hyperparameter selection, KPCA is trained using Bayesian Optimization (BO) iterations with a small amount of labeled data and a large amount of unlabeled data. This step can obtain the optimal hyperparameter to greatly improve the model fault detection capability, which is beneficial for detecting both early faults and minor faults. In addition, a unified contribution graph based on the partial differentiation of KPCA was adopted to build a reasonable isolation scheme. The semi-supervised model of KPCA based on Bayesian Optimization and contribution graph is developed to reveal the relationship between fault and variable. Finally, the proposed method is fully tested on four fault datasets and the results prove the excellent detection capability in the early stage of faults compared with other methods and the accurate fault isolation capability from the occurrence to the end of the fault.

Correlated Chained Gaussian Processes for Datasets With Multiple Annotators.

The labeling process within a supervised learning task is usually carried out by an expert, which provides the ground truth (gold standard) for each sample. However, in many real-world applications, we typically have access to annotations provided by crowds holding different and unknown expertise levels. Learning from crowds (LFC) intends to configure machine learning paradigms in the presence of multilabelers, residing on two key assumptions: the labeler's performance does not depend on the input space, and independence among the annotators is imposed. Here, we propose the correlated chained Gaussian processes from the multiple annotators (CCGPMA) approach, which models each annotator's performance as a function of the input space and exploits the correlations among experts. Experimental results associated with classification and regression tasks show that our CCGPMA performs better modeling of the labelers' behavior, indicating that it consistently outperforms other state-of-the-art LFC approaches.

Learning Instance-Level Label Correlation Distribution for Multilabel Classification With Fuzzy Rough Sets

In multi-label learning, research on label correlation provides an effective solution to compress the hypothesis space of classifiers. However, the current works focus on the label correlation adapted to overall data, while ignoring the locally targeted information presented by some instances. The lack exploration on the distribution of local label correlation in multi-label instance space undoubtedly limits the in-depth application of label correlation in multi-label learning. Based on fuzzy rough set theory, the instance-level label correlation distribution is first proposed in this paper and applied to design a novel multi-label learner. For each multi-label instance, the local importance of features to label is quantitatively analyzed, by considering the decisive influence of input information on decision-making. According to coincidence degree between local feature weight distribution for different labels, the instance-level label correlation is constructed. In order to reflect the internal relationship between label variables objectively, the instance-level label correlation distribution is integrated into the empirical label relevance. On the basis, the label relevance matrix is used to define the constraints of optimization function in a new form. The relative position of sub-separating hyperplanes in input space is quantitatively characterized to reduce the complexity of multi-label classifier and improve learning performance. The experiment results on eighteen multi-label datasets illustrate the effectiveness of our algorithm. The impact of core parameters on performance is also dissected.

Multi-view kernel PCA for time series forecasting

In this paper, we propose a kernel principal component analysis model for multi-variate time series forecasting, where the training and prediction schemes are derived from the multi-view formulation of Restricted Kernel Machines. The training problem is simply an eigenvalue decomposition of the summation of two kernel matrices corresponding to the views of the input and output data. When a linear kernel is used for the output view, it is shown that the forecasting equation takes the form of kernel ridge regression. When that kernel is non-linear, a pre-image problem has to be solved to forecast a point in the input space. We evaluate the model on several standard time series datasets, perform ablation studies, benchmark with closely related models and discuss its results.

BGG Sequences with Weak Regularity and Applications

We investigate some Bernstein–Gelfand–Gelfand complexes consisting of Sobolev spaces on bounded Lipschitz domains in Rn\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${\\mathbb {R}}^{n}$$\\end{document}. In particular, we compute the cohomology of the conformal deformation complex and the conformal Hessian complex in the Sobolev setting. The machinery does not require algebraic injectivity/surjectivity conditions between the input spaces, and allows multiple input complexes. As applications, we establish a conformal Korn inequality in two space dimensions with the Cauchy–Riemann operator and an additional third-order operator with a background in Möbius geometry. We show that the linear Cosserat elasticity model is a Hodge–Laplacian problem of a twisted de Rham complex. From this cohomological perspective, we propose potential generalizations of continuum models with microstructures.

DreamFace: Progressive Generation of Animatable 3D Faces under Text Guidance

Emerging Metaverse applications demand accessible, accurate and easy-to-use tools for 3D digital human creations in order to depict different cultures and societies as if in the physical world. Recent large-scale vision-language advances pave the way for novices to conveniently customize 3D content. However, the generated CG-friendly assets still cannot represent the desired facial traits for human characteristics. In this paper, we present Dream-Face, a progressive scheme to generate personalized 3D faces under text guidance. It enables layman users to naturally customize 3D facial assets that are compatible with CG pipelines, with desired shapes, textures and fine-grained animation capabilities. From a text input to describe the facial traits, we first introduce a coarse-to-fine scheme to generate the neutral facial geometry with a unified topology. We employ a selection strategy in the CLIP embedding space to generate coarse geometry, and subsequently optimize both the detailed displacements and normals using Score Distillation Sampling (SDS) from the generic Latent Diffusion Model (LDM). Then, for neutral appearance generation, we introduce a dual-path mechanism, which combines the generic LDM with a novel texture LDM to ensure both the diversity and textural specification in the UV space. We also employ a two-stage optimization to perform SDS in both the latent and image spaces to significantly provide compact priors for fine-grained synthesis. It also enables learning the mapping from the compact latent space into physically-based textures (diffuse albedo, specular intensity, normal maps, etc.). Our generated neutral assets naturally support blendshapes-based facial animations, thanks to the unified geometric topology. We further improve the animation ability with personalized deformation characteristics. To this end, we learn the universal expression prior in a latent space with neutral asset conditioning using the cross-identity hypernetwork, we subsequently train a neural facial tracker from video input space into the pre-trained expression space for personalized fine-grained animation. Extensive qualitative and quantitative experiments validate the effectiveness and generalizability of DreamFace. Notably, DreamFace can generate realistic 3D facial assets with physically-based rendering quality and rich animation ability from video footage, even for fashion icons or exotic characters in cartoons and fiction movies.

Prompt-Based Tuning of Transformer Models for Multi-Center Medical Image Segmentation of Head and Neck Cancer.

Medical image segmentation is a vital healthcare endeavor requiring precise and efficient models for appropriate diagnosis and treatment. Vision transformer (ViT)-based segmentation models have shown great performance in accomplishing this task. However, to build a powerful backbone, the self-attention block of ViT requires large-scale pre-training data. The present method of modifying pre-trained models entails updating all or some of the backbone parameters. This paper proposes a novel fine-tuning strategy for adapting a pretrained transformer-based segmentation model on data from a new medical center. This method introduces a small number of learnable parameters, termed prompts, into the input space (less than 1% of model parameters) while keeping the rest of the model parameters frozen. Extensive studies employing data from new unseen medical centers show that the prompt-based fine-tuning of medical segmentation models provides excellent performance regarding the new-center data with a negligible drop regarding the old centers. Additionally, our strategy delivers great accuracy with minimum re-training on new-center data, significantly decreasing the computational and time costs of fine-tuning pre-trained models. Our source code will be made publicly available.

An Active Learning Strategy for Joint Surrogate Models Construction with Compatibility Conditions: Application to VPP

Static Velocity Prediction Programs (VPP) are standard tools in sailing yachts’ design and performance assessment. Predicting the maximal steady velocity of a yacht involves resolving constrained optimization problems. These problems have a prohibitive computational cost when using high-fidelity global modeling of the yacht. This difficulty has motivated the introduction of modular approaches, decomposing the global model into subsystems modeled independently and approximated by surrogate models (response surfaces). The maximum boat speed for prescribed conditions solves an optimization problem for the trimming parameters of the model constrained by compatibility conditions between the subsystems’ surrogate solution (e.g., the yacht equilibrium). The accuracy of the surrogates is then critical for the quality of the resulting VPP. This paper relies on Gaussian Process (GP) models of the subsystems and introduces an original sequential Active Learning Method (ALM) for their joint construction. Our ALM exploits the probabilistic nature of the GP models to decide the enrichment of the training sets using an infilling criterion that combines the predictive uncertainty of the surrogate models and the likelihood of equilibrium at every input point. The resulting strategy enables the concentration of the computational effort around the manifolds where equilibrium is satisfied. The results presented compare ALM with a standard (uninformed) Quasi-Monte Carlo method, which samples the input space of the subsystems uniformly. ALM surrogates have higher accuracy in the equilibrium regions for equal construction cost, with improved mean prediction and reduced prediction uncertainty. We further investigate the effect of the prediction uncertainty on the numerical VPP and in a routing problem.

Robust Reinforcement Learning-Based Multiple Inputs and Multiple Outputs Controller for Wind Turbines

The control of variable-speed wind turbines that generate electricity from the kinetic energy of the wind involves subsystems that need to be controlled simultaneously, namely, the blade pitch angle controllers and the generator torque controllers. The presented study solves the control problem with multiple inputs and multiple outputs (MIMO), using the method of reinforcement learning–based Trust Region Policy Optimization, through which the control parameters of both subsystems are simultaneously optimized. In this case, the robust control problem is transformed into a constrained optimal control problem with an appropriate choice of value functions for the nominal system. The study aims to synthesize a robust controller, with the aim of maximizing the generated energy (power) and minimizing unwanted forces (thrust). The innovative control architecture uses an extended input space, which allows fine-tuning of parameters for each operating state. Test calculations carried out in simulation experiments using models of the 5 MW NREL wind turbine and the 4 MW Enercon E-126 EP3 wind turbine are presented to illustrate the performance and practicality of the proposed approach.

Statistical Methods for the Study of Computer Experiment Failures: Application to a Fuel-Coolant-Interaction Simulation Code

In the framework of risk assessment in nuclear accidents, simulation tools are widely used to understand and model physical phenomena. These simulation tools take into account a large number of uncertain input parameters. We often use Monte Carlo–type methods to explore their range of variation: The input space is randomly sampled, and a code run is performed on each sampled point. However, some of these code runs may fail to converge. Analyzing these code failures to understand which of the inputs have the most influence on them leads to a better understanding of how the code works. It also intends to improve the robustness of the simulation software and code computations. For this purpose, we propose two complementary approaches performing a statistical analysis of the code failures. The first approach is based on goodness-of-fit tests and compares conditional probability distributions according to code failures to a reference one. A second approach, based on a dependence measure named the Hilbert-Schmidt Independence Criterion, provides another way to measure the global dependence between the inputs and the code failures. The development of this methodology is carried out in the context of severe nuclear accidents. More especially, the presented methods are applied for the study of the simulation code MC3D, which simulates the fuel-coolant interaction in a severe nuclear accident context.

Probabilistic volcanic mass flow hazard assessment using statistical surrogates of deterministic simulations

Probabilistic volcanic hazard assessments require (1) an identification of the hazardous volcanic source; (2) estimation of the magnitude-frequency relationship for the volcanic process; (3) quantification of the dependence of hazard on magnitude and external conditions; and (4) estimation of hazard exceedance from the magnitude-frequency and hazard intensity relationship. For volcanic mass flows, quantification of the hazard is typically undertaken through the use of computationally expensive mass flow simulators. However, this computational expense restricts the number of samples that can be used to produce a probabilistic assessment and limits the ability to rapidly update hazard assessments in response to changing source probabilities. We develop an alternate approach to defining hazard intensity through a surrogate model that provides a continuous estimate of simulation outputs at negligible computational expense, demonstrated through a probabilistic hazard assessment of dome collapse (block-and-ash) flows at Taranaki volcano, New Zealand. A Gaussian Process emulator trained on a database of simulations is used as the surrogate model of hazard intensity across the input space of possible dome collapse volumes and configurations, which is then sampled using a volume-frequency relationship of dome collapse flows. The demonstrated technique is a tractable solution to the problem of probabilistic volcanic hazard assessment, with the surrogates providing a good approximation of the simulator, and is generally applicable to volcanic hazard and geo-hazard assessments that are limited by the demands of numerical simulations and changing source probabilities.

On the sensitivity analysis of the DEM oedometer experiment

The discrete element method (DEM) is frequently used to investigate the behaviour of granular media (Bravo in Simulation of soil and tillage-tool interaction by the discrete element method, 2013; Tijskens et al. in J Sound Vib 266:493–514, 2003; Langston et al. in Chem Eng Sci 50:967–987, 1995; Kohring et al. in Comput Methods Appl Mech Eng 124:273–281, 1995; Stahl et al. in Granul Matter 13:417–428, 2011). The parameter calibration is a challenging task due to the large number of input parameters and the computational effort. Sometimes, this is performed with a trial-and-error approach as mentioned in Roessler et al. (Powder Technol 343:803–812, 2019), Rackl and Hanley (Powder Technol 307:73–83, 2017) based on laboratory tests, e.g. the pile experiment, the oedometer experiment and the shear test. To achieve a more suitable calibration, a better model understanding is necessary in which the influence of the DEM parameters is analysed. Consequently, the calibration can be focused on specific parameters, which have a significant influence on thef model response. If parameters with a negligibly small influence exist, the number of calibration parameters can be reduced. On this basis, it is possible to decide whether the laboratory test is suitable for the calibration of specific parameters or not. This is demonstrated with a sensitivity analysis based on Sobol’ indices for the oedometer laboratory test. In order to reduce the computational effort, the sensitivity analysis is performed with different metamodels of the oedometer simulation. The metamodels are fitted and validated with two separate sampling point sets. It is shown that the Young’s modulus for the investigated input space is the most significant parameter. This knowledge can be used to only focus the calibration on this significant parameter which enables an easier calibration and makes clear that for calibrating of other parameters this laboratory test is inappropriate. An algorithm of a force-driven plate is developed and shown which prevents non-physical states in which the interaction force between the particles and the loadplate exceeds the applied force.

A neuro-structural framework for bankruptcy prediction

We develop a framework to simultaneously compute the unobservable parameters underlying the structural-parametric models for bankruptcy prediction. More specifically, we compute the unobservable parameters such as, asset value and asset volatility, through learning by embedding in the structural models a neural network that maps the neural network’s input space (e.g. companies’ observable financial and market data) to the unobservable parameter space. Within such a ‘neuro-structural’ framework, the neural network and the structural model work together as a one unit during the learning phase by providing to each other forward and backward information, respectively, until the weights of the neural network are optimized according to a merit function. Empirical results show that structural models, like the Black-Scholes-Merton and the Down-and-Out option models, with parameters computed with our approach, perform better than alternative specifications of the structural models, out of sample, in terms of discriminatory power, information content and economic impact. Importantly, they also perform better than a standard neural network, suggesting that the co-joint dynamics between the neural network and the structural model are useful during the learning phase and can improve the prediction performance (and the training efficiency) of neural networks. Finally, our approach provides methodological (and empirical) enhancements over logit specifications such as, Campbell et al. [In search of distress risk. J Finance, 2008, 63, 2899–2939]. There, financial and market data are the inputs, and the output is the probability of bankruptcy whereas our approach includes an intermediary step to obtain the unobservable parameters and subsequently the probability of bankruptcy.

Dynamic mutual information-based feature selection for multi-label learning

In classification problems, feature selection is used to identify important input features to reduce the dimensionality of the input space while improving or maintaining classification performance. Traditional feature selection algorithms are designed to handle single-label learning, but classification problems have recently emerged in multi-label domain. In this study, we propose a novel feature selection algorithm for classifying multi-label data. This proposed method is based on dynamic mutual information, which can handle redundancy among features controlling the input space. We compare the proposed method with some existing problem transformation and algorithm adaptation methods applied to real multi-label datasets using the metrics of multi-label accuracy and hamming loss. The results show that the proposed method demonstrates more stable and better performance for nearly all multi-label datasets.

Sequential Designs for Filling Output Spaces

Space-filling designs are commonly used in computer experiments to fill the space of inputs so that the input–output relationship can be accurately estimated. However, in certain applications such as inverse design or feature-based modeling, the aim is to fill the response or feature space. In this article, we propose a new experimental design framework that aims to sequentially fill the space of the outputs (responses or features). Several examples are given to show the advantages of the proposed method over the traditional input space-filling designs.

A principal component analysis assisted machine learning modeling and validation of methanol formation over Cu-based catalysts in direct CO2 hydrogenation

With the advent of powerful machine learning algorithms strongly supporting complex non-linear regression modeling, catalyst design features for a wide and customized set of catalysts used for a specific reaction have been made easy. Herein, we use these techniques in the methanol synthesis by CO2 reduction over Cu-based binary and ternary catalysts with the help of three machine learning algorithms: artificial neural network, support vector machine regression, and gaussian process regression. 227 catalytic performance dataset points from existing literature on CO2 hydrogenation to methanol were compiled and initially accessed by Principal Component Analysis (PCA) for training and preliminary evaluation of the algorithms, which was further guided using a 10-fold cross-validation method. The predictive model and its insights were validated experimentally over 30 datasets derived from experimental runs of this reaction over a ternary Cu/ZnO/ZrO2 laboratory-synthesized catalyst at varying conditions of temperature, pressure, and space velocity in a continuous mode fixed-bed plug-flow reactor. The assessment of the space of input and laboratory data was aided by Principal Component Analysis (PCA), scores, and loadings plot. This work shows how experimentalists can predict typical heterogeneous catalytic reaction outputs (R2 greater than 0.9 for three variables) with fair accuracy using a combination of machine learning and PCA.

Unsupervised deep consistency learning adaptation network for cardiac cross-modality structural segmentation.

Deep neural networks have recently been succeessful in the field of medical image segmentation; however, they are typically subject to performance degradation problems when well-trained models are tested in another new domain with different data distributions. Given that annotated cross-domain images may inaccessible, unsupervised domain adaptation methods that transfer learnable information from annotated source domains to unannotated target domains with different distributions have attracted substantial attention. Many methods leverage image-level or pixel-level translation networks to align domain-invariant information and mitigate domain shift issues. However, These methods rarely perform well when there is a large domain gap. A new unsupervised deep consistency learning adaptation network, which adopts input space consistency learning and output space consistency learning to realize unsupervised domain adaptation and cardiac structural segmentation, is introduced in this paper The framework mainly includes a domain translation path and a cross-modality segmentation path. In domain translation path, a symmetric alignment generator network with attention to cross-modality features and anatomy is introduced to align bidirectional domain features. In the segmentation path, entropy map minimization, output probability map minimization and segmentation prediction minimization are leveraged to align the output space features. The model conducts supervised learning to extract source domain features and conducts unsupervised deep consistency learning to extract target domain features. Through experimental testing on two challenging cross-modality segmentation tasks, our method has robust performance compared to that of previous methods. Furthermore, ablation experiments are conducted to confirm the effectiveness of our framework.