Input Space Research Articles

Increased interpretation of deep learning models using hierarchical cluster-based modelling

Linear prediction models based on data with large inhomogeneity or abrupt non-linearities often perform poorly because relationships between groups in the data dominate the model. Given that the data is locally linear, this can be overcome by splitting the data into smaller clusters and creating a local model within each cluster. In this study, the previously published Hierarchical Cluster-based Partial Least Squares Regression (HC-PLSR) procedure was extended to deep learning, in order to increase the interpretability of the deep learning models through local modelling. Hierarchical Cluster-based Convolutional Neural Networks (HC-CNNs), Hierarchical Cluster-based Recurrent Neural Networks (HC-RNNs) and Hierarchical Cluster-based Support Vector Regression models (HC-SVRs) were implemented and tested on spectroscopic data consisting of Fourier Transform Infrared (FT-IR) measurements of raw material dry films, for prediction of average molecular weight during hydrolysis and a simulated data set constructed to contain three clusters of observations with different non-linear relationships between the independent variables and the response. HC-CNN, HC-RNN and HC-SVR outperformed HC-PLSR for the simulated data set, showing the disadvantage of PLSR for highly non-linear data, but for the FT-IR data set there was little to gain in prediction ability from using more complex models than HC-PLSR. Local modelling can ease the interpretation of deep learning models through highlighting differences in feature importance between different regions of the input or output space. Our results showed clear differences between the feature importance for the various local models, which demonstrate the advantages of a local modelling approach with regards to interpretation of deep learning models.

A Neural Space-Time Representation for Text-to-Image Personalization

A key aspect of text-to-image personalization methods is the manner in which the target concept is represented within the generative process. This choice greatly affects the visual fidelity, downstream editability, and disk space needed to store the learned concept. In this paper, we explore a new text-conditioning space that is dependent on both the denoising process timestep (time) and the denoising U-Net layers (space) and showcase its compelling properties. A single concept in the space-time representation is composed of hundreds of vectors, one for each combination of time and space , making this space challenging to optimize directly. Instead, we propose to implicitly represent a concept in this space by optimizing a small neural mapper that receives the current time and space parameters and outputs the matching token embedding. In doing so, the entire personalized concept is represented by the parameters of the learned mapper, resulting in a compact, yet expressive, representation. Similarly to other personalization methods, the output of our neural mapper resides in the input space of the text encoder. We observe that one can significantly improve the convergence and visual fidelity of the concept by introducing a textual bypass , where our neural mapper additionally outputs a residual that is added to the output of the text encoder. Finally, we show how one can impose an importance-based ordering over our implicit representation, providing users control over the reconstruction and editability of the learned concept using a single trained model. We demonstrate the effectiveness of our approach over a range of concepts and prompts, showing our method's ability to generate high-quality and controllable compositions without fine-tuning any parameters of the generative model itself.

MG-Trans: Multi-Scale Graph Transformer With Information Bottleneck for Whole Slide Image Classification.

Multiple instance learning (MIL)-based methods have become the mainstream for processing the megapixel-sized whole slide image (WSI) with pyramid structure in the field of digital pathology. The current MIL-based methods usually crop a large number of patches from WSI at the highest magnification, resulting in a lot of redundancy in the input and feature space. Moreover, the spatial relations between patches can not be sufficiently modeled, which may weaken the model's discriminative ability on fine-grained features. To solve the above limitations, we propose a Multi-scale Graph Transformer (MG-Trans) with information bottleneck for whole slide image classification. MG-Trans is composed of three modules: patch anchoring module (PAM), dynamic structure information learning module (SILM), and multi-scale information bottleneck module (MIBM). Specifically, PAM utilizes the class attention map generated from the multi-head self-attention of vision Transformer to identify and sample the informative patches. SILM explicitly introduces the local tissue structure information into the Transformer block to sufficiently model the spatial relations between patches. MIBM effectively fuses the multi-scale patch features by utilizing the principle of information bottleneck to generate a robust and compact bag-level representation. Besides, we also propose a semantic consistency loss to stabilize the training of the whole model. Extensive studies on three subtyping datasets and seven gene mutation detection datasets demonstrate the superiority of MG-Trans.

Spatial predictions of groundwater potential using automated machine learning (AutoML): a comparative study of feature selection and training sample size in Qinghai Province, China.

Predicting groundwater potential is crucial for identifying the spatial distribution of groundwater in a region. It serves as an essential guide for the development, utilization, and protection of groundwater resources. Previous studies have primarily emphasized finding the most accurate prediction model for groundwater potential while giving less attention to the selection of training features and sample sizes. This study aims to predict groundwater potential within Qinghai Province using automated machine learning technology and assess the influence of sample sizes and feature selection on prediction accuracy. Sixteen groundwater conditioning factors were categorized into categorical and numerical variables. Four feature selection modes were utilized as input in training the model. The results indicated that, except for correlations between evaporation and landforms (- 0.8) and precipitation and normalized difference vegetation index (0.8), the Pearson correlation coefficients among the remaining sixteen factors were ≤ 0.5 or ≥ - 0.5. The models XGB-ALL, RF-Entropy, ET-CRITIC, and XGB-PCA yielded accuracy scores of 0.783, 0.685, 0.745, and 0.703, and area under curve (AUC) of 0.819, 0.724, 0.779, and 0.747, respectively. If enough samples are available with the tree model, an increased number of features can improve prediction accuracy. The principal component analysis method showed difficulty in reducing the dimensionality of the input space, while the Entropy method proved efficient. The accuracy and AUC value of the prediction model improved with an increasing number of samples. Training with 8 features and 200 data points achieved an accuracy of 0.745, sufficient to evaluate regional groundwater potential. As for training with 600 samples, the model's performance accuracy rose to 0.9, enabling precise groundwater potential prediction. The outputs of this research can provide decision-makers in groundwater resource management in Qinghai Province with crucial theoretical and practical support. The lessons learned can have future applications in similar situations.

A novel hybrid feature selection and ensemble-based machine learning approach for botnet detection

In the age of sophisticated cyber threats, botnet detection remains a crucial yet complex security challenge. Existing detection systems are continually outmaneuvered by the relentless advancement of botnet strategies, necessitating a more dynamic and proactive approach. Our research introduces a ground-breaking solution to the persistent botnet problem through a strategic amalgamation of Hybrid Feature Selection methods—Categorical Analysis, Mutual Information, and Principal Component Analysis—and a robust ensemble of machine learning techniques. We uniquely combine these feature selection tools to refine the input space, enhancing the detection capabilities of the ensemble learners. Extra Trees, as the ensemble technique of choice, exhibits exemplary performance, culminating in a near-perfect 99.99% accuracy rate in botnet classification across varied datasets. Our model not only surpasses previous benchmarks but also demonstrates exceptional adaptability to new botnet phenomena, ensuring persistent accuracy in a landscape of evolving threats. Detailed comparative analyses manifest our model's superiority, consistently achieving over 99% True Positive Rates and an unprecedented False Positive Rate close to 0.00%, thereby setting a new precedent for reliability in botnet detection. This research signifies a transformative step in cybersecurity, offering unprecedented precision and resilience against botnet infiltrations, and providing an indispensable blueprint for the development of next-generation security frameworks.

TransformEHR: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records

Deep learning transformer-based models using longitudinal electronic health records (EHRs) have shown a great success in prediction of clinical diseases or outcomes. Pretraining on a large dataset can help such models map the input space better and boost their performance on relevant tasks through finetuning with limited data. In this study, we present TransformEHR, a generative encoder-decoder model with transformer that is pretrained using a new pretraining objective—predicting all diseases and outcomes of a patient at a future visit from previous visits. TransformEHR’s encoder-decoder framework, paired with the novel pretraining objective, helps it achieve the new state-of-the-art performance on multiple clinical prediction tasks. Comparing with the previous model, TransformEHR improves area under the precision–recall curve by 2% (p < 0.001) for pancreatic cancer onset and by 24% (p = 0.007) for intentional self-harm in patients with post-traumatic stress disorder. The high performance in predicting intentional self-harm shows the potential of TransformEHR in building effective clinical intervention systems. TransformEHR is also generalizable and can be easily finetuned for clinical prediction tasks with limited data.

Testing Causality in Scientific Modelling Software

From simulating galaxy formation to viral transmission in a pandemic, scientific models play a pivotal role in developing scientific theories and supporting government policy decisions that affect us all. Given these critical applications, a poor modelling assumption or bug could have far-reaching consequences. However, scientific models possess several properties that make them notoriously difficult to test, including a complex input space, long execution times, and non-determinism, rendering existing testing techniques impractical. In fields such as epidemiology, where researchers seek answers to challenging causal questions, a statistical methodology known as Causal inference has addressed similar problems, enabling the inference of causal conclusions from noisy, biased, and sparse data instead of costly experiments. This article introduces the causal testing framework: a framework that uses causal inference techniques to establish causal effects from existing data, enabling users to conduct software testing activities concerning the effect of a change, such as metamorphic testing, a posteriori . We present three case studies covering real-world scientific models, demonstrating how the causal testing framework can infer metamorphic test outcomes from reused, confounded test data to provide an efficient solution for testing scientific modelling software.

Locally linear attributes of ReLU neural networks.

A ReLU neural network functions as a continuous piecewise linear map from an input space to an output space. The weights in the neural network determine a partitioning of the input space into convex polytopes, where each polytope is associated with a distinct affine mapping. The structure of this partitioning, together with the affine map attached to each polytope, can be analyzed to investigate the behavior of the associated neural network. We investigate simple problems to build intuition on how these regions act and both how they can potentially be reduced in number and how similar structures occur across different networks. To validate these intuitions, we apply them to networks trained on MNIST to demonstrate similarity between those networks and the potential for them to be reduced in complexity.

Performance evaluation of KPCA pre-imaging methods for speech signal denoising

&lt;p class="Abstract"&gt;Kernel principal component analysis (KPCA) has gained wider interest amongst the researchers in nonlinear dimensionality reduction, data compression, feature extraction, and denoising applications. In KPCA, data from low dimensional input space implicitly mapped to higher dimensional feature space where linear PCA is enforced. However, for denoising application data need to invert back to input space, which is impossible and known as pre-imaging problems. Over recent years, several pre-imaging methods have proposed each with its benefits and disadvantages. In this paper, we evaluated the performance for selected pre-imaging methods for denoising speech signal, whose intelligibility and quality degraded by background noises. Further, we extend our work by comparing the performance of these pre-imaging methods by objective evaluation of denoised speech signal on generated toy examples and NOIZUES database.&lt;/p&gt;

Input‐response space‐filling designs incorporating response uncertainty

AbstractTraditionally space‐filling designs have focused on the characteristics of the design in the input space ensuring uniform spread throughout the region. Input‐response space‐filling designs considered scenarios when having good spread throughout the range or region of the responses is also of interest. This paper acknowledges that there is typically uncertainty associated with the values of the response(s) and hence proposes a method, Input‐Response Space‐Filling Designs with Uncertainty (IRSFwU), to incorporate this into the design construction. The Pareto front of designs offers alternatives that balance input and response space filling, while prioritizing input combinations with lower associated response uncertainty. These lower uncertainty choices improve the chances of observing the desired response values. We describe the new approach with an uncertainty‐adjusted distance to measure the response space filling, the Pareto aggregate point exchange algorithm to populate the set of promising designs, and illustrate the method with three examples of different input and response relationships and dimensions.

Three-way imbalanced learning based on fuzzy twin SVM

Three-way decision (3WD) is a powerful tool for granular computing to deal with uncertain data, commonly used in information systems, decision-making, and medical care. Three-way decision gets much research in traditional rough set models. However, three-way decision is rarely combined with the currently popular field of machine learning to expand its research. In this paper, three-way decision is connected with SVM, a standard binary classification model in machine learning, for solving imbalanced classification problems that SVM needs to improve. A new three-way fuzzy membership function and a new fuzzy twin support vector machine with three-way membership (TWFTSVM) are proposed. The new three-way fuzzy membership function is defined to increase the certainty of uncertain data in both input space and feature space, which assigns higher fuzzy membership to minority samples compared with majority samples. To evaluate the effectiveness of the proposed model, comparative experiments are designed for forty-seven different datasets with varying imbalance ratios. In addition, datasets with different imbalance ratios are derived from the same dataset to further assess the proposed model’s performance. The results show that the proposed model significantly outperforms other traditional SVM-based methods.

Variance-based reliability sensitivity with dependent inputs using failure samples

Reliability sensitivity analysis is concerned with measuring the influence of a system’s uncertain input parameters on its probability of failure. Statistically dependent inputs present a challenge in both computing and interpreting these sensitivity indices; such dependencies require discerning between variable interactions produced by the probabilistic model describing the system inputs and the computational model describing the system itself. To accomplish such a separation of effects in the context of reliability sensitivity analysis we extend on an idea originally proposed by Mara and Tarantola (2012) for model outputs unrelated to rare events. We compute the independent (influence via computational model) and full (influence via both computational and probabilistic model) contributions of all inputs to the variance of the indicator function of the rare event. We compute this full set of variance-based sensitivity indices of the rare event indicator using a single set of failure samples. This is possible by considering d different hierarchically structured isoprobabilistic transformations of this set of failure samples from the original d-dimensional space of dependent inputs to standard-normal space. The approach facilitates computing the full set of variance-based reliability sensitivity indices with a single set of failure samples obtained as the byproduct of a single run of a sample-based rare event estimation method. That is, no additional evaluations of the computational model are required. We demonstrate the approach on a test function and two engineering problems.

Improving Reliability of Magnetic Localization Using Input Space Transformation.

Body motion tracking for medical applications has the potential to improve quality of life for people with physical or speech motor disorders. Current solutions available in the market are either inaccurate, not affordable, or are impractical for a medical setting or at home. Magnetic localization can address these issues thanks to its high accuracy, simplicity of use, wearability, and use of inexpensive sensors such as magnetometers. However, sources of unreliability affect magnetometers to such an extent that the localization model trained in a controlled environment might exhibit poor tracking accuracy when deployed to end users. Traditional magnetic calibration methods, such as ellipsoid fit (EF), do not sufficiently attenuate the effect of these sources of unreliability to reach a positional accuracy that is both consistent and satisfactory for our target applications. To improve reliability, we developed a calibration method called post-deployment input space transformation (PDIST) that reduces the distribution shift in the magnetic measurements between model training and deployment. In this paper, we focused on change in magnetization or magnetometer as sources of unreliability. Our results show that PDIST performs better than EF in decreasing positional errors by a factor of ~3 when magnetization is distorted, and up to ~7 when our localization model is tested on a different magnetometer than the one it was trained with. Furthermore, PDIST is shown to perform reliably by providing consistent results across all our data collection that tested various combinations of the sources of unreliability.

Integrating geometries of ReLU feedforward neural networks.

This paper investigates the integration of multiple geometries present within a ReLU-based neural network. A ReLU neural network determines a piecewise affine linear continuous map, M, from an input space ℝm to an output space ℝn. The piecewise behavior corresponds to a polyhedral decomposition of ℝm. Each polyhedron in the decomposition can be labeled with a binary vector (whose length equals the number of ReLU nodes in the network) and with an affine linear function (which agrees with M when restricted to points in the polyhedron). We develop a toolbox that calculates the binary vector for a polyhedra containing a given data point with respect to a given ReLU FFNN. We utilize this binary vector to derive bounding facets for the corresponding polyhedron, extraction of "active" bits within the binary vector, enumeration of neighboring binary vectors, and visualization of the polyhedral decomposition (Python code is available at https://github.com/cglrtrgy/GoL_Toolbox). Polyhedra in the polyhedral decomposition of ℝm are neighbors if they share a facet. Binary vectors for neighboring polyhedra differ in exactly 1 bit. Using the toolbox, we analyze the Hamming distance between the binary vectors for polyhedra containing points from adversarial/nonadversarial datasets revealing distinct geometric properties. A bisection method is employed to identify sample points with a Hamming distance of 1 along the shortest Euclidean distance path, facilitating the analysis of local geometric interplay between Euclidean geometry and the polyhedral decomposition along the path. Additionally, we study the distribution of Chebyshev centers and related radii across different polyhedra, shedding light on the polyhedral shape, size, clustering, and aiding in the understanding of decision boundaries.

Quantifying uncertainties in direct numerical simulations of a turbulent channel flow

Direct numerical simulation (DNS) provides unrivalled levels of detail and accuracy for simulating turbulent flows. However, like all numerical methods, DNS is subject to uncertainties arising from the numerical scheme and input parameters (e.g. mesh resolution). While uncertainty quantification (UQ) techniques are being employed more and more to provide a systematic analysis of uncertainty for lower-fidelity models, their application to DNS is still relatively rare. In light of this, the aim of this work is to apply UQ and sensitivity analysis to the DNS of a canonical wall-bounded turbulent channel flow at low Reynolds number (Reτ=180). To compute the DNS, Incompact3d – a highly scalable open-source framework based on high-order compact finite differences and a spectral Poisson solver – is used as a black-box solver. Stochastic collocation is used to propagate the input uncertainties through Incompact3d to the output quantities of interest (QOIs). To facilitate the non-intrusive forward UQ analysis, the open-source EasyVVUQ package is used to provide integrated capability for sampling, pre-processing, execution, post-processing, and analysis of the computational campaign. Three separate UQ campaigns are conducted. The first two examine the effect of domain size and the numerical parameters (e.g. mesh resolution, time step, sample time), respectively, and adopt Gaussian quadrature rules combined via tensor products to sample the multi-dimensional input space. Finally, the third campaign investigates the performance of a dimension-adaptive sampling strategy that significantly reduces the computational cost compared to the full tensor product approach. The analysis focuses on the cross-channel statistical moments of the QOIs, as well as local and global sensitivity analyses to assess the sensitivity of each QOI with respect to each individual input. This enables an assessment of the robustness and sensitivity of DNS to the user-defined numerical parameters for wall-bounded turbulent flows, and provides an indication of suitable ranges for defining the values of these parameters.

AdjointBackMapV2: Precise reconstruction of arbitrary CNN unit’s activation via adjoint operators

Adjoint operators have been found to be effective in the exploration of CNN’s inner workings (Wan and Choe, 2022). However, the previous no-bias assumption restricted its generalization. We overcome the restriction via embedding input images into an extended normed space that includes bias in all CNN layers as part of the extended space and propose an adjoint-operator-based algorithm that maps high-level weights back to the extended input space for reconstructing an effective hypersurface. Such hypersurface can be computed for an arbitrary unit in the CNN, and we prove that this reconstructed hypersurface, when multiplied by the original input (through an inner product), will precisely replicate the output value of each unit. We show experimental results based on the CIFAR-10 and CIFAR-100 data sets where the proposed approach achieves near 0 activation value reconstruction error.

PRIME: Posterior Reconstruction of the Input for Model Explanations

Artificial intelligence (AI) has developed rapidly in the past few decades, but the corresponding explaining techniques do not keep up with the pace of development. This state of low interpretability leads developers to improve model performance by continuously increasing the use of features when training machine learning models, but it is difficult for them to judge the contribution of features to the prediction. Therefore, it is particularly important to evaluate features. Most of the existing research on model explaining focuses on characterizing the output space and ignored the research on the input space. In this paper, we propose a model-agnostic method to evaluate the feature importance by analyzing the change in the input space. The motivation is to come up with a flexible, general, and computationally efficient method of model explaining. The experiments on the synthetic datasets, standard datasets, and industrial dataset verify the effectiveness of our algorithm. At the same time, we compare our methods to the classic and late model explaining methods, such as SHAP, LIME, and Marginal Contribution Feature Importance (MCI) to further demonstrate the feasibility of our method.

Automated black-box boundary value detection

Software systems typically have an input domain that can be subdivided into sub-domains, each of which generates similar or related outputs. Testing it on the boundaries between these sub-domains is critical to ensure high-quality software. Therefore, boundary value analysis and testing have been a fundamental part of the software testing toolbox for a long time and are typically taught early to software engineering students. Despite its many argued benefits, boundary value analysis for a given software specification or application is typically described in abstract terms. This allows for variation in how testers apply it and in the benefits they see. Additionally, its adoption has been limited since it requires a specification or model to be analysed. We propose an automated black-box boundary value detection method to support software testers in performing systematic boundary value analysis. This dynamic method can be utilized even without a specification or model. The proposed method is based on a metric referred to as the program derivative, which quantifies the level of boundariness of test inputs. By combining this metric with search algorithms, we can identify and rank pairs of inputs as good boundary candidates, i.e., inputs that are in close proximity to each other but with outputs that are far apart. We have implemented the AutoBVA approach and evaluated it on a curated dataset of example programs. Furthermore, we have applied the approach broadly to a sample of 613 functions from the base library of the Julia programming language. The approach could identify boundary candidates that highlight diverse boundary behaviours in over 70% of investigated systems under test. The results demonstrate that even a simple variant of the program derivative, combined with broad sampling and search over the input space, can identify interesting boundary candidates for a significant portion of the functions under investigation. In conclusion, we also discuss the future extension of the approach to encompass more complex systems under test cases and datatypes.

Application of Generative Autoencoders in the Detection of Anomalies in Hypercompressors

Hypercompressors are essential assets that compress high-flow-rates of ethylene to pressures between 100~350 MPa in the LDPE industry. They are sources of essential risks and costs. This work proposes an unsupervised and univariate monitoring method for detecting anomalies in hypercompressors through data collected from an online monitoring system in an actual installation. A variational autoencoder learns the process of generating shapelets associated with vibrational patterns. A combination of matrix profile algorithms automatically selects the training data set. A β-VAE composed of MLP layers is trained and applied on the input space so that a voting operation and a box-cox transformation on the absolute residual errors between the inputs and outputs lead to the upper outlier detection threshold, obtained by the Tukey fence method. The model detected suspicious vibration patterns classified a priori as potential anomalies.

Kernel Geometric Mean Metric Learning

Geometric mean metric learning (GMML) algorithm is a novel metric learning approach proposed recently. It has many advantages such as unconstrained convex objective function, closed form solution, faster computational speed, and interpretability over other existing metric learning technologies. However, addressing the nonlinear problem is not effective enough. The kernel method is an effective method to solve nonlinear problems. Therefore, a kernel geometric mean metric learning (KGMML) algorithm is proposed. The basic idea is to transform the input space into a high-dimensional feature space through nonlinear transformation, and use the integral representation of the weighted geometric mean and the Woodbury matrix identity in new feature space to generalize the analytical solution obtained in the GMML algorithm as a form represented by a kernel matrix, and then the KGMML algorithm is obtained through operations. Experimental results on 15 datasets show that the proposed algorithm can effectively improve the accuracy of the GMML algorithm and other metric algorithms.