Output Space Research Articles

Adversarial Robustness with Partial Isometry.

Despite their remarkable performance, deep learning models still lack robustness guarantees, particularly in the presence of adversarial examples. This significant vulnerability raises concerns about their trustworthiness and hinders their deployment in critical domains that require certified levels of robustness. In this paper, we introduce an information geometric framework to establish precise robustness criteria for l2 white-box attacks in a multi-class classification setting. We endow the output space with the Fisher information metric and derive criteria on the input-output Jacobian to ensure robustness. We show that model robustness can be achieved by constraining the model to be partially isometric around the training points. We evaluate our approach using MNIST and CIFAR-10 datasets against adversarial attacks, revealing its substantial improvements over defensive distillation and Jacobian regularization for medium-sized perturbations and its superior robustness performance to adversarial training for large perturbations, all while maintaining the desired accuracy.

ANOVA-GP Modeling for High-Dimensional Bayesian Inverse Problems

Markov chain Monte Carlo (MCMC) stands out as an effective method for tackling Bayesian inverse problems. However, when dealing with computationally expensive forward models and high-dimensional parameter spaces, the challenge of repeated sampling becomes pronounced. A common strategy to address this challenge is to construct an inexpensive surrogate of the forward model, which cuts the computational cost of individual samples. While the Gaussian process (GP) is widely used as a surrogate modeling strategy, its applicability can be limited when dealing with high-dimensional input or output spaces. This paper presents a novel approach that combines the analysis of variance (ANOVA) decomposition method with Gaussian process regression to handle high-dimensional Bayesian inverse problems. Initially, the ANOVA method is employed to reduce the dimension of the parameter space, which decomposes the original high-dimensional problem into several low-dimensional sub-problems. Subsequently, principal component analysis (PCA) is utilized to reduce the dimension of the output space on each sub-problem. Finally, a Gaussian process model with a low-dimensional input and output is constructed for each sub-problem. In addition to this methodology, an adaptive ANOVA-GP-MCMC algorithm is proposed, which further enhances the adaptability and efficiency of the method in the Bayesian inversion setting. The accuracy and computational efficiency of the proposed approach are validated through numerical experiments. This innovative integration of ANOVA and Gaussian processes provides a promising solution to address challenges associated with high-dimensional parameter spaces and computationally expensive forward models in Bayesian inference.

Optimal transport-based domain adaptation for semantic segmentation of remote sensing images

ABSTRACT Thanks to its great power in feature representation, deep learning (DL) is widely used in semantic segmentation tasks. However, the requirements for high distribution consistency of different domains are too tight to be met by large-scale remote sensing tasks due to the domain shift in imaging modes and geographic environments. In this case, trained models in a source domain can hardly achieve sufficient accuracy in a target domain with domain shift. To address this issue, a novel unsupervised domain adaptation (UDA) method driven by optimal transport (OT) with two-stage training is proposed to alleviate domain shift in remote sensing images (RSIs). In the first stage, a colour distribution alignment (CDA) module and a feature joint alignment (FJA) module based on OT were designed to mitigate the discrepancy between different domains. CDA transports source-domain distribution according to the target-domain colour style, and FJA aligns source and target domains in both feature and output spaces by minimizing OT-based losses. In the second stage, self-training with pseudo-label denoising (STPD) was proposed, which alleviated the interference of noises in pseudo-labels based on a joint OT distance. For the experiments, the Potsdam, Vaihingen and UAVid datasets were employed. Based on the characteristics of these datasets, five UDA tasks were introduced. The results of these UDA experiments indicate the superiority of our method. Our code will be available at https://github.com/Hcshenziyang/OT-Domain-Adaptation-Semantic-Segmentation.

Automatic Seabed Target Segmentation of AUV via Multilevel Adversarial Network and Marginal Distribution Adaptation

The real-time segmentation results of side-scan sonar (SSS) images can realize the intelligent perception of autonomous underwater vehicles (AUVs). However, there remains a compelling challenge that the models trained with training data may not generalize well in practical applications due to the various marine conditions/working devices. This article proposes a real-time cross-domain segmentation adaptation scheme for SSS images based on 1) adversarial training via min-max training to distribute SSS images in the output space and 2) marginal distribution adaption to improve the segmentation performance by minimizing the measured distribution distance. To further enhance the adapted model, the multi-layer feature discriminators are established to realize the fine-grained alignment. Experiments on the real-world SSS datasets demonstrate that our proposed model outperforms state-of-the-art methods and achieves the actual industrial application for AUV.

Ferry: Toward Better Understanding of Input/Output Space for Data Wrangling Scripts.

Understanding the input and output of data wrangling scripts is crucial for various tasks like debugging code and onboarding new data. However, existing research on script understanding primarily focuses on revealing the process of data transformations, lacking the ability to analyze the potential scope, i.e., the space of script inputs and outputs. Meanwhile, constructing input/output space during script analysis is challenging, as the wrangling scripts could be semantically complex and diverse, and the association between different data objects is intricate. To facilitate data workers in understanding the input and output space of wrangling scripts, we summarize ten types of constraints to express table space and build a mapping between data transformations and these constraints to guide the construction of the input/output for individual transformations. Then, we propose a constraint generation model for integrating table constraints across multiple transformations. Based on the model, we develop Ferry, an interactive system that extracts and visualizes the data constraints describing the input and output space of data wrangling scripts, thereby enabling users to grasp the high-level semantics of complex scripts and locate the origins of faulty data transformations. Besides, Ferry provides example input and output data to assist users in interpreting the extracted constraints and checking and resolving the conflicts between these constraints and any uploaded dataset. Ferry's effectiveness and usability are evaluated through two usage scenarios and two case studies, including understanding, debugging, and checking both single and multiple scripts, with and without executable data. Furthermore, an illustrative application is presented to demonstrate Ferry's flexibility.

DECISION THEORETIC BOOTSTRAPPING

The design and testing of supervised machine learning models combine two fundamental distributions: (1) the training data distribution and (2) the testing data distribution. Although these two distributions are identical and identifiable when the data set is infinite, they are imperfectly known when the data are finite (and possibly corrupted), and this uncertainty must be taken into account for robust uncertainty quantification (UQ). An important case is when the test distribution is coming from a modal or localized area of the finite sample distribution. We present a general decision theoretic bootstrapping solution to this problem: (1) partition the available data into a training subset and a UQ subset; (2) take m subsampled subsets of the training set and train m models; (3) partition the UQ set into n sorted subsets and take a random fraction of them to define <i>n</i> corresponding empirical distributions μ<sub>j</sub>; (4) consider the adversarial game where Player I selects a model i ∈ {1,.....,m}, Player II selects the UQ distribution μ<sub>j</sub>, and Player I receives a loss defined by evaluating the model <i>i</i> against data points sampled from μ<sub>j</sub>; (5) identify optimal mixed strategies (probability distributions over models and UQ distributions) for both players. These randomized optimal mixed strategies provide optimal model mixtures, and UQ estimates given the adversarial uncertainty of the training and testing distributions represented by the game. The proposed approach provides (1) some degree of robustness to in-sample distribution localization/concentration and (2) conditional probability distributions on the output space forming aleatory representations of the uncertainty on the output as a function of the input variable.

CCN+: A neuro-symbolic framework for deep learning with requirements

For their outstanding ability of finding hidden patterns in data, deep learning models have been extensively applied in many different domains. However, recent works have shown that, if a set of requirements expressing inherent knowledge about the problem at hand is given, then neural networks often fail to comply with them. This represents a major drawback for deep learning models, as requirements compliance is normally considered a necessary condition for standard software deployment. In this paper, we propose a novel neuro-symbolic framework able to make any neural network compliant by design to a given set of requirements over the output space expressed in full propositional logic. This framework, called CCN+, integrates the requirements into the output layer of the neural network by applying multiple inference rules that ensure compliance with the requirements and adapts the standard binary cross-entropy loss function to the requirement output layer. As a result, not only the outputted predictions are guaranteed to be compliant with the requirements, but the neural network itself learns how to exploit the domain knowledge expressed by the requirements to get better performance. We conduct an extensive experimental evaluation of CCN+ on 19 real-world multi-label classification datasets with propositional logic requirements, including a challenging dataset for autonomous driving. Our experimental analysis confirms that CCN+ is able to outperform both its neural counterparts and the state-of-the-art models.

Toward Accurate Cardiac MRI Segmentation With Variational Autoencoder-Based Unsupervised Domain Adaptation.

Accurate myocardial segmentation is crucial in the diagnosis and treatment of myocardial infarction (MI), especially in Late Gadolinium Enhancement (LGE) cardiac magnetic resonance (CMR) images, where the infarcted myocardium exhibits a greater brightness. However, segmentation annotations for LGE images are usually not available. Although knowledge gained from CMR images of other modalities with ample annotations, such as balanced-Steady State Free Precession (bSSFP), can be transferred to the LGE images, the difference in image distribution between the two modalities (i.e., domain shift) usually results in a significant degradation in model performance. To alleviate this, an end-to-end Variational autoencoder based feature Alignment Module Combining Explicit and Implicit features (VAMCEI) is proposed. We first re-derive the Kullback-Leibler (KL) divergence between the posterior distributions of the two domains as a measure of the global distribution distance. Second, we calculate the prototype contrastive loss between the two domains, bringing closer the prototypes of the same category across domains and pushing away the prototypes of different categories within or across domains. Finally, a domain discriminator is added to the output space, which indirectly aligns the feature distribution and forces the extracted features to be more favorable for segmentation. In addition, by combining CycleGAN and VAMCEI, we propose a more refined multi-stage unsupervised domain adaptation (UDA) framework for myocardial structure segmentation. We conduct extensive experiments on the MSCMRSeg 2019, MyoPS 2020 and MM-WHS 2017 datasets. The experimental results demonstrate that our framework achieves superior performances than state-of-the-art methods.

Deep Optimized Broad Learning System for Applications in Tabular Data Recognition.

The broad learning system (BLS) is a versatile and effective tool for analyzing tabular data. However, the rapid expansion of big data has resulted in an overwhelming amount of tabular data, necessitating the development of specialized tools for effective management and analysis. This article introduces an optimized BLS (OBLS) specifically tailored for big data analysis. In addition, a deep-optimized BLS (DOBLS) network is developed further to enhance the performance and efficiency of the OBLS. The main contributions of this article are: 1) by retracing the network's error from the output space to the latent space, the OBLS adjusts parameters in the feature and enhancement node layers. This process aims to achieve more resilient representations, resulting in improved performance; 2) the DOBLS is a multilayered structure consisting of multiple OBLSs, wherein each OBLS connects to the input and output layers, enabling direct data propagation. This design helps reduce information loss between layers, ensuring an efficient flow of information throughout the network; and 3) the proposed methods demonstrate robustness across various applications, including multiview feature embedding, one-class classification (OCC), camera model identification, electroencephalogram (EEG) signal processing, and radar signal analysis. Experimental results validate the effectiveness of the proposed models. To ensure reproducibility, the source code is available at https://github.com/1027051515/OBLS_DOBLS.

Scalable Label Distribution Learning for Multi-Label Classification.

Multi-label classification (MLC) refers to the problem of tagging a given instance with a set of relevant labels. Most existing MLC methods are based on the assumption that the correlation of two labels in each label pair is symmetric, which is violated in many real-world scenarios. Moreover, most existing methods design learning processes associated with the number of labels, which makes their computational complexity a bottleneck when scaling up to large-scale output space. To tackle these issues, we propose a novel method named scalable label distribution learning (SLDL) for MLC, which can describe different labels as distributions in a latent space, where the label correlation is asymmetric and the dimension is independent of the number of labels. Specifically, SLDL first converts labels into continuous distributions within a low-dimensional latent space and leverages the asymmetric metric to establish the correlation between different labels. Then, it learns the mapping from the feature space to the latent space, resulting in the computational complexity is no longer related to the number of labels. Finally, SLDL leverages a nearest neighbor-based strategy to decode the latent representations and obtain the final predictions. Extensive experiments illustrate that SLDL achieves very competitive classification performances with little computational consumption.

SRNSD: Structure-Regularized Night-Time Self-Supervised Monocular Depth Estimation for Outdoor Scenes.

Deep CNNs have achieved impressive improvements for night-time self-supervised depth estimation form a monocular image. However, the performance degrades considerably compared to day-time depth estimation due to significant domain gaps, low visibility, and varying illuminations between day and night images. To address these challenges, we propose a novel night-time self-supervised monocular depth estimation framework with structure regularization, i.e., SRNSD, which incorporates three aspects of constraints for better performance, including feature and depth domain adaptation, image perspective constraint, and cropped multi-scale consistency loss. Specifically, we utilize adaptations of both feature and depth output spaces for better night-time feature extraction and depth map prediction, along with high- and low-frequency decoupling operations for better depth structure and texture recovery. Meanwhile, we employ an image perspective constraint to enhance the smoothness and obtain better depth maps in areas where the luminosity jumps change. Furthermore, we introduce a simple yet effective cropped multi-scale consistency loss that utilizes consistency among different scales of depth outputs for further optimization, refining the detailed textures and structures of predicted depth. Experimental results on different benchmarks with depth ranges of 40m and 60m, including Oxford RobotCar dataset, nuScenes dataset and CARLA-EPE dataset, demonstrate the superiority of our approach over state-of-the-art night-time self-supervised depth estimation approaches across multiple metrics, proving our effectiveness.

Some thoughts about transfer learning. What role for the source domain?

Transfer learning is called for when the training and test data do not share the same input distributions (PXS≠PXT) or/and not the same conditional ones (PY|XS≠PY|XT). In the most general case, the input spaces and/or output spaces can be different: XS≠XT and/or YS≠YT. However, most work assume that XS=XT. Furthermore, a common held assumption is that it is necessary that the source hypothesis be good on the source training data and that the “distance” between the source and the target domains be as small as possible in order to get a good (transferred) target hypothesis.This paper revisits the reasons for these beliefs and discusses the relevance of these conditions. An algorithm is presented which can deal with transfer learning problems where XS≠XT, and that furthermore brings a fresh perspective on the role of the source hypothesis (it does not have to be good) and on what is important in the distance between the source and the target domains (translations between them should belong to a limited set). Experiments illustrate the properties of the method and confirm the theoretical analysis.Determining beforehand a relevant source hypothesis remains an open problem, but the vista provided here helps understanding its role.

Self-ensembling with mask-boundary domain adaptation for optic disc and cup segmentation

Due to different retinal fundus image acquisition devices having various imaging principles, domain shift often occurs between different datasets. Hence, a segmentation network well-trained on one dataset (i.e., source domain) usually obtains very poor performance on another dataset (i.e., target domain), which results in us having to annotate the new dataset (target domain) to train the segmentation network again. However, annotating a new dataset is usually time-consuming and laborious. To address this problem, we proposed a novel unsupervised domain adaptation method for optic disc and cup segmentation. To be specific, we first utilized a domain adaptation method based on self-ensembling to effectively align the features of the source domain and target domain. Then, we designed a novel backbone network (MBU-Net) to make full use of the mask and boundary information to improve the segmentation performance of self-ensembling. Finally, we proposed an output-level adversarial domain adaptation (OADA) to address the domain shift problem of the structured output space in self-ensembling. In experiments, we test our proposed method on three different target domain datasets including Target Domain 1 (RIM-ONE_r3 dataset), Target Domain 2 (Drishti-GS dataset) and Target Domain 3 (REFUGE dataset). The experimental results demonstrate that our proposed method outperforms the compared state-of-the-art methods in the optic disc and cup segmentation tasks.

Increased interpretation of deep learning models using hierarchical cluster-based modelling

Linear prediction models based on data with large inhomogeneity or abrupt non-linearities often perform poorly because relationships between groups in the data dominate the model. Given that the data is locally linear, this can be overcome by splitting the data into smaller clusters and creating a local model within each cluster. In this study, the previously published Hierarchical Cluster-based Partial Least Squares Regression (HC-PLSR) procedure was extended to deep learning, in order to increase the interpretability of the deep learning models through local modelling. Hierarchical Cluster-based Convolutional Neural Networks (HC-CNNs), Hierarchical Cluster-based Recurrent Neural Networks (HC-RNNs) and Hierarchical Cluster-based Support Vector Regression models (HC-SVRs) were implemented and tested on spectroscopic data consisting of Fourier Transform Infrared (FT-IR) measurements of raw material dry films, for prediction of average molecular weight during hydrolysis and a simulated data set constructed to contain three clusters of observations with different non-linear relationships between the independent variables and the response. HC-CNN, HC-RNN and HC-SVR outperformed HC-PLSR for the simulated data set, showing the disadvantage of PLSR for highly non-linear data, but for the FT-IR data set there was little to gain in prediction ability from using more complex models than HC-PLSR. Local modelling can ease the interpretation of deep learning models through highlighting differences in feature importance between different regions of the input or output space. Our results showed clear differences between the feature importance for the various local models, which demonstrate the advantages of a local modelling approach with regards to interpretation of deep learning models.

Multi-Task Probabilistic Regression With Overlap Maximization for Visual Tracking

Recent researches made a breakthrough in visual tracking accuracy. Many trackers benefit from the object state representations and network loss functions, which mine the output space and improve the power of supervision, respectively. Probabilistic regression method models the noises and uncertainties in the annotations. However, advanced trackers with probabilistic regression are not studied sufficiently in the aspect of supervision and the aspect of robustness of evaluation maximization. In this paper, an overlap maximization network in the manner of probabilistic regression is proposed to improve the learning ability of the network and the discriminative ability in the evaluation maximization. Firstly, the probabilistic regression is extended with the intersection over union (IoU) evaluation, which is normalized as a probability density in the regression space. Secondly, the classification probability is added as a branch of the iterative evaluation module to improve the ability of distinguishing objects in the evaluation maximization. Moreover, the two branches are constructed into a joint probabilistic regression task of IoU evaluation, which makes the network learn from two types of ground truth and provide a consistent result with multi-branch outputs. For feature interpretation, the strip pooling network and the space-time memory network are introduced to encode long-range context and provide dynamic features, respectively. Compared to the state-of-the-art probabilistic regression trackers and other advanced trackers, the experiments show that the proposed tracker achieves outstanding performance across the six datasets, including GOT-10k, LaSOT, TrackingNet, UAV123, OTB-100 and VOT2018.

SIN: Semantic Inference Network for Few-Shot Streaming Label Learning.

Streaming label learning aims to model newly emerged labels for multilabel classification systems, which requires plenty of new label data for training. However, in changing environments, only a small amount of new label data can practically be collected. In this work, we formulate and study few-shot streaming label learning (FSLL), which models emerging new labels with only a few annotated examples by utilizing the knowledge learned from past labels. We propose a meta-learning framework, semantic inference network (SIN), which can learn and infer the semantic correlation between new labels and past labels to adapt FSLL tasks from a few examples effectively. SIN leverages label semantic representation to regularize the output space and acquires labelwise meta-knowledge based on gradient-based meta-learning. Moreover, SIN incorporates a novel label decision module with a meta-threshold loss to find the optimal confidence thresholds for each new label. Theoretically, we illustrate that the proposed semantic inference mechanism could constrain the complexity of hypotheses space to reduce the risk of overfitting and achieve better generalizability. Experimentally, extensive empirical results and ablation studies demonstrate the performance of SIN is superior to the prior state-of-the-art methods on FSLL.

A Novel Affine Relaxation-Based Algorithm for Minimax Affine Fractional Program

This paper puts forward a novel affine relaxation-based algorithm for solving the minimax affine fractional program problem (MAFPP) over a polyhedron set. First of all, some new variables are introduced for deriving the equivalence problem (EP) of the MAFPP. Then, for the EP, the affine relaxation problem (ARP) is established by using the two-stage affine relaxation method. The method provides a lower bound by solving the ARP in the branch-and-bound searching process. By subdividing the output space rectangle and solving a series of ARPs continuously, the feasible solution sequence generated by the algorithm converges to a global optimal solution of the initial problem. In addition, the algorithmic maximum iteration in the worst case is estimated by complexity analysis for the first time. Lastly, the practicability and effectiveness of the algorithm have been verified by numerical experimental results.

Limits to Fiscal and Monetary Policy in Small Open Economies

ABSTRACT In this paper we investigate the impact of the balance-of-payments constraint on output, income distribution and policy space in a small economy open to both commercial and financial flows. We do so by extending a Sraffian supermultiplier growth model to consider current and capital accounts and endogenous money. Expansionary fiscal policies increase output and its long run growth rate but have a negative impact on the balance-of-payments. Based on these results, we derive the value of government spending that maximises output while maintaining external equilibrium. Monetary policy also influences the balance-of-payments by attracting foreign financial flows through higher interest rates. This relationship is non-monotonic, since these flows can finance the external deficit resulting from a high output level but can also lead to a greater interest burden. This trade off allows us to find the level of the interest rate that maximises external space. It also implies that, with limits, raising interest rates might allow for higher real wages. Lastly, we examine the effects of the currency denomination of debt. We find that, in case of a depreciation, debt in domestic currency allows for a less contractionary adjustment than foreign currency-denominated debt but does not lead to higher economic growth.

Locally linear attributes of ReLU neural networks.

A ReLU neural network functions as a continuous piecewise linear map from an input space to an output space. The weights in the neural network determine a partitioning of the input space into convex polytopes, where each polytope is associated with a distinct affine mapping. The structure of this partitioning, together with the affine map attached to each polytope, can be analyzed to investigate the behavior of the associated neural network. We investigate simple problems to build intuition on how these regions act and both how they can potentially be reduced in number and how similar structures occur across different networks. To validate these intuitions, we apply them to networks trained on MNIST to demonstrate similarity between those networks and the potential for them to be reduced in complexity.

Bayesian optimal experimental design for constitutive model calibration

Computational simulation is increasingly relied upon for high/consequence engineering decisions, which necessitates a high confidence in the calibration of and predictions from complex material models. However, the calibration and validation of material models is often a discrete, multi-stage process that is decoupled from material characterization activities, which means the data collected does not always align with the data that is needed. To address this issue, an integrated workflow for delivering an enhanced characterization and calibration procedure—Interlaced Characterization and Calibration (ICC)—is introduced and demonstrated. This framework leverages Bayesian optimal experimental design (BOED), which creates a line of communication between model calibration needs and data collection capabilities in order to optimize the information content gathered from the experiments for model calibration.Eventually, the ICC framework will be used in quasi real-time to actively control experiments of complex specimens for the calibration of a high-fidelity material model. This work presents the critical first piece of algorithm development and a demonstration in determining the optimal load path of a cruciform specimen with simulated data. Calibration results, obtained via Bayesian inference, from the integrated ICC approach are compared to calibrations performed by choosing the load path a priori based on human intuition, as is traditionally done. The calibration results are communicated through parameter uncertainties which are propagated to the model output space (i.e. stress–strain). In these exemplar problems, data generated within the ICC framework resulted in calibrated model parameters with reduced measures of uncertainty compared to the traditional approaches.