General Learning Framework Research Articles

Meta-Modulation: A General Learning Framework for Cross-Task Adaptation.

Building learning systems possessing adaptive flexibility to different tasks is critical and challenging. In this article, we propose a novel and general meta-learning framework, called meta-modulation (MeMo), to foster the adaptation capability of a base learner across different tasks where only a few training data are available per task. For one independent task, MeMo proceeds like a "feedback regulation system", which achieves an adaptive modulation on the so-called definitive embeddings of query data to maximize the corresponding task objective. Specifically, we devise a type of efficient feedback information, definitive embedding feedback (DEF), to mathematize and quantify the unsuitability between the few training data and the base learner as well as the promising adjustment direction to reduce this unsuitability. The DEFs are encoded into high-level representation and temporarily stored as task-specific modulator templates by a modulation encoder. For coming query data, we develop an attention mechanism acting upon these modulator templates and combine both task/data-level modulation to generate the final data-specific meta-modulator. This meta-modulator is then used to modulate the query's embedding for correct decision-making. Our framework is scalable for various base learner models like multi-layer perceptron (MLP), long short-term memory (LSTM), convolutional neural network (CNN), and transformer, and applicable to different learning problems like language modeling and image recognition. Experimental results on a 2-D point synthetic dataset and various benchmarks in language and vision domains demonstrate the effectiveness and competitiveness of our framework.

Mobility knowledge graph: review and its application in public transport

AbstractUnderstanding human mobility in urban areas is crucial for transportation planning, operations, and online control. The availability of large-scale and diverse mobility data (e.g., smart card data, GPS data), provides valuable insights into human mobility patterns. However, organizing and analyzing such data pose significant challenges. Knowledge graph (KG), a graph-based knowledge representation method, has been successfully applied in various domains but has limited applications in urban mobility. This paper aims to address this gap by reviewing existing KG studies, introducing the concept of a mobility knowledge graph (MKG), and proposing a general learning framework to construct MKG from smart card data. The MKG represents hidden travel activities between public transport stations, with stations as nodes and their relations as edges. Two decomposition approaches, rule-based and neural network-based models, are developed to extract MKG relations from smart card data, capturing latent spatiotemporal travel dependencies. The case study is conducted using smart card data from a heavily used urban railway system to validate the effectiveness of MKG in predicting individual trip destinations. The results demonstrate the significance of establishing an MKG database, as it assists in a typical problem of predicting individual trip destinations for public transport systems with only tap-in records. Additionally, the MKG framework offers potential for efficient data management and applications such as individual mobility prediction and personalized travel recommendations.

General deep learning framework for emissivity engineering

Wavelength-selective thermal emitters (WS-TEs) have been frequently designed to achieve desired target emissivity spectra, as a typical emissivity engineering, for broad applications such as thermal camouflage, radiative cooling, and gas sensing, etc. However, previous designs require prior knowledge of materials or structures for different applications and the designed WS-TEs usually vary from applications to applications in terms of materials and structures, thus lacking of a general design framework for emissivity engineering across different applications. Moreover, previous designs fail to tackle the simultaneous design of both materials and structures, as they either fix materials to design structures or fix structures to select suitable materials. Herein, we employ the deep Q-learning network algorithm, a reinforcement learning method based on deep learning framework, to design multilayer WS-TEs. To demonstrate the general validity, three WS-TEs are designed for various applications, including thermal camouflage, radiative cooling and gas sensing, which are then fabricated and measured. The merits of the deep Q-learning algorithm include that it can (1) offer a general design framework for WS-TEs beyond one-dimensional multilayer structures; (2) autonomously select suitable materials from a self-built material library and (3) autonomously optimize structural parameters for the target emissivity spectra. The present framework is demonstrated to be feasible and efficient in designing WS-TEs across different applications, and the design parameters are highly scalable in materials, structures, dimensions, and the target functions, offering a general framework for emissivity engineering and paving the way for efficient design of nonlinear optimization problems beyond thermal metamaterials.

Unsupervised Local Discrimination for Medical Images.

Contrastive learning, which aims to capture general representation from unlabeled images to initialize the medical analysis models, has been proven effective in alleviating the high demand for expensive annotations. Current methods mainly focus on instance-wise comparisons to learn the global discriminative features, however, pretermitting the local details to distinguish tiny anatomical structures, lesions, and tissues. To address this challenge, in this paper, we propose a general unsupervised representation learning framework, named local discrimination (LD), to learn local discriminative features for medical images by closely embedding semantically similar pixels and identifying regions of similar structures across different images. Specifically, this model is equipped with an embedding module for pixel-wise embedding and a clustering module for generating segmentation. And these two modules are unified by optimizing our novel region discrimination loss function in a mutually beneficial mechanism, which enables our model to reflect structure information as well as measure pixel-wise and region-wise similarity. Furthermore, based on LD, we propose a center-sensitive one-shot landmark localization algorithm and a shape-guided cross-modality segmentation model to foster the generalizability of our model. When transferred to downstream tasks, the learned representation by our method shows a better generalization, outperforming representation from 18 state-of-the-art (SOTA) methods and winning 9 out of all 12 downstream tasks. Especially for the challenging lesion segmentation tasks, the proposed method achieves significantly better performance. The source codes are publicly available at https://github.com/HuaiChen-1994/LDLearning.

A general hypergraph learning algorithm for drug multi-task predictions in micro-to-macro biomedical networks.

The powerful combination of large-scale drug-related interaction networks and deep learning provides new opportunities for accelerating the process of drug discovery. However, chemical structures that play an important role in drug properties and high-order relations that involve a greater number of nodes are not tackled in current biomedical networks. In this study, we present a general hypergraph learning framework, which introduces Drug-Substructures relationship into Molecular interaction Networks to construct the micro-to-macro drug centric heterogeneous network (DSMN), and develop a multi-branches HyperGraph learning model, called HGDrug, for Drug multi-task predictions. HGDrug achieves highly accurate and robust predictions on 4 benchmark tasks (drug-drug, drug-target, drug-disease, and drug-side-effect interactions), outperforming 8 state-of-the-art task specific models and 6 general-purpose conventional models. Experiments analysis verifies the effectiveness and rationality of the HGDrug model architecture as well as the multi-branches setup, and demonstrates that HGDrug is able to capture the relations between drugs associated with the same functional groups. In addition, our proposed drug-substructure interaction networks can help improve the performance of existing network models for drug-related prediction tasks.

Training feedforward neural nets in Hopfield-energy-based configuration: A two-step approach

We introduce Hopfield-Energy-Based Learning, a general learning framework that is inspired by energy-based models, to train feedforward neural nets. Our approach includes two training phases applied iteratively: first, the minimization of the internal energy, which captures dependencies between input samples and network parameters, is carried out in an unsupervised manner; second, the problem-dependent supervised external energy (e.g., cross-entropy loss) combined with partially reversed internal energy gradients are back-propagated in a standard manner. The intuition is that the first stage helps parameters to settle into the state that simply partitions data into clusters; while in the second stage, the network is allowed to deviate from that clustering a bit (hence gradient reversal) in order to converge to parameters that ultimately perform well on the task at hand. Notably, the data used for the two steps might not be one and the same (e.g., can come from different domains) and the approach naturally tailors itself to solve unsupervised domain adaptation problems without adopting any distribution alignment techniques. We also show that the proposed training strategy substantially improves the performance of several ConvNets on standard supervised classification tasks; showing improvements of at least 1.2% (2.64% on CIFAR-10, 4.5% on CIFAR-100, and 1.35% on ImageNet). Our formulation is general, performs well in practice, and holds promise for scenarios where labeled data is limited.

Iterative residual policy: For goal-conditioned dynamic manipulation of deformable objects

This paper tackles the task of goal-conditioned dynamic manipulation of deformable objects. This task is highly challenging due to its complex dynamics (introduced by object deformation and high-speed action) and strict task requirements (defined by a precise goal specification). To address these challenges, we present Iterative Residual Policy (IRP), a general learning framework applicable to repeatable tasks with complex dynamics. IRP learns an implicit policy via delta dynamics—instead of modeling the entire dynamical system and inferring actions from that model, IRP learns delta dynamics that predict the effects of delta action on the previously observed trajectory. When combined with adaptive action sampling, the system can quickly optimize its actions online to reach a specified goal. We demonstrate the effectiveness of IRP on two tasks: whipping a rope to hit a target point and swinging a cloth to reach a target pose. Despite being trained only in simulation on a fixed robot setup, IRP is able to efficiently generalize to noisy real-world dynamics, new objects with unseen physical properties, and even different robot hardware embodiments, demonstrating its excellent generalization capability relative to alternative approaches.

Dynamic Treatment Regimes Using Bayesian Additive Regression Trees for Censored Outcomes

To achieve the goal of providing the best possible care to each individual under their care, physicians need to customize treatments for individuals with the same health state, especially when treating diseases that can progress further and require additional treatments, such as cancer. Making decisions at multiple stages as a disease progresses can be formalized as a dynamic treatment regime (DTR). Most of the existing optimization approaches for estimating dynamic treatment regimes including the popular method of Q-learning were developed in a frequentist context. Recently, a general Bayesian machine learning framework that facilitates using Bayesian regression modeling to optimize DTRs has been proposed. In this article, we adapt this approach to censored outcomes using Bayesian additive regression trees (BART) for each stage under the accelerated failure time modeling framework, along with simulation studies and a real data example that compare the proposed approach with Q-learning. We also develop an R wrapper function that utilizes a standard BART survival model to optimize DTRs for censored outcomes. The wrapper function can easily be extended to accommodate any type of Bayesian machine learning model.

Machine learning and genetic algorithm-based framework for the life-cycle cost-based optimal design of self-centering building structures

Self-centering systems have emerged as innovative lateral force-resisting mechanisms aimed at improving the earthquake resilience of building structures. The fully self-centering behavior effectively eliminates the residual deformation, thereby improving the post-earthquake repairability of building structures. However, this behavior does intensify floor acceleration responses that contribute to nonstructural damage and lead to increased initial costs. This research aims to develop a general machine learning and genetic algorithm-based framework for optimizing the design of self-centering building structures and to strike a balance between the initial costs and seismic losses by minimizing the overall life-cycle cost. The artificial neural network (ANN) based StructureNet model was developed for accelerating the calculation of life-cycle costs associated with self-centering structures. The genetic algorithm was adopted to determine the optimal design parameters for self-centering structures. The framework was applied to 8 design cases, with the analysis centering on two factors: the maximum allowable residual inter-story drift for controlling demolition and the initial construction cost of self-centering members. The study examined their influences on the optimization design results of self-centering building structures. The findings highlight a preference for a design approach that leans toward partially self-centering behavior as opposed to fully self-centering behavior, indicating the higher life-cycle benefits of partially self-centering behavior.

A machine learning framework for neighbor generation in metaheuristic search

This paper presents a methodology for integrating machine learning techniques into metaheuristics for solving combinatorial optimization problems. Namely, we propose a general machine learning framework for neighbor generation in metaheuristic search. We first define an efficient neighborhood structure constructed by applying a transformation to a selected subset of variables from the current solution. Then, the key of the proposed methodology is to generate promising neighbors by selecting a proper subset of variables that contains a descent of the objective in the solution space. To learn a good variable selection strategy, we formulate the problem as a classification task that exploits structural information from the characteristics of the problem and from high-quality solutions. We validate our methodology on two metaheuristic applications: a Tabu Search scheme for solving a Wireless Network Optimization problem and a Large Neighborhood Search heuristic for solving Mixed-Integer Programs. The experimental results show that our approach is able to achieve a satisfactory trade-offs between the exploration of a larger solution space and the exploitation of high-quality solution regions on both applications.

FEAT: A general framework for feature-aware multivariate time-series representation learning

Multivariate time-series is complex and uncertain. The overall temporal patterns change dynamically over time, and each feature is often observed to have a unique pattern. Therefore, it is challenging to model a framework that can flexibly learn feature-specific unique patterns as well as dynamically changing temporal patterns simultaneously. We propose a general framework for FEature-Aware multivariate Time-series representation learning, called FEAT. Unlike previous methods that only focus on training the overall temporal dependencies, we focus on training feature-specific as well as feature-agnostic representations in a data-driven manner. Specifically, we introduce a feature-wise encoder to explicitly model the feature-specific information and design an element-wise gating layer that learns the influence of feature-specific patterns per dataset in general. FEAT outperforms the benchmark models in average accuracy on 29 UEA multivariate time-series classification datasets and in MSE and MAE on four multivariate time-series forecasting datasets.

I’m Me, We’re Us, and I’m Us: Tri-directional Contrastive Learning on Hypergraphs

Although machine learning on hypergraphs has attracted considerable attention, most of the works have focused on (semi-)supervised learning, which may cause heavy labeling costs and poor generalization. Recently, contrastive learning has emerged as a successful unsupervised representation learning method. Despite the prosperous development of contrastive learning in other domains, contrastive learning on hypergraphs remains little explored. In this paper, we propose TriCL (Tri-directional Contrastive Learning), a general framework for contrastive learning on hypergraphs. Its main idea is tri-directional contrast, and specifically, it aims to maximize in two augmented views the agreement (a) between the same node, (b) between the same group of nodes, and (c) between each group and its members. Together with simple but surprisingly effective data augmentation and negative sampling schemes, these three forms of contrast enable TriCL to capture both node- and group-level structural information in node embeddings. Our extensive experiments using 14 baseline approaches, 10 datasets, and two tasks demonstrate the effectiveness of TriCL, and most noticeably, TriCL almost consistently outperforms not just unsupervised competitors but also (semi-)supervised competitors mostly by significant margins for node classification. The code and datasets are available at https://github.com/wooner49/TriCL.

Incomplete Multi-View Multi-Label Learning via Label-Guided Masked View- and Category-Aware Transformers

As we all know, multi-view data is more expressive than single-view data and multi-label annotation enjoys richer supervision information than single-label, which makes multi-view multi-label learning widely applicable for various pattern recognition tasks. In this complex representation learning problem, three main challenges can be characterized as follows: i) How to learn consistent representations of samples across all views? ii) How to exploit and utilize category correlations of multi-label to guide inference? iii) How to avoid the negative impact resulting from the incompleteness of views or labels? To cope with these problems, we propose a general multi-view multi-label learning framework named label-guided masked view- and category-aware transformers in this paper. First, we design two transformer-style based modules for cross-view features aggregation and multi-label classification, respectively. The former aggregates information from different views in the process of extracting view-specific features, and the latter learns subcategory embedding to improve classification performance. Second, considering the imbalance of expressive power among views, an adaptively weighted view fusion module is proposed to obtain view-consistent embedding features. Third, we impose a label manifold constraint in sample-level representation learning to maximize the utilization of supervised information. Last but not least, all the modules are designed under the premise of incomplete views and labels, which makes our method adaptable to arbitrary multi-view and multi-label data. Extensive experiments on five datasets confirm that our method has clear advantages over other state-of-the-art methods.

Hierarchical Consistent Contrastive Learning for Skeleton-Based Action Recognition with Growing Augmentations

Contrastive learning has been proven beneficial for self-supervised skeleton-based action recognition. Most contrastive learning methods utilize carefully designed augmentations to generate different movement patterns of skeletons for the same semantics. However, it is still a pending issue to apply strong augmentations, which distort the images/skeletons’ structures and cause semantic loss, due to their resulting unstable training. In this paper, we investigate the potential of adopting strong augmentations and propose a general hierarchical consistent contrastive learning framework (HiCLR) for skeleton-based action recognition. Specifically, we first design a gradual growing augmentation policy to generate multiple ordered positive pairs, which guide to achieve the consistency of the learned representation from different views. Then, an asymmetric loss is proposed to enforce the hierarchical consistency via a directional clustering operation in the feature space, pulling the representations from strongly augmented views closer to those from weakly augmented views for better generalizability. Meanwhile, we propose and evaluate three kinds of strong augmentations for 3D skeletons to demonstrate the effectiveness of our method. Extensive experiments show that HiCLR outperforms the state-of-the-art methods notably on three large-scale datasets, i.e., NTU60, NTU120, and PKUMMD. Our project is publicly available at: https://jhang2020.github.io/Projects/HiCLR/HiCLR.html.

Sound and complete causal identification with latent variables given local background knowledge

Great efforts have been devoted to causal discovery from observational data, and it is well known that introducing some background knowledge attained from experiments or human expertise can be very helpful. However, it remains unknown that what causal relations are identifiable given background knowledge in the presence of latent confounders. In this paper, we solve the problem with sound and complete orientation rules when the background knowledge is given in a local form. Furthermore, based on the solution to the problem, this paper proposes two applications that are of independent interests. One is that we give a maximal ancestral graph (MAG) listing algorithm, to output all the MAGs consistent to the observational data in the presence of latent variables. The other application is that we present a general active learning framework for causal discovery in the presence of latent confounders, where we propose a baseline criterion to select the intervention variable with a Metropolis-Hastings MAG-sampling method. Experiments validate the efficiency of the proposed MAG listing method and the effectiveness of the active learning framework.

GTRF: A general deep learning framework for tuples recognition towards supervised, semi-supervised and unsupervised paradigms

Tuple, featured as sequences of elements are regarded as one of the most predominant data forms in structural health monitoring (SHM). Activated by powerful capabilities of deep learning (DL) techniques, the DL-driven tuple recognition regime has facilitated plenty of problems settlement in SHM practice by mapping tuples with structural patterns, whereas the determined feature extraction strategies, designated network architectures and specified learning schemas (i.e., supervision paradigms) degrade severely to the model’s transferability and generalizability. Thereby, this study devises a novel General Tuple Recognition Framework (GTRF) towards supervised (SL), unsupervised (UL) and semi-supervised learning (SSL) paradigms. In the framework, pattern-sensitive features (PSFs) are quantitatively defined via a novel feature extractor intervened by deep autoencoder for downstream label propagation via an optimized fuzzy clustering algorithm in SL, UL and SSL paradigms. With sophisticated networks integrated and exquisite novelties embedded, the proposed GTRF is competent for various tuple recognition tasks in diverse learning paradigms regardless of the measurement types or lengths. Multi-level experimental tasks implementation representative in SHM scope were conducted for validations, varying in vibration SL-recognition of a prototype skyscraper, damage UL-detection of a laboratory RC beam and condition SSL-assessment of a full-scale building model. The results comprehensively confirmed the effectiveness and generality of proposed GTRF as well as comparable superiority in recognition accuracy and model adaptability. With flexible paradigm specialization, broad application and great space for optimization, the proposed GTRF framework can promisingly be a prototype for bridging the gap for DL algorithms fusion and models integration of different learning paradigms.

An Optimal Transport Analysis on Generalization in Deep Learning.

Deep neural networks (DNNs) have achieved state-of-the-art performance in various learning tasks, such as computer vision, natural language processing, and speech recognition. However, the fundamental theory of generalization still remains obscure in deep learning-why DNN models can generalize well, despite that they are heavily overparametrized in both depth and width? Recently, some work shows that traditional theory of analyzing the generalization error of learning models fails to explain the generalization of DNNs. The failure is mainly because of one simple fact that the worse case analysis of generalization error for learning models would be too loose for models with large parameter space, such as DNNs. In this work, we propose a new analysis of generalization in DNNs from an optimal transport perspective. Unlike traditional worse-case uniform convergence analysis in learning theory, our analysis of generalization error is dependent on both the learning algorithm and the data distribution and is the average-case analysis. Thus, our theory can be more practical and accurate to describe the generalization behavior of DNNs. More specifically, in this article, we try to answer a fundamental yet unsolved question in deep learning-why deeper models can generalize well than shallow models? The main contribution of this article can be summarized in four aspects. First, under a general learning framework, we derive upper bounds on the generalization error of learning algorithms by their algorithmic transport cost: the expected Wasserstein distance between the output hypothesis and the output hypothesis conditioned on an input example. We further provide several upper bounds on the algorithmic transport cost in terms of total variation distance, relative entropy, and Vapnik-Chervonenkis (VC) dimension. Moreover, we also study different conditions for loss functions under which the generalization error of a learning algorithm can be upper bounded by different probability metrics between distributions relating to the output hypothesis and/or the input data. Finally, under our established framework, we obtain our main results, showing that the generalization error in DNNs decreases exponentially to zero as the number of layers increases.

Active learning applied to automated physical systems increases the rate of discovery

Active machine learning is widely used in computational studies where repeated numerical simulations can be conducted on high performance computers without human intervention. But translation of these active learning methods to physical systems has proven more difficult and the accelerated pace of discoveries aided by these methods remains as yet unrealized. Through the presentation of a general active learning framework and its application to large-scale boundary layer wind tunnel experiments, we demonstrate that the active learning framework used so successfully in computational studies is directly applicable to the investigation of physical experimental systems and the corresponding improvements in the rate of discovery can be transformative. We specifically show that, for our wind tunnel experiments, we are able to achieve in approximately 300 experiments a learning objective that would be impossible using traditional methods.

The general framework for few-shot learning by kernel HyperNetworks

Few-shot models aim at making predictions using a minimal number of labeled examples from a given task. The main challenge in this area is the one-shot setting, where only one element represents each class. We propose the general framework for few-shot learning via kernel HyperNetworks—the fusion of kernels and hypernetwork paradigm. Firstly, we introduce the classical realization of this framework, dubbed HyperShot. Compared to reference approaches that apply a gradient-based adjustment of the parameters, our models aim to switch the classification module parameters depending on the task’s embedding. In practice, we utilize a hypernetwork, which takes the aggregated information from support data and returns the classifier’s parameters handcrafted for the considered problem. Moreover, we introduce the kernel-based representation of the support examples delivered to hypernetwork to create the parameters of the classification module. Consequently, we rely on relations between the support examples’ embeddings instead of the backbone models’ direct feature values. Thanks to this approach, our model can adapt to highly different tasks. While such a method obtains very good results, it is limited by typical problems such as poorly quantified uncertainty due to limited data size. We further show that incorporating Bayesian neural networks into our general framework, an approach we call BayesHyperShot, solves this issue.

Metal3D: a general deep learning framework for accurate metal ion location prediction in proteins

Metal ions are essential cofactors for many proteins and play a crucial role in many applications such as enzyme design or design of protein-protein interactions because they are biologically abundant, tether to the protein using strong interactions, and have favorable catalytic properties. Computational design of metalloproteins is however hampered by the complex electronic structure of many biologically relevant metals such as zinc . In this work, we develop two tools - Metal3D (based on 3D convolutional neural networks) and Metal1D (solely based on geometric criteria) to improve the location prediction of zinc ions in protein structures. Comparison with other currently available tools shows that Metal3D is the most accurate zinc ion location predictor to date with predictions within 0.70 ± 0.64 Å of experimental locations. Metal3D outputs a confidence metric for each predicted site and works on proteins with few homologes in the protein data bank. Metal3D predicts a global zinc density that can be used for annotation of computationally predicted structures and a per residue zinc density that can be used in protein design workflows. Currently trained on zinc, the framework of Metal3D is readily extensible to other metals by modifying the training data.