Meta Learning Research Articles

Digital Twin Generalization with Meta and Geometric Deep Learning

Deep digital twins (DDTs) are deep neural networks that encode the behavior of complex physical systems. DDTs are excellent system representations due to their ability to continuously adapt to operational changes and their capability to capture complex relationships between system components and processes that cannot be explicitly modeled. For this challenge, DDTs benefit greatly from recent success in geometric deep learning (GDL) which allows the integration of information from multiple systems based on schematic representations. A major challenge in training DDTs is their dependence on the quality and representativeness of training data, especially under the dynamic conditions typical in prognostics and health management (PHM). Recent developments in differentiable simulation present new opportunities for optimizing the training data representativeness. In this thesis, we propose a novel meta-learning framework that trains DDTs using the output from differentiable simulators. This setup enables active optimization of training data sampling through gradient computation, enhancing training speed, robustness, and data representativeness. We extend this framework to address challenges in multi-system data integration in power grids and fault detection in railway traction networks. By applying our framework, we aim to tackle significant challenges in forecasting, anomaly detection and sensor-fault analysis using advanced data fusion techniques. Our approach promises substantial improvements in DDT robustness and operational efficiency, with its effectiveness to be demonstrated through empirical studies on both simple and complex case studies within the power systems domain.

Application of Machine Learning Models for Malware Classification With Real and Synthetic Datasets

Stacking of multiple Machine Learning (ML) classifiers have gained popularity in addressing anomalous data classification along with Deep Learning (DL) algorithms. This study compares traditional ML classifiers, multi-layer stacking ML classifiers, and DL classifiers using an open-source malware dataset-containing equal numbers of benign and malware samples. The results on the realistic dataset indicate that the DL classifier, utilizing a Bidirectional Long Short-Term Memory (BiLSTM) model, outperformed the stacked classifiers with Logistic Regression (LR) and Support Vector Machine (SVM) as Meta learners by 36.78% and 39.69%, respectively, in terms of classification accuracy and performance. The research work was extended to study the impact of Generative Adversarial Network (GAN) based synthetic dataset of relatively smaller size on deep learning models. It was observed that the Deep Learning Multi-Layer Perceptron (DLMLP) Model had relatively superior performance as compared to complex deep learning models like Long Short-Term Memory LSTM and BiLSTM

A scheme of combining DFO and channel estimation scheme for mobile OFDM systems

AbstractTo reduce the impact of residual Doppler frequency offset (RDFO), a joint DFO estimation and CE scheme is proposed for the OFDM systems under the high‐speed mobile environment. In the paper, the expression of interference power caused by the RDFO is first derived, and the effect of RDFO on time‐varying characteristic of channel is analysed. Then, a joint DFO and channel estimation scheme is presented. Specifically, a high‐precision DFO estimator based on the convolutional neural network with anti‐noise is firstly designed. Due to its ability to use fewer samples to adapt well to the new environments, the meta learning is adopted to estimate the time‐varying channel. Moreover, to improve the practicality of the algorithm, the non‐ideal values rather than ideal values are used as the training targets in the two neural networks. Additionally, the proposed method is only based on the received signal and does not require any pilots or training sequences, which has higher transmission efficiency compared to the existing algorithms. The research results indicate that the proposed method has good estimation performance and good practicality, and it is suitable for high‐speed mobile scenarios.

Prediction of electron-solid interaction parameters using machine learning.

Electron backscattering coefficient and electron-stopping power are essential concepts in many disciplines, from radiation to materials science, semiconductor manufacturing, and space exploration. They enable precise calculations, measurements, and simulations of electron interactions with matter, which contribute to advancing science, technology, and safety in a variety of applications. The availability of these data is fundamental to scientific research to validate hypotheses, conduct experiments, and explore new theories. A relatively novel machine learning approach has demonstrated notable success in enhancing data quality and completeness, significantly contributing to the facilitation of data discovery. Using fundamental material property data, the stacking ensemble machine learning (EML) technique was established in this study to generate electron-solid interaction parameters for any target material over a wide range of energies. The final stacking EML was built using the base and meta learners bagging regressor (BR), K-nearest neighbors (k-NN), random forest (RF), support vector regression (SVR), and eXtreme Gradient Boosting (XGB). In this study, two publicly available databases with a total of 4030 data points were used. Training datasets have 785 and 525 data points for electron backscattering coefficient and stopping power, respectively, whereas testing datasets contain 262 and 175 data points. Fivefeatures were used as input variables to train different individual algorithms and their combinations. On both the training and test datasets, the model was evaluated using different error metrics, including R-squared (R2), mean-absolute-error (MAE), root-mean-squared-error (RMSE), and mean-absolute-percentage-error (MAPE). Our model evaluation tests revealed that combining RF and XGB with a k-NN meta-learner outperformed other algorithms. The analysis of error metrics demonstrated a very close fit to all samples in each training dataset. Furthermore, predictions made by the model on unseen test data indicated accurate estimations of new backscattering and stopping power data. The developed model achieved high prediction accuracy for various target materials across the broad electron energy spectrum. The outcomes demonstrate the effectiveness of machine learning methodology and the chosen models' suitability for addressing substantial physics challenges.

Natural Gas Consumption Forecasting Based on Homoheterogeneous Stacking Ensemble Learning

Natural gas consumption is an important indicator of energy utilization and demand, and its scientific and high-accuracy prediction plays a key role in energy policy formulation. With the development of deep neural networks and ensemble learning, a homoheterogeneous stacking ensemble learning method is proposed for natural gas consumption forecasting. Firstly, to obtain the potential data characteristics, a nonlinear concave and convex transformation-based data dimension enhancement method is designed. Then, with the aid of a stacking ensemble learning framework, the multiscale autoregressive integrated moving average (ARIMA) and high-order fuzzy cognitive map (HFCM) methods are chosen as the base learner models, while the meta learner model is constructed via a well-designed deep neural network with long short-term memory (LSTM) cells. Finally, with the natural gas energy consumption data of national and 30 provinces (where the data of Xizang are unavailable) of China from 2000 to 2019, the numerical results show the proposed algorithm has a better forecasting performance in accuracy, robustness to noise, and sensitivity to data variations than the seven compared traditional and ensemble methods, and the corresponding model applicability rate could achieve more than 90%.

Meta Learning to Rank for Sparsely Supervised Queries

Supervisory signals are a critical resource for training learning to rank models. In many real-world search and retrieval scenarios, these signals may not be readily available or could be costly to obtain for some queries. The examples include domains where labeling requires professional expertise, applications with strong privacy constraints, and user engagement information that are too scarce. We refer to these scenarios as sparsely supervised queries which pose significant challenges to traditional learning to rank models. In this work, we address sparsely supervised queries by proposing a novel meta learning to rank framework which leverages fast learning and adaption capability of meta-learning. The proposed approach accounts for the fact that different queries have different optimal parameters for their rankers, in contrast to traditional learning to rank models which only learn a global ranking model applied to all the queries. In consequence, the proposed method would yield significant advantages especially when new queries are of different characteristics with the training queries. Moreover, the proposed meta learning to rank framework is generic and flexible. We conduct a set of comprehensive experiments on both public datasets and a real-world e-commerce dataset. The results demonstrate that the proposed meta-learning approach can significantly enhance the performance of learning to rank models with sparsely labeled queries.

Efficient data-driven nonlinear system identification for structural health monitoring: A proof-of-principle study

Structural health monitoring (SHM) is the process of detecting damage in structures. Typically, the damage-sensitive feature identification is based on fitting some models to the measured system response data. The parameters of these models or the prediction errors associated with these models then become the desired damage-sensitive features. Real-world structures usually exhibit significant nonlinear behaviors. Data-driven nonlinear system identification (NSI) is essential to obtain accurate models of structural dynamics for reliable damage detection. However, on the one hand, the data acquisition of real-world structures for damage detection is usually difficult or scarce. On the other hand, NSI generally needs to solve a nonlinear optimization problem that requires more computational time for convergence than linear system identification. Data efficiency and computational efficiency become two critical challenges in data-driven NSI. To address these challenges, this work presents a proof-of-principle study on developing a meta learning-based method for efficient data-driven NSI. Specifically, the meta learning algorithm leverages a previously collected database of similar but different systems to learn the meta-knowledge about how to identify a new system, enabling a fast identification/modeling with limited data. Numerical simulations are performed to validate the method on fundamental nonlinear dynamical systems widely seen in civil engineering. It is observed that the presented meta learning-based method outperforms the conventional method in terms of both data and computational efficiency. Limitations of this work are further discussed.

Estimation of Carbon Stock in Chinese Fir Stands Based on Stacking Ensemble Learning and LSTM Models

Abstract Forest carbon sinks, a critical component of the global carbon cycle, constitute nearly half of the total terrestrial carbon pool. This study employed correlation analysis and factor effect analysis to quantify the influences of various factors on the volume of Chinese fir stands. A novel modeling framework was developed using the stacking ensemble learning and LSTM (LongShort-TermMemory) models. This framework incorporated diverse base learners, including XGBoost, Adaboost, KNN, and DT, which are intelligently ensemble via GBDT as a meta learner. The model was further optimized by comparing activation functions and optimizers, with ReLU selected as the activation function and Adam as the optimizer. Model accuracy was evaluated using RMSE, MAE, and R2 metrics, significantly enhancing its learning ability and generalization performance. The findings are as follows: (1) Stand stocking showed strong positive correlations with depression, age group, average diameter at breast height (ADBH), average tree height (ATH), slope, and elevation. Conversely, it exhibited significant negative correlations with origin, stand density, and slope position. In investigating Chinese fir growth on slopes, no significant growth differences were observed between downslopes and midslopes; however, both differed significantly from upper slopes. (2) The stacking ensemble learning method constructed here surpassed all existing single models in terms of estimation and assessment indices, demonstrating superior comprehensive performance. (3) Among the LSTM models, Adam-LSTM performed the best (R2=0.844), followed by Sigmoid-LSTM (R2=0.656), while the RMSprop-LSTM model performed the worst (R2=0.618). Combined with artificial intelligence methods, our optimized carbon stock estimation model can help to improve the ability of forest land management and provide a theoretical basis for the scientific management of forest areas.

MT-Net: Single image dehazing based on meta learning, knowledge transfer and contrastive learning

Single image dehazing is becoming increasingly important as its results impact the efficiency of subsequent computer vision tasks. While many methods have been proposed to address this challenge, existing dehazing approaches often exhibit limited adaptability to different types of images and lack future learnability. In light of this, we propose a dehazing network based on meta-learning, knowledge transfer, and contrastive learning, abbreviated as MT-Net. In our approach, we combine knowledge transfer with meta-learning to tackle these challenges, thus enhancing the network’s generalization performance. We refine the structure of knowledge transfer by introducing a two-phases approach to facilitate learning under the guidance of teacher networks and learning committee networks. We also optimize the negative examples of contrastive learning to reduce the contrast space. Extensive experiments conducted on synthetic and real datasets demonstrate the remarkable performance of our method in both quantitative and qualitative comparisons. The code has been released on https://github.com/71717171fan/MT-Net.

MBI-TCMR: A Traditional Chinese Formula Recommendation Deep Learning Model Based on Multi-Task Meta Learning and Bi-Ological Network Structure

MBI-TCMR: A Traditional Chinese Formula Recommendation Deep Learning Model Based on Multi-Task Meta Learning and Bi-Ological Network Structure

Metafeature Selection via Multivariate Sparse-Group Lasso Learning for Automatic Hyperparameter Configuration Recommendation.

The performance of classification algorithms is mainly governed by the hyperparameter settings deployed in applications, and the search for desirable hyperparameter configurations usually is quite challenging due to the complexity of datasets. Metafeatures are a group of measures that characterize the underlying dataset from various aspects, and the corresponding recommendation algorithm fully relies on the appropriate selection of metafeatures. Metalearning (MtL), aiming to improve the learning algorithm itself, requires development in integrating features, models, and algorithm learning to accomplish its goal. In this article, we develop a multivariate sparse-group Lasso (SGLasso) model embedded with MtL capacity in recommending suitable configurations via learning. The main idea is to select the principal metafeatures by removing those redundant or irregular ones, promoting both efficiency and performance in the hyperparameter configuration recommendation. To be specific, we first extract the metafeatures and classification performance of a set of configurations from the collection of historical datasets, and then, a metaregression task is established through SGLasso to capture the main characteristics of the underlying relationship between metafeatures and historical performance. For a new dataset, the classification performance of configurations can be estimated through the selected metafeatures so that the configuration with the highest predictive performance in terms of the new dataset can be generated. Furthermore, a general MtL architecture combined with our model is developed. Extensive experiments are conducted on 136 UCI datasets, demonstrating the effectiveness of the proposed approach. The empirical results on the well-known SVM show that our model can effectively recommend suitable configurations and outperform the existing MtL-based methods and the well-known search-based algorithms, such as random search, Bayesian optimization, and Hyperband.

Loan Defaults Prediction Based on Stacked Models Trained by Personalized Features

Training one machine learning model with features that all clients have will result in a waste of features, which is likely to adversely affect the model’s performance. To solve the problem, the study attempts a new method, which it to train an individual stacked model for each loan client based on personalised features. Data used contains information of about fifteen million loan applicants, their default status, and 468 features in all. 41 of the features that can be quantitively analysed are selected according to the feature importance output by a Random Forest model. Default prediction of every client is made by a stacked model trained with all selected features he/she has. The stacked model consists of two layers, in which a Light Gradient-Boosting Machine (LGBM) classifier is the base learner, and a Logistic Regression model is the meta learner. As the defaulters account for only 3.14%, which is significantly unbalanced, Area Under the Curve (AUC) and F1 scores are employed to evaluate the method, instead of accuracy. Test results show that models trained by personalised features perform better than the ones trained by shared features. Additionally, the stacked model outperforms individual Logistic Regression model, but performs nearly the same as individual LGBM Classifier. In detailed, the stacked models trained with personalised features result in AUC=0.772 and F1=0.188. Due to data unbalance, although the method’s F1 score is relatively low, it’s considered to be passable. In the future, stacked models combining different models will be attempted.

Marginal debiased network for fair visual recognition

Deep neural networks (DNNs) are often prone to learn the spurious correlations between target classes and bias attributes, like gender and race, inherent in a major portion of training data (bias-aligned samples), thus showing unfair behavior and arising controversy in the modern pluralistic and egalitarian society. In this paper, we propose a novel marginal debiased network (MDN) to learn debiased representations. More specifically, a marginal softmax loss (MSL) is designed by introducing the idea of margin penalty into the fairness problem, which assigns a larger margin for bias-conflicting samples (data without spurious correlations) than for bias-aligned ones, so as to deemphasize the spurious correlations and improve generalization on unbiased test criteria. To determine the margins, our MDN is optimized through a meta learning framework. We propose a meta equalized loss (MEL) to perceive the model fairness, and adaptively update the margin parameters by meta-optimization which requires the trained model guided by the optimal margins should minimize MEL computed on an unbiased meta-validation set. Extensive experiments on BiasedMNIST, Corrupted CIFAR-10, CelebA and UTK-Face datasets demonstrate that our MDN can achieve a remarkable performance on under-represented samples and obtain superior debiased results against the previous approaches.

Using the MSFNet Model to Explore the Temporal and Spatial Evolution of Crop Planting Area and Increase Its Contribution to the Application of UAV Remote Sensing

Remote sensing technology can be used to monitor changes in crop planting areas to guide agricultural production management and help achieve regional carbon neutrality. Agricultural UAV remote sensing technology is efficient, accurate, and flexible, which can quickly collect and transmit high-resolution data in real time to help precision agriculture management. It is widely used in crop monitoring, yield prediction, and irrigation management. However, the application of remote sensing technology faces challenges such as a high imbalance of land cover types, scarcity of labeled samples, and complex and changeable coverage types of long-term remote sensing images, which have brought great limitations to the monitoring of cultivated land cover changes. In order to solve the abovementioned problems, this paper proposed a multi-scale fusion network (MSFNet) model based on multi-scale input and feature fusion based on cultivated land time series images, and further combined MSFNet and Model Diagnostic Meta Learning (MAML) methods, using particle swarm optimization (PSO) to optimize the parameters of the neural network. The proposed method is applied to remote sensing of crops and tomatoes. The experimental results showed that the average accuracy, F1-score, and average IoU of the MSFNet model optimized by PSO + MAML (PSML) were 94.902%, 91.901%, and 90.557%, respectively. Compared with other schemes such as U-Net, PSPNet, and DeepLabv3+, this method has a better effect in solving the problem of complex ground objects and the scarcity of remote sensing image samples and provides technical support for the application of subsequent agricultural UAV remote sensing technology. The study found that the change in different crop planting areas was closely related to different climatic conditions and regional policies, which helps to guide the management of cultivated land use and provides technical support for the realization of regional carbon neutrality.

Meta learning regression framework for energy consumption prediction in retrofitted buildings: A case study of South Korea

Building energy consumption (BEC) prediction is crucial for optimizing energy efficiency in building retrofit projects, particularly during the early design phase. However, existing machine learning (ML) models often face challenges with overfitting and complexity, limiting their accuracy and generalizability. This study addresses these issues by proposing a novel meta-learning regression framework for energy prediction in green retrofitted buildings. The framework employs optimized base regressors using various ML models and integrates them through stacking regression to enhance predictive accuracy and generalization. The model’s performance is evaluated against advanced methods like Deep Neural Networks and Ensemble Learning Regression using data from over 811 green retrofitting projects in South Korea. The proposed method demonstrates superior efficiency, achieving the lowest mean absolute error (19.899 for Primary Energy Consumption and 11.301 for Energy Required Amount), lowest root mean squared error (29.494 for Primary Energy Consumption and 19.977 for Energy Required Amount), and highest R-squared score (0.786 for Primary Energy Consumption and 0.542 for Energy Required Amount). This research contributes a novel approach to ensemble modeling for BEC prediction, showcasing its superior accuracy, generalizability, and practical applicability in the context of green building retrofits.

Enhancing Coherence and Diversity in Multi-class Slogan Generation Systems

Many problems related to natural language processing are solved by neural networks and big data. Researchers have previously focused on single-task supervised goals with limited data management to train slogan classification. A multi-task learning framework is used to learn jointly across several tasks related to generating multi-class slogan types. This study proposes a multi-task model named slogan generative adversarial network systems (Slo-GAN) to enhance coherence and diversity in slogan generation, utilizing generative adversarial networks and recurrent neural networks (RNN). Slo-GAN generates a new text slogan-type corpus, and the training generalization process is improved. We explored active learning (AL) and meta-learning (ML) for dataset labeling efficiency. AL reduced annotations by 10% compared to ML but still needed about 70% of the full dataset for baseline performance. The whole framework of Slo-GAN is supervised and trained together on all of these tasks. The text with the higher reporting score level is filtered by Slo-GAN, and a classification accuracy of 87.2% is achieved. We leveraged relevant datasets to perform a cross-domain experiment, reinforcing our assertions regarding both the distinctiveness of our dataset and the challenges of adapting bilingual dialects to one another.

Affective, meta and collaborative STEAM learning

The purpose of this research was to identify synergies in affective, meta and collaborative STEAM learning, and the level of professional development structure and collaborative support needed for teachers to achieve these. Three project case studies (KIKS, STEAMTEACH and STEAMCONNECT) are examined to do this. In the KIKS (Kids Inspiring Kids in STEAM) project, student groups were issued a very open-ended challenge: “How would you get your fellow students to love STEAM?” They undertook a series of student-led intensive workshops to deliver prototypes and other teaching materials. The intensive, collaborative problem-solving Hothousing workshops focusing on Expectation shock perception enhancement, were inspired by industry practice. The successful outcomes evidenced affective, meta and collaborative learning and suggested the need for further research to develop an associated teacher professional development framework (STEAMTEACH) and also to develop further the collaborative aspect beyond the students (STEAMCONNECT). The projects together delivered over forty project outcomes judged high quality by teachers, students and researchers alike benefitted from affective, meta and collaborative, inclusive learning on many levels from students to teachers to parents to wider community collaboration including local community, business and academic experts. It was thus found that STEAM education can synergically enhance affective, meta and collaborative learning, and for this to happen, should be supported by a STEAM professional development framework and extended networking between teachers, schools, parents, education and industry experts, and community from all abilities and backgrounds.

Meta Learning With Graph Attention Networks for Low-Data Drug Discovery.

Finding candidate molecules with favorable pharmacological activity, low toxicity, and proper pharmacokinetic properties is an important task in drug discovery. Deep neural networks have made impressive progress in accelerating and improving drug discovery. However, these techniques rely on a large amount of label data to form accurate predictions of molecular properties. At each stage of the drug discovery pipeline, usually, only a few biological data of candidate molecules and derivatives are available, indicating that the application of deep neural networks for low-data drug discovery is still a formidable challenge. Here, we propose a meta learning architecture with graph attention network, Meta-GAT, to predict molecular properties in low-data drug discovery. The GAT captures the local effects of atomic groups at the atom level through the triple attentional mechanism and implicitly captures the interactions between different atomic groups at the molecular level. GAT is used to perceive molecular chemical environment and connectivity, thereby effectively reducing sample complexity. Meta-GAT further develops a meta learning strategy based on bilevel optimization, which transfers meta knowledge from other attribute prediction tasks to low-data target tasks. In summary, our work demonstrates how meta learning can reduce the amount of data required to make meaningful predictions of molecules in low-data scenarios. Meta learning is likely to become the new learning paradigm in low-data drug discovery. The source code is publicly available at: https://github.com/lol88/Meta-GAT.

A universal knowledge model and cognitive architectures for prototyping AGI

The article identified 56 cognitive architectures for creating general artificial intelligence (AGI) and proposed a set of interrelated functional blocks that an agent approaching AGI in its capabilities should possess. Since the required set of blocks is not found in any of the existing architectures, the article proposes a reference cognitive architecture for intelligent systems approaching AGI in their capabilities. As one of the key solutions within the framework of the architecture, a universal method of knowledge representation is proposed, which allows combining various non-formalized, partially and fully formalized methods of knowledge representation in a single knowledge base, such as texts in natural languages, images, audio and video recordings, graphs, algorithms, databases, neural networks, knowledge graphs, ontologies, frames, essence-property-relation models, production systems, predicate calculus models, conceptual models, and others. To combine and structure various fragments of knowledge, archigraph model are used, constructed as a development of annotated metagraphs. As other components, the reference cognitive architecture being developed includes following modules: machine consciousness, machine subconsciousness, interaction with the external environment, a goal management, an emotional control, social interaction, reflection, ethics, worldview, learning, monitoring, statement problems, solving problems, self-organization and meta learning. Based on the composition of the proposed reference architecture modules, existing cognitive architectures containing the following modules were analyzed: machine consciousness, machine subconsciousness, reflection, worldview.

GCReID: Generalized continual person re-identification via meta learning and knowledge accumulation

Person re-identification (ReID) has made good progress in stationary domains. The ReID model must be retrained to adapt to new scenarios (domains) as they emerge unexpectedly, which leads to catastrophic forgetting. Continual learning trains the model in the order of domain emergence to alleviate catastrophic forgetting. However, generalization ability of the model is still limited due to the distribution difference between training and testing domains. To address the above problem, we propose the generalized continual person re-Identification (GCReID) model to continuously train an anti-forgetting and generalizable model. We endeavor to increase the diversity of samples by prior to simulate unseen domains. Meta-train and meta-test are adopted to enhance generalization of the model. Universal knowledge extracted from all seen domains and the simulated domains is stored in a set of feature embeddings. The knowledge is continually updated and applied to guide meta-train and meta-test via a graph attention network. Extensive experiments on 12 benchmark datasets and comparisons with 6 representative models demonstrate the effectiveness of the proposed model GCReID in enhancing generalization performance on unseen domains and alleviating catastrophic forgetting of seen domains. The code will be available at https://github.com/DFLAG-NEU/GCReID if our work is accepted.