Supervised Learning Framework Research Articles

Learning for Online Mixed-Integer Model Predictive Control With Parametric Optimality Certificates

We propose a supervised learning framework for computing solutions of multi-parametric Mixed Integer Linear Programs (MILPs) that arise in Model Predictive Control. Our approach also quantifies sub-optimality for the computed solutions. Inspired by Branch-and-Bound techniques, the key idea is to train a Neural Network or Random Forest which, for a given parameter, predicts a strategy consisting of (1) a set of Linear Programs (LPs) such that their feasible sets form a partition of the feasible set of the MILP and (2) an integer solution. For control computation and sub-optimality quantification, we solve a set of LPs online in parallel. We demonstrate our approach for a motion planning example and compare against various commercial and open-source mixed-integer programming solvers.

Cut Selection in MILP based on Supervised Learning

With the increasing popularity of deep learning techniques, there has been a growing interest in combining learning methods with Mixed-Integer Linear Programming (MILP) solving processes. A promising approach is to incorporate the learning model as a module in traditional methods. Cut selection is a fundamental subroutine in modern MILP solvers used to select a subset of generated cuts and enhance solver performance. In this work, we present a supervised learning framework to improve the effectiveness of cut selection. Cut selection scoring rules are typically weighted sums of different metrics, and we have developed weighted cut selection metrics based on Machine Learning (ML) techniques for different problems. We propose a novel Neural Network (NN) architecture that incorporates a Graph Convolutional Neural Network (GCN) with a self-attention mechanism to determine appropriate weights. The resulting model serves as a component of the solver and is evaluated through controlled experiments on real-world MILPs. The numerical results demonstrate that our approach outperforms the standard SCIP cut selection strategy, especially on datasets containing the same class of problems.

Towards Enabling Binary Decomposition for Partial Multi-Label Learning.

Partial multi-label learning (PML) is an emerging weakly supervised learning framework, where each training example is associated with multiple candidate labels which are only partially valid. To learn the multi-label predictive model from PML training examples, most existing approaches work by identifying valid labels within candidate label set via label confidence estimation. In this paper, a novel strategy towards partial multi-label learning is proposed by enabling binary decomposition for handling PML training examples. Specifically, the widely used error-correcting output codes (ECOC) techniques are adapted to transform the PML learning problem into a number of binary learning problems, which refrains from using the error-prone procedure of estimating labeling confidence of individual candidate label. In the encoding phase, a ternary encoding scheme is utilized to balance the definiteness and adequacy of the derived binary training set. In the decoding phase, a loss weighted scheme is applied to consider the empirical performance and predictive margin of derived binary classifiers. Extensive comparative studies against state-of-the-art PML learning approaches clearly show the performance advantage of the proposed binary decomposition strategy for partial multi-label learning.

Porosity and permeability prediction using a transformer and periodic long short-term network

Effective reservoir parameter prediction is important for subsurface characterization and understanding fluid migration. However, conventional methods for obtaining porosity and permeability are based on either core measurements or mathematical/petrophysical modeling, which are expensive or inefficient. In this study, we develop a reliable and low-cost deep learning (DL) framework for reservoir permeability and porosity prediction from real logging data at different regions. We leverage an advanced learning architecture (i.e., the transformer model) and design a new regression network (RPTransformer) that is sensitive to the depth period change of the logging data. The RPTransformer is composed of 1D convolutional, long short-term memory (LSTM), and transformer layers. First, we use a 1D convolutional layer for the first layer of the network to extract significant features from the logging data. Then, the nonlinear mapping relationships between logging data and reservoir parameters are established using several LSTM layers with a period parameter. Afterward, we use the encoder in the vision transformer with the self-attention mechanism to further extract logging data features. The developed network is a data-driven supervised learning framework and indicates highly accurate and robust prediction results when applied to different geographic regions. To demonstrate the reliable prediction performance of our network, we compare it with several classic machine learning and state-of-the-art DL methods, e.g., random forest, multilayer LSTM, and long short-term time-series network (LSTNet). More importantly, we find the generalization and uncertainty of the network in real-world applications through comprehensive numerical experiments.

An Efficient Time-Domain End-to-End Single-Channel Bird Sound Separation Network.

Bird sounds have obvious characteristics per species, and they are an important way for birds to communicate and transmit information. However, the recorded bird sounds in the field are usually mixed, which making it challenging to identify different bird species and to perform associated tasks. In this study, based on the supervised learning framework, we propose a bird sound separation network, a dual-path tiny transformer network, to directly perform end-to-end mixed species bird sound separation in the time-domain. This separation network is mainly composed of the dual-path network and the simplified transformer structure, which greatly reduces the computational resources required of the network. Experimental results show that our proposed separation network has good separation performance (SI-SNRi reaches 19.3 dB and SDRi reaches 20.1 dB), but compared with DPRNN and DPTNet, its parameters and floating point operations are greatly reduced, which means a higher separation efficiency and faster separation speed. The good separation performance and high separation efficiency indicate that our proposed separation network is valuable for distinguishing individual birds and studying the interaction between individual birds, as well as for realizing the automatic identification of bird species on a variety of mobile devices or edge computing devices.

Learning High-Dimensional Evolving Data Streams With Limited Labels.

In the context of streaming data, learning algorithms often need to confront several unique challenges, such as concept drift, label scarcity, and high dimensionality. Several concept drift-aware data stream learning algorithms have been proposed to tackle these issues over the past decades. However, most existing algorithms utilize a supervised learning framework and require all true class labels to update their models. Unfortunately, in the streaming environment, requiring all labels is unfeasible and not realistic in many real-world applications. Therefore, learning data streams with minimal labels is a more practical scenario. Considering the problem of the curse of dimensionality and label scarcity, in this article, we present a new semisupervised learning technique for streaming data. To cure the curse of dimensionality, we employ a denoising autoencoder to transform the high-dimensional feature space into a reduced, compact, and more informative feature representation. Furthermore, we use a cluster-and-label technique to reduce the dependency on true class labels. We employ a synchronization-based dynamic clustering technique to summarize the streaming data into a set of dynamic microclusters that are further used for classification. In addition, we employ a disagreement-based learning method to cope with concept drift. Extensive experiments performed on many real-world datasets demonstrate the superior performance of the proposed method compared to several state-of-the-art methods.

GraphDPI: Partial label disambiguation by graph representation learning via mutual information maximization

Partial label learning (PLL) is a weakly supervised learning framework where each training instance is associated with more than one candidate label, and only one of them is the true label. Most of the existing PLL algorithms directly disambiguate the candidate labels according to the instance feature similarity, but fail to discover the latent semantic relationship over the entire dataset. In this paper, method GraphDPI, an innovative deep partial label disambiguation by graph representation via mutual information maximization, is proposed. This method can capture the semantic clusters with the most unique information in the latent space and automatically adapt to different feature distributions. Specifically, a new sampling method based on the graph is proposed to estimate mutual information, extending GCN to the field of weakly supervised learning. Therefore, the graph representation of the data can contain more distinguishing information to disambiguate candidate labels by maximizing the mutual information of the local graph representation and the global one. Furthermore, the triplet loss is introduced to fully exploit the relationship between instances and extract the latent embedding representation over the entire dataset. It thereby can make the model output as large as possible on the inter-class variation and as small as possible on the intra-class variation. Finally, the candidate labels can be disambiguated by the difference between semantic clusters. Experiments reveal the overwhelming performances of GraphDPI.

A CNN Regression Approach for Real-Time Estimation of Dielectrophoretic Force

This research presents a supervised learning framework with a Convolutional Neural Network (CNN) to precisely quantify dielectrophoretic force () in microfluidic devices using a regression approach. As opposed to using a classification paradigm in our prior work, here we recast the estimation task as a regression problem. Micrographs of pearl-chain arrangement in a DEP sensing device were processed using a using modified CNN architectures of AlexNet, ResNet-50, MobileNetV2, and GoogLeNet in order to predict the induced on yeast cells and polystyrene microbeads. Micrographs obtained from our dielectrophoresis (DEP) experiments with varied input voltages were preprocessed and used in building these deep regression models. prediction accuracy of the models was tested for using Mean Absolute Error (MAE), Mean Absolute Relative (MRE), Mean Squared Error (MSE), R-squared, and Root Mean Square Error (RMSE) as evaluation metrics. The results from the experiments show that the performance of our regression models had better prediction accuracy and generalization ability when compared to our classification models. ResNet-50 with RMSPROP gave the best performance, with a validation RMSE of 0.0918 on yeast cells while AlexNet with ADAM optimizer gave the best performance, with a validation RMSE of 0.1745 on microbeads. This provides a baseline for further studies in the application of deep learning and dielectrophoresis for DEP force estimation. Figure 1

Boosting single-cell gene regulatory network reconstruction via bulk-cell transcriptomic data.

Computational recovery of gene regulatory network (GRN) has recently undergone a great shift from bulk-cell towards designing algorithms targeting single-cell data. In this work, we investigate whether the widely available bulk-cell data could be leveraged to assist the GRN predictions for single cells. We infer cell-type-specific GRNs from both the single-cell RNA sequencing data and the generic GRN derived from the bulk cells by constructing a weakly supervised learning framework based on the axial transformer. We verify our assumption that the bulk-cell transcriptomic data are a valuable resource, which could improve the prediction of single-cell GRN by conducting extensive experiments. Our GRN-transformer achieves the state-of-the-art prediction accuracy in comparison to existing supervised and unsupervised approaches. In addition, we show that our method can identify important transcription factors and potential regulations for Alzheimer's disease risk genes by using the predicted GRN. Availability: The implementation of GRN-transformer is available at https://github.com/HantaoShu/GRN-Transformer.

Learning to Equalize OTFS

Orthogonal Time Frequency Space (OTFS) is a novel framework that processes modulation symbols via a time-independent channel characterized by the delay-Doppler domain. The conventional waveform, orthogonal frequency division multiplexing (OFDM), requires tracking frequency selective fading channels over the time, whereas OTFS benefits from full time-frequency diversity by leveraging appropriate equalization techniques. In this paper, we consider a neural network-based supervised learning framework for OTFS equalization. Learning of the introduced neural network is conducted in each OTFS frame fulfilling an online learning framework: the training and testing datasets are within the same OTFS-frame over the air. Utilizing reservoir computing, a special recurrent neural network, the resulting one-shot online learning is sufficiently flexible to cope with channel variations among different OTFS frames (e.g., due to the link/rank adaptation and user scheduling in cellular networks). The proposed method does not require explicit channel state information (CSI) and simulation results demonstrate a lower bit error rate (BER) than conventional equalization methods in the low signal-to-noise (SNR) regime under large Doppler spreads. When compared with its neural network-based counterparts for OFDM, the introduced approach for OTFS will lead to a better tradeoff between the processing complexity and the equalization performance.

PageNet: Towards End-to-End Weakly Supervised Page-Level Handwritten Chinese Text Recognition

Handwritten Chinese text recognition (HCTR) has been an active research topic for decades. However, most previous studies solely focus on the recognition of cropped text line images, ignoring the error caused by text line detection in real-world applications. Although some approaches aimed at page-level text recognition have been proposed in recent years, they either are limited to simple layouts or require very detailed annotations including expensive line-level and even character-level bounding boxes. To this end, we propose PageNet for end-to-end weakly supervised page-level HCTR. PageNet detects and recognizes characters and predicts the reading order between them, which is more robust and flexible when dealing with complex layouts including multi-directional and curved text lines. Utilizing the proposed weakly supervised learning framework, PageNet requires only transcripts to be annotated for real data; however, it can still output detection and recognition results at both the character and line levels, avoiding the labor and cost of labeling bounding boxes of characters and text lines. Extensive experiments conducted on five datasets demonstrate the superiority of PageNet over existing weakly supervised and fully supervised page-level methods. These experimental results may spark further research beyond the realms of existing methods based on connectionist temporal classification or attention. The source code is available at https://github.com/shannanyinxiang/PageNet .

Addressing label ambiguity imbalance in candidate labels: Measures and disambiguation algorithm

Partial Label Learning (PLL) is a weakly supervised learning framework where each training instance is associated with more than one candidate label. However, label ambiguity in the case of label imbalance has not been well solved. In this paper, a novel label imbalance-aware algorithm named PL-LIMAP is proposed to effectively solve this problem. Firstly, a partial manifold constrained low-rank representation method is proposed to learn the instance affinity matrix, which can identify instances with the same label by fully mining the manifold structure of the data. Moreover, for making full use of the information hidden in candidate labels, the Jaccard distance is introduced to measure the similarity between candidate labels to obtain the label affinity matrix. The combination of these two affinity matrices can achieve the purpose of disambiguation through complementary information. Meanwhile, aiming to determine the imbalance of ambiguous labels, specialized measures are proposed to evaluate the imbalance level in PLL. According to the imbalance level of different labels, an alternative weighting scheme is proposed to mitigate the imbalance of ambiguous labels. As a consequence, the ground-truth label of an instance can be found out even though its label information is not available in advance. Experiments reveal the overwhelming performances of PL-LIMAP.

Partial Label Learning Based on Disambiguation Correction Net With Graph Representation

Partial Label Learning (PLL) is a weakly supervised learning framework where each training instance is associated with more than one candidate label. This learning method is dedicated to finding out the true label for each training instance. Most of the current PLL algorithms directly disambiguate the candidate labels without correcting the disambiguated results, making the algorithms vulnerable to the influence of instances easily misjudged. In this paper, GraphDCN, an innovative disambiguation correction net with inductive graph representation learning model is proposed. GraphDCN consists of a disambiguation model and a correction model. For a given instance, the disambiguation model tries to fit its underlying ground-truth label through the candidate label distribution of the instances connected with the given one, while the correction model tries to maximize the distance between the disambiguated labels and non-candidate labels, and uses the label probability thresholds to correct the disambiguated labels that may be wrong. As the training goes on, both the disambiguation and correction models alternately and iteratively boost their performance. Moreover, when considering the implementation of the disambiguation model, a partial cross entropy formulation is proposed to estimate the ground-truth label loss by updating the ambiguity confidence matrix, which can be proved satisfying convergence in PLL. Simulation results reveal the overwhelming performance of GraphDCN.

Pneumonia Classification in Lung X-ray Images Using CNN Technique

Abstract— A fundamental phase in the technique of pneumonia diagnosis is the evaluation and categorization of lung disorders utilizing X-raypictures, especially during a crucial era such as the COVID19 pandemic, which is a kind of pneumonia. As a result of the growing number of cases, an automated approach with high classification accuracy is required to classify lung disorders. Due to its quickness and effectiveness when it comes to visual recognition tasks, CNN based segmentation has acquired a lot of traction in recent years. We present an implementation of CNN-based classifiertechniques utilize a domain adaptation approach to identify pneumonia and analyze the outcomes to choose the best model for the job depending on specific parameters in this paper. There are various models because this is a rapidly growing topic, and we will concentrate on the bestperforming methods relying on their structure, tier length and style, and categorization task assessment criteria. To begin, we look at the existing traditional approaches and supervised learning frameworks for segmentation. Then, depending on the reliability and damage functionality of the constructed models, we undertake a detailed evaluation and analysis. A rigorous examination of the findings is carried out in order to highlight the major concerns that need to be addressed. Keywords— Covid19, CNN, pneumonia, X-rays.

Feature Importance Explanations for Temporal Black-Box Models

Models in the supervised learning framework may capture rich and complex representations over the features that are hard for humans to interpret. Existing methods to explain such models are often specific to architectures and data where the features do not have a time-varying component. In this work, we propose TIME, a method to explain models that are inherently temporal in nature. Our approach (i) uses a model-agnostic permutation-based approach to analyze global feature importance, (ii) identifies the importance of salient features with respect to their temporal ordering as well as localized windows of influence, and (iii) uses hypothesis testing to provide statistical rigor.

ScEpiLock: A Weakly Supervised Learning Framework for cis-Regulatory Element Localization and Variant Impact Quantification for Single-Cell Epigenetic Data.

Recent advances in single-cell transposase-accessible chromatin using a sequencing assay (scATAC-seq) allow cellular heterogeneity dissection and regulatory landscape reconstruction with an unprecedented resolution. However, compared to bulk-sequencing, its ultra-high missingness remarkably reduces usable reads in each cell type, resulting in broader, fuzzier peak boundary definitions and limiting our ability to pinpoint functional regions and interpret variant impacts precisely. We propose a weakly supervised learning method, scEpiLock, to directly identify core functional regions from coarse peak labels and quantify variant impacts in a cell-type-specific manner. First, scEpiLock uses a multi-label classifier to predict chromatin accessibility via a deep convolutional neural network. Then, its weakly supervised object detection module further refines the peak boundary definition using gradient-weighted class activation mapping (Grad-CAM). Finally, scEpiLock provides cell-type-specific variant impacts within a given peak region. We applied scEpiLock to various scATAC-seq datasets and found that it achieves an area under receiver operating characteristic curve (AUC) of ~0.9 and an area under precision recall (AUPR) above 0.7. Besides, scEpiLock’s object detection condenses coarse peaks to only ⅓ of their original size while still reporting higher conservation scores. In addition, we applied scEpiLock on brain scATAC-seq data and reported several genome-wide association studies (GWAS) variants disrupting regulatory elements around known risk genes for Alzheimer’s disease, demonstrating its potential to provide cell-type-specific biological insights in disease studies.

Unsupervised disparity estimation from light field using plug-and-play weighted warping loss

We investigated disparity estimation from a light field using a convolutional neural network (CNN). Most of the methods implemented a supervised learning framework, where the predicted disparity map was compared directly to the corresponding ground-truth disparity map in the training stage. However, light field data accompanied with ground-truth disparity maps were insufficient and rarely available for real-world scenes. The lack of training data resulted in limited generality of the methods trained with them. To tackle this problem, we took a simple Figure-and-play approach to remake a supervised method into an unsupervised (self-supervised) one. We replaced the loss function of the original method with one that does not depend on the ground-truth disparity maps. More specifically, our loss function is designed to indirectly evaluate the accuracy of the disparity map by using warping errors among the input light field views. We designed pixel-wise weights to properly evaluate the warping errors in the presence of occlusions, and an edge loss to encourage edge alignment between the image and the disparity map. As a result of this unsupervised learning framework, our method can use more abundant training datasets (even those without ground-truth disparity maps) than the original supervised method. Our method was evaluated on computer-generated scenes (4D Light Field Benchmark) and real-world scenes captured by Lytro Illum cameras. Our method achieved the state-of-the-art performance as an unsupervised method on the benchmark. We also demonstrated that our method can estimate disparity maps more accurately than the original supervised method for various real-world scenes.

Peer-to-Peer Variational Federated Learning Over Arbitrary Graphs

This paper proposes a federated supervised learning framework over a general peer-to-peer network with agents that act in a variational Bayesian fashion. The proposed framework consists of local agents where each of which keeps a local “posterior probability distribution” over the parameters of a global model; the updating of the posterior over time happens in a local fashion according to two subroutines of: 1) variational model training given (a batch of) local labeled data, and 2) asynchronous communication and model aggregation with the 1-hop neighbors. Inspired by the popular federated learning (model averaging), the framework allows the training data to remain distributed on mobile devices while utilizing a peer-to-peer model aggregation in a social network. The proposed framework is shown to allow for a systematic treatment of model aggregation over any arbitrary connected graph with consistent (in general, non-iid) local labeled data. Specifically, under mild technical conditions, the proposed algorithm allows agents with local data to learn a shared model explaining the global training data in a decentralized fashion over an arbitrary peering/connectivity graph. Furthermore, the rate of convergence is characterized and shown to be a function of each individual agent’s data quality weighted by its eigenvector centrality. Empirically, the proposed methodology is shown to work well with efficient variation Bayesian inference techniques to train Bayesian neural networks in a decentralized manner even when the local data batches are not identically distributed.

약한 지도학습 기반의 Deep SVDD를 통한 주편 절단 공정의 불꽃 비산 감지

In the continuous casting process, which is one of the steel manufacturing processes, a slab is cut by a torch cutter. During cutting, sparks are generated around the slab; if a problem occurs, the sparks are scattered. The failure in recognizing this problem early on can lead to huge losses and serious accidents. Using learning-based network model, this paper proposes a method of spark scattering detection that can be used in the steel manufacturing process. However, since the spark scattering occurs very intermittently, it becomes difficult to secure sufficient data for network learning. To resolve this issue, this study uses an anomaly detection method through Deep SVDD. This study also applies a weakly supervised learning framework using a small amount of spark scattering data so that the network can extract discriminative features for normal and abnormal situations.

Colorectal cancer subtype identification from differential gene expression levels using minimalist deep learning

BackgroundCancer molecular subtyping plays a critical role in individualized patient treatment. In previous studies, high-throughput gene expression signature-based methods have been proposed to identify cancer subtypes. Unfortunately, the existing ones suffer from the curse of dimensionality, data sparsity, and computational deficiency.MethodsTo address those problems, we propose a computational framework for colorectal cancer subtyping without any exploitation in model complexity and generality. A supervised learning framework based on deep learning (DeepCSD) is proposed to identify cancer subtypes. Specifically, based on the differentially expressed genes under cancer consensus molecular subtyping, we design a minimalist feed-forward neural network to capture the distinct molecular features in different cancer subtypes. To mitigate the overfitting phenomenon of deep learning as much as possible, L1 and L2 regularization and dropout layers are added.ResultsFor demonstrating the effectiveness of DeepCSD, we compared it with other methods including Random Forest (RF), Deep forest (gcForest), support vector machine (SVM), XGBoost, and DeepCC on eight independent colorectal cancer datasets. The results reflect that DeepCSD can achieve superior performance over other algorithms. In addition, gene ontology enrichment and pathology analysis are conducted to reveal novel insights into the cancer subtype identification and characterization mechanisms.ConclusionsDeepCSD considers all subtype-specific genes as input, which is pathologically necessary for its completeness. At the same time, DeepCSD shows remarkable robustness in handling cross-platform gene expression data, achieving similar performance on both training and test data without significant model overfitting or exploitation of model complexity.