Small Training Set Sizes Research Articles

MEF-AlloSite: an accurate and robust Multimodel Ensemble Feature selection for the Allosteric Site identification model

A crucial mechanism for controlling the actions of proteins is allostery. Allosteric modulators have the potential to provide many benefits compared to orthosteric ligands, such as increased selectivity and saturability of their effect. The identification of new allosteric sites presents prospects for the creation of innovative medications and enhances our comprehension of fundamental biological mechanisms. Allosteric sites are increasingly found in different protein families through various techniques, such as machine learning applications, which opens up possibilities for creating completely novel medications with a diverse variety of chemical structures. Machine learning methods, such as PASSer, exhibit limited efficacy in accurately finding allosteric binding sites when relying solely on 3D structural information.Scientific ContributionPrior to conducting feature selection for allosteric binding site identification, integration of supporting amino-acid–based information to 3D structural knowledge is advantageous. This approach can enhance performance by ensuring accuracy and robustness. Therefore, we have developed an accurate and robust model called Multimodel Ensemble Feature Selection for Allosteric Site Identification (MEF-AlloSite) after collecting 9460 relevant and diverse features from the literature to characterise pockets. The model employs an accurate and robust multimodal feature selection technique for the small training set size of only 90 proteins to improve predictive performance. This state-of-the-art technique increased the performance in allosteric binding site identification by selecting promising features from 9460 features. Also, the relationship between selected features and allosteric binding sites enlightened the understanding of complex allostery for proteins by analysing selected features. MEF-AlloSite and state-of-the-art allosteric site identification methods such as PASSer2.0 and PASSerRank have been tested on three test cases 51 times with a different split of the training set. The Student’s t test and Cohen’s D value have been used to evaluate the average precision and ROC AUC score distribution. On three test cases, most of the p-values (<0.05\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$< 0.05$$\\end{document}) and the majority of Cohen’s D values (>0.5\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$> 0.5$$\\end{document}) showed that MEF-AlloSite’s 1–6% higher mean of average precision and ROC AUC than state-of-the-art allosteric site identification methods are statistically significant.

Self-supervised contrastive learning improves machine learning discrimination of full thickness macular holes from epiretinal membranes in retinal OCT scans.

There is a growing interest in using computer-assisted models for the detection of macular conditions using optical coherence tomography (OCT) data. As the quantity of clinical scan data of specific conditions is limited, these models are typically developed by fine-tuning a generalized network to classify specific macular conditions of interest. Full thickness macular holes (FTMH) present a condition requiring urgent surgical repair to prevent vision loss. Other works on automated FTMH classification have tended to use supervised ImageNet pre-trained networks with good results but leave room for improvement. In this paper, we develop a model for FTMH classification using OCT B-scans around the central foveal region to pre-train a naïve network using contrastive self-supervised learning. We found that self-supervised pre-trained networks outperform ImageNet pre-trained networks despite a small training set size (284 eyes total, 51 FTMH+ eyes, 3 B-scans from each eye). On three replicate data splits, 3D spatial contrast pre-training yields a model with an average F1-score of 1.0 on holdout data (50 eyes total, 10 FTMH+), compared to an average F1-score of 0.831 for FTMH detection by ImageNet pre-trained models. These results demonstrate that even limited data may be applied toward self-supervised pre-training to substantially improve performance for FTMH classification, indicating applicability toward other OCT-based problems.

Application of a Machine Learning-Based Classification Approach for Developing Host Protein Diagnostic Models for Infectious Disease

In recent years, infectious disease diagnosis has increasingly turned to host-centered approaches as a complement to pathogen-directed ones. The former, however, typically requires the interpretation of complex multiple biomarker datasets to arrive at an informative diagnostic outcome. This report describes a machine learning (ML)-based classification workflow that is intended as a template for researchers seeking to apply ML approaches for developing host-based infectious disease biomarker classifiers. As an example, we built a classification model that could accurately distinguish between three disease etiology classes: bacterial, viral, and normal in human sera using host protein biomarkers of known diagnostic utility. After collecting protein data from known disease samples, we trained a series of increasingly complex Auto-ML models until arriving at an optimized classifier that could differentiate viral, bacterial, and non-disease samples. Even when limited to a relatively small training set size, the model had robust diagnostic characteristics and performed well when faced with a blinded sample set. We present here a flexible approach for applying an Auto-ML-based workflow for the identification of host biomarker classifiers with diagnostic utility for infectious disease, and which can readily be adapted for multiple biomarker classes and disease states.

Layer-by-layer reconstruction of fatigue damages in composites from thermal images by a Residual U-Net

In this paper a deep learning model is used to fully reconstruct the 3D distribution of arbitrarily shaped subsurface fatigue damages in a fiber/epoxy composite from synthetic thermal surface images. Synthetic thermal surface images (TIs) of self-heating damage hotspots are produced by thermal finite element analysis which are consequently used to train a Residual U-Net based on recent architectures designed for image segmentation. Different augmentation techniques are employed to mitigate the computational cost of generating training data through thermal finite element analysis. The Residual U-Net model accurately reconstructed – layer by layer – the ground truths and thereby enabled the quantitative assessment of location, size, shape, depth and gradient of an internal fatigue damage distribution. Moreover, the Residual U-Net achieved good predictions for comparatively small training set sizes.

A prediction model for blood-brain barrier penetrating peptides based on masked peptide transformers with dynamic routing.

Blood-brain barrier penetrating peptides (BBBPs) are short peptide sequences that possess the ability to traverse the selective blood-brain interface, making them valuable drug candidates or carriers for various payloads. However, the in vivo or in vitro validation of BBBPs is resource-intensive and time-consuming, driving the need for accurate in silico prediction methods. Unfortunately, the scarcity of experimentally validated BBBPs hinders the efficacy of current machine-learning approaches in generating reliable predictions. In this paper, we present DeepB3P3, a novel framework for BBBPs prediction. Our contribution encompasses four key aspects. Firstly, we propose a novel deep learning model consisting of a transformer encoder layer, a convolutional network backbone, and a capsule network classification head. This integrated architecture effectively learns representative features from peptide sequences. Secondly, we introduce masked peptides as a powerful data augmentation technique to compensate for small training set sizes in BBBP prediction. Thirdly, we develop a novel threshold-tuning method to handle imbalanced data by approximating the optimal decision threshold using the training set. Lastly, DeepB3P3 provides an accurate estimation of the uncertainty level associated with each prediction. Through extensive experiments, we demonstrate that DeepB3P3 achieves state-of-the-art accuracy of up to 98.31% on a benchmarking dataset, solidifying its potential as a promising computational tool for the prediction and discovery of BBBPs.

Evaluation of a deep learning-enabled automated computational heart modelling workflow for personalized assessment of ventricular arrhythmias.

Personalized, image-based computational heart modelling is a powerful technology that can be used to improve patient-specific arrhythmia risk stratification and ventricular tachycardia (VT) ablation targeting. However, most state-of-the-art methods still require manual interactions by expert users. The goal of this study is to evaluate the feasibility of an automated, deep learning-based workflow for reconstructing personalized computational electrophysiological heart models to guide patient-specific treatment of VT. Contrast-enhanced computed tomography (CE-CT) images with expert ventricular myocardium segmentations were acquired from 111 patients across five cohorts from three different institutions. A deep convolutional neural network (CNN) for segmenting left ventricular myocardium from CE-CT was developed, trained and evaluated. From both CNN-based and expert segmentations in a subset of patients, personalized electrophysiological heart models were reconstructed and rapid pacing was used to induce VTs. CNN-based and expert segmentations were more concordant in the middle myocardium than in the heart's base or apex. Wavefront propagation during pacing was similar between CNN-based and original heart models. Between most sets of heart models, VT inducibility was the same, the number of induced VTs was strongly correlated, and VT circuits co-localized. Our results demonstrate that personalized computational heart models reconstructed from deep learning-based segmentations even with a small training set size can predict similar VT inducibility and circuit locations as those from expertly-derived heart models. Hence, a user-independent, automated framework for simulating arrhythmias in personalized heart models could feasibly be used in clinical settings to aid VT risk stratification and guide VT ablation therapy. KEY POINTS: Personalized electrophysiological heart modelling can aid in patient-specific ventricular tachycardia (VT) risk stratification and VT ablation targeting. Current state-of-the-art, image-based heart models for VT prediction require expert-dependent, manual interactions that may not be accessible across clinical settings. In this study, we develop an automated, deep learning-based workflow for reconstructing personalized heart models capable of simulating arrhythmias and compare its predictions with that of expert-generated heart models. The number and location of VTs was similar between heart models generated from the deep learning-based workflow and expert-generated heart models. These results demonstrate the feasibility of using an automated computational heart modelling workflow to aid in VT therapeutics and has implications for generalizing personalized computational heart technology to a broad range of clinical centres.

A Bio-Inspired Hierarchical Spiking Neural Network With Reward-Modulated STDP Learning Rule for AER Object Recognition

The inherent spike-based and event-driven computation makes spiking neural networks (SNNs) naturally suitable to provide efficient and low-latency solution in neuromorphic vision processing. In this work we propose an address-event representation (AER) object recognition system which consists of a noise filter, an event stream segmentation module and a hierarchical convolutional SNN. In order to improve the feature learning capacity of bio-inspired shallow hierarchical SNNs for AER object recognition, we apply a convolutional SNN with R-STDP learning rule and first-spike coding to further extract features after primary feature learning using Gabor filter-based event-driven convolution. The R-STDP learning rule enables the network to adjust weights by combining local STDP learning rule with a global reward signal and to make classification without using an external classifier. Experimental results show that our method significantly outperforms the existing hierarchical SNN-based recognition methods on four popular AER datasets. Moreover, our method has much better recognition ability when using very short input event stream, and can effectively learn from small size training sets. In addition, the robustness of the network to input noise is improved. Our method is very beneficial for developing neuromorphic-vision object recognition algorithm in resource-constrained applications when event streams are short and training samples are limited.

A Machine Learning Algorithm to Discriminating Between Bipolar and Major Depressive Disorders Based on Resting EEG Data.

Distinguishing major depressive disorder (MDD) from bipolar disorder (BD) is a crucial clinical challenge due to the lack of known biomarkers. Conventional methods of diagnosis rest exclusively on symptomatic presentation, and personal and family history. As a result, BD-depressed episode (BD-DE) is often misdiagnosed as MDD, and inappropriate therapy is given. Electroencephalography (EEG) has been widely studied as a potential source of biomarkers to differentiate these disorders. Previous attempts using machine learning (ML) methods have delivered insufficient sensitivity and specificity for clinical use, likely as a consequence of the small training set size, and inadequate ML methodology. We hope to overcome these limitations by employing a training dataset of resting-state EEG from 71 MDD and 71 BD patients. We introduce a robust 3 steps ML technique: 1) a multi-step preprocessing method is used to improve the quality of the EEG signal 2) symbolic transfer entropy (STE), which is an effective connectivity measure, is applied to the resultant EEG signals 3) the ML algorithm uses the extracted STE features to distinguish MDD from BD patients. Clinical Relevance--- The accuracy of our algorithm, derived from a large sample of patients, suggests that this method may hold significant promise as a clinical tool. The proposed method delivered total accuracy, sensitivity, and specificity of 84.9%, 83.4%, and 87.1%, respectively.

Deep Ensemble CNN Method Based on Sample Expansion for Hyperspectral Image Classification

With the continuous progress of computer deep learning technology, convolutional neural network (CNN), as a representative approach, provides a unique solution for hyperspectral image (HSI) classification. However, the parameters of CNN can not be well-tuned when the number of training samples is insufficient, resulting in unsatisfactory classification performance. To tackle the thorny problem, a deep ensemble CNN method based on sample expansion for HSI classification is studied in this paper. Specially, spatial information is first extracted and fused with original spectral bands to help classifiers obtain discriminant spectral-spatial features. Then we use the pixel-pair feature (PPF) to expand the number of training samples so that the parameters of CNN structure can be fully trained. In addition, deep ensemble CNN is employed in this paper, enabling the trained model to obtain better generalization ability and more robust classification results. Ultimately, the proposed method is applied to classify four widely used hyperspectral data sets. Experimental results show that the studied approach yields higher classification accuracy than some CNN-based methods even under the condition of small-size training set.

The algorithm for generating the training set for the problem of elastoplastic deformation of the metal rod

The paper proposes an algorithm for forming a small training set, which will provide a reasonable quality of a surrogate ML-model for the problem of elastoplastic deformation of a metal rod under the action of a longitudinal load pulse. This dynamic physical problem is computationally simple and convenient for testing various approaches, but at the same time it is physically quite complex, because it contains a significant range of effects. So, the methods tested on this problem can be further applied to other areas. This work demonstrates the possibility of a surrogate ML-model to provide a reasonable prediction quality for a dynamic physical problem with a small training set size.

Machine learning RBF-based surrogate models for uncertainty quantification of age and time-dependent fracture mechanics

This paper proposes a machine learning strategy using radial basis function (RBF) as surrogate models for uncertainty quantification of age and time-dependent fracture mechanics problems. The RBF surrogate models are trained to replace the time-consuming evaluations of the mapping integral of the time-dependent energy release rate. The probabilistic problem considers input random variables of geometry, loading, and material parameters for a concrete plate section with the presence of an initial surface crack. The performance of the RBF surrogates was evaluated through cross-validation and Monte Carlo reference solutions that expensively evaluated the integrals. The results of mean square errors and the Kolmogorov-Smirnov goodness-of-fit tests on the predicted probability and cumulative density functions show that, the RBF surrogates reached good accuracy even for a small training set size, rather than second-order polynomial models which could reasonably perform only for bigger training set sizes. The RBF surrogates accurately predict even long-tail distributions for farther prediction times with relatively small training sets. The proposed RBF surrogate models presented the generalization capabilities to predict thousands of unseen data points from Monte Carlo simulations and successfully assess the uncertainty quantification of the time-dependent energy release rate.

The Implementation of DCGAN in the Data Augmentation for the Sperm Morphology Datasets

A large amount of data is the key requirement in order to train a neural network efficiently. Using a small size training set in network training causes low accuracy for model performance over the testing set and also hard to implement the model in practice. Similar to many other problems, sperm morphology datasets are also limited for training the neural network-based deep networks in order to provide an automatic evaluation of sperm morphometry. Data augmentation mitigates this problem by utilizing actual data more effectively. The standard data augmentation techniques focus on only spatial changes over the images and can only produce a restricted number of useful informative and disjunctive data. Therefore, in order to create more distinctive and diverse data than the regular spatial domain-based augmentation techniques, a deep learning-based data augmentation technique which is known as the generative model, is trained in this study for the sperm morphology datasets. The deep convolutional generative adversarial network (DCGAN) was optimized and utilized in this study for three well-known sperm morphometry datasets as SMIDS, HuSHeM, and SCIAN-Morpho. Each dataset was individually augmented to a 1000 sample size by the proposed approach. In order to optimize the network with different parameters and observe the generated data, a graphical user interface has been designed. For the similarity evaluation of the generated images to original images, the Fréchet Inception Distance (FID) score was utilized. The FID results indicate that the most similar generated images have been obtained for SMIDS with an average of 29.06 FID score. The worst performance (Average FID = 53.46) was obtained for the SCIAN-Morpho dataset, which has low resolution and data imbalance problems. Lastly, DCGAN based proposed approach resulted in an average of 44.25 FID score for the HuSHeM dataset.

Modeling extra-deep electromagnetic logs using a deep neural network

Modern geosteering is heavily dependent on real-time interpretation of deep electromagnetic (EM) measurements. We have developed a methodology to construct a deep neural network (DNN) model trained to reproduce a full set of extra-deep EM logs consisting of 22 measurements per logging position. The model is trained in a 1D layered environment consisting of up to seven layers with different resistivity values. A commercial simulator provided by a tool vendor is used to generate a training data set. The data set size is limited because the simulator provided by the vendor is optimized for sequential execution. Therefore, we design a training data set that embraces the geologic rules and geosteering specifics supported by the forward model. We use this data set to produce an EM simulator based on a DNN without access to the proprietary information about the EM tool configuration or the original simulator source code. Despite using a relatively small training set size, the resulting DNN forward model is quite accurate for the considered examples: a multilayer synthetic case and a section of a published historical operation from the Goliat field. The observed average evaluation time of 0.15 ms per logging position makes it also suitable for future use as part of evaluation-hungry statistical and/or Monte Carlo inversion algorithms within geosteering workflows.

Quantum adiabatic machine learning by zooming into a region of the energy surface

Recent work has shown that quantum annealing for machine learning, referred to as QAML, can perform comparably to state-of-the-art machine learning methods with a specific application to Higgs boson classification. We propose QAML-Z, a novel algorithm that iteratively zooms in on a region of the energy surface by mapping the problem to a continuous space and sequentially applying quantum annealing to an augmented set of weak classifiers. Results on a programmable quantum annealer show that QAML-Z matches classical deep neural network performance at small training set sizes and reduces the performance margin between QAML and classical deep neural networks by almost 50% at large training set sizes, as measured by area under the ROC curve. The significant improvement of quantum annealing algorithms for machine learning and the use of a discrete quantum algorithm on a continuous optimization problem both opens a new class of problems that can be solved by quantum annealers and suggests the approach in performance of near-term quantum machine learning towards classical benchmarks.

One Step Is Not Enough: A Multi-Step Procedure for Building the Training Set of a Query by String Keyword Spotting System to Assist the Transcription of Historical Document

Digital libraries offer access to a large number of handwritten historical documents. These documents are available as raw images and therefore their content is not searchable. A fully manual transcription is time-consuming and expensive while a fully automatic transcription is cheaper but not comparable in terms of accuracy. The performance of automatic transcription systems is strictly related to the composition of the training set. We propose a multi-step procedure that exploits a Keyword Spotting system and human validation for building up a training set in a time shorter than the one required by a fully manual procedure. The multi-step procedure was tested on a data set made up of 50 pages extracted from the Bentham collection. The palaeographer that transcribed the data set with the multi-step procedure instead of the fully manual procedure had a time gain of 52.54%. Moreover, a small size training set that allowed the keyword spotting system to show a precision value greater than the recall value was built with the multi-step procedure in a time equal to 35.25% of the time required for annotating the whole data set.

Comparative study between deep learning and QSAR classifications for TNBC inhibitors and novel GPCR agonist discovery

Machine learning is a well-known approach for virtual screening. Recently, deep learning, a machine learning algorithm in artificial neural networks, has been applied to the advancement of precision medicine and drug discovery. In this study, we performed comparative studies between deep neural networks (DNN) and other ligand-based virtual screening (LBVS) methods to demonstrate that DNN and random forest (RF) were superior in hit prediction efficiency. By using DNN, several triple-negative breast cancer (TNBC) inhibitors were identified as potent hits from a screening of an in-house database of 165,000 compounds. In broadening the application of this method, we harnessed the predictive properties of trained model in the discovery of G protein-coupled receptor (GPCR) agonist, by which computational structure-based design of molecules could be greatly hindered by lack of structural information. Notably, a potent (~ 500 nM) mu-opioid receptor (MOR) agonist was identified as a hit from a small-size training set of 63 compounds. Our results show that DNN could be an efficient module in hit prediction and provide experimental evidence that machine learning could identify potent hits in silico from a limited training set.

Spectral Mixture Model Inspired Network Architectures for Hyperspectral Unmixing

In many statistical hyperspectral unmixing approaches, the unmixing task is essentially an optimization problem given a defined linear or nonlinear spectral mixture model. However, most of the model inference algorithms require a time-consuming iterative procedure. On the other hand, neural networks have been recently used to estimate abundances given some training samples, or directly estimate endmembers and abundances simultaneously in an unsupervised setting. However, their disadvantages are clear: lack of interpretability and reliance on the large training set. Model-inspired neural networks are constructed by the problem model and its corresponding inference algorithm. It incorporates the prior knowledge of physical model and algorithm into network architecture, combining the advantages of model-based and learning-based methods. This article deeply unfolds the linear mixture model and the corresponding iterative shrinkage-thresholding algorithm (ISTA) to build two unmixing network architectures. The first assumes that the set of endmembers are known, and the deep unfolded ISTA model is only for abundance estimation; and the second is used for blind unmixing to estimate both endmembers and abundances at the same time. The networks can be trained by supervised and unsupervised schemes, respectively, with a small-size training set, and then, unmixing becomes a feedforward process, which is very fast since no iteration is required. The experimental results show their competitive performance compared with the state-of-the-art unmixing approaches.

A distributionally robust area under curve maximization model

Area under ROC curve (AUC) is a performance measure for classification models. We propose new distributionally robust AUC models (DR-AUC) that rely on the Kantorovich metric and approximate AUC with the hinge loss function, and derive convex reformulations using duality. The DR-AUC models outperform deterministic AUC and support vector machine models and have superior worst-case out-of-sample performance, thereby showing their robustness. The results are encouraging since the numerical experiments are conducted with small-size training sets conducive to low out-of-sample performance.

Towards automated generation of curated datasets in radiology: Application of natural language processing to unstructured reports exemplified on CT for pulmonary embolism

PurposeTo design and evaluate a self-trainable natural language processing (NLP)-based procedure to classify unstructured radiology reports. The method enabling the generation of curated datasets is exemplified on CT pulmonary angiogram (CTPA) reports. MethodWe extracted the impressions of CTPA reports created at our institution from 2016 to 2018 (n = 4397; language: German). The status (pulmonary embolism: yes/no) was manually labelled for all exams. Data from 2016/2017 (n = 2801) served as a ground truth to train three NLP architectures that only require a subset of reference datasets for training to be operative. The three architectures were as follows: a convolutional neural network (CNN), a support vector machine (SVM) and a random forest (RF) classifier. Impressions of 2018 (n = 1377) were kept aside and used for general performance measurements. Furthermore, we investigated the dependence of classification performance on the amount of training data with multiple simulations. ResultsThe classification performance of all three models was excellent (accuracies: 97 %–99 %; F1 scores 0.88–0.97; AUCs: 0.993–0.997). Highest accuracy was reached by the CNN with 99.1 % (95 % CI 98.5–99.6 %). Training with 470 labelled impressions was sufficient to reach an accuracy of > 93 % with all three NLP architectures. ConclusionOur NLP-based approaches allow for an automated and highly accurate retrospective classification of CTPA reports with manageable effort solely using unstructured impression sections. We demonstrated that this approach is useful for the classification of radiology reports not written in English. Moreover, excellent classification performance is achieved at relatively small training set sizes.

Multisource Region Attention Network for Fine-Grained Object Recognition in Remote Sensing Imagery

Fine-grained object recognition concerns the identification of the type of an object among a large number of closely related sub-categories. Multisource data analysis, that aims to leverage the complementary spectral, spatial, and structural information embedded in different sources, is a promising direction towards solving the fine-grained recognition problem that involves low between-class variance, small training set sizes for rare classes, and class imbalance. However, the common assumption of co-registered sources may not hold at the pixel level for small objects of interest. We present a novel methodology that aims to simultaneously learn the alignment of multisource data and the classification model in a unified framework. The proposed method involves a multisource region attention network that computes per-source feature representations, assigns attention scores to candidate regions sampled around the expected object locations by using these representations, and classifies the objects by using an attention-driven multisource representation that combines the feature representations and the attention scores from all sources. All components of the model are realized using deep neural networks and are learned in an end-to-end fashion. Experiments using RGB, multispectral, and LiDAR elevation data for classification of street trees showed that our approach achieved 64.2% and 47.3% accuracies for the 18-class and 40-class settings, respectively, which correspond to 13% and 14.3% improvement relative to the commonly used feature concatenation approach from multiple sources.