Full Training Dataset Research Articles

Graph-based Text Classification by Contrastive Learning with Text-level Graph Augmentation

Text Classification (TC) is a fundamental task in the information retrieval community. Nowadays, the mainstay TC methods are built on the deep neural networks, which can learn much more discriminative text features than the traditional shallow learning methods. Among existing deep TC methods, the ones based on Graph Neural Network (GNN) have attracted more attention due to the superior performance. Technically, the GNN-based TC methods mainly transform the full training dataset to a graph of texts; however, they often neglect the dependency between words, so as to miss potential semantic information of texts, which may be significant to exactly represent them. To solve the aforementioned problem, we generate graphs of words instead, so as to capture the dependency information of words. Specifically, each text is translated into a graph of words, where neighboring words are linked. We learn the node features of words by a GNN-like procedure and then aggregate them as the graph feature to represent the current text. To further improve the text representations, we suggest a contrastive learning regularization term. Specifically, we generate two augmented text graphs for each original text graph, we constrain the representations of the two augmented graphs from the same text close and the ones from different texts far away. We propose various techniques to generate the augmented graphs. Upon those ideas, we develop a novel deep TC model, namely Text-level Graph Networks with Contrastive Learning (TGN cl ). We conduct a number of experiments to evaluate the proposed TGN cl model. The empirical results demonstrate that TGN cl can outperform the existing state-of-the-art TC models.

Hierarchical and Bidirectional Joint Multi-Task Classifiers for Natural Language Understanding

The MASSIVE dataset is a spoken-language comprehension resource package for slot filling, intent classification, and virtual assistant evaluation tasks. It contains multi-language utterances from human beings communicating with a virtual assistant. In this paper, we exploited the relationship between intent classification and slot filling to improve the exact match accuracy by proposing five models with hierarchical and bidirectional architectures. There are two variants for hierarchical architectures and three variants for bidirectional architectures. These are the hierarchical concatenation model, the hierarchical attention-based model, the bidirectional max-pooling model, the bidirectional LSTM model, and the bidirectional attention-based model. The results of our models showed a significant improvement in the averaged exact match accuracy. The hierarchical attention-based model improved the accuracy by 1.01 points for the full training dataset. As for the zero-shot setup, we observed that the exact match accuracy increased from 53.43 to 53.91. In this study, we observed that, for multi-task problems, utilizing the relevance between different tasks can help in improving the model’s overall performance.

Non-intrusive personal thermal comfort modeling: A machine learning approach using infrared face recognition

At present, non-intrusive personal thermal comfort models are receiving more and more attention. Non-intrusive sensing technology is used to accurately capture the real-time thermal state of occupants indoors so as to construct personal thermal comfort models by data-driven methods. This study developed a non-intrusive personal thermal comfort model using machine learning techniques combined with infrared facial recognition. Firstly, the Charlotte-ThermalFace database was used to extract the temperatures from six regions of interest on the face using infrared face recognition and key point extraction algorithms. Subsequently, the feature importance of the variables was calculated by random forest (RF) and gradient boosting decision tree (GBDT) respectively to explore the key parameters influencing the prediction performance of personal thermal preferences. Finally, the performance of 12 machine learning models was systematically compared, including 6 traditional models, 5 ensemble models, and 1 broad model, based on precision, recall, F1 score and macro-F1 score. The results show that the ensemble learning models and the broad learning (BL) model perform better than the traditional models by using the full training dataset size. Secondly, the BL model is applied for the first time as an alternative to deep network models for thermal preference prediction, with a prediction precision of 90.44%. Compared with traditional deep neural networks (DNN) model, it has lower computational complexity and faster training speed. Furthermore, BL and deep cascade forest (DCF) have significant advantages over other models in predicting thermal preference with different data subsets. Overall, the results of this study provide a reference for non-intrusive personal thermal comfort modeling that can be used to optimize building thermal environments.

Dictionary-enabled efficient training of ConvNets for image classification

Convolutional networks (ConvNets) are computationally expensive but well known for their performance on image data. One way to reduce their complexity is to explore inherited data sparsity. However, since the gradients involved in ConvNets require dynamic updates, applying data sparsity in the training step is not straightforward. Dictionary-based learning methods can be useful since they encode the original data in a sparse form. This paper proposes a new dictionary-based training paradigm for ConvNets by exploiting redundancy in the training data while keeping the distinctive features intact. The ConvNet is then trained on the reduced, sparse dataset. The new approach significantly reduces the training time without compromising accuracy.To the best of our knowledge, this is the first implementation of ConvNet on dictionary-based sparse training data. The proposed method is validated on three publicly available datasets –MNIST, USPS, and MNIST FASHION. The experimental results show a significant reduction of 4.5 times in the overall computational burden of vanilla ConvNet for all the datasets. Whereas the accuracy is intact at 97.21% for MNIST, 96.81% for USPS, and 88.4% for FASHION datasets. These results are comparable to state-of-the-art algorithms, such as ResNet-{18,34,50}, trained on the full training dataset.

Optimization-Based Post-Training Quantization With Bit-Split and Stitching.

Deep neural networks have shown great promise in various domains. Meanwhile, problems including the storage and computing overheads arise along with these breakthroughs. To solve these problems, network quantization has received increasing attention due to its high efficiency and hardware-friendly property. Nonetheless, most existing quantization approaches rely on the full training dataset and the time-consuming fine-tuning process to retain accuracy. Post-training quantization does not have these problems, however, it has mainly been shown effective for 8-bit quantization. In this paper, we theoretically analyze the effect of network quantization and show that the quantization loss in the final output layer is bounded by the layer-wise activation reconstruction error. Based on this analysis, we propose an Optimization-based Post-training Quantization framework and a novel Bit-split optimization approach to achieve minimal accuracy degradation. The proposed framework is validated on a variety of computer vision tasks, including image classification, object detection, instance segmentation, with various network architectures. Specifically, we achieve near-original model performance even when quantizing FP32 models to 3-bit without fine-tuning.

Question classification using limited labelled data

Question classification (QC) involves classifying given question based on the expected answer type and is an important task in the Question Answering(QA) system. Existing approaches for question classification use full training dataset to fine-tune the models. It is expensive and requires more time to develop labelled datasets in huge size. Hence, there is a need to develop approaches that can achieve comparable or state of the art performance using limited training instances. In this paper, we propose an approach that uses data augmentation as a tool to generate additional training instances. We evaluate our proposed approach on two question classification datasets namely TREC and ICHI datasets. Experimental results show that our proposed approach reduces the requirement of labelled instances (a) up to 81.7% and achieves new state of the art accuracy of 98.11 on TREC dataset and (b) up to 75% and achieves 67.9 on ICHI dataset.

External validation: a simulation study to compare cross-validation versus holdout or external testing to assess the performance of clinical prediction models using PET data from DLBCL patients

AimClinical prediction models need to be validated. In this study, we used simulation data to compare various internal and external validation approaches to validate models.MethodsData of 500 patients were simulated using distributions of metabolic tumor volume, standardized uptake value, the maximal distance between the largest lesion and another lesion, WHO performance status and age of 296 diffuse large B cell lymphoma patients. These data were used to predict progression after 2 years based on an existing logistic regression model. Using the simulated data, we applied cross-validation, bootstrapping and holdout (n = 100). We simulated new external datasets (n = 100, n = 200, n = 500) and simulated stage-specific external datasets (1), varied the cut-off for high-risk patients (2) and the false positive and false negative rates (3) and simulated a dataset with EARL2 characteristics (4). All internal and external simulations were repeated 100 times. Model performance was expressed as the cross-validated area under the curve (CV-AUC ± SD) and calibration slope.ResultsThe cross-validation (0.71 ± 0.06) and holdout (0.70 ± 0.07) resulted in comparable model performances, but the model had a higher uncertainty using a holdout set. Bootstrapping resulted in a CV-AUC of 0.67 ± 0.02. The calibration slope was comparable for these internal validation approaches. Increasing the size of the test set resulted in more precise CV-AUC estimates and smaller SD for the calibration slope. For test datasets with different stages, the CV-AUC increased as Ann Arbor stages increased. As expected, changing the cut-off for high risk and false positive- and negative rates influenced the model performance, which is clearly shown by the low calibration slope. The EARL2 dataset resulted in similar model performance and precision, but calibration slope indicated overfitting.ConclusionIn case of small datasets, it is not advisable to use a holdout or a very small external dataset with similar characteristics. A single small testing dataset suffers from a large uncertainty. Therefore, repeated CV using the full training dataset is preferred instead. Our simulations also demonstrated that it is important to consider the impact of differences in patient population between training and test data, which may ask for adjustment or stratification of relevant variables.

Cross-Domain Activity Recognition Using Shared Representation in Sensor Data

Existing models based on sensor data for human activity recognition are reporting state-of-the-art performances. Most of these models are conducted based on single-domain learning in which for each domain a model is required to be trained. However, the generation of adequate labelled data and a learning model for each domain separately is often time-consuming and computationally expensive. Moreover, the deployment of multiple domain-wise models is not scalable as it obscures domain distinctions, introduces extra computational costs, and limits the usefulness of training data. To mitigate this, we propose a multi-domain learning network to transfer knowledge across different but related domains and alleviate isolated learning paradigms using a shared representation. The proposed network consists of two identical causal convolutional sub-networks that are projected to a shared representation followed by a linear attention mechanism. The proposed network can be trained using the full training dataset of the source domain and a dataset of restricted size of the target training domain to reduce the need of large labelled training datasets. The network processes the source and target domains jointly to learn powerful and mutually complementary features to boost the performance in both domains. The proposed multi-domain learning network on six real-world sensor activity datasets outperforms the existing methods by applying only 50% of the labelled data. This confirms the efficacy of the proposed approach as a generic model to learn human activities from different but related domains in a joint effort, to reduce the number of required models and thus improve system efficiency.

Federated Learning for Face Recognition with Gradient Correction

With increasing appealing to privacy issues in face recognition, federated learning has emerged as one of the most prevalent approaches to study the unconstrained face recognition problem with private decentralized data. However, conventional decentralized federated algorithm sharing whole parameters of networks among clients suffers from privacy leakage in face recognition scene. In this work, we introduce a framework, FedGC, to tackle federated learning for face recognition and guarantees higher privacy. We explore a novel idea of correcting gradients from the perspective of backward propagation and propose a softmax-based regularizer to correct gradients of class embeddings by precisely injecting a cross-client gradient term. Theoretically, we show that FedGC constitutes a valid loss function similar to standard softmax. Extensive experiments have been conducted to validate the superiority of FedGC which can match the performance of conventional centralized methods utilizing full training dataset on several popular benchmark datasets.

Predicting Nanoparticle Delivery to Tumors Using Machine Learning and Artificial Intelligence Approaches.

BackgroundLow delivery efficiency of nanoparticles (NPs) to the tumor is a critical barrier in the field of cancer nanomedicine. Strategies on how to improve NP tumor delivery efficiency remain to be determined.MethodsThis study analyzed the roles of NP physicochemical properties, tumor models, and cancer types in NP tumor delivery efficiency using multiple machine learning and artificial intelligence methods, using data from a recently published Nano-Tumor Database that contains 376 datasets generated from a physiologically based pharmacokinetic (PBPK) model.ResultsThe deep neural network model adequately predicted the delivery efficiency of different NPs to different tumors and it outperformed all other machine learning methods; including random forest, support vector machine, linear regression, and bagged model methods. The adjusted determination coefficients (R2) in the full training dataset were 0.92, 0.77, 0.77 and 0.76 for the maximum delivery efficiency (DEmax), delivery efficiency at 24 h (DE24), at 168 h (DE168), and at the last sampling time (DETlast). The corresponding R2 values in the test dataset were 0.70, 0.46, 0.33 and 0.63, respectively. Also, this study showed that cancer type was an important determinant for the deep neural network model in predicting the tumor delivery efficiency across all endpoints (19–29%). Among all physicochemical properties, the Zeta potential and core material played a greater role than other properties, such as the type, shape, and targeting strategy.ConclusionThis study provides a quantitative model to improve the design of cancer nanomedicine with greater tumor delivery efficiency. These results help to improve our understanding of the causes of low NP tumor delivery efficiency. This study demonstrates the feasibility of integrating artificial intelligence with PBPK modeling approaches to study cancer nanomedicine.

Combining Self-supervised Learning and Active Learning for Disfluency Detection

Spoken language is fundamentally different from the written language in that it contains frequent disfluencies or parts of an utterance that are corrected by the speaker. Disfluency detection (removing these disfluencies) is desirable to clean the input for use in downstream NLP tasks. Most existing approaches to disfluency detection heavily rely on human-annotated data, which is scarce and expensive to obtain in practice. To tackle the training data bottleneck, in this work, we investigate methods for combining self-supervised learning and active learning for disfluency detection. First, we construct large-scale pseudo training data by randomly adding or deleting words from unlabeled data and propose two self-supervised pre-training tasks: (i) a tagging task to detect the added noisy words and (ii) sentence classification to distinguish original sentences from grammatically incorrect sentences. We then combine these two tasks to jointly pre-train a neural network. The pre-trained neural network is then fine-tuned using human-annotated disfluency detection training data. The self-supervised learning method can capture task-special knowledge for disfluency detection and achieve better performance when fine-tuning on a small annotated dataset compared to other supervised methods. However, limited in that the pseudo training data are generated based on simple heuristics and cannot fully cover all the disfluency patterns, there is still a performance gap compared to the supervised models trained on the full training dataset. We further explore how to bridge the performance gap by integrating active learning during the fine-tuning process. Active learning strives to reduce annotation costs by choosing the most critical examples to label and can address the weakness of self-supervised learning with a small annotated dataset. We show that by combining self-supervised learning with active learning, our model is able to match state-of-the-art performance with just about 10% of the original training data on both the commonly used English Switchboard test set and a set of in-house annotated Chinese data.

Predicting healthcare-seeking behavior based on stated readiness to act: development and validation of a prediction model.

A starting point of many digital health interventions informed by the Stages of Change Model of behavior change is assessing a person's readiness to change. In this paper, we use the concept of readiness to develop and validate a prediction model of health-seeking behavior in the context of family planning. We conducted a secondary analysis of routinely collected, anonymized health data submitted by 4,088 female users of a free health chatbot in Kenya. We developed a prediction model of (future) self-reported action by randomly splitting the data into training and test data sets (80/20, stratified by the outcome). We further split the training data into 10 folds for cross-validating the hyperparameter tuning step in model selection. We fit nine different classification models and selected the model that maximized the area under the receiver operator curve. We then fit the selected model to the full training dataset and evaluated the performance of this model on the holdout test data. The model predicted who will visit a family planning provider in the future with high precision (0.93) and moderate recall (0.75). Using the Stages of Change framework, we concluded that 29% of women were in the "Preparation" stage, 21% were in the "Contemplation" stage, and 50% were in the "Pre-Contemplation" stage. We demonstrated that it is possible to accurately predict future healthcare-seeking behavior based on information learned during the initial encounter. Models like this may help intervention developers to tailor strategies and content in real-time.

Feature space approximation for kernel-based supervised learning

We propose a method for the approximation of high- or even infinite-dimensional feature vectors, which play an important role in supervised learning. The goal is to reduce the size of the training data, resulting in lower storage consumption and computational complexity. Furthermore, the method can be regarded as a regularization technique, which improves the generalizability of learned target functions. We demonstrate significant improvements in comparison to the computation of data-driven predictions involving the full training data set. The method is applied to classification and regression problems from different application areas such as image recognition, system identification, and oceanographic time series analysis.

An Inverse-Free and Scalable Sparse Bayesian Extreme Learning Machine for Classification Problems

Sparse Bayesian Extreme Learning Machine (SBELM) constructs an extremely sparse and probabilistic model with low computational cost and high generalization. However, the update rule of hyperparameters (ARD prior) in SBELM involves using the diagonal elements from the inversion of the covariance matrix with the full training dataset, which raises the following two issues. Firstly, inverting the Hessian matrix may suffer ill-conditioning issues in some cases, which hinders SBELM from converging. Secondly, it may result in the memory-overflow issue with computational memory O(L <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> ) ( L: number of hidden nodes) to invert the big covariance matrix for updating the ARD priors. To address these issues, an inverse-free SBELM called QN-SBELM is proposed in this paper, which integrates the gradient-based Quasi-Newton (QN) method into SBELM to approximate the inverse covariance matrix. It takes O(L <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) computational complexity and is simultaneously scalable to large problems. QN-SBELM was evaluated on benchmark datasets of different sizes. Experimental results verify that QN-SBELM achieves more accurate results than SBELM with a sparser model, and also provides more stable solutions and a great extension to large-scale problems.

Feature Selection Methods Simultaneously Improve the Detection Accuracy and Model Building Time of Machine Learning Classifiers

The detection accuracy and model building time of machine learning (ML) classifiers are vital aspects for an intrusion detection system (IDS) to predict attacks in real life. Recently, researchers have introduced feature selection methods to increase the detection accuracy and minimize the model building time of a limited number of ML classifiers. Therefore, identifying more ML classifiers with very high detection accuracy and the lowest possible model building time is necessary. In this study, the authors tested six supervised classifiers on a full NSL-KDD training dataset (a benchmark record for Internet traffic) using 10-fold cross-validation in the Weka tool with and without feature selection/reduction methods. The authors aimed to identify more options to outperform and secure classifiers with the highest detection accuracy and lowest model building time. The results show that the feature selection/reduction methods, including the wrapper method in combination with the discretize filter, the filter method in combination with the discretize filter, and the discretize filter, can significantly decrease model building time without compromising detection accuracy. The suggested ML algorithms and feature selection/reduction methods are automated pattern recognition approaches to detect network attacks, which are within the scope of the Symmetry journal.

Decision trees using local support vector regression models for large datasets

ABSTRACTOur proposed decision trees using local support vector regression models (tSVR, rtSVR) aim to efficiently handle the regression task for large datasets. The learning algorithm tSVR of regression models is done by two main steps. The first one is to construct a decision tree regressor for partitioning the full training dataset into k terminal-nodes (subsets), followed which the second one is to learn the SVR model from each terminal-node to predict the data locally in a parallel way on multi-core computers. The algorithm rtSVR learns the random forest of decision trees with local SVR models for improving the prediction correctness against the tSVR model alone. The performance analysis shows that our algorithms tSVR, rtSVR are efficient in terms of the algorithmic complexity and the generalization ability compared to the classical SVR. The experimental results on five large datasets from UCI repository showed that proposed tSVR and rtSVR algorithms are faster than the standard SVR in training the non-linear regression model from large datasets while achieving the high correctness in the prediction. Typically, the average training time of tSVR and rtSVR are 1282.66 and 482.29 times faster than the standard SVR; Furthermore, tSVR and rtSVR improve 59.43%, 63.70% of the relative prediction correctness compared to the standard SVR.

CapsCarcino: A novel sparse data deep learning tool for predicting carcinogens

Determining chemical carcinogenicity in the early stages of drug discovery is fundamentally important to prevent the adverse effect of carcinogens on human health. There has been a recent surge of interest in developing computational approaches to predict chemical carcinogenicity. However, the predictive power of many existing approaches is limited, and there is plenty of room for improvement. Here, we develop a new deep learning architecture, termed CapsCarcino, to distinguish between carcinogens and noncarcinogens. CapsCarcino is constructed based on a dynamic routing algorithm that requires less data, extracts more comprehensive information, and does not require feature selection. We find that CapsCarcino provides a significantly improved predictive and generalization ability over, and outperforms five other machine learning models. Specifically, the best model of CapsCarcino achieves an accuracy of 85.0% on an external validation dataset. In addition, we discover that the enhanced predictive capability of CapsCarcino over that of the other methods is robust and can be achieved using sparse datasets. Training on merely 20% of the dataset, CapsCarcino performs comparably to the other methods based on the full training dataset. Further mechanism analysis indicates that CapsCarcino could efficiently learn the characteristics of carcinogens even if structural alerts are insufficiently represented. The results indicate that CapsCarcino should be helpful for carcinogen risk assessment.

SeSe-Net: Self-Supervised deep learning for segmentation

Deep convolutional networks have become ubiquitous for image segmentation. But relatively little is known on how to leverage unlabeled data to further improve their performance. Here we introduce a new approach named as Self-supervised deep learning for Segmentation (SeSe-Net) that uses a ‘Worker’ (W) neural network to segment input images and a ‘Supervisor’ (S) neural network to evaluate the quality of the segmentation results. We further propose a two-stage training process. In the first stage, W learns to segment with standard dataset and generates training sets for S. Based on this, S learns how well the W performs for the segmentation task. In the second stage, S transfers the knowledge from the first stage to supervise the learning process of W on unlabeled datasets towards better segmentation performance. We show that SeSe-Net leads to a significant performance boosting with additional unlabeled data. Furthermore, with less than 5% of labeled training dataset, SeSe-Net performs on par with challenging baseline trained on full training dataset in terms of Dice Metric (DM) on three different datasets. Theoretical analysis is also given to establish the convergence of SeSe-Net.

MiRTaW: An Algorithm for Atmospheric Temperature and Water Vapor Profile Estimation from ATMS Measurements Using a Random Forests Technique

A new algorithm for the estimation of atmospheric temperature (T) and water vapor (WV) vertical profiles in nonprecipitating conditions is presented. The microwave random forest temperature and water vapor (MiRTaW) profiling algorithm is based on the random forest (RF) technique and it uses microwave (MW) sounding from the Advanced Technology Microwave Sounder (ATMS) onboard the Suomi National Polar-orbiting Partnership (SNPP) satellite. Three different data sources were chosen for both training and validation purposes, namely, the ERA-Interim from the European Centre for Medium-Range Weather Forecasts (ECMWF), the Infrared Atmospheric Sounding Interferometer Atmospheric Temperature Water Vapour and Surface Skin Temperature (IASI L2 v6) from the Meteorological Operational satellites of the European Organization for the Exploitation of Meteorological Satellites (EUMETSAT), and the radiosonde observations from the Integrated Global Radiosonde Archive (IGRA). The period from 2012 to 2016 was considered in the training dataset; particular attention was paid to the instance selection procedure, in order to reduce the full training dataset with negligible information loss. The out-of-bag (OOB) error was computed and used to select the optimal RF parameters. Different RFs were trained, one for each vertical level: 32 levels for T (within 10–1000 hPa) and 23 levels for WV (200–1000 hPa). The validation of the MiRTaW profiling algorithm was conducted on a dataset from 2017. The mean bias error (MBE) of T vertical profiles ranges within about (−0.4–0.4) K, while for the WV mixing ratio, the MBE starts at ~0.5 g/kg near the surface and decreases to ~0 g/kg at 200 hPa level, in line with the expectations.

A strategy on the definition of applicability domain of model based on population analysis

In recent years, there have been growing concerns about quality evaluation of predictions of developed quantitative structure-activity relationship (QSAR) models. Well-defined applicability domain (AD) is very crucial in the validation of QSAR models as stated in the third principle of Organization for Economic Co-operation and Development (OECD). In this study, a new perspective on defining AD of model based on population analysis (PA) strategy, including model population analysis (MPA) and approach population analysis (APA), was proposed. MPA employed classical AD approaches to define AD with a vast amount of sub-datasets derived from training set. On the basis of MPA, the classical AD approaches could distinguish part of the samples that cannot be distinguished by full training samples. APA was then used to get a union of all results generated by the used AD approaches to give a consensus list of samples as falling outside the AD. In order to investigate the performance of PA strategy in defining AD with the classical AD approaches, two QSAR datasets were used. The results show that implementing PA strategy can assist three classical AD approaches to distinguish the additional samples that cannot be distinguished using full training dataset. When excluding the additional samples, the root mean square error of prediction of test set decreased, suggesting that PA strategy has a potential to distinguish the samples that cannot be reliably predicted.