High-quality Training Datasets Research Articles

Methods, New Software Tools, and Best Practices for Developing High-quality Training Data for Machine Learning-based Image Analysis in Biodiversity Research

Recent progress in using deep learning techniques to automate the analysis of complex image data is opening up exciting new avenues for research in biodiversity science. However, potential applications of machine learning methods in biodiversity research are often limited by the relative scarcity of data suitable for training machine learning models. Development of high-quality training data sets can be a surprisingly challenging task that can easily consume hundreds of person-hours of time. In this talk, we present the results of our recent work implementing and comparing several different methods for generating annotated, biodiversity-oriented image data for training machine learning models, including collaborative expert scoring, local volunteer image annotators with on-site training, and distributed, remote image annotation via citizen science platforms. We discuss error rates, among-annotator variance, and depth of coverage required to ensure highly reliable image annotations. We also discuss time considerations and efficiency of the various methods. Finally, we present new software, called ImageAnt (currently under development), that supports efficient, highly flexible image annotation workflows. ImageAnt was created primarily in response to the challenges we discovered in our own efforts to generate image-based training data for machine learning models. ImageAnt features a simple user interface and can be used to implement sophisticated, adaptive scripting of image annotation tasks.

Application of neural networks in multiphase flow through porous media: Predicting capillary pressure and relative permeability curves

Artificial Neural Networks (ANN) are trained to simulate two-phase capillary pressure and relative permeability data in bundles of capillary tubes with non-uniform arbitrary wettability conditions and cross-sectional shapes of different irregular convex polygons. All polygons with variable number of corners are randomly generated for a given range of inscribed radii, shape, and elongation factors. To generate the data for the training of ANNs, the minimization of Helmholtz free energy and Mayer-Stowe-Princen (MS-P) method are combined to find thermodynamically consistent threshold capillary pressures for two-phase flow. These capillary pressures are then used to determine the sequence of displacements in different capillary tubes. We calculate saturations and phase conductance at each quasi steady-state condition where no more displacements can be done for a given capillary pressure. The generated two-phase capillary pressure and relative permeability curves are then used for the training of ANNs. We test different designs of ANNs to find the optimal workflow for the training and predicting of petrophysical properties related to multiphase flow. In this work, we present the results of two different neural network structures. In the first structure, we use ANN to predict threshold capillary pressures of different capillary tubes during a drainage process (i.e., oil-to-water displacements). In the second structure, we predict capillary pressure and relative permeability curves for an arbitrary bundle of capillary tubes. The first structure of ANNs simulates a fixed property for a given capillary tube, whereas the second structure simulates time-series data format (i.e., for a given bundle of capillary tubes calculated properties vary with saturation). To do so, we have generated multi-phase flow properties for two large datasets consisting of 40,000 and 60,000 capillary tubes each. High-quality training datasets are critical in the training of high-fidelity ANN models. These models can then learn the impact of a wide variety of pore geometries (i.e., shape factors and elongations). Additionally, feature selection and preprocessing of the input data could significantly impact ANN's predictions. The multi-layer perceptron (MLP) neural network with three hidden layers with four outputs is adequate for predicting capillary pressure and relative permeability curves during drainage. This model is approximately an order of magnitude faster than conventional direct calculations using a desktop computer with four cores CPU. Such improvement in the speed of calculations becomes significant when dealing with larger models, more dimensions, and/or introducing pore connectivity in 3D.

A low-cost photorealistic CG dataset rendering pipeline for facial landmark localization

Face analysis has been a hot research field in computer vision for decades. The dataset is of vital importance for modern machine learning methods. The paper proposes a flexible CG (Computer Graphics) rendering pipe-line for creating facial image datasets together with automatic ground truth labelling. The proposed pipe-line could produce a huge amount of labelled data fast and in low cost compared to traditional dataset creation methods which need high cost hardware and longtime manual ground truth labelling. The paper also proposes a data capture setup in the CG environment for creating the dataset for facial landmark localization. The effectiveness of the proposed method is verified by cross validation with Multi-PIE dataset. For creating a high quality training dataset, some of the varying factors of the dataset should be considered. The paper analyzes a few varying factors for accurate eye landmark localization, such as eye closure levels, eye and eyebrow shapes and wearing glasses. Based on the benefits of the proposed CG rendering pipe-line, the paper implemented a facial landmark localization system across large face rotation by integrating off-the-shelves algorithms. The experiments on Multi-PIE and real persons show that the implemented system could localize facial landmarks accurately across [−90°, +90°] in yaw rotation in real time.

Cooperative Hybrid Semi-Supervised Learning for Text Sentiment Classification

A large-scale and high-quality training dataset is an important guarantee to learn an ideal classifier for text sentiment classification. However, manually constructing such a training dataset with sentiment labels is a labor-intensive and time-consuming task. Therefore, based on the idea of effectively utilizing unlabeled samples, a synthetical framework that covers the whole process of semi-supervised learning from seed selection, iterative modification of the training text set, to the co-training strategy of the classifier is proposed in this paper for text sentiment classification. To provide an important basis for selecting the seed texts and modifying the training text set, three kinds of measures—the cluster similarity degree of an unlabeled text, the cluster uncertainty degree of a pseudo-label text to a learner, and the reliability degree of a pseudo-label text to a learner—are defined. With these measures, a seed selection method based on Random Swap clustering, a hybrid modification method of the training text set based on active learning and self-learning, and an alternately co-training strategy of the ensemble classifier of the Maximum Entropy and Support Vector Machine are proposed and combined into our framework. The experimental results on three Chinese datasets (COAE2014, COAE2015, and a Hotel review, respectively) and five English datasets (Books, DVD, Electronics, Kitchen, and MR, respectively) in the real world verify the effectiveness of the proposed framework.

An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets.

Owing to improvements in image recognition via deep learning, machine-learning algorithms could eventually be applied to automated medical diagnoses that can guide clinical decision-making. However, these algorithms remain a 'black box' in terms of how they generate the predictions from the input data. Also, high-performance deep learning requires large, high-quality training datasets. Here, we report the development of an understandable deep-learning system that detects acute intracranial haemorrhage (ICH) and classifies five ICH subtypes from unenhanced head computed-tomography scans. By using a dataset of only 904 cases for algorithm training, the system achieved a performance similar to that of expert radiologists in two independent test datasets containing 200 cases (sensitivity of 98% and specificity of 95%) and 196 cases (sensitivity of 92% and specificity of 95%). The system includes an attention map and a prediction basis retrieved from training data to enhance explainability, and an iterative process that mimics the workflow of radiologists. Our approach to algorithm development can facilitate the development of deep-learning systems for a variety of clinical applications and accelerate their adoption into clinical practice.

The Optimal ANN Model for Predicting Bearing Capacity of Shallow Foundations trained on Scarce Data

This study is focused on determining the potential of using deep neural networks (DNNs) to predict the ultimate bearing capacity of shallow foundation in situations when the experimental data which may be used to train networks is scarce. Two experiments involving testing over 17000 networks were conducted. The first experiment was aimed at comparing the accuracy of shallow neural networks and DNNs predictions. It shows that when the experimental dataset used for preparing models is small then DNNs have a significant advantage over shallow networks. The second experiment was conducted to compare the performance of DNNs consisting of different number of neurons and layers. Obtained results indicate that the optimal number of layers varies between 5 to 7. Networks with less and - surprisingly - more layers obtain lower accuracy. Moreover, the number of neurons in DNN has a lower impact on the prediction accuracy than the number of DNN's layers. DNNs perform very well, even when trained with only 6 samples. Basing on the results it seems that when predicting the ultimate bearing capacity with ANN models obtaining small but high-quality experimental training datasets instead of large training datasets affected by a higher error is an advisable approach.

HMMER Cut-off Threshold Tool (HMMERCTTER): Supervised classification of superfamily protein sequences with a reliable cut-off threshold.

BackgroundProtein superfamilies can be divided into subfamilies of proteins with different functional characteristics. Their sequences can be classified hierarchically, which is part of sequence function assignation. Typically, there are no clear subfamily hallmarks that would allow pattern-based function assignation by which this task is mostly achieved based on the similarity principle. This is hampered by the lack of a score cut-off that is both sensitive and specific.ResultsHMMER Cut-off Threshold Tool (HMMERCTTER) adds a reliable cut-off threshold to the popular HMMER. Using a high quality superfamily phylogeny, it clusters a set of training sequences such that the cluster-specific HMMER profiles show cluster or subfamily member detection with 100% precision and recall (P&R), thereby generating a specific threshold as inclusion cut-off. Profiles and thresholds are then used as classifiers to screen a target dataset. Iterative inclusion of novel sequences to groups and the corresponding HMMER profiles results in high sensitivity while specificity is maintained by imposing 100% P&R self detection. In three presented case studies of protein superfamilies, classification of large datasets with 100% precision was achieved with over 95% recall. Limits and caveats are presented and explained.ConclusionsHMMERCTTER is a promising protein superfamily sequence classifier provided high quality training datasets are used. It provides a decision support system that aids in the difficult task of sequence function assignation in the twilight zone of sequence similarity. All relevant data and source codes are available from the Github repository at the following URL: https://github.com/BBCMdP/HMMERCTTER.

An Unsupervised Approach to Inferring the Localness of People Using Incomplete Geotemporal Online Check-In Data

Inferring the localness of people is to classify people who are local residents in a city from people who visit the city by analyzing online check-in points that are contributed by online users. This information is critical for the urban planning, user profiling, and localized recommendation systems. Supervised learning approaches have been developed to infer the location of people in a city by assuming the availability of high-quality training datasets with complete geotemporal information. In this article, we develop an unsupervised model to accurately identify local people in a city by using the incomplete online check-in data that are publicly available. In particular, we develop an incomplete geotemporal expectation maximization (IGT-EM) scheme, which incorporates a set of hidden variables to represent the localness of people and a set of estimation parameters to represent the likelihood of venues to attract local and nonlocal people, respectively. Our solution can accurately classify local people from nonlocal nones without requiring any training data. We also implement a parallel IGT-EM algorithm by leveraging the computing power of a graphic processing unit (GPU) that consists of 2,496 cores. In the evaluation, we compare our new approach with the existing solutions through four real-world case studies using data from the New York City, Chicago, Boston, and Washington, DC. The results show that our approach can identify the local people and significantly outperform the compared baselines in estimation accuracy and execution time.

Multi-level hybrid support vector machine and extreme learning machine based on modified K-means for intrusion detection system

Intrusion detection has become essential to network security because of the increasing connectivity between computers. Several intrusion detection systems have been developed to protect networks using different statistical methods and machine learning techniques. This study aims to design a model that deals with real intrusion detection problems in data analysis and classify network data into normal and abnormal behaviors. This study proposes a multi-level hybrid intrusion detection model that uses support vector machine and extreme learning machine to improve the efficiency of detecting known and unknown attacks. A modified K-means algorithm is also proposed to build a high-quality training dataset that contributes significantly to improving the performance of classifiers. The modified K-means is used to build new small training datasets representing the entire original training dataset, significantly reduce the training time of classifiers, and improve the performance of intrusion detection system. The popular KDD Cup 1999 dataset is used to evaluate the proposed model. Compared with other methods based on the same dataset, the proposed model shows high efficiency in attack detection, and its accuracy (95.75%) is the best performance thus far.

Genome-wide prediction of prokaryotic two-component system networks using a sequence-based meta-predictor.

BackgroundTwo component systems (TCS) are signalling complexes manifested by a histidine kinase (receptor) and a response regulator (effector). They are the most abundant signalling pathways in prokaryotes and control a wide range of biological processes. The pairing of these two components is highly specific, often requiring costly and time-consuming experimental characterisation. Therefore, there is considerable interest in developing accurate prediction tools to lessen the burden of experimental work and cope with the ever-increasing amount of genomic information.ResultsWe present a novel meta-predictor, MetaPred2CS, which is based on a support vector machine. MetaPred2CS integrates six sequence-based prediction methods: in-silico two-hybrid, mirror-tree, gene fusion, phylogenetic profiling, gene neighbourhood, and gene operon. To benchmark MetaPred2CS, we also compiled a novel high-quality training dataset of experimentally deduced TCS protein pairs for k-fold cross validation, to act as a gold standard for TCS partnership predictions. Combining individual predictions using MetaPred2CS improved performance when compared to the individual methods and in comparison with a current state-of-the-art meta-predictor.ConclusionWe have developed MetaPred2CS, a support vector machine-based metapredictor for prokaryotic TCS protein pairings. Central to the success of MetaPred2CS is a strategy of integrating individual predictors that improves the overall prediction accuracy, with the in-silico two-hybrid method contributing most to performance. MetaPred2CS outperformed other available systems in our benchmark tests, and is available online at http://metapred2cs.ibers.aber.ac.uk, along with our gold standard dataset of TCS interaction pairs.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-015-0741-7) contains supplementary material, which is available to authorized users.

An unsupervised hierarchical clustering based heuristic algorithm for facilitated training of electricity consumption disaggregation systems

Provision of training data sets is one of the core requirements for event-based supervised NILM (Non-Intrusive Load Monitoring) algorithms. Due to diversity in appliances’ technologies, in-situ training by users is often required. This process might require continuous user-interaction to ensure that a high quality training data set is provided. Pre-populating a training data set could potentially reduce the need for user-system interaction. In this study, a heuristic unsupervised clustering algorithm is presented and evaluated to enable autonomous partitioning of appliances signature space (i.e. feature space) for applications in electricity consumption disaggregation. The algorithm is based on hierarchical clustering and uses the characteristics of a cluster binary tree to determine the distance threshold for pruning the tree without a priori information. The algorithm determines the partition of a feature space recursively to account for multi-scale nature of the binary cluster tree. Evaluation of the algorithm was carried out using metrics for accuracy and cluster quality (proposed in this study) on a fully labeled data set that was collected and processed in a real residential setting. The algorithm performance in accurate partitioning of the feature space and the effect of different feature extraction techniques were presented and discussed.

Neural network Jacobian analysis for high-resolution profiling of the atmosphere

Neural networks have been widely used to provide retrievals of geophysical parameters from spectral radiance measurements made remotely by air-, ground-, and space-based sensors. The advantages of retrievals based on neural networks include speed of execution, simplicity of the trained algorithm, and ease of error analysis, and the proliferation of high quality training data sets derived from models and/or operational measurements has further facilitated their use. In this article, we provide examples of geophysical retrieval algorithms based on neural networks with a focus on Jacobian analysis. We examine a hypothetical 80-channel hyperspectral microwave atmospheric sounder (HyMAS) and construct examples comparing neural network water vapor retrieval performance with simple regressions. Jacobians (derivatives of the outputs with respect to the network weights and with respect to the inputs) are also presented and discussed. Finally, a discussion of the Jacobian operating points is provided.

A novel data selection method based on shadowed sets

One of the main factors that affect the supervised learning performance of neural network (NN) is training data set. However, for real world problems, it is not always easy to obtain high quality training data sets. In this paper, a novel training data selection method based on shadowed sets is proposed that can select an informative and representative subset of training data to ameliorate the supervised learning performance of NN. The main goal of this work is to improve the generalization ability and diminish misclassification errors of classifier of NN. This paper firstly introduces central idea of shadowed sets. Then the algorithm of proposed data selection method is described in detail. Finally, taking LVQ model as an example, some experiments are made to test validity of this method. The experiment results indicate that using selected sample data to train NN can lessen computation consumption, save training time, guarantee generalization ability, which verify the effectiveness and applicability of proposed data selection method.

Learning from imperfect data

For a supervised learning method, the quality of the training data or the training supervisor is very important in generating reliable neural networks. However, for real world problems, it is not always easy to obtain high quality training data sets. In this research, we propose a learning method for a neural network ensemble model that can be trained with an imperfect training data set, which is a data set containing erroneous training samples. With a competitive training mechanism, the ensemble is able to exclude erroneous samples from the training process, thus generating a reliable neural network. Through the experiment, we show that the proposed model is able to tolerate the existence of erroneous training samples in generating a reliable neural network. The ability of the neural network to tolerate the existence of erroneous samples in the training data lessens the costly task of analyzing and arranging the training data, thus increasing the usability of the neural networks for real world problems.