Unlabeled Learning Research Articles

Boosting Positive and Unlabeled Learning for Anomaly Detection With Multi-Features

One of the key challenges of machine learning-based anomaly detection relies on the difficulty of obtaining anomaly data for training, which is usually rare, diversely distributed, and difficult to collect. To address this challenge, we formulate anomaly detection as a Positive and Unlabeled (PU) learning problem where only labeled positive (normal) data and unlabeled (normal and anomaly) data are required for learning an anomaly detector. As a semi-supervised learning method, it does not require providing labeled anomaly data for the training, thus it is easily deployed to various applications. As the unlabeled data can be extremely unbalanced, we introduce a novel PU learning method, which can tackle the situation where an unlabeled data set is mostly composed of positive instances. We start by using a linear model to extract the most reliable negative instances followed by a self-learning process to add reliable negative and positive instances with different speeds based on the estimated positive class prior. Furthermore, when feedback is available, we adopt boosting in the self-learning process to advantageously exploit the instability characteristic of PU learning. The classifiers in the self-learning process are weighted combined based on the estimated error rate to build the final classifier. Extensive experiments on six real datasets and one synthetic dataset show that our methods have better results under different conditions compared to existing methods.

Multi-Manifold Positive and Unlabeled Learning for Visual Analysis

Positive and Unlabeled (PU) learning has attracted intensive research interests in recent years, which is capable of training a binary classifier solely based on positive and unlabeled examples when the negative data are absent or too are diverse. However, the existing PU learning methods largely overlook the relationship between the examples when handling the unlabeled data, leading to insufficient exploitation of data structure which actually contains useful distribution information. Therefore, by following the multi-manifold assumption which is observed in many real-world vision problems, this paper proposes a novel algorithm termed “Multi-Manifold PU learning” (MMPU), which assumes that the data belonging to different classes lie on different underlying manifolds. As such, the structural information revealed by the dataset is deployed, which is helpful in deciding the labels of unlabeled examples. Our MMPU contains two main steps, namely, multi-manifold exploration and positive confidence training, where the former is accomplished by computing the local similarity, structural similarity, and semantic similarity of pairwise data, and the latter establishes a binary classifier in reproducing kernel Hilbert space based on the real-valued confidence level of each example to be positive. Experimentally, we not only test the proposed MMPU on five highly nonlinear synthetic datasets but also apply MMPU to various typical computer vision tasks, including handwritten digit recognition, violent behavior detection, and hyperspectral image classification. The results demonstrate that MMPU can obtain a superior performance compared to the state-of-the-art PU learning methodologies.

Large-Margin Label-Calibrated Support Vector Machines for Positive and Unlabeled Learning.

Positive and unlabeled learning (PU learning) aims to train a binary classifier based on only PU data. Existing methods usually cast PU learning as a label noise learning problem or a cost-sensitive learning problem. However, none of them fully take the data distribution information into consideration when designing the model, which hinders them from acquiring more encouraging performance. In this paper, we argue that the clusters formed by positive examples and potential negative examples in the feature space should be critically utilized to establish the PU learning model, especially when the negative data are not explicitly available. To this end, we introduce a hat loss to discover the margin between data clusters, a label calibration regularizer to amend the biased decision boundary to the potentially correct one, and propose a novel discriminative PU classifier termed "Large-margin Label-calibrated Support Vector Machines" (LLSVM). Our LLSVM classifier can work properly in the absence of negative training examples and effectively achieve the max-margin effect between positive and negative classes. Theoretically, we derived the generalization error bound of LLSVM which reveals that the introduction of PU data does help to enhance the algorithm performance. Empirically, we compared LLSVM with state-of-the-art PU methods on various synthetic and practical data sets, and the results confirm that the proposed LLSVM is more effective than other compared methods on dealing with PU learning tasks.

Family profile mining in retailing

In the era of personalized marketing, the ability to leverage data analysis to recommend tailored products is a key competitive advantage for retailers. Family profiles are an essential aspect of customer information to boost the performance of knowledge-based recommendations. This study integrates positive and unlabeled learning and feature selection techniques to design a novel and flexible family profiling algorithm that tags target customers based on unlabeled transaction data. The empirical evaluation shows that our algorithm outperforms other algorithms in terms of the recall rate of families of the target tag. The knowledge of inferred family profiles also enhances the product recommendation performance. Our algorithm can help retailers efficiently target customers based on family profiles for better marketing performance.

Efficient Detection of Environmental Violators: A Big Data Approach

The detection of environmental violators is critical to the long-term adoption of sustainability in supply chain management. However, there exist manufacturing facilities that report false environmental monitoring data, thereby seriously hampering governments’ efforts to identify true offenders and to properly intervene. We integrate waste gas data from the world’s largest Continuous Emission Monitoring System (CEMS) with a publicly available Violation and Punishment Dataset to build prediction models for the identification of environmental violators. We utilize and create innovative machine learning approaches to overcome analytical challenges associated with empirical data. First, we use a feature engineering approach to generate features from the raw, and possibly fraudulent, reporting data. This overcomes the challenges associated with low fidelity, irregularity, and the presence of extreme values in the raw dataset. Second, while building prediction models, we develop new approaches to positive and unlabeled learning to overcome the challenges posed by sparsity and mislabeled data. Our prediction model achieves satisfactory results in a related field test. Our study develops new techniques for big data analytics, which greatly improve the efficiency and effectiveness in the detection of environmental violators and enhance operational outcomes of environmental protection agencies. This research is a joint effort between academia and practitioners, as evidenced by the participation of the Ministry of Ecology and Environment of People’s Republic of China. The Ministry kindly granted us direct data access, as well as opportunities to interview Subject Matter Experts at the Ministry, which led to research insights incorporated in this manuscript. Our research findings have global implications, as CEMS devices are universally adopted to monitor waste gas emissions.

МОДЕЛЬ ТА АЛГОРИТМ НАВЧАННЯ СИСТЕМИ ДЕТЕКТУВАННЯ МАЛОРОЗМІРНИХ ОБ’ЄКТІВ ДЛЯ МАЛОГАБАРИТНИХ БЕЗПІЛОТНИХ ЛІТАЛЬНИХ АПАРАТІВ

The efficient model and learning algorithm of the small object detection system for compact aerial vehicle under conditions of restricted computing resources and the limited volume of the labeled learning set are developed. The four-stage learning algorithm of the object detector is proposed. At the first stage, selecting the type of deep convolutional neural network and the number of low-level layers that is pretrained on the ImageNet dataset for reusing takes place. The second stage involves unsupervised learning of high-level convolutional sparse coding layers using the modification of growing neural gas to automatically determine the required number of neurons and provide optimal distributions of the neurons over the data. Its application makes it possible to utilize the unlabeled learning datasets for the adaptation of the high-level feature description to the domain application area. At the third stage, the output feature map is formed by concatenation of feature maps from the different level of the deep convolutional neural network. At that, there is a reduction of output feature map using principal component analysis and followed by the building of decision rules. In order to perform the classification analysis of output, feature map is proposed to use information-extreme classifier learning on principles of boosting. Besides that, the orthogonal incremental extreme learning machine is used to build the regression model for the predict bounding box of the detected small object. The last stage involves fine-tuning of high-level layers of deep network using simulated annealing metaheuristic algorithm in order to approximate the global optimum of the complex criterion of learning efficiency of detection model. As a result of the use of proposed approach has been achieved 96% correctly detection of objects on the images of the open test dataset which indicates the suitability of the model and learning algorithm for practical use. In this case, the size of the learning dataset that has been used to construct the model was 500 unlabeled and 200 labeled learning samples

A testing data validity assessment method and testing data validation platform based on SOA

In modern manufacturing, ensuring the quality of component testing data is highly valued by both product manufacturers and component suppliers. However, in common component quality analysis processes, testing data are assumed to be valid, which might not be true. Therefore, assessing the validity of component testing data would be important. Many existing data analysis platforms are separated from enterprises’ own systems, which makes the inspection data analysis incoherent to their business process. In this paper, we propose a testing data quality assessment method and a testing data validation platform based on SOA. The platform provides reliable third-party testing data validation service via RESTful APIs, so that the services can be seamlessly integrated to enterprise systems. The testing data validity assessment method, which is the core of the platform, is implemented by detecting illegal behavior in data recording. The detection is a combination of behavior analysis and a positive and unlabeled learning process.

Twenty years of bioinformatics research for protease-specific substrate and cleavage site prediction: a comprehensive revisit and benchmarking of existing methods.

The roles of proteolytic cleavage have been intensively investigated and discussed during the past two decades. This irreversible chemical process has been frequently reported to influence a number of crucial biological processes (BPs), such as cell cycle, protein regulation and inflammation. A number of advanced studies have been published aiming at deciphering the mechanisms of proteolytic cleavage. Given its significance and the large number of functionally enriched substrates targeted by specific proteases, many computational approaches have been established for accurate prediction of protease-specific substrates and their cleavage sites. Consequently, there is an urgent need to systematically assess the state-of-the-art computational approaches for protease-specific cleavage site prediction to further advance the existing methodologies and to improve the prediction performance. With this goal in mind, in this article, we carefully evaluated a total of 19 computational methods (including 8 scoring function-based methods and 11 machine learning-based methods) in terms of their underlying algorithm, calculated features, performance evaluation and software usability. Then, extensive independent tests were performed to assess the robustness and scalability of the reviewed methods using our carefully prepared independent test data sets with 3641 cleavage sites (specific to 10 proteases). The comparative experimental results demonstrate that PROSPERous is the most accurate generic method for predicting eight protease-specific cleavage sites, while GPS-CCD and LabCaS outperformed other predictors for calpain-specific cleavage sites. Based on our review, we then outlined some potential ways to improve the prediction performance and ease the computational burden by applying ensemble learning, deep learning, positive unlabeled learning and parallel and distributed computing techniques. We anticipate that our study will serve as a practical and useful guide for interested readers to further advance next-generation bioinformatics tools for protease-specific cleavage site prediction.

A positive and unlabeled learning framework based on extreme learning machine for drug-drug interactions discovery

Drug-drug interactions (DDIs) lead to Adverse Drug Reactions (ADRs) in most cases, which increase medical costs tremendously, and may cause medical negligence or even fatal accidents. Many researchers introduce binary classification methods into DDIs discovery task to avoid expensive and inefficient experimental trials. However, we find that drug-drug pairs without DDI labels cannot be simply viewed as negative samples in practice, since unobserved DDIs may be discovered from unlabeled pairs. Therefore, different from traditional positive-negative classification, in this paper, we treat DDIs discovery as a positive-unlabeled learning (PU learning) problem, in which each drug-drug pair is either a positive sample (labeled as DDI) or an unlabeled one (labeled as non-DDI). We propose a PU learning framework based on Extreme Learning Machine (ELM) named PU-ELM, which consists of two components, namely reliable negative extraction and classifier learning. To improve the quality of reliable negative set, we propose a negative extraction method named OC-ELM-RN (One-Class ELM for Reliable Negative extraction). As to the learning module, we first apply the semi-supervised learning strategy in PU-ELM (denoted as PU-ELM-SS) to achieve extremely fast learning speed, and then we propose an entropy based weighted learning method named PU-ELM-EW (PU-ELM with Entropy Weighted learning) to improve the PU learning performance. Extensive experiments are conducted on a synthetic dataset with varied settings. We also apply PU-ELM to a real-world drug dataset. The results indicate that PU-ELM achieves better DDIs discovery ability compared with state-of-the-art methods.

Efficient Training for Positive Unlabeled Learning.

Positive unlabeled (PU) learning is useful in various practical situations, where there is a need to learn a classifier for a class of interest from an unlabeled data set, which may contain anomalies as well as samples from unknown classes. The learning task can be formulated as an optimization problem under the framework of statistical learning theory. Recent studies have theoretically analyzed its properties and generalization performance, nevertheless, little effort has been made to consider the problem of scalability, especially when large sets of unlabeled data are available. In this work we propose a novel scalable PU learning algorithm that is theoretically proven to provide the optimal solution, while showing superior computational and memory performance. Experimental evaluation confirms the theoretical evidence and shows that the proposed method can be successfully applied to a large variety of real-world problems involving PU learning.

C-PUGP: A cluster-based positive unlabeled learning method for disease gene prediction and prioritization

Disease gene detection is an important stage in the understanding disease processes and treatment. Some candidate disease genes are identified using many machine learning methods Although there are some differences in these methods including feature vector of genes, the method used to selecting reliable negative data (non-disease genes), and the classification method, the lack of negative data is the most significant challenge of them. Recently, candidate disease genes are identified by semi-supervised learning methods based on positive and unlabeled data. These methods are reasonably accurate and achieved more desirable results versus preceding methods. In this article, we propose a novel Positive Unlabeled (PU) learning technique based upon clustering and One-Class classification algorithm. In this regard, unlike existing methods, we make a more Reliable Negative (RN) set in three steps: (1) Clustering positive data, (2) Learning One-Class classifier models using the clusters, and (3) Selecting intersection set of negative data as the Reliable Negative set. Next, we attempt to identify and rank the candidate disease genes using a binary classifier based on support vector machine (SVM) algorithm. Experimental results indicate that the proposed method yields to the best results, that is 92.8, 93.6, and 93.1 in terms of precision, recall, and F-measure respectively. Compared to the existing methods, the increase of performances of our proposed method is 11.7 percent better than the best method in terms of F-measure. Also, results show about 6% increase in the prioritization results.

Controllable-Domain-Based Fuzzy Rule Extraction for Copper Removal Process Control

In copper removal process control, the commonly used technique is the so-called rule-based control, which is largely dependent upon the operators' experience, likely leading to unstable process production due to each individual's characters and favors. In this paper, to enhance the effectiveness of process control, a controllable-domain-based fuzzy rule extraction strategy is proposed. New definitions of representative controlled samples are introduced, by which the input variable space is divided into several controllable domains by applying positive and unlabeled learning algorithm. Also, the unreasonable removed and the controllable domains are accordingly determined. Then, support vector machine method is employed to extract fuzzy control rules for different domains. Finally, an industrial experiment is presented to demonstrate the effectiveness and advantages of the developed new design scheme.

An Ensemble of Classifiers Based on Positive and Unlabeled Data in One-Class Remote Sensing Classification

One-class remote sensing classification refers to the situations when users are only interested in one specific land type without considering other types. The positive and unlabeled learning (PUL) algorithm, which trains a binary classifier from positive and unlabeled data, has been shown to be promising in one-class classification. The implementation of PUL by a single classifier has been investigated. However, implementing PUL using multiple classifiers and creating classifier ensembles based on PUL have not been studied. In this research, we investigate the implementations of PUL using several classifiers, including generalized linear model, generalized additive model, multivariate adaptive regression splines, maximum entropy, backpropagation neural network, and support vector machine, as well as three ensemble methods based on majority vote, weighted average, and weighted vote combination rules. These methods are applied in classifying the urban areas from four remote sensing imagery of different spatial resolutions, including aerial photograph, Landsat 8, WorldView-3, and Gaofen-1. Experimental results show that classifiers can successfully extract the urban areas with high accuracies, and the ensemble methods based on weighted average and weighted vote generally outperform the individual classifiers on different datasets. We conclude that PUL is a promising method in one-class remote sensing classification, and the classifier ensemble based on PUL can significantly improve the accuracy.

Positive and Unlabeled Learning for User Behavior Analysis Based on Mobile Internet Traffic Data

With the rapid development of wireless communication and mobile Internet, mobile phone becomes ubiquitous and functions as a versatile and a smart system, on which people frequently interact with various mobile applications (Apps). Understanding human behaviors using mobile phone is significant for mobile system developers, for human-centered system optimization and better service provisioning. In this paper, we focus on mobile user behavior analysis and prediction based on mobile Internet traffic data. Real traffic flow data is collected from the public network of Internet service providers, by high-performance network traffic monitors. We construct a User-App bipartite network to represent the traffic interaction pattern between users and App servers. After mining the explicit and implicit features from the User-App bipartite network, we propose two positive and unlabeled (PU) learning methods, including Spy-based PU learning and K-means-based PU learning, for App usage prediction and mobile video traffic identification. We first use the traffic flow data of QQ, a very famous messaging and social media application possessing high market share in China, as the experimental data set for App usage prediction task. Then, we use the traffic flow data from six popular Apps, including video intensive Apps (Youku, Baofeng, LeTV, and Tudou) and other Apps (Meituan and Apple), as the experimental data set for mobile video traffic identification task. Experimental results show that our proposed PU learning methods perform well in both tasks.

Detecting Spammer Groups From Product Reviews: A Partially Supervised Learning Model

Nowadays, online product reviews play a crucial role in the purchase decision of consumers. A high proportion of positive reviews will bring substantial sales growth, while negative reviews will cause sales loss. Driven by the immense financial profits, many spammers try to promote their products or demote their competitors’ products by posting fake and biased online reviews. By registering a number of accounts or releasing tasks in crowdsourcing platforms, many individual spammers could be organized as spammer groups to manipulate the product reviews together and can be more damaging. Existing works on spammer group detection extract spammer group candidates from review data and identify the real spammer groups using unsupervised spamicity ranking methods. Actually, according to the previous research, labeling a small number of spammer groups is easier than one assumes, however, few methods try to make good use of these important labeled data. In this paper, we propose a partially supervised learning model (PSGD) to detect spammer groups. By labeling some spammer groups as positive instances, PSGD applies positive unlabeled learning (PU-Learning) to study a classifier as spammer group detector from positive instances (labeled spammer groups) and unlabeled instances (unlabeled groups). Specifically, we extract reliable negative set in terms of the positive instances and the distinctive features. By combining the positive instances, extracted negative instances and unlabeled instances, we convert the PU-Learning problem into the well-known semi-supervised learning problem, and then use a Naive Bayesian model and an EM algorithm to train a classifier for spammer group detection. Experiments on real-life Amazon.cn data set show that the proposed PSGD is effective and outperforms the state-of-the-art spammer group detection methods.

Positive unlabeled learning for building recommender systems in a parliamentary setting

Our goal is to learn about the political interests and preferences of Members of Parliament (MPs) by mining their parliamentary activity in order to develop a recommendation/filtering system to determine how relevant documents should be distributed among MPs. We propose the use of positive unlabeled learning to tackle this problem since we only have information about relevant documents (the interventions of each MP in debates) but not about irrelevant documents and so it is not possible to use standard binary classifiers which have been trained with positive and negative examples. Additionally, we have also developed a new positive unlabeled learning algorithm that compares favorably with: (a) a baseline approach which assumes that every intervention by any other MP is irrelevant; (b) another well-known positive unlabeled learning method; and (c) an approach based on information retrieval methods that matches documents and legislators’ representations. The experiments have been conducted with data from the regional Spanish Andalusian Parliament.

Global and local learning from positive and unlabeled examples

In common binary classification scenarios, learning algorithms assume the presence of both positive and negative examples. Unfortunately, in many practical areas, only limited labeled positive examples and large amounts of unlabeled examples are available, but there are no negative examples. In such cases, the algorithm that only exploits positive and unlabeled examples is needed. Such learning is termed as positive and unlabeled (PU) learning. In this paper, a novel classifier called global and local learning classifier (GLLC) for PU learning is proposed. The advantages of GLLC are as follows: (1) both intrinsic geometric structure and accurate positive information of PU data are exploited from global learning. (2) The smoothness and manifold of data are reflected sufficiently from local learning. (3) The algorithm of GLLC has faster training speed because the linear equations are solved. (4) The experiments on both synthetic and real datasets verify the above opinions and show that the classification result of GLLC is much better than those popular methods, such as LUHC, Pulce, BSVM, NB and so on.

Identifying Leading Indicators of Product Recalls from Online Reviews Using Positive Unlabeled Learning and Domain Adaptation

Consumer protection agencies are charged with safeguarding the public from hazardous products, but the thousands of products under their jurisdiction make it challenging to identify and respond to consumer complaints quickly. In this paper, we propose a system to mine Amazon.com reviews to identify products that may pose safety or health hazards. Since labeled data for this task are scarce, our approach combines positive unlabeled learning with domain adaptation to train a classifier from consumer complaints submitted to an online government portal. We find that our approach results in an absolute F1 score improvement of 8% over the best competing baseline. Furthermore, when we apply the classifier to Amazon reviews of known recalled products, we identify safety hazard reports prior to the recall date for 45% of the products. This suggests that the system may be able to provide an early warning system to alert consumers to hazardous products before an official recall is announced.

Bayesian belief network for positive unlabeled learning with uncertainty

The current state-of-art for tackling the problem of classification of static uncertain data under PU learning (Positive Unlabeled Learning) scenario, is UPNB. It is based on the Bayesian assumption, which does not hold for real-life applications, and hence it may depress the classification performance of UPNB. In this paper, we propose UPTAN (Uncertain Positive Tree Augmented Naive Bayes), a Bayesian network algorithm, so as to utilize the dependence information among uncertain attributes for classification. We propose uncertain conditional mutual information (UCMI) for measuring the mutual information between uncertain attributes, and then use it to learn the tree structure of Bayesian network. Furthermore, we give our approach for estimating the parameters of the Bayesian network for uncertain data without negative training examples. Our experiments on 20 UCI datasets show that UPTAN has excellent classification performance, with average F1 being 0.8257, which outperforms UPNB by 3.73%.

Biological Network Derivation by Positive Unlabeled Learning Algorithms

Background: In cases where only a single group (or class) of samples is available for a given problem, positive unlabeled learning algorithms can be applied. One such case is the interactions between various biological/chemical entity pairs, where only the set of interacting entities can be collected, not the “noninteracting” ones. Objective: We aim to improve the performance of deriving protein-protein and protein-ligand interactions. We argue that the positive-unlabeled learning algorithms can be applied to this problem. Method: In this paper, we propose some modifications to two of the existing methods for protein-protein and protein-ligand interaction network derivation. First, we extend the algorithms to use Random Forests and then we devise an ensemble classifier from these two based on voting. Results: We report the evaluation results of the proposed algorithms in comparison to the original methods and well-known biological network derivation algorithms. We achieved significant improvements in terms of different metrics. Conclusion: The results are promising in the sense that proposed methods either perform competitively or better than previous methods. This motivates us in applying the proposed methods to other data sets and similar problems. Keywords: Binary classification, positive unlabeled learning, protein-ligand interaction networks, protein-protein interaction networkS, random forests, support vector machines.