Learning From Label Proportions Research Articles

LLP-GAN: A GAN-Based Algorithm for Learning From Label Proportions.

Learning from label proportions (LLP) is a widespread and important learning paradigm: only the bag-level proportional information of the grouped training instances is available for the classification task, instead of the instance-level labels in the fully supervised scenario. As a result, LLP is a typical weakly supervised learning protocol and commonly exists in privacy protection circumstances due to the sensitivity in label information for real-world applications. In general, it is less laborious and more efficient to collect label proportions as the bag-level supervised information than the instance-level one. However, the hint for learning the discriminative feature representation is also limited as a less informative signal directly associated with the labels is provided, thus deteriorating the performance of the final instance-level classifier. In this article, delving into the label proportions, we bypass this weak supervision by leveraging generative adversarial networks (GANs) to derive an effective algorithm LLP-GAN. Endowed with an end-to-end structure, LLP-GAN performs approximation in the light of an adversarial learning mechanism without imposing restricted assumptions on distribution. Accordingly, the final instance-level classifier can be directly induced upon the discriminator with minor modification. Under mild assumptions, we give the explicit generative representation and prove the global optimality for LLP-GAN. In addition, compared with existing methods, our work empowers LLP solvers with desirable scalability inheriting from deep models. Extensive experiments on benchmark datasets and a real-world application demonstrate the vivid advantages of the proposed approach.

LLP-AAE: Learning from label proportions with adversarial autoencoder

This paper presents an effective weakly supervised learning algorithm LLP-AAE to leverage the adversarial autoencoder (AAE) for learning from label proportions (LLP), in which only the bag-level proportional information is available. Our LLP-AAE utilizes an autoencoder backbone and performs adversarial training in latent space to match the aggregated posterior distribution of hidden coding with the prior distributions. In this way, apart from the reconstruction task, the encoder is also dedicated to producing fake samples, in order to deceive discriminators as far as possible. Ultimately, the encoder is employed as a competent label predictor for unseen data. In addition to the LLP classifier, our model can also achieve controllable samples generation by feeding the decoder with gradually changing latent code, which is proven to be useful for a better LLP performance. We also provide a panoramic explanation for LLP-AAE by regarding the LLP problem as an alternative learning procedure between proportion-based pseudo label generation and discriminative reconstruction. Experiments on six benchmark image datasets demonstrate the advantage of our method both in style manipulation with the latent feature representation and comparable multi-class LLP performance with the state-of-the-art models.

Learning from Label Proportions with Prototypical Contrastive Clustering

The use of priors to avoid manual labeling for training machine learning methods has received much attention in the last few years. One of the critical subthemes in this regard is Learning from Label Proportions (LLP), where only the information about class proportions is available for training the models. While various LLP training settings verse in the literature, most approaches focus on bag-level label proportions errors, often leading to suboptimal solutions. This paper proposes a new model that jointly uses prototypical contrastive learning and bag-level cluster proportions to implement efficient LLP classification. Our proposal explicitly relaxes the equipartition constraint commonly used in prototypical contrastive learning methods and incorporates the exact cluster proportions into the optimal transport algorithm used for cluster assignments. At inference time, we compute the clusters' assignment, delivering instance-level classification. We experimented with our method on two widely used image classification benchmarks and report a new state-of-art LLP performance, achieving results close to fully supervised methods.

SELF-LLP: Self-supervised learning from label proportions with self-ensemble

In this paper, we tackle the problem called learning from label proportions (LLP), where the training data is arranged into various bags, with only the proportions of different categories in each bag available. Existing efforts mainly focus on training a model with only the limited proportion information in a weakly supervised manner, thus result in apparent performance gap to supervised learning, as well as computational inefficiency. In this work, we propose a multi-task pipeline called SELF-LLP to make full use of the information contained in the data and model themselves. Specifically, to intensively learn representation from the data, we leverage the self-supervised learning as a plug-in auxiliary task to learn better transferable visual representation. The main insight is to benefit from the self-supervised representation learning with deep model, as well as improving classification performance by a large margin. Meanwhile, in order to better leverage the implicit benefits from the model itself, we incorporate the self-ensemble strategy to guide the training process with an auxiliary supervision information, which is constructed by aggregating multiple previous network predictions. Furthermore, a ramp-up mechanism is further employed to stabilize the training process. In the extensive experiments, our method demonstrates compelling advantages in both accuracy and efficiency over several state-of-the-art LLP approaches.

Active learning from label proportions via pSVM

Learning from label proportions (LLP), in which the training data is divided into different bags and only the proportions of samples belonging to certain categories in each bag are known, has attracted widespread interest in many application scenarios. However, the existing researches ignore the importance of part instances in which the labels can be easily obtained. In this paper, we propose a novel method called active-pSVM, which incorporates active learning into the large margin framework to solve LLP problem. In detail, we first label the most valuable instances that are selected by QUIRE or uncertainty sampling query strategy. Then, the obtained labeled instances and the given unlabeled data are simultaneously used to construct the prediction model under the large margin framework. Extensive experiments on benchmark datasets show that the proposed active-pSVM outperforms the state-of-the-art approaches for solving LLP problem, i.e. InvCal, alter-∝SVM and LLP-NPSVM.

Deep learning from label proportions with labeled samples

Learning from label proportions (LLP), where the training data is in form of bags, and only the proportions of classes in each bag are available, has attracted wide interest in machine learning community. In general, most LLP algorithms adopt random sampling to obtain the proportional information of different categories, which correspondingly obtains some labeled samples in each bag. However, LLP training process always fails to leverage these labeled samples, which may contain essential data distribution information. To address this issue, in this paper, we propose end-to-end LLP solver based on convolutional neural networks (ConvNets), called LLP with labeled samples (LLP-LS). First, we reshape the cross entropy loss in ConvNets, so that it can combine the proportional information and labeled samples in each bag. Second, in order to comply with the training data in a bag manner, ADAM based on batch is employed to train LLP-LS. Hence, the batch size in training process is in accordance with the bag size. Compared with up-to-date methods on multi-class problem, our algorithm can obtain the state-of-the-art on several image datasets.

Unsupervised learning in a BCI chess application using label proportions and expectation-maximization

ABSTRACT The online usage of brain-computer interfaces (BCI) generates unlabeled data. This data in combination with the rich structure contained in BCI applications based on event-related potentials allow to design novel unsupervised classification approaches like learning from label proportions (LLP) or its combination with expectation-maximization (EM) into a mixed model. In this work, we explore the feasibility of unsupervised classification in a BCI chess application. We propose an LLP extension based on weighted least squares regression. It requires randomization of timing parameters but overcomes the dependency on additional symbols. Simulations on electroencephalogram data obtained from six subjects playing BCI-controlled chess show that a combination of unsupervised LLP with EM (despite not using any labels) by constant adaptation quickly reaches and on the long run outperforms the average performance level of non-adaptive supervised classifiers. With our contribution, we increase the scope for which unsupervised learning methods can successfully be applied in BCI.

A Fast Algorithm for Multi-Class Learning from Label Proportions

Learning from label proportions (LLP) is a new kind of learning problem which has attracted wide interest in machine learning. Different from the well-known supervised learning, the training data of LLP is in the form of bags and only the proportion of each class in each bag is available. Actually, many modern applications can be successfully abstracted to this problem such as modeling voting behaviors and spam filtering. However, time-consuming training is still a challenge for LLP, which becomes a bottleneck especially when addressing large bags and bag sizes. In this paper, we propose a fast algorithm called multi-class learning from label proportions by extreme learning machine (LLP-ELM), which takes advantage of an extreme learning machine with fast learning speed to solve multi-class learning from label proportions. Firstly, we reshape the hidden layer output matrix and the training data target matrix of an extreme learning machine to adapt to the proportion information instead of the real labels. Secondly, a robust loss function with a regularization term is formulated and two efficient solutions are provided to different cases. Finally, various experiments demonstrate the significant speed-up of the proposed model with better accuracies on different datasets compared with several state-of-the-art methods.

Multi-Class Learning from Label Proportions for Bank Customer Classification

In this paper, we study multi-class learning from label proportions, and apply it to bank customer classification, in order to provide advice for banks to better manage customer relationships. We attempt to apply the multi-class extreme learning machine (ELM) to learning from label proportions (LLP). With a structure similar to neural network, ELM possesses higher computational speed and better generalization ability. As a result, it is suitable to deal with large-scale and multi-class problems. LLP is a learning problem, which classifies training data into bags where only the label proportions of each class in each bag is available. Furthermore, in order to maintain the stable model accuracy when the bag sizes increasing, we manage to add small number of labeled samples in our model, called LLP-ELM with in a semi-supervised learning framework. The experiments prove that our improvement has advantages in the case of large bag sizes. In practical, it is worth to consider to apply the proposed algorithm to multi-class learning from label proportions for bank customer classification and other multi-class scenarios.

A Study on Mobile Customer Churn Based on Learning from Soft Label Proportions

In real-life scenarios, manual labeling is costly and inefficient. besides sometimes there is privacy restrictions on the user’s information, which makes it difficult to detect customers losses in the mobile customer field. We cannot directly obtain information about a user is lost or not. However, the business manager can estimate the proportion of lost users based on his business experience. Besides, considering that the proportion information we obtain in reality is estimated and not completely accurate, we propose a soft label proportional learning method based on the learning from label proportions (LLP). In detail, we modify proportional loss term in the loss function by introducing a slack variable. In this way, the proportional loss within a certain range can be regarded as zero when the estimated proportional information is uncertain. The proportional information can be determined in a range by adjusting the parameter. The experimental results show the effectiveness of the proposed method, which provides a new way to effectively solve the problem of customer churn analysis in the field of mobile communication.

Weakly Supervised Classification For High Energy Physics

We present a classification algorithm that applies the machine learning paradigm of Learning from Label Proportions (LLP) [1] to enable learning on unlabelled data. Our algorithm, Weakly Supervised Classification, receives as its only input the class proportions of batches of data but makes per-instance classification decisions matching the performance of fully supervised approaches. We apply our model to the problem of Quark-Gluon tagging and show that it is robust to underlying mismodelling of the simulated data unlike fully supervised learning.

Learning from label proportions on high-dimensional data

Learning from label proportions (LLP), in which the training data is in the form of bags and only the proportion of each class in each bag is available, has attracted wide interest in machine learning. However, how to solve high-dimensional LLP problem is still a challenging task. In this paper, we propose a novel algorithm called learning from label proportions based on random forests (LLP-RF), which has the advantage of dealing with high-dimensional LLP problem. First, by defining the hidden class labels inside target bags as random variables, we formulate a robust loss function based on random forests and take the corresponding proportion information into LLP-RF by penalizing the difference between the ground truth and estimated label proportion. Second, a simple but efficient alternating annealing method is employed to solve the corresponding optimization model. At last, various experiments demonstrate that our algorithm can obtain the best accuracies on high-dimensional data compared with several recently developed methods.

Learning from label proportions in brain-computer interfaces: Online unsupervised learning with guarantees.

ObjectiveUsing traditional approaches, a brain-computer interface (BCI) requires the collection of calibration data for new subjects prior to online use. Calibration time can be reduced or eliminated e.g., by subject-to-subject transfer of a pre-trained classifier or unsupervised adaptive classification methods which learn from scratch and adapt over time. While such heuristics work well in practice, none of them can provide theoretical guarantees. Our objective is to modify an event-related potential (ERP) paradigm to work in unison with the machine learning decoder, and thus to achieve a reliable unsupervised calibrationless decoding with a guarantee to recover the true class means.MethodWe introduce learning from label proportions (LLP) to the BCI community as a new unsupervised, and easy-to-implement classification approach for ERP-based BCIs. The LLP estimates the mean target and non-target responses based on known proportions of these two classes in different groups of the data. We present a visual ERP speller to meet the requirements of LLP. For evaluation, we ran simulations on artificially created data sets and conducted an online BCI study with 13 subjects performing a copy-spelling task.ResultsTheoretical considerations show that LLP is guaranteed to minimize the loss function similar to a corresponding supervised classifier. LLP performed well in simulations and in the online application, where 84.5% of characters were spelled correctly on average without prior calibration.SignificanceThe continuously adapting LLP classifier is the first unsupervised decoder for ERP BCIs guaranteed to find the optimal decoder. This makes it an ideal solution to avoid tedious calibration sessions. Additionally, LLP works on complementary principles compared to existing unsupervised methods, opening the door for their further enhancement when combined with LLP.

Improving zero-training brain-computer interfaces by mixing model estimators

Objective. Brain-computer interfaces (BCI) based on event-related potentials (ERP) incorporate a decoder to classify recorded brain signals and subsequently select a control signal that drives a computer application. Standard supervised BCI decoders require a tedious calibration procedure prior to every session. Several unsupervised classification methods have been proposed that tune the decoder during actual use and as such omit this calibration. Each of these methods has its own strengths and weaknesses. Our aim is to improve overall accuracy of ERP-based BCIs without calibration. Approach. We consider two approaches for unsupervised classification of ERP signals. Learning from label proportions (LLP) was recently shown to be guaranteed to converge to a supervised decoder when enough data is available. In contrast, the formerly proposed expectation maximization (EM) based decoding for ERP–BCI does not have this guarantee. However, while this decoder has high variance due to random initialization of its parameters, it obtains a higher accuracy faster than LLP when the initialization is good. We introduce a method to optimally combine these two unsupervised decoding methods, letting one method’s strengths compensate for the weaknesses of the other and vice versa. The new method is compared to the aforementioned methods in a resimulation of an experiment with a visual speller. Main results. Analysis of the experimental results shows that the new method exceeds the performance of the previous unsupervised classification approaches in terms of ERP classification accuracy and symbol selection accuracy during the spelling experiment. Furthermore, the method shows less dependency on random initialization of model parameters and is consequently more reliable. Significance. Improving the accuracy and subsequent reliability of calibrationless BCIs makes these systems more appealing for frequent use.

Learning With Label Proportions via NPSVM.

Recently, learning from label proportions (LLPs), which seeks generalized instance-level predictors merely based on bag-level label proportions, has attracted widespread interest. However, due to its weak label scenario, LLP usually falls into a transductive learning framework accounting for an intractable combinatorial optimization issue. In this paper, we propose a brand new algorithm, called LLPs via nonparallel support vector machine (LLP-NPSVM), to facilitate this dilemma. To harness satisfactory data adaption, instead of transductive learning fashion, our scheme determined instance labels according to two nonparallel hyper-planes under the supervision of label proportion information. In a geometrical view, our approach can be interpreted as an alternative competitive method benefiting from large margin clustering. In practice, LLP-NPSVM can be efficiently addressed by applying two fast sequential minimal optimization paths iteratively. To rationally support the effectiveness of our method, finite termination and monotonic decrease of the proposed LLP-NPSVM procedure were essentially analyzed. Various experiments demonstrated our algorithm enjoys rapid convergence and robust numerical stability, along with best accuracies among several recently developed methods in most cases.