RAndom k-labELsets Research Articles

Prediction of Drug Pathway-based Disease Classes using Multiple Properties of Drugs

Background: Drug repositioning now is an important research area in drug discovery as it can accelerate the procedures of discovering novel effects of existing drugs. However, it is challenging to screen out possible effects for given drugs. Designing computational methods are a quick and cheap way to complete this task. Most existing computational methods infer the relationships between drugs and diseases. The pathway-based disease classification reported in KEGG provides us a new way to investigate drug repositioning as such classification can be applied to drugs. A predicted class of a given drug suggests latent diseases it can treat. Objective: The purpose of this study is to set up efficient multi-label classifiers to predict the classes of drugs. Methods: We adopt three types of drug information to generate drug features, including drug pathway information, label information and drug network. For the first two types, drugs are first encoded into binary vectors, which are further processed by singular value decomposition. For the third type, the network embedding algorithm, Mashup, is employed to yield drug features. Above features are combined and fed into RAndom k-labELsets (RAKEL) to construct multi-label classifiers, where support vector machine is selected as the base classification algorithm. Results: The ten-fold cross-validation results show that the classifiers provide high performance with accuracy higher than 0.95 and absolute true higher than 0.92. The case study indicates the novel effects of three drugs, i.e., they may treat new diseases. Conclusion: The proposed classifiers have high performance and are superiority to the classifiers with other classic algorithms and drug information. Furthermore, they have the ability to discover new effects of drugs.

PredictEFC: a fast and efficient multi-label classifier for predicting enzyme family classes

BackgroundEnzymes play an irreplaceable and important role in maintaining the lives of living organisms. The Enzyme Commission (EC) number of an enzyme indicates its essential functions. Correct identification of the first digit (family class) of the EC number for a given enzyme is a hot topic in the past twenty years. Several previous methods adopted functional domain composition to represent enzymes. However, it would lead to dimension disaster, thereby reducing the efficiency of the methods. On the other hand, most previous methods can only deal with enzymes belonging to one family class. In fact, several enzymes belong to two or more family classes.ResultsIn this study, a fast and efficient multi-label classifier, named PredictEFC, was designed. To construct this classifier, a novel feature extraction scheme was designed for processing functional domain information of enzymes, which counting the distribution of each functional domain entry across seven family classes in the training dataset. Based on this scheme, each training or test enzyme was encoded into a 7-dimenion vector by fusing its functional domain information and above statistical results. Random k-labelsets (RAKEL) was adopted to build the classifier, where random forest was selected as the base classification algorithm. The two tenfold cross-validation results on the training dataset shown that the accuracy of PredictEFC can reach 0.8493 and 0.8370. The independent test on two datasets indicated the accuracy values of 0.9118 and 0.8777.ConclusionThe performance of PredictEFC was slightly lower than the classifier directly using functional domain composition. However, its efficiency was sharply improved. The running time was less than one-tenth of the time of the classifier directly using functional domain composition. In additional, the utility of PredictEFC was superior to the classifiers using traditional dimensionality reduction methods and some previous methods, and this classifier can be transplanted for predicting enzyme family classes of other species. Finally, a web-server available at http://124.221.158.221/ was set up for easy usage.

PMPTCE-HNEA: Predicting Metabolic Pathway Types of Chemicals and Enzymes with a Heterogeneous Network Embedding Algorithm

Background: Metabolic chemical reaction is one of the main types of fundamental processes to maintain life. Generally, each reaction needs an enzyme. The metabolic pathway collects a series of chemical reactions at the system level. As compounds and enzymes are two important components in each metabolic pathway, identification of metabolic pathways that a given compound or enzyme can participate is the first important step for understanding the mechanism of metabolic pathways. Objective: The purpose of this study was to build efficient computational methods to predict the metabolic pathways of compounds and enzymes. Methods: Novel multi-label classifiers were proposed to identify metabolic pathway types, reported in KEGG, of compounds and enzymes. Three heterogeneous networks defining compounds and enzymes as nodes were constructed. To extract more informative features of compounds and enzymes, we generalized the powerful network embedding algorithm, Mashup, to its heterogeneous network version, named MashupH. RAndom k-labELsets (RAKEL) was employed to build the classifiers and support vector machine or random forest was selected as the base classification algorithm. Results: The 10-fold cross-validation results indicated the good performance of the proposed classifiers and such performance was superior to the previous classifier that adopted features yielded by Mashup. Furthermore, some key parameters of MashupH that might contribute to or influence the classifiers were analyzed. Conclusion: The features yielded by MashupH were more informative than those produced by Mashup on heterogeneous networks. This was the main reason the new classifiers were superior to those using features yielded by Mashup.

Improved RAkEL's Fault Diagnosis Method for High-Speed Train Traction Transformer.

The traction system is very important to ensure the safe operation of high-speed trains, and the failure of the traction transformer is the most likely fault in the traction system. Fault diagnosis in actual work relies largely on manual experience. This paper proposes an improved RAkEL (Random k-Labelsets) algorithm for the fault diagnosis of high-speed train traction transformers. Firstly, this article starts from the large amount of "sleeping" fault maintenance data accumulated by the railway department, takes a single maintenance record as an instance, uses specific monitoring values to construct an instance vector, and uses the fault phenomena corresponding to the monitoring indicators as labels. Then, this paper improves the step of selecting k-labelsets in RAkEL, and extracts associated faults using the Relief algorithm. Finally, this paper excavates and uses the association rules between data and faults to identify traction transformer faults. The results showed that the improved RAkEL diagnostic method had a significant improvement in the evaluation indicators. Compared with other multi-label classification algorithms, including BR (Binary Relevance) and CLR (Calibrated Label Ranking), this method performs well on multiple evaluation indicators. It can further help engineers perform timely maintenance work in the future.

An Ordinal Multi-Dimensional Classification (OMDC) for Predictive Maintenance

Predictive Maintenance is a type of condition-based maintenance that assesses the equipment's states and estimates its failure probability and when maintenance should be performed. Although machine learning techniques have been frequently implemented in this area, the existing studies disregard to the natural order between the target attribute values of the historical sensor data. Thus, these methods cause losing the inherent order of the data that positively affects the prediction performances. To deal with this problem, a novel approach, named Ordinal Multi-dimensional Classification (OMDC), is proposed for estimating the conditions of a hydraulic system's four components by taking into the natural order of class values. To demonstrate the prediction ability of the proposed approach, eleven different multi-dimensional classification algorithms (traditional Binary Relevance (BR), Classifier Chain (CC), Bayesian Classifier Chain (BCC), Monte Carlo Classifier Chain (MCC), Probabilistic Classifier Chain (PCC), Classifier Dependency Network (CDN), Classifier Trellis (CT), Classifier Dependency Trellis (CDT), Label Powerset (LP), Pruned Sets (PS), and Random k-Labelsets (RAKEL)) were implemented using the Ordinal Class Classifier (OCC) algorithm. Besides, seven different classification algorithms (Multilayer Perceptron (MLP), Support Vector Machine (SVM), k-Nearest Neighbour (kNN), Decision Tree (C4.5), Bagging, Random Forest (RF), and Adaptive Boosting (AdaBoost)) were chosen as base learners for the OCC algorithm. The experimental results present that the proposed OMDC approach using binary relevance multi-dimensional classification methods predicts the conditions of a hydraulic system's multiple components with high accuracy. Also, it is clearly seen from the results that the OMDC models that utilize ensemble-based classification algorithms give more reliable prediction performances with an average Hamming score of 0.853 than the others that use traditional algorithms as base learners.

Improved multi-label classifiers for predicting protein subcellular localization.

Protein functions are closely related to their subcellular locations. At present, the prediction of protein subcellular locations is one of the most important problems in protein science. The evident defects of traditional methods make it urgent to design methods with high efficiency and low costs. To date, lots of computational methods have been proposed. However, this problem is far from being completely solved. Recently, some multi-label classifiers have been proposed to identify subcellular locations of human, animal, Gram-negative bacterial and eukaryotic proteins. These classifiers adopted the protein features derived from gene ontology information. Although they provided good performance, they can be further improved by adopting more powerful machine learning algorithms. In this study, four improved multi-label classifiers were set up for identification of subcellular locations of the above four protein types. The random k-labelsets (RAKEL) algorithm was used to tackle proteins with multiple locations, and random forest was used as the basic prediction engine. All classifiers were tested by jackknife test, indicating their high performance. Comparisons with previous classifiers further confirmed the superiority of the proposed classifiers.

Identifying Functions of Proteins in Mice With Functional Embedding Features.

In current biology, exploring the biological functions of proteins is important. Given the large number of proteins in some organisms, exploring their functions one by one through traditional experiments is impossible. Therefore, developing quick and reliable methods for identifying protein functions is necessary. Considerable accumulation of protein knowledge and recent developments on computer science provide an alternative way to complete this task, that is, designing computational methods. Several efforts have been made in this field. Most previous methods have adopted the protein sequence features or directly used the linkage from a protein–protein interaction (PPI) network. In this study, we proposed some novel multi-label classifiers, which adopted new embedding features to represent proteins. These features were derived from functional domains and a PPI network via word embedding and network embedding, respectively. The minimum redundancy maximum relevance method was used to assess the features, generating a feature list. Incremental feature selection, incorporating RAndom k-labELsets to construct multi-label classifiers, used such list to construct two optimum classifiers, corresponding to two key measurements: accuracy and exact match. These two classifiers had good performance, and they were superior to classifiers that used features extracted by traditional methods.

Multiple Protein Subcellular Locations Prediction Based on Deep Convolutional Neural Networks with Self-Attention Mechanism.

As an important research field in bioinformatics, protein subcellular location prediction is critical to reveal the protein functions and provide insightful information for disease diagnosis and drug development. Predicting protein subcellular locations remains a challenging task due to the difficulty of finding representative features and robust classifiers. Many feature fusion methods have been widely applied to tackle the above issues. However, they still suffer from accuracy loss due to feature redundancy. Furthermore, multiple protein subcellular locations prediction is more complicated since it is fundamentally a multi-label classification problem. The traditional binary classifiers or even multi-class classifiers cannot achieve satisfactory results. This paper proposes a novel method for protein subcellular location prediction with both single and multiple sites based on deep convolutional neural networks. Specifically, we first obtain the integrated features by simultaneously considering the pseudo amino acid, amino acid index distribution, and physicochemical property. We then adopt deep convolutional neural networks to extract high-dimensional features from the fused feature, removing the redundant preliminary features and gaining better representations of the raw sequences. Moreover, we use the self-attention mechanism and a customized loss function to ensure that the model is more inclined to positive data. In addition, we use randomk-label sets to reduce the number of prediction labels. Meanwhile, we employ a hybrid strategy of over-sampling and under-sampling to tackle the data imbalance problem. We compare our model with three representative classification alternatives. The experiment results show that our model achieves the best performance in terms of accuracy, demonstrating the efficacy of the proposed model.

IMPT-FDNPL: Identification of Membrane Protein Types with Functional Domains and a Natural Language Processing Approach.

Membrane protein is an important kind of proteins. It plays essential roles in several cellular processes. Based on the intramolecular arrangements and positions in a cell, membrane proteins can be divided into several types. It is reported that the types of a membrane protein are highly related to its functions. Determination of membrane protein types is a hot topic in recent years. A plenty of computational methods have been proposed so far. Some of them used functional domain information to encode proteins. However, this procedure was still crude. In this study, we designed a novel feature extraction scheme to obtain informative features of proteins from their functional domain information. Such scheme termed domains as words and proteins, represented by its domains, as sentences. The natural language processing approach, word2vector, was applied to access the features of domains, which were further refined to protein features. Based on these features, RAndom k-labELsets with random forest as the base classifier was employed to build the multilabel classifier, namely, iMPT-FDNPL. The tenfold cross-validation results indicated the good performance of such classifier. Furthermore, such classifier was superior to other classifiers based on features derived from functional domains via one-hot scheme or derived from other properties of proteins, suggesting the effectiveness of protein features generated by the proposed scheme.

K-Labelsets Method for Multi-Label ECG Signal Classification Based on SE-ResNet

Cardiovascular diseases are the leading cause of death globally. The ECG is the most commonly used tool for diagnosing cardiovascular diseases, and, recently, there are a number of attempts to use deep learning to analyze ECG. In this study, we propose a method for performing multi-label classification on standard ECG (12-lead with duration of 10 s) data. We used the ResNet model that can perform residual learning as a base model for classification in this work, and we tried to improve performance through SE-ResNet, which added squeeze and excitation blocks on the plain ResNet. As a result of the experiment, it was possible to induce overall performance improvement through squeeze and excitation blocks. In addition, the random k-labelsets (RAKEL) algorithm was applied to improve the performance in multi-label classification problems. As a result, the model that applied soft voting through the RAKEL algorithm to SE-ResNet-34 represented the best performance, and the average performances according to the number of label divisions k were achieved 0.99%, 88.49%, 92.43%, 90.54%, and 93.40% in exact match, accuracy, F1-score, precision, and recall, respectively.

IMPTCE-Hnetwork: A Multilabel Classifier for Identifying Metabolic Pathway Types of Chemicals and Enzymes with a Heterogeneous Network

Metabolic pathway is an important type of biological pathways. It produces essential molecules and energies to maintain the life of living organisms. Each metabolic pathway consists of a chain of chemical reactions, which always need enzymes to participate in. Thus, chemicals and enzymes are two major components for each metabolic pathway. Although several metabolic pathways have been uncovered, the metabolic pathway system is still far from complete. Some hidden chemicals or enzymes are not discovered in a certain metabolic pathway. Besides the traditional experiments to detect hidden chemicals or enzymes, an alternative pipeline is to design efficient computational methods. In this study, we proposed a powerful multilabel classifier, called iMPTCE-Hnetwork, to uniformly assign chemicals and enzymes to metabolic pathway types reported in KEGG. Such classifier adopted the embedding features derived from a heterogeneous network, which defined chemicals and enzymes as nodes and the interactions between chemicals and enzymes as edges, through a powerful network embedding algorithm, Mashup. The popular RAndom k-labELsets (RAKEL) algorithm was employed to construct the classifier, which incorporated the support vector machine (polynomial kernel) as the basic classifier. The ten-fold cross-validation results indicated that such a classifier had good performance with accuracy higher than 0.800 and exact match higher than 0.750. Several comparisons were done to indicate the superiority of the iMPTCE-Hnetwork.

Self-evoluting framework of deep convolutional neural network for multilocus protein subcellular localization.

In the present paper, deep convolutional neural network (DCNN) is applied to multilocus protein subcellular localization as it is more suitable for multi-class classification. There are two main problems with this application. First, the appropriate features for correlation between multiple sites are hard to find. Second, the classifier structure is difficult to determine as it is greatly affected by the distribution of classified data. To solve these problems, a self-evoluting framework using DCNNs for multilocus protein subcellular localization is proposed. It has three characteristics that the previous algorithms do not. The first is that it combines the ant colony algorithm with the DCNN to form a self-evoluting algorithm for multilocus protein subcellular localization. The second is that it randomly groups subcellular sites using a limited random k-labelsets multi-label classification method. It also solves complex problems in a divide-and-conquer approach and proposes a flexible expansion model. The third is that it realizes the random selection feature extraction method in the positioning process and avoids the defects in individual feature extraction methods. The algorithm in the present paper is tested on the human database, and the overall correct rate is 67.17%, which is higher than that for the stacked self-encoder (SAE), support vector machine (SVM), random forest classifier (RF), or single deep convolutional neural network.Graphical abstract The algorithm mentioned in the present paper mainly includes four parts. They are protein sequence data preprocessing, integrated DCNN model construction, finding optimal DCNN combination by ant colony optimization, and protein subcellular localization for sequences. These parts are sequential relationships and the data obtained in the previous part is the basis for the latter part of the function. In the part of data preprocessing, the limited RAkEL multi-label classification method is used to randomly group subcellular sites. At the same time, the feature fusion of protein sequences is carried out by using multiple feature extraction methods. Each combination including features and sites information corresponds to a DCNN model. In the part of finding optimal DCNN combination by ant colony optimization, the main purpose is to find the best combination of DCNN models through the global optimization ability of the ant colony algorithm. The positioning of sequences is mainly to obtain multilocus subcellular localization by the optimal model combination.

IMPT-FRAKEL: A Simple Multi-label Web-server that Only Uses Fingerprints to Identify which Metabolic Pathway Types Compounds can Participate In

Background: Metabolic pathway is one of the most basic biological pathways in living organisms. It consists of a series of chemical reactions and provides the necessary molecules and energies for organisms. To date, lots of metabolic pathways have been detected. However, there still exist hidden participants (compounds and enzymes) for some metabolic pathways due to the complexity and diversity of metabolic pathways. It is necessary to develop quick, reliable, and non-animal-involved prediction model to recognize metabolic pathways for any compound. Methods: In this study, a multi-label classifier, namely iMPT-FRAKEL, was developed for identifying which metabolic pathway types that compounds can participate in. Compounds and 12 metabolic pathway types were retrieved from KEGG. Each compound was represented by its fingerprints, which was the most widely used form for representing compounds and can be extracted from its SMILES format. A popular multi-label classification scheme, Random k-Labelsets (RAKEL) algorithm, was adopted to build the classifier. Classic machine learning algorithm, Support Vector Machine (SVM) with RBF kernel, was selected as the basic classification algorithm. Ten-fold cross-validation was used to evaluate the performance of the iMPT-FRAKEL. In addition, a web-server version of such classifier was set up, which can be assessed at http://cie.shmtu.edu.cn/impt/index. Results: iMPT-FRAKEL yielded the accuracy of 0.804, exact match of 0.745 and hamming loss of 0.039. Comparison results indicated that such classifier was superior to other models, including models with Binary Relevance (BR) or other classification algorithms. Conclusion: The proposed classifier employed limited prior knowledge of compounds but gives satisfying performance for recognizing metabolic pathways of compounds.

Active k-labelsets ensemble for multi-label classification

The random k-labelsets ensemble (RAkEL) is a multi-label learning strategy that integrates many single-label learning models. Each single-label model is constructed using a label powerset (LP) technique based on a randomly generated size-k label subset. Although RAkEL can improve the generalization capability and reduce the complexity of the original LP method, the quality of the randomly generated label subsets could be low. On the one hand, the transformed classes may be difficult to separate in the feature space, negatively affecting the performance; on the other hand, the classes might be highly imbalanced, resulting in difficulties in using the existing single-label algorithms. To solve these problems, we propose an active k-labelsets ensemble (ACkEL) paradigm. Borrowing the idea of active learning, a label-selection criterion is proposed to evaluate the separability and balance level of the classes transformed from a label subset. Subsequently, by randomly selecting the first label or label subset, the remaining ones are iteratively chosen based on the proposed criterion. ACkEL can be realized in both the disjoint and overlapping modes, which adopt pool-based and stream-based frameworks, respectively. Experimental comparisons demonstrate the feasibility and effectiveness of the proposed methods.

Comparison of base classifiers for multi-label learning

Multi-label learning methods can be categorised into algorithm adaptation, problem transformation and ensemble methods. Some of these methods depend on a base classifier and the relationship is not well understood. In this paper the sensitivity of five problem transformation and two ensemble methods to four types of classifiers is studied. Their performance across 11 benchmark datasets is measured using 16 evaluation metrics. The best classifier is shown to depend on the method: Support Vector Machines (SVM) for binary relevance, classifier chains, calibrated label ranking, quick weighted multi-label learning and RAndom k-labELsets; k-Nearest Neighbours (k-NN) and Naïve Bayes (NB) for Hierarchy Of Multilabel classifiERs; and Decision Trees (DT) for ensemble of classifier chains. The statistical performance of a classifier is also found to be generally consistent across the metrics for any given method. Overall, DT and SVM have the best performance–computational time trade-off followed by k-NN and NB.

A classification model for prediction of clinical severity level using qSOFA medical score

PurposeThe purpose of this study is to develop an efficient prediction model using vital signs and standard medical score systems, which predicts the clinical severity level of the patient in advance based on the quick sequential organ failure assessment (qSOFA) medical score method.Design/methodology/approachTo predict the clinical severity level of the patient in advance, the authors have formulated a training dataset that is constructed based on the qSOFA medical score method. Further, along with the multiple vital signs, different standard medical scores and their correlation features are used to build and improve the accuracy of the prediction model. It is made sure that the constructed training set is suitable for the severity level prediction because the formulated dataset has different clusters each corresponding to different severity levels according to qSOFA score.FindingsFrom the experimental result, it is found that the inclusion of the standard medical scores and their correlation along with multiple vital signs improves the accuracy of the clinical severity level prediction model. In addition, the authors showed that the training dataset formulated from the temporal data (which includes vital signs and medical scores) based on the qSOFA medical scoring system has the clusters which correspond to each severity level in qSOFA score. Finally, it is found that RAndom k-labELsets multi-label classification performs better prediction of severity level compared to neural network-based multi-label classification.Originality/valueThis paper helps in identifying patient' clinical status.

An Energy Prediction Approach for a Nonintrusive Load Monitoring in Home Appliances

Home energy monitoring by appliance-level information can provide consumers awareness on energy saving. The system can be implemented through a smart meter which requires an efficient data analysis algorithm for providing an accurate energy consumption profile, the purpose for proper home energy management. This article proposes a set of data analysis procedures for extracting appliances power state from its power consumption data. The approach is based on multitarget classification, a new data learning framework for nonintrusive load monitoring. The procedures include: 1) partitioning the appliance power data into an effective number of power states using K-means clustering, and 2) determining the optimal number of power states using the Area Under the ROC Curve index. The design objective is to obtain the optimal predictive performance for identification of the appliance power state which could result in a proper power and energy prediction. Applying the multitarget classification algorithm of RAndom k-labELsets by disjoint subsets with the decision tree, the identification of appliance power state achieved F-score and accuracy values greater than 89% for high-power loads such as A/C and water heater. The normalized error values of power prediction outperformed the use of Factorial Hidden Markov Model and binary state modeling system.

IATC-NRAKEL: an efficient multi-label classifier for recognizing anatomical therapeutic chemical classes of drugs.

The anatomical therapeutic chemical (ATC) classification system plays an increasingly important role in drug repositioning and discovery. The correct identification of classes in each level of such system that a given drug may belong to is an essential problem. Several multi-label classifiers have been proposed in this regard. Although they provided satisfactory performance, the feature extraction procedures were still rough. More refined features may further improve the predicted quality. In this article, we provide a novel multi-label classifier, called iATC-NRAKEL, to predict drug ATC classes in the first level. To obtain more informative drug features, we employed the drug association information in STITCH and KEGG, which was organized by seven drug networks. The powerful network embedding algorithm, Mashup, was adopted to extract informative drug features. The obtained features were fed into the RAndom k-labELsets (RAKEL) algorithm with support vector machine as the basic classification algorithm to construct the classifier. The 10-fold cross-validation of the benchmark dataset with 3883 drugs showed that the accuracy and absolute true were 76.56 and 74.51%, respectively. The comparison results indicated that iATC-NRAKEL was much superior to all previous reported classifiers. Finally, the contribution of each network was analyzed. The codes of iATC-NRAKEL are available at https://github.com/zhou256/iATC-NRAKEL.

Predicting emergency department orders with multilabel machine learning techniques and simulating effects on length of stay.

Emergency departments (EDs) continue to pursue optimal patient flow without sacrificing quality of care. The speed with which a healthcare provider receives pertinent information, such as results from clinical orders, can impact flow. We seek to determine if clinical ordering behavior can be predicted at triage during an ED visit. Using data available during triage, we trained multilabel machine learning classifiers to predict clinical orders placed during an ED visit. We benchmarked 4 classifiers with 2 multilabel learning frameworks that predict orders independently (binary relevance) or simultaneously (random k-labelsets). We evaluated algorithm performance, calculated variable importance, and conducted a simple simulation study to examine the effects of algorithm implementation on length of stay and cost. Aggregate performance across orders was highest when predicting orders independently with a multilayer perceptron (median F1 score = 0.56), but prediction frameworks that simultaneously predict orders for a visit enhanced predictive performance for correlated orders. Visit acuity was the most important predictor for most orders. Simulation results indicated that direct implementation of the model would increase ordering costs (from $21 to $45 per visit) but reduce length of stay (from 158 minutes to 151 minutes) over all visits. Simulated implementations of the predictive algorithm decreased length of stay but increased ordering costs. Optimal implementation of these predictions to reduce patient length of stay without incurring additional costs requires more exploration. It is possible to predict common clinical orders placed during an ED visit with data available at triage.

Automatic Classification of Open-Ended Questions: Check-All-That-Apply Questions

Text data from open-ended questions in surveys are challenging to analyze and are often ignored. Open-ended questions are important though because they do not constrain respondents’ answers. Where open-ended questions are necessary, often human coders manually code answers. When data sets are large, it is impractical or too costly to manually code all answer texts. Instead, text answers can be converted into numerical variables, and a statistical/machine learning algorithm can be trained on a subset of manually coded data. This statistical model is then used to predict the codes of the remainder. We consider open-ended questions where the answers are coded into multiple labels (all-that-apply questions). For example, in the open-ended question in our Happy example respondents are explicitly told they may list multiple things that make them happy. Algorithms for multilabel data take into account the correlation among the answer codes and may therefore give better prediction results. For example, when giving examples of civil disobedience, respondents talking about “minor nonviolent offenses” were also likely to talk about “crimes.” We compare the performance of two different multilabel algorithms (random k-labelsets [RAKEL], classifier chains [CC]) to the default method of binary relevance (BR) which applies single-label algorithms to each code separately. Performance is evaluated on data from three open-ended questions (Happy, Civil Disobedience, and Immigrant). We found weak bivariate label correlations in the Happy data (90th percentile: 7.6%), and stronger bivariate label correlations in the Civil Disobedience (90th percentile: 17.2%) and Immigrant (90th percentile: 19.2%) data. For the data with stronger correlations, we found both multilabel methods performed substantially better than BR using 0/1 loss (“at least one label is incorrect”) and had little effect when using Hamming loss (average error). For data with weak label correlations, we found no difference in performance between multilabel methods and BR. We conclude that automatic classification of open-ended questions that allow multiple answers may benefit from using multilabel algorithms for 0/1 loss. The degree of correlations among the labels may be a useful prognostic tool.