Weak Learning Algorithm Research Articles

Bagging-based cross-media retrieval algorithm

It is hard to come up with a strong learning algorithm with high cross-media retrieval accuracy, but finding a weak learning algorithm with slightly higher accuracy than random prediction is simple. This paper proposes an innovative Bagging-based cross-media retrieval algorithm (called BCMR) based on this concept. First, we use bootstrap sampling to take a random sample from the original set. The amount of sample abstracted by bootstrapping is set to be the same as the amount of sample abstracted by the original dataset. Secondly, 50 bootstrap replicates are used to train 50 weak classifiers. In our experiments, we used homogeneous individual classifiers and eight different baseline methods. Last but not least, we generate the final strong classifier from 50 weak classifiers using the sample voting integration strategy. Using collective wisdom, we can get rid of bad decisions, giving the integrated model a much better generalization ability. Extensive experiments on three datasets show that BCMR can significantly improve cross-media retrieval accuracy.

Ensemble Approach for Detecting COVID-19 Propaganda on Online Social Networks

COVID-19 affected the entire world due to the unavailability of the vaccine. The social distancing was a contributing factor that gave rise to the usage of Online Social Networks. It has been seen that people share the information that comes to them without verifying its source . One of the common forms of information that is disseminated that have a radical purpose is propaganda. Propaganda is organized and conscious method of molding conclusions and impacting an individual's contemplations to accomplish the ideal aim of proselytizer. For this paper, different propagandistic tweets were shared in the COVID-19 Era. Data regarding COVID-19 propaganda was extracted from Twitter. Labelling of data was performed manually using different propaganda identification techniques and Hybrid feature engineering was used to select the essential features. Ensemble machine learning classifiers were used for performing the binary classification. Adaboost shows an accuracy of 98.7%, which learns from a weak learning algorithm by updating the weights.

Modeling the organic matter of water using the decision tree coupled with bootstrap aggregated and least-squares boosting

The purpose of the work is to investigate the use of decision tree (DT) enhanced by bootstrap aggregates (Bag) and least-squares boosting (Lsboost) in modeling the organic matter of water according to its physicochemical parameters. An entire database of 500 samples of 21 physicochemical parameters, including organic matter, was used to build the DT, DT_Bag, and DT_Lsboost models. Training data (364 data points) is resampled using a bootstrap technique to form different training datasets to train different models. The models built were validated by a dataset of 91 samples. The predicted outputs obtained from the developed DT models are then combined by simple averaging. On the other hand, the data was also boosted with the Lsboost technical aid to increase the strength of a weak learning algorithm. The model trains the first weak learner with equal weight across all data points in the training set, then trains all other weak learners based on the updated weight aimed at the validation result to minimize the squared error medium. Good agreement between the predicted and experimental organic matter concentrations for the DT_Lsboost model was obtained (the correlation coefficient for the validation dataset was 0.9992), followed by the DT-Bag model with a correlation coefficient of 0.9949. The comparison between DT, DT_Bag, and DT_Lsboost revealed the superiority of the DT_Lsboost model (the mean root of the squared errors for the dataset were 0.1295 for the DT_Lsboost, 0.1664 for the DT_Bag, and 0.5444 for the DT). These results show that Lsboost technology dramatically improved the DT model. This result is also confirmed by the results of tests on models (interpolation data of 45 points). It should also be noted that the Bag technique was also very effective in optimizing the DT model, as the results obtained with this technique were very close to the DT_Lsboost model. • The concentration of organic matter in the water was predicted using various models. • The decision tree (DT) model was coupled by bootstrap aggregates (Bag) and least-squares boosting (Lsboost) techniques. • Significant Physico-chemical parameters on the organic matter were identified using the decision tree (DT) and the ensemble trees model (DT_Bag and DT_Lsboost). • The decision of tree (DT) and the ensemble trees model made it possible to highlight the significant interactions. • Bootstrap aggregates (Bag) and least-squares boosting (Lsboost) techniques have improved and developed the decision tree model.

An Efficient AdaBoost Algorithm with the Multiple Thresholds Classification

Adaptive boost (AdaBoost) is a prominent example of an ensemble learning algorithm that combines weak classifiers into strong classifiers through weighted majority voting rules. AdaBoost’s weak classifier, with threshold classification, tries to find the best threshold in one of the data dimensions, dividing the data into two categories-1 and 1. However, in some cases, this Weak Learning algorithm is not accurate enough, showing poor generalization performance and a tendency to over-fit. To solve these challenges, we first propose a new Weak Learning algorithm that classifies examples based on multiple thresholds, rather than only one, to improve its accuracy. Second, in this paper, we make changes to the weight allocation scheme of the Weak Learning algorithm based on the AdaBoost algorithm to use potential values of other dimensions in the classification process, while the theoretical identification is provided to show its generality. Finally, comparative experiments between the two algorithms on 18 datasets on UCI show that our improved AdaBoost algorithm has a better generalization effect in the test set during the training iteration.

Identification of Colorectal Cancer Using Near-Infrared Spectroscopy and Adaboost with Decision Stump

ABSTRACTRapid and objective detection of cancer is crucial for successful treatment. Near-infrared (NIR) spectroscopy is a vibrational technique capable of optically probing molecular changes associated with disease. The purpose of this study was to explore NIR spectroscopy for discriminating cancer from normal colorectal tissues. A total of 110 tissue samples from patients who underwent operations were characterized in this study. The popular ensemble technique AdaBoost was used to construct the diagnostic model. A decision stump was used as the weak learning algorithm. Adaboost with decision stump, an ensemble of weak classifiers, was compared with the most suitable single model, a strong classifier. Only the 20 most significant variables were selected as inputs for the model based on measured defined variable importance. Using an independent test set, the single strong classifier provided diagnostic accuracy of 89.1%, sensitivity of 100%, and specificity of 78.6%, whereas the ensemble of weak stumps provided accuracy of 96.3%, sensitivity of 96.3%, and specificity of 96.3% for distinguishing cancer from normal colorectal tissues. Therefore, NIR spectroscopy in combination with AdaBoost with decision stumps has demonstrated potential for rapid and objective diagnosis of colorectal cancer.

Conspiracies between learning algorithms, circuit lower bounds, and pseudorandomness

We prove several results giving new and stronger connections between learning theory, circuit complexity and pseudorandomness. Let ℭ be any typical class of Boolean circuits, and ℭ[s(n)] denote n-variable ℭ-circuits of size ≤ s(n). We show:Learning Speedups. If ℭ[poly(n)] admits a randomized weak learning algorithm under the uniform distribution with membership queries that runs in time 2n/nω(1), then for every k ≥ 1 and e > 0 the class ℭ[nk] can be learned to high accuracy in time O(2ne). There is e > 0 such that ℭ[2ne] can be learned in time 2n/nω(1) if and only if ℭ[poly(n)] can be learned in time 2(logn)o(1).Equivalences between Learning Models. We use learning speedups to obtain equivalences between various randomized learning and compression models, including sub-exponential time learning with membership queries, sub-exponential time learning with membership and equivalence queries, probabilistic function compression and probabilistic average-case function compression.A Dichotomy between Learnability and Pseudorandomness. In the non-uniform setting, there is non-trivial learning for ℭ[poly(n)] if and only if there are no exponentially secure pseudorandom functions computable in ℭ[poly(n)].Lower Bounds from Nontrivial Learning. If for each k ≥ 1, (depth-d)-ℭ[nk] admits a randomized weak learning algorithm with membership queries under the uniform distribution that runs in time 2n/nω(1), then for each k ≥ 1, BPE n (depth-d)-ℭ[nk]. If for some e > 0 there are P-natural proofs useful against ℭ[2ne], then ZPEXP n ℭ[poly(n)].Karp-Lipton Theorems for Probabilistic Classes. If there is a k > 0 such that BPE ⊆ i.o.Circuit[nk], then BPEXP ⊆ i.o.EXP/O(log n). If ZPEXP ⊆ i.o.Circuit[2n/3], then ZPEXP ⊆ i.o.ESUBEXP.Hardness Results for MCSP. All functions in non-uniform NC1 reduce to the Minimum Circuit Size Problem via truth-table reductions computable by TC0 circuits. In particular, if MCSP e TC0 then NC1 = TC0.

Quantum AdaBoost algorithm via cluster state

The principle and theory of quantum computation are investigated by researchers for many years, and further applied to improve the efficiency of classical machine learning algorithms. Based on physical mechanism, a quantum version of AdaBoost (Adaptive Boosting) training algorithm is proposed in this paper, of which purpose is to construct a strong classifier. In the proposed scheme with cluster state in quantum mechanism is to realize the weak learning algorithm, and then update the corresponding weight of examples. As a result, a final classifier can be obtained by combining efficiently weak hypothesis based on measuring cluster state to reweight the distribution of examples.

The Application of Boosting Algorithm in Data Mining

An algorithm which named Boosting algorithm occurred in the last ten years. It can raise the learning algorithm accuracy with multiple learning and obviously improve the efficiency of learning algorithm by adopting the principle of “comprehensive optimizing”. In addition, it can effectively develop the “weak learning algorithm” with low efficiency into a “strong learning algorithm” with high efficiency. The boosting, a new learning method to integration machine, is based in learning theory and show its good qualities in many fields. The paper elaborates and summarizes the basic ideas of Boosting algorithm and applies it into Data Mining (DM). In recent years, the DM has been extensively adopted out of labs such as in commerce, technological research and engineering technology. In this background, the paper tries improving traditional DM algorithm to solve these problems in industrial application.

On Boosting Sparse Parities

While boosting has been extensively studied, considerablyless attention has been devoted to the task of designing good weaklearning algorithms. In this paper we consider the problem of designing weak learners thatare especially adept to the boosting procedure and specifically the AdaBoost algorithm. First we describe conditions desirable for a weak learning algorithm. We then propose using sparse parity functions as weak learners, which have many of our desired properties, as weak learners in boosting. Our experimental tests show the proposed weak learners tobe competitive with the most widely used ones: decisionstumps and pruned decision trees.

Advance and Prospects of AdaBoost Algorithm

AdaBoost is one of the most excellent Boosting algorithms. It has a solid theoretical basis and has made great success in practical applications. AdaBoost can boost a weak learning algorithm with an accuracy slightly better than random guessing into an arbitrarily accurate strong learning algorithm, bringing about a new method and a new design idea to the design of learning algorithm. This paper first introduces how Boosting, just a conjecture when proposed, was proved right, and how this proof led to the origin of AdaBoost algorithm. Second, training and generalization error of AdaBoost are analyzed to explain why AdaBoost can successfully improve the accuracy of a weak learning algorithm. Third, different theoretical models to analyze AdaBoost are given. Meanwhile, many variants derived from these models are presented. Fourth, extensions of binary-class AdaBoost to multiclass AdaBoost are described. Besides, applications of AdaBoost algorithm are also introduced. Finally, interested directions which need to be further studied are discussed. For Boosting theory, these directions include deducing a tighter generalization error bound and finding a more precise weak learning condition in multiclass problem. For AdaBoost, the stopping conditions, the way to enhance anti-noise capability and how to improve the accuracy by optimizing the diversity of the base classifiers, are good questions to be in-depth researched.

KKL, Kruskal--Katona, and Monotone Nets

We generalize the Kahn--Kalai--Linial (KKL) theorem to random walks on Cayley and Schreier graphs, making progress on an open problem of Hoory, Linial, and Wigderson. In our generalization, the underlying group need not be abelian so long as the generating set is a union of conjugacy classes. An example corollary is that for every $f : \binom{[n]}{k} \to \{0,1\}$ with ${\bf E}[f]$ and $k/n$ bounded away from $0$ and $1$, there is a pair $1 \leq i < j \leq n$ such that ${\cal I}_{ij}(f) \geq \Omega(\frac{\log n}{n})$. Here ${\cal I}_{ij}(f)$ denotes the “influence” on $f$ of swapping the $i$th and $j$th coordinates. Using this corollary we obtain a “robust” version of the Kruskal--Katona theorem: Given a constant-density subset $A$ of a middle slice of the Hamming $n$-cube, the density of $\partial A$ is greater by at least $\Omega(\frac{\log n}{n})$, unless $A$ is noticeably correlated with a single coordinate. As an application of these results, we show that the set of functions $\{0, 1, x_1, \dots, x_n, \mathrm{Maj}\}$ is a $(1/2 - \gamma)$-net for the set of all $n$-bit monotone Boolean functions, where $\gamma = \Omega(\frac{\log n}{\sqrt{n}})$. This distance is optimal for polynomial-size nets and gives an optimal weak-learning algorithm for monotone functions under the uniform distribution, solving a problem of Blum, Burch, and Langford.

Detecting the Authors of Texts by Neural Network Committee Machines

This paper proposes a means of using a boosting by filtering algorithm in artificial neural networks to identify the author of a text. This approach involves filtering the training examples by different versions of a weak learning algorithm. It assures the availability of a large source of examples, with the examples being either discarded or kept during training. An advantage of this approach is that it allows for a small memory requirement. Once the network has been trained, its hidden layer activations are recorded as a representation of the selected lexical descriptors of an author. This stored information can then be used to identify the texts written by the same author. Texts studied are literary works of two Bosnian writers, Ivo Andrić (1892-1975) and M. Meša Selimović (1910-1982). The data collected by counting syntactic characteristics in 1466 paragraphs of "na drini ćupria" by Ivo Andrić, and "derviš i smirt" by M. Meša Selimović each.

An Improved LS-SVR Ensemble Learning in Internet Traffic Prediction

In this paper, we present a new method for internet traffic forecasting based on a boosting LS-SVR algorithm. AdaBoost has been proved to be an effective method for improving the performance of weak learning algorithms and widely applied to classification problems. Inspired by it, we use LS-SVR to complete the initial training; and pay more attention on the “high error areas” in the time series; then, we use an ensemble learning algorithm to learn these areas.

Intrusion detection method based on rough set and adaptive boost

To solve the problem of high rate of false negatives and false positives of IDS, an intrusion detection method was proposed in this paper, which combined Rough Set and Adaboost algorithm. Rough set was used to reduce amount of Adaboost' training data and improve running speed. Adaboost was a learning algorithm for constructing accurate classifiers. It can obtain a strong learning algorithm by combining a series of weak learning algorithms through some rules. The experimental results show that the model has high detection rate and detection efficiency.

Boosting in the presence of noise

Boosting algorithms are procedures that “boost” low-accuracy weak learning algorithms to achieve arbitrarily high accuracy. Over the past decade boosting has been widely used in practice and has become a major research topic in computational learning theory. In this paper we study boosting in the presence of random classification noise, giving both positive and negative results. We show that a modified version of a boosting algorithm due to Mansour and McAllester (J. Comput. System Sci. 64(1) (2002) 103) can achieve accuracy arbitrarily close to the noise rate. We also give a matching lower bound by showing that no efficient black-box boosting algorithm can boost accuracy beyond the noise rate (assuming that one-way functions exist). Finally, we consider a variant of the standard scenario for boosting in which the “weak learner” satisfies a slightly stronger condition than the usual weak learning guarantee. We give an efficient algorithm in this framework which can boost to arbitrarily high accuracy in the presence of classification noise.

The Dynamics of AdaBoost: Cyclic Behavior and Convergence of Margins

In order to study the convergence properties of the AdaBoost algorithm, we reduce AdaBoost to a nonlinear iterated map and study the evolution of its weight vectors. This dynamical systems approach allows us to understand AdaBoost's convergence properties completely in certain cases; for these cases we find stable cycles, allowing us to explicitly solve for AdaBoost's output.Using this unusual technique, we are able to show that AdaBoost does not always converge to a maximum margin combined classifier, answering an open question. In addition, we show that non-optimal AdaBoost (where the weak learning algorithm does not necessarily choose the best weak classifier at each iteration) may fail to converge to a maximum margin classifier, even if optimal AdaBoost produces a maximum margin. Also, we show that if AdaBoost cycles, it cycles among support vectors, i.e., examples that achieve the same smallest margin.

Three papers on boosting: an introduction

The notion of boosting originated in the Machine Learning literature in the 1980's [VALIANT, L.G. (1984). A theory of the learnable. In Proc. 16th Annual ACM Symposium on Theory of Computing 436-445. ACM Press, New York]. The goal of boosting is to improve the generalization performance of weak (or base) learning algorithms by combining them in a certain way. The first algorithm of this type was discovered by Schapire [SCHAPIRE, R.E. (1990). The strength of weak learnability. Machine Learning 5 197-227] and then the second one by Freund [FREUND, Y. (1995). Boosting a weak learning algorithm by majority. Inform. and Comput. 121 256-285]. Schapire and Freund [FREUND, Y. and Schapire. R.E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. System. Sci. 55 119-139] came up with the idea of a more practical version of boosting and invented the algorithm called AdaBoost that combines simple classification rules into much more powerful and precise classification algorithms. For a fixed number of iterations, AdaBoost runs the weak (or base) learning algorithm on resampled original data sets in a sequential manner and then combines the resulting learning algorithms through a weighted summation at the end of the iteration. Gradually, it became clear that AdaBoost is a special case of a more general statistical methodology of combining simple estimates in classification or regression into more complex and more precise ones. The study of statistical properties of these methods has been conducted in several directions since then in both the machine learning and statistics communities. The problem of consistency of AdaBoost is posed by Leo Breiman in the first paper in this issue of The Annals of Statistics. Breiman studies one ingredient needed to prove the consistency, the convergence properties of AdaBoost as a numerical method in the population case. This paper has been circulated for a couple of years as a preprint and its results were also covered in the Wald Lectures delivered by Breiman at the IMS Annual Meeting in 2002 in Banff, Canada. The papers by Jiang, Lugosi and Vayatis, and Zhang, published below with discussions, consider various versions of boosting and give answers to the consistency question posed by Breiman.

A geometric approach to leveraging weak learners

AdaBoost is a popular and effective leveraging procedure for improving the hypotheses generated by weak learning algorithms. AdaBoost and many other leveraging algorithms can be viewed as performing a constrained gradient descent over a potential function. At each iteration the distribution over the sample given to the weak learner is proportional to the direction of steepest descent. We introduce a new leveraging algorithm based on a natural potential function. For this potential function, the direction of steepest descent can have negative components. Therefore, we provide two techniques for obtaining suitable distributions from these directions of steepest descent. The resulting algorithms have bounds that are incomparable to AdaBoost's. The analysis suggests that our algorithm is likely to perform better than AdaBoost on noisy data and with weak learners returning low confidence hypotheses. Modest experiments confirm that our algorithm can perform better than AdaBoost in these situations.

General Bounds on Statistical Query Learning and PAC Learning with Noise via Hypothesis Boosting

We derive general bounds on the complexity of learning in the statistical query (SQ) model and in the PAC model with classification noise. We do so by considering the problem of boosting the accuracy of weak learning algorithms which fall within the SQ model. This new model was introduced by Kearns to provide a general framework for efficient PAC learning in the presence of classification noise. We first show a general scheme for boosting the accuracy of weak SQ learning algorithms, proving that weak SQ learning is equivalent to strong SQ learning. The boosting is efficient and is used to show our main result of the first general upper bounds on the complexity of strong SQ learning. Since all SQ algorithms can be simulated in the PAC model with classification noise, we also obtain general upper bounds on learning in the presence of classification noise for classes which can be learned in the SQ model.

A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting

In the first part of the paper we consider the problem of dynamically apportioning resources among a set of options in a worst-case on-line framework. The model we study can be interpreted as a broad, abstract extension of the well-studied on-line prediction model to a general decision-theoretic setting. We show that the multiplicative weight-update Littlestone–Warmuth rule can be adapted to this model, yielding bounds that are slightly weaker in some cases, but applicable to a considerably more general class of learning problems. We show how the resulting learning algorithm can be applied to a variety of problems, including gambling, multiple-outcome prediction, repeated games, and prediction of points in Rn. In the second part of the paper we apply the multiplicative weight-update technique to derive a new boosting algorithm. This boosting algorithm does not require any prior knowledge about the performance of the weak learning algorithm. We also study generalizations of the new boosting algorithm to the problem of learning functions whose range, rather than being binary, is an arbitrary finite set or a bounded segment of the real line.