Low Learning Rate Research Articles

Fine-tuning Pretrained Transformers for Sentiment Analysis on Twitter Data

Due to the noise and informal language present in Twitter data, it is difficult to perform sentiment analysis on the platform. In recent years, a number of transformer models have been developed that can perform well in this type of task. This study aims to analyze the performance of these models on Twitter data. The study utilizes a publicly-available dataset of tweets with neutral, positive, or negative sentiment. It preprocesses the data and tokenizes it using WordPiece. Three transformer models are then tuned using the labeled tweets' pre-defined weights and the models' training weights from large language modeling projects. The models are trained on a 5-phased scale. The three models' performance was evaluated using various metrics, such as accuracy, recall, and F1 score. The results show that the models performed well overall, with ELECTRA leading the way with an accuracy of 85.8%, followed by XLNet and BERT with 84.3% and 84.5% accuracy, respectively. The study also looked into the hyperparameters' impact on the performance. It revealed that batch sizes and learning rates have a significant effect on the models' performance. The results indicate that the models performed better with larger batch sizes and lower learning rates. The study concluded that the three pre-trained transformer models, namely XLNet, ELECTRA, and BERT, were able to perform well in terms of their performance when it came to analyzing Twitter data. Their findings can be beneficial for those working in the field of sentiment analysis on social media platforms.

The Use of 3D Convolutional Autoencoder in Fault and Fracture Network Characterization

Conventional pattern recognition methods directly use 1D poststack data or 2D prestack data for the statistical pattern recognition of fault and fracture network, thereby ignoring the spatial structure information in 3D seismic data. As a result, the generated fault and fracture network is not distinguishable and has poor continuity. In this paper, a fault and fracture network characterization method based on 3D convolutional autoencoder is proposed. First, in the autoencoder training frame, 3D prestack data are used as input, and the 3D convolution operation is used to mine the spatial structure information to the maximum and gradually reduce the spatial dimension of the input. Then, the residual network is used to recover the input’s details and the corresponding spatial dimension. Lastly, the hidden features extracted by the encoders are recognized via k -means, SOM, and two-step clustering analysis. The validity of the method is verified by testing the seismic simulation data and applying real seismic data. The 3D convolution can directly process the seismic data and maximize the prestack texture attributes and spatial structure information provided by 3D seismic data without dimensionality reduction and other preprocessing operations. The interleaving convolution layer and residual block overcome low learning and accuracy rates due to the deepening of networks.

Learning Multi-Granularity Neural Network Encoding Image Classification Using DCNNs for Easter Africa Community Countries

Remote sensing scene classification is a fundamental responsibility of earth observation, aiming at identifying information granular for land cover classification. The multi-granular land use for multi-source remotely sensed image categories is now a principal task in remote sensing data augmentation and data selection. Understanding image representations are meaningful for the scene classification task. Training deep learning model-based approaches has to do with scene classification and brings about fantastic achievement. At the same time, these high-level approaches are computationally expensive and time–consuming. This paper introduces multi-granularity Neural Network Encoding architecture base on InceptionV3, InceptionReseNetV2, VGG16, and DenseNet201 architecture into remote sensing scene classification. To improve performance and to solve intra-class variation for multi-class scene issues remote sensing dataset. By using pre-trained CNN, activation function and ensemble learning have been adopting. InceptionV3 and VGG16 are used to extract features. InceptionResNetV2 use for fine-tuning, which consists of unfreezing the entre model and retraining the new data with a lower learning rate. The proposed fine-tune whole pre-trained model produces better results of test set up to 97.84 % than features extracted by InceptionResNetV2. Also, we use DCNNs ensemble average and weighted average to achieve better outcomes for 97.36% and 99.10%. In our experiments, we attempted to fine-tune the deep convolutional neural networks (DCNNs) training method for remote sensing scene classification on two public datasets UCM, SIRI-WHU, and one dataset I collected through the google earth engine from East Africa Community Countries (EACC) within nine classes within total 2112 labeled images. The results indicate that our proposed fine-tuning of the pre-trained model with few epochs and less computational time increases accuracy.

Neural Network for Handwriting Recognition

Today, in the digital age, the problem of pattern recognition is very relevant. In particular, the task of text recognition is important in banking, for the automatic reading of documents and their control; in video control systems, for example, to identify the license plate of a car that violated traffic rules; in security systems, for example, to check banknotes at an ATM and in many other areas. A large number of methods are known for solving the problem of pattern recognition, but the main advantage of neural networks over other methods is their learning ability. It is this feature that makes neural networks attractive to study. The article proposes a basic neural network model. The main algorithms are considered and a programming model is implemented in the Python programming language. In the course of research, the following shortcomings of the basic model were revealed: low learning rate (the number of correctly recognized digits in the first epochs of learning); retraining - the network has not learned to generalize the knowledge gained; low probability of recognition - 95.13%.To solve the above disadvantages, various techniques were used that increase the accuracy and speed of work, as well as reduce the effect of network retraining.

Autonomic responses to choice outcomes: Links to task performance and reinforcement-learning parameters

Previous work has shown that autonomic responses to choice feedback can predict subsequent decision-making. In this study, we tested whether skin conductance responses (SCRs) and heart rate (HR) decelerations following the presentation of choice outcomes predict Iowa Gambling Task performance in nonclinical participants (n = 64). We also examined how these signals related to parameters of a reinforcement-learning (RL) model. Feedback SCRs and HR decelerations were greater following outcomes that included losses and choices from the bad decks defined by their negative expected value. In addition, SCRs predicted task performance. A hierarchical Bayesian RL model indicated that greater feedback SCR for the bad decks compared to good decks was associated with stronger loss aversion and a lower learning rate, both of which predicted higher performance. These results suggest that feedback-related SCRs are linked to individual differences in outcome evaluation and learning processes that guide reinforcement-learning.

Imprecise action selection in substance use disorder: Evidence for active learning impairments when solving the explore-exploit dilemma

BackgroundSubstance use disorders (SUDs) are a major public health risk. However, mechanisms accounting for continued patterns of poor choices in the face of negative life consequences remain poorly understood. MethodsWe use a computational (active inference) modeling approach, combined with multiple regression and hierarchical Bayesian group analyses, to examine how treatment-seeking individuals with one or more SUDs (alcohol, cannabis, sedatives, stimulants, hallucinogens, and/or opioids; N = 147) and healthy controls (HCs; N = 54) make choices to resolve uncertainty within a gambling task. A subset of SUDs (N = 49) and HCs (N = 51) propensity-matched on age, sex, and verbal IQ were also compared to replicate larger group findings. ResultsResults indicate that: (a) SUDs show poorer task performance than HCs (p = 0.03, Cohen’s d = 0.33), with model estimates revealing less precise action selection mechanisms (p = 0.004, d = 0.43), a lower learning rate from losses (p = 0.02, d = 0.36), and a greater learning rate from gains (p = 0.04, d = 0.31); and (b) groups do not differ significantly in goal-directed information seeking. ConclusionsFindings suggest a pattern of inconsistent behavior in response to positive outcomes in SUDs combined with a tendency to attribute negative outcomes to chance. Specifically, individuals with SUDs fail to settle on a behavior strategy despite sufficient evidence of its success. These learning impairments could help account for difficulties in adjusting behavior and maintaining optimal decision-making during and after treatment.

Exploiting Retraining-Based Mixed-Precision Quantization for Low-Cost DNN Accelerator Design.

For successful deployment of deep neural networks (DNNs) on resource-constrained devices, retraining-based quantization has been widely adopted to reduce the number of DRAM accesses. By properly setting training parameters, such as batch size and learning rate, bit widths of both weights and activations can be uniformly quantized down to 4 bit while maintaining full precision accuracy. In this article, we present a retraining-based mixed-precision quantization approach and its customized DNN accelerator to achieve high energy efficiency. In the proposed quantization, in the middle of retraining, an additional bit (extra quantization level) is assigned to the weights that have shown frequent switching between two contiguous quantization levels since it means that both quantization levels cannot help to reduce quantization loss. We also mitigate the gradient noise that occurs in the retraining process by taking a lower learning rate near the quantization threshold. For the proposed novel mixed-precision quantized network (MPQ-network), we have implemented a customized accelerator using a 65-nm CMOS process. In the accelerator, the proposed processing elements (PEs) can be dynamically reconfigured to process variable bit widths from 2 to 4 bit for both weights and activations. The numerical results show that the proposed quantization can achieve 1.37 × better compression ratio for VGG-9 using CIFAR-10 data set compared with a uniform 4-bit (both weights and activations) model without loss of classification accuracy. The proposed accelerator also shows 1.29× of energy savings for VGG-9 using the CIFAR-10 data set over the state-of-the-art accelerator.

Reinforcement learning as an intermediate phenotype in psychosis? Deficits sensitive to illness stage but not associated with polygenic risk of schizophrenia in the general population

BackgroundSchizophrenia is a complex disorder in which the causal relations between risk genes and observed clinical symptoms are not well understood and the explanatory gap is too wide to be clarified without considering an intermediary level. Thus, we aimed to test the hypothesis of a pathway from molecular polygenic influence to clinical presentation occurring via deficits in reinforcement learning. MethodsWe administered a reinforcement learning task (Go/NoGo) that measures reinforcement learning and the effect of Pavlovian bias on decision making. We modelled the behavioural data with a hierarchical Bayesian approach (hBayesDM) to decompose task performance into its underlying learning mechanisms. Study 1 included controls (n = 29, F|M = 0.81), At Risk Mental State for psychosis (ARMS, n = 23, F|M = 0.35) and FEP (First-episode psychosis, n = 26, F|M = 0.18). Study 2 included healthy adolescents (n = 735, F|M = 1.06), 390 of whom had their polygenic risk scores for schizophrenia (PRSs) calculated. ResultsPatients with FEP showed significant impairments in overriding Pavlovian conflict, a lower learning rate and a lower sensitivity to both reward and punishment. Less widespread deficits were observed in ARMS. PRSs did not significantly predict performance on the task in the general population, which only partially correlated with measures of psychopathology. ConclusionsReinforcement learning deficits are observed in first episode psychosis and, to some extent, in those at clinical risk for psychosis, and were not predicted by molecular genetic risk for schizophrenia in healthy individuals. The study does not support the role of reinforcement learning as an intermediate phenotype in psychosis.

Altered Reinforcement Learning From Reward and Punishment in Anorexia Nervosa: Evidence From Computational Modeling

OBJECTIVES Anorexia nervosa (AN) is associated with altered sensitivity to reward and punishment. Few studies have investigated whether this results in aberrant learning. The ability to learn from rewarding and aversive experiences is essential for flexibly adapting to changing environments, yet individuals with AN tend to demonstrate cognitive inflexibility, difficulty set-shifting and altered decision-making. Deficient reinforcement learning may contribute to repeated engagement in maladaptive behavior. METHODS This study investigated learning in AN using a probabilistic associative learning task that separated learning of stimuli via reward from learning via punishment. Forty-two individuals with Diagnostic and Statistical Manual of Mental Disorders (DSM)-5 restricting-type AN were compared to 38 healthy controls (HCs). We applied computational models of reinforcement learning to assess group differences in learning, thought to be driven by violations in expectations, or prediction errors (PEs). Linear regression analyses examined whether learning parameters predicted BMI at discharge. RESULTS AN had lower learning rates than HC following both positive and negative PE (p < .02), and were less likely to exploit what they had learned. Negative PE on punishment trials predicted lower discharge BMI (p < .001), suggesting individuals with more negative expectancies about avoiding punishment had the poorest outcome. CONCLUSIONS This is the first study to show lower rates of learning in AN following both positive and negative outcomes, with worse punishment learning predicting less weight gain. An inability to modify expectations about avoiding punishment might explain persistence of restricted eating despite negative consequences, and suggests that treatments that modify negative expectancy might be effective in reducing food avoidance in AN.

Achieving grid parity of wind power in China – Present levelized cost of electricity and future evolution

China has adopted an ambitious plan for wind power to achieve grid parity with the on-grid price of coal-fired power in 2020. Whether this target can be achieved is a great concern for policy makers as well as potential investors. To address this issue, we first estimate the future levelized cost of electricity (LCOE) of wind power using a learning curve method, and then determine whether grid parity can be achieved by comparing it with the on-grid price of coal-fired power. Specially, the effect of carbon pricing on the grid-parity is explored, and a sensitivity analysis on how the discount rates, learning rates, and curtailment rates affect grid parity is conducted. The learning rate of onshore wind power is estimated using a panel dataset consisting of information of 2059 onshore wind projects in China from 2006 to 2015. Based on this learning rate, the future LCOE of Chinese onshore wind power from 2016 to 2025 is calculated. The results show that the LCOE of onshore wind power decreases by 13.91% from 0.40 RMB/kWh in 2016 to 0.34 RMB/kWh in 2025. By comparing the LCOE with the on-grid price of coal-fired power, the grid parity of onshore wind power may be achieved in 2019. With the implementation of the carbon pricing policy, the grid parity will be achieved earlier. More specifically, with the carbon price reaching 10, 35, and 60 RMB/t CO2, the grid parity can be achieved in 2019, 2017, and 2016, respectively. The results of the sensitivity analysis show that in the case of high discount rates, low learning rates, high curtailment rates, high O&M cost and low capacity factor, the grid parity will be delayed, and a high carbon price will be required to achieve the grid parity.

Biases in estimating the balance between model-free and model-based learning systems due to model misspecification

Accumulating studies support the existence of dual learning systems in decision making: one is the model-free learning system, which updates values based solely on experience; and the other is the model-based learning system, which calculates values using a complex environmental structure. A two-stage decision task and its computational model have been widely used to distinguish the effects of these systems on choices. However, the computational model is often used as a tool without doubting its assumptions. In this study, we examined the possible biases in model parameter estimation due to model misspecification of a computational model. In particular, we focused on two features related to choice behavior, the existence of which was implied by the actual choice data but has not been assumed in the widely used computational models. One feature is the forgetting process, which assumes a change in unchosen option values. The other feature is gradual perseveration, which assumes that actions are positively autocorrelated with multiple preceding actions. We simulated cases in which these features relate to the choice process, but the obtained data were fit using a model that does not assume these features. We revealed that such misspecification of a fitting model can cause systematic biases in the estimation of the relative contributions of the model-free and model-based systems, implying that previous findings using the standard computational model might have been distorted by some biases. The possibility of estimation biases discussed in this study is important because the assumptions of the forgetting process and gradual perseveration, which can be combined with any reinforcement learning model, are not included in most existing models. In addition, the discussed mechanisms of the biases are widely related to basic model parameters. Using experimental data from the two-stage decision task (N = 39), we examined the associations between obsessive compulsivity and the weighting parameter between the model-free and model-based systems. Although a negative association has been consistently reported, we could not replicate this association when we used the model with forgetting and gradual perseveration. Instead, we found that individuals with greater obsessive–compulsive tendencies had both lower learning rates and lower forgetting rates than those with lesser obsessive–compulsive tendencies.

Internet Game Overuse Is Associated With an Alteration of Fronto-Striatal Functional Connectivity During Reward Feedback Processing.

Internet gaming disorder is associated with abnormal reward processing in the reward circuit, which is known to interact with other brain regions during feedback learning. Kim et al. (1) observed that individuals with internet game overuse (IGO) exhibit altered behavior and neural activity for non-monetary reward, but not for monetary reward. Here, we extend our analysis of IGO to the functional connectivity of the reward network. Functional MRI data were obtained during a stimulus-response association learning task from 18 young males with IGO and 20 age-matched controls, where either monetary or non-monetary rewards were given as positive feedback for a correct response. Group differences in task-dependent functional connectivity were examined for the ventromedial prefrontal cortex (vmPFC) and ventral striatum (VS), which are known for reward evaluation and hedonic response processing, respectively, using a generalized form of the psychophysiological interaction approach. For non-monetary reward processing, no differences in functional connectivity were found. In contrast, for monetary reward, connectivity of the vmPFC with the left caudate nucleus was weaker for the IGO group relative to controls, while vmPFC connectivity with the right nucleus accumbens (NAcc) was elevated. The strength of vmPFC-NAcc functional connectivity appeared to be behaviorally relevant, because individuals with stronger vmPFC-NAcc connectivity showed lower learning rates for monetary reward. In addition, the IGO group showed weaker ventral striatum functional connectivity with various brain regions, including the right ventrolateral prefrontal cortex, dorsal anterior cingulate regions, and left pallidum. Thus, for monetary reward, the IGO group exhibited stronger functional connectivity within the brain regions involved in motivational salience, whereas they showed reduced functional connectivity the widely distributed brain areas involved in learning or attention. These differences in functional connectivity of reward networks, along with related behavioral impairments of reward learning, suggest that internet gaming disorder is associated with the increased incentive salience or “wanting” of addiction disorders, and may serve as the neurobiological mechanisms underlying the impaired goal-directed behavior.

Computer aided diagnosis with boosted learning for anomaly detection in microwave tomography

Most recently developed Computer Aided Diagnosis (CAD) systems and their related research is based on medical images that are usually obtained through conventional imaging techniques such as Magnetic Resonance Imaging (MRI), x-ray mammography, and ultrasound. With the development of a new imaging technology called Microwave Tomography Imaging (MTI), it has become inevitable to develop a CAD system that can show improved performance using the new format of data. In this paper, we propose the Multilayer Perceptron (MLP) model taking MT data as an input and we boost the learning process by using Dynamic Learning Rate (DLR) and momentum. In order to eliminate the indeterminate equation problem, we optimize the number of weights to the number of the training data. To make the model capable of escaping the local minima problem, we assign the learning rate dynamically. Since training a network with data having uneven distribution for the all possible classes can cause the network to be biased to the majority class, our model assign the low learning rate if unseen data belongs to the majority class. Along with this strategy, to speed up a back-propagation, the model employs the momentum optimizer to reduce the convergence time. In experiment, we train the model with two different datasets, 15 and 30, and evaluate the performance by the following measures; precision, recall, specificity, accuracy, and Matthews Correlation Coefficient (MCC). Differences in each measure are assessed by paired t-tests at the significance level of 0.01 for validation purposes. The results show that the proposed model outperforms the conventional model for the overall measures but only precision, accuracy, and MCC are statistically significant.

Прогнозування екологічного стану міської транспортної системи з використанням нейронних мереж

Мета: Метою цієї роботи є розробка ефективної моделі оцінювання екологічного стану міської транспортної системи з використанням загальних принципів нейронних мереж. Методи дослідження: Запропонована модель основана на роботі двох нейронних мереж з урахуванням впливу щільності транспортного потоку та рівня транзитного потенціалу на урбанізовані райони. Результати: На основі синтезу загальних принципів теорії нечітких множин та нейронних мереж розроблено модель оцінки рівня екологічного стану міської транспортної системи. Дано графічне представлення моделі. Високий ступінь достовірності прогнозу забезпечується навіть при низьких швидкостях навчання і високій динаміці змін статистичних даних в умовах транзиту транспортних потоків міста. Висновки: Використання нечітких нейронних мереж забезпечує можливість встановлювати повну відповідність між математичним представленням процедури нечіткого висновку і структурою міської транспортної системи. Запропонована модель дозволяє сформулювати чіткі екологічні орієнтири при прийнятті рішень в області транзитних перевезень з урахуванням інтересів підприємств, транспорту і населення міста, з подальшим перерозподілом транспортних потоків в тимчасовому і географічному просторі промислових районів міста.

Stuttering Thoughts: Negative Self-Referent Thinking Is Less Sensitive to Aversive Outcomes in People with Higher Levels of Depressive Symptoms

Learning theories of depression have proposed that depressive cognitions, such as negative thoughts with reference to oneself, can develop through a reinforcement learning mechanism. This negative self-reference is considered to be positively reinforced by rewarding experiences such as genuine support from others after negative self-disclosure, and negatively reinforced by avoidance of potential aversive situations. The learning account additionally predicts that negative self-reference would be maintained by an inability to adjust one’s behavior when negative self-reference no longer leads to such reward. To test this prediction, we designed an adapted version of the reversal-learning task. In this task, participants were reinforced to choose and engage in either negative or positive self-reference by probabilistic economic reward and punishment. Although participants were initially trained to choose negative self-reference, the stimulus-reward contingencies were reversed to prompt a shift toward positive self-reference (Study 1) and a further shift toward negative self-reference (Study 2). Model-based computational analyses showed that depressive symptoms were associated with a low learning rate of negative self-reference, indicating a high level of reward expectancy for negative self-reference even after the contingency reversal. Furthermore, the difficulty in updating outcome predictions of negative self-reference was significantly associated with the extent to which one possesses negative self-images. These results suggest that difficulty in adjusting action-outcome estimates for negative self-reference increases the chance to be faced with negative aspects of self, which may result in depressive symptoms.

Evaluating relative benefits of different types of R&D for clean energy technologies

Clean energy technologies that cost more than fossil fuel technologies require support through research and development (R&D). Learning-by-doing relates historical cost decreases to accumulation of experience. A learning investment is the amount of subsidy that is required to reach cost parity between a new technology and a conventional technology. We use learning investments to compare the relative impacts of two stylized types of R&D. We define curve-following R&D to be R&D that lowers costs by producing knowledge that would have otherwise been gained through learning-by-doing. We define curve-shifting R&D to be R&D that lowers costs by producing innovations that would not have occurred through learning-by-doing. We show that if an equal investment in curve-following or curve-shifting R&D would produce the same reduction in cost, the curve-shifting R&D would be more effective at reducing the learning investment needed to make the technology competitive. The relative benefit of curve-shifting over curve-following R&D is greater with a high starting cost and low learning rate. Our analysis suggests that, other things equal, investments in curve-shifting R&D have large benefits relative to curve-following R&D. In setting research policy, governments should consider the greater benefits of cost reductions brought about by transformational rather than incremental change.

How to Train a Compact Binary Neural Network with High Accuracy?

How to train a binary neural network (BinaryNet) with both high compression rate and high accuracy on large scale dataset? We answer this question through a careful analysis of previous work on BinaryNets, in terms of training strategies, regularization, and activation approximation. Our findings first reveal that a low learning rate is highly preferred to avoid frequent sign changes of the weights, which often makes the learning of BinaryNets unstable. Secondly, we propose to use PReLU instead of ReLU in a BinaryNet to conveniently absorb the scale factor for weights to the activation function, which enjoys high computation efficiency for binarized layers while maintains high approximation accuracy. Thirdly, we reveal that instead of imposing L2 regularization, driving all weights to zero which contradicts with the setting of BinaryNets, we introduce a regularization term that encourages the weights to be bipolar. Fourthly, we discover that the failure of binarizing the last layer, which is essential for high compression rate, is due to the improper output range. We propose to use a scale layer to bring it to normal. Last but not least, we propose multiple binarizations to improve the approximation of the activations. The composition of all these enables us to train BinaryNets with both high compression rate and high accuracy, which is strongly supported by our extensive empirical study.

Robust techniques for abandoned and removed object detection based on Markov random field

This paper presents a novel framework for detecting abandoned objects by introducing a fully-automatic GrabCut object segmentation. GrabCut seed initialization is treated as a background (BG) modelling problem that focuses only on unhanded objects and objects that become immobile. The BG distribution is constructed with dual Gaussian mixtures that are comprised of high and low learning rate models. We propose a primitive BG model-based removed object validation and Haar feature-based cascade classifier for still-people detection once a candidate for a released object has been detected. Our system can obtain more robust and accurate results for real environments based on evaluations of realistic scenes from CAVIAR, PETS2006, CDnet 2014, and our own datasets.

Particle Swarm Optimization for Creating Self Organized Evolutionary Back Propagation Artificial Neural Networks

A novel technique for developing a self organized Back Propagation Artificial Neural Networks (BPANNs) or simply Back Propagation Neural Network (BPN) with the help of swarm Intelligence (SI) technique. This paper proposes an overlapping swarm intelligence algorithm for training of neural networks in which a particle swarm is assigned to each neuron to search for that neuron’s weights. The training of ANN is a difficult task when it’s get into the local minima and causes the low learning rates; it is required of an evolutionary multi dimensional algorithm to seek optimal weight values, the positional optimum values and the dimensional optimum in values in the dynamic problem space. The approach discussed throughout in this paper is the credit assignment process by first focusing on updating weights and biases swarms Intelligence and then evaluating the fitness of the particles using a network. With the proper adaptation of the SI-BPN process, the proposed method can develop an optimum network within an architecture space for a particular problem. Additionally, it provides a class list of all other potential configurations. This algorithm will provide superior learning ability to the traditional Back-Propagation (BP) method in terms of accuracy and speed.

Cluster-Based Boosting

Boosting is an iterative process that improves the predictive accuracy for supervised (machine) learning algorithms. Boosting operates by learning multiple functions with subsequent functions focusing on incorrect instances where the previous functions predicted the wrong label. Despite considerable success, boosting still has difficulty on data sets with certain types of problematic training data (e.g., label noise) and when complex functions overfit the training data. We propose a novel cluster-based boosting (CBB) approach to address limitations in boosting for supervised learning systems. Our CBB approach partitions the training data into clusters containing highly similar member data and integrates these clusters directly into the boosting process. CBB boosts selectively (using a high learning rate, low learning rate, or not boosting) on each cluster based on both the additional structure provided by the cluster and previous function accuracy on the member data. Selective boosting allows CBB to improve predictive accuracy on problematic training data. In addition, boosting separately on clusters reduces function complexity to mitigate overfitting. We provide comprehensive experimental results on 20 UCI benchmark data sets with three different kinds of supervised learning systems. These results demonstrate the effectiveness of our CBB approach compared to a popular boosting algorithm, an algorithm that uses clusters to improve boosting, and two algorithms that use selective boosting without clustering.