Partitioning Of Feature Space Research Articles

Convolutional Channel-Wise Competitive Learning for the Forward-Forward Algorithm

The Forward-Forward (FF) Algorithm has been recently proposed to alleviate the issues of backpropagation (BP) commonly used to train deep neural networks. However, its current formulation exhibits limitations such as the generation of negative data, slower convergence, and inadequate performance on complex tasks. In this paper we take the main ideas of FF and improve them by leveraging channel-wise competitive learning in the context of convolutional neural networks for image classification tasks. A layer-wise loss function is introduced that promotes competitive learning and eliminates the need for negative data construction. To enhance both the learning of compositional features and feature space partitioning, a channel-wise feature separator and extractor block is proposed that complements the competitive learning process. Our method outperforms recent FF-based models on image classification tasks, achieving testing errors of 0.58%, 7.69%, 21.89%, and 48.77% on MNIST, Fashion-MNIST, CIFAR-10 and CIFAR-100 respectively. Our approach bridges the performance gap between FF learning and BP methods, indicating the potential of our proposed approach to learn useful representations in a layer-wise modular fashion, enabling more efficient and flexible learning. Our source code and supplementary material are available at https://github.com/andreaspapac/CwComp.

Brain tumour microstructure is associated with post-surgical cognition

Brain tumour microstructure is potentially predictive of changes following treatment to cognitive functions subserved by the functional networks in which they are embedded. To test this hypothesis, intra-tumoural microstructure was quantified from diffusion-weighted MRI to identify which tumour subregions (if any) had a greater impact on participants’ cognitive recovery after surgical resection. Additionally, we studied the role of tumour microstructure in the functional interaction between the tumour and the rest of the brain. Sixteen patients (22–56 years, 7 females) with brain tumours located in or near speech-eloquent areas of the brain were included in the analyses. Two different approaches were adopted for tumour segmentation from a multishell diffusion MRI acquisition: the first used a two-dimensional four group partition of feature space, whilst the second used data-driven clustering with Gaussian mixture modelling. For each approach, we assessed the capability of tumour microstructure to predict participants’ cognitive outcomes after surgery and the strength of association between the BOLD signal of individual tumour subregions and the global BOLD signal. With both methodologies, the volumes of partially overlapped subregions within the tumour significantly predicted cognitive decline in verbal skills after surgery. We also found that these particular subregions were among those that showed greater functional interaction with the unaffected cortex. Our results indicate that tumour microstructure measured by MRI multishell diffusion is associated with cognitive recovery after surgery.

Towards Codebook-Free Deep Probabilistic Quantization for Image Retrieval.

As a classical feature compression technique, quantization is usually coupled with inverted indices for scalable image retrieval. Most quantization methods explicitly divide feature space into Voronoi cells, and quantize feature vectors in each cell into the centroids learned from data distribution. However, Voronoi decomposition is difficult to achieve discriminative space partition for semantic image retrieval. In this paper, we explore semantic-aware feature space partition by deep neural network instead of Voronoi cells. To this end, we propose a new deep probabilistic quantization method, abbreviated as DeepIndex, which constructs inverted indices without explicit centroid learning. In our method, the deep neural network takes an image as input and outputs its probability of being put into each inverted index list. During training, we progressively quantize each image into the inverted lists with the top- T maximal probabilities, and calculate the reward of each trial based on retrieval accuracy. We optimize the deep neural network to maximize the probability of the inverted list with maximal reward. In this way, the retrieval performance is directly optimized, leading to a more semantically discriminative space partition than other quantization methods. The experiments on public image datasets demonstrate the effectiveness of our DeepIndex method on semantic image retrieval.

Incremental few-shot learning via implanting and consolidating

This work focuses on tackling the challenging but realistic visual task of Incremental Few-Shot Learning (IFSL), which requires a model to continually learn novel classes from only a few examples while not forgetting the base classes on which it was pre-trained. Our study reveals that the challenges of IFSL lie in both inter-class separation and novel-class representation. Due to intra-class variation, a novel class may implicitly leverage the knowledge from multiple base classes to construct its feature representation. Hence, simply reusing the pre-trained embedding space could lead to a scattered feature distribution and result in category confusion. To address such issues, we propose a two-step learning strategy referred to as Implanting and Compressing (IMCO), which optimizes both feature space partition and novel class reconstruction in a systematic manner. Specifically, in the Implanting step, we propose to mimic the data distribution of novel classes with the assistance of data-abundant base set, so that a model could learn semantically-rich features that are beneficial for discriminating between the base and other unseen classes. In the Consolidating step, we adapt the feature extractor to precisely represent each novel class for enhancing intra-class compactness, together with a regularized parameter updating rule for preventing aggressive model updating. Finally, we demonstrate that IMCO outperforms competing baselines with a significant margin on widely used benchmarks.

A Multi-DOF Manipulator Joint Trajectory Tracking and Monitoring Method Based on Decision Tree

Understanding trajectory tracking control concerns is crucial for industrial-grade manipulators to provide precise and risk-free operations for the safe environment. Consequently, the robot arms need precise aim for tracking a given target trajectory by the trajectory control input driving torque which can use smart AI-based techniques for precision. Similarly, a decision tree is a soft computing-based method of feature space partitioning which can certainly allow the movement of robots in an accurate manner. The control of robot arms is an important aspect for automating the process of sustainable development. Aiming at the problem of poor tracking accuracy of traditional Multi-Degree-of-Freedom Manipulator Joint Trajectory Monitoring (Multi-DoF MJTM) and long monitoring delay, this article proposes a Multi-DOF manipulator joint trajectory tracking method based on decision tree. The Multi-DoF manipulator is developed for the adaptive control object of the working machine, and it is combined with the output response feature to construct the kinematics model of the Multi-DoF manipulator mechanism of the walking machine. The joint trajectory reconstruction of the running trajectory is used to obtain the joint trajectory deviation of the Multi-DoF manipulator’s running trajectory through the multi-measurement system. Based on this, the Multi-DoF manipulator’s running trajectory joint trajectory tracking control equation is obtained to realize the joint trajectory tracking and monitoring of the manipulator. The features for a safe environment are also integrated. The experimental results show that the proposed method has high accuracy in tracking the trajectory of Multi-DoF manipulator joints, and the time delay for tracking and monitoring the trajectory of manipulator’s joints is also optimized.

Feature space partition: a local–global approach for classification

Feature space partition: a local–global approach for classification

Subspace-based decision trees integration

The ensemble methods are meta-algorithms that combine several base machine learning techniques to produce one optimal predictive model. Many existing committees of classifiers use feature space partitioning to determine the decision regions in which the selection of base classifiers occur or to learn a diverse ensemble of base classifiers. Furthermore, the division of the feature space into subspaces does not depend on the predictive model that defines the decision boundaries. Therefore, we propose a novel ensemble learning algorithm based on the feature space partitioning defined by decision tree boundaries. In addition, in a proposed framework, the feature subspace selection occurs taking class label imbalance ratio into account. The proposed method defines the ensemble class label based on selected previously feature subspaces and a neighborhood of feature subspaces defined by its reference point. Experimental results indicate that our proposed method is more effective than state-of-the-art ensemble methods on twenty-seven benchmark datasets regarding seven representative classification performance measures.

Trainable monotone combiner

We consider a binary classification problem in which the class label is given in the form of a discriminant function that satisfies a monotone constraint. That is, the degree of confidence that an object belongs to a class can not decrease as one of the input features increases. This manuscript examines how such a discriminant function can be trained on the basis of a labeled data set. Two alternative quality measures are considered. One of them is the AUC, which is based on the ROC analysis. The second is encouraged by the Neyman-Pearson lemma, which aims to maximize the ratio of correctly classified to misclassified examples. We propose an approach in which feature space is partitioned into quality layers that can then effectively compute the discriminant function. We prove that the resulting discriminant function is optimal with respect to the two quality measures mentioned, which indicates, among other things, the equivalence of these two quality measures. We also show that the associated optimal partitioning of feature space is unique, and provide a polynomial training algorithm for generating this partitioning.

A Parallel Multilevel Feature Selection algorithm for improved cancer classification

Biological data is prone to grow exponentially, which consumes more resources, time and manpower. Parallelization of algorithms could reduce overall execution time. There are two main challenges in parallelizing computational methods. (1) Biological data is multi-dimensional in nature. (2). Parallel algorithms reduce execution time, but with the penalty of reduced prediction accuracy. This research paper targets these two issues and proposes the following approaches. (1) Vertical partitioning of data along feature space and horizontal partitioning along samples in order to ease the task of data parallelism. (2) Parallel Multilevel Feature Selection (M-FS) algorithm to select optimal and important features for improved classification of cancer sub-types. The selected features are evaluated using parallel Random Forest on Spark, compared with previously reported results and also with the results of sequential execution of same algorithms. The proposed parallel M-FS algorithm was compared with existing parallel feature selection algorithms in terms of accuracy and execution time. The results reveal that parallel multilevel feature selection algorithm improved cancer classification resulting into prediction accuracy ranging from ∼85% to ∼99% with very high speed up in terms of seconds. On the other hand, existing sequential algorithms yielded prediction accuracy of ∼65% to ∼99% with execution time of more than 24 hours.

Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons

An activation boundary for a neuron refers to a separating hyperplane that determines whether the neuron is activated or deactivated. It has been long considered in neural networks that the activations of neurons, rather than their exact output values, play the most important role in forming classificationfriendly partitions of the hidden feature space. However, as far as we know, this aspect of neural networks has not been considered in the literature of knowledge transfer. In this paper, we propose a knowledge transfer method via distillation of activation boundaries formed by hidden neurons. For the distillation, we propose an activation transfer loss that has the minimum value when the boundaries generated by the student coincide with those by the teacher. Since the activation transfer loss is not differentiable, we design a piecewise differentiable loss approximating the activation transfer loss. By the proposed method, the student learns a separating boundary between activation region and deactivation region formed by each neuron in the teacher. Through the experiments in various aspects of knowledge transfer, it is verified that the proposed method outperforms the current state-of-the-art.

Ensemble classification for imbalanced data based on feature space partitioning and hybrid metaheuristics

One of the most challenging issues when facing a classification problem is to deal with imbalanced datasets. Recently, ensemble classification techniques have proven to be very successful in addressing this problem. We present an ensemble classification approach based on feature space partitioning for imbalanced classification. A hybrid metaheuristic called GACE is used to optimize the different parameters related to the feature space partitioning. To assess the performance of the proposal, an extensive experimentation over imbalanced and real-world datasets compares different configurations and base classifiers. Its performance is competitive with that of reference techniques in the literature.

Estimation of Algorithms Efficiency in the Task of Biological Objects Clustering

Background. The task of determining the functional relationship between biophysical parameters is an integral part of the actual problem of finding the optimal impact on a biological object and is currently not completely resolved. One of the important tasks in this area is the partitioning of the original feature space into such areas (clusters) that relate to different functional relationships linking biophysical parameters and have, in general, an arbitrary shape. Such clusters in the future is logical to call functional. To obtain and analyze the functional clusters, there are a number of algorithms, each of which has its advantages and disadvantages. At the same time, the solution of a certain practical problem requires an evaluation of the efficiency of the algorithms in terms of the cluster separation adequacy.Objective. In this paper, for a general example of the biological objects clustering problem (Fischer’s Iris Data Set), the efficiency of a typical clustering tools series is evaluated. The application of k-means classical algorithm, the Ward algorithm and developed in this work the fuzzy version of clustering for the k-means algorithm with a limited mass of the working area for the clusters’ formation was considered.Methods. The algorithm includes a procedure for a priori estimation of the clusters quantity. The estimation is carried out according to the frequency histogram. To determine the optimal number of the histogram columns, the application of the Scott formula is justified. The algorithm allows forming clusters of arbitrary configuration with obtaining the value of the object's membership measure for each of the clusters. The comparative testing of the above algorithms was carried out on Fisher’s Iris Data Set.Results. The best value of F1-score is obtained for the algorithm proposed in this paper: F1 = 0.92, the value F1 = 0.90 is obtained for the Ward method and the value F1 = 0.88 – for the classical k-means algorithm.Conclusions. The obtained test results on the analysis problem of arbitrary-shaped clusters made it possible to give preference to the version of fuzzy k-means with a limited mass of the working area for the clusters’ formation. The calculating of the membership measure value allows us to obtain additional information on the structure of cluster formations, as well as to correct the result of clustering of k-means with a limited mass, which is especially important since the formation of clusters occurs in a single pass. Comparing the computational resources required for computing algorithms with relatively close test results also makes it possible to give preference to the developed algorithm. Compared with the Ward algorithm, it requires fewer computing resources since no additional memory is needed to store the distance matrix and no time is required to recalculate it.

Balanced Mixture of Deformable Part Models With Automatic Part Configurations

This paper presents a method to improve the traditional mixture of deformable part models (MDPM) method from the learning perspective. First, an object part configuration learning algorithm based on group sparsity constraint is introduced to automatically discover the object part number, size, and location. The algorithm imposes two additional regularization terms in addition to the standard hinge loss function. The first term focuses on automatic part selection and the second term focuses on automatic part placement. Second, this paper introduces an improved MDPM training framework. The framework applies a learned transformation to normalize the prediction score from each individual deformable part model (DPM) into a pseudo probability such that the partition of the entire object appearance feature space becomes less sensitive to the prior distributions of different DPMs. Finally, the two proposed improvements are combined and formulated under the expectation-maximization framework. We evaluate our method mainly using the PASCAL VOC2007 and VOC2010 detection benchmarks and show that the proposed learning algorithms could increase the detection mean AP score by 2.4% and 0.9%, respectively, on these two data sets when using the proposed part selection method and the training algorithm. We also present further in-depth analysis of the proposed algorithm in the experiments.

Online Classification via Self-Organizing Space Partitioning

The authors study online supervised learning under the empirical zero-one loss and introduce a novel classification algorithm with strong theoretical guarantees. The proposed method is a highly dynamical self-organizing decision tree structure, which adaptively partitions the feature space into small regions and combines (takes the union of) the local simple classification models specialized in those regions. The authors’ approach sequentially and directly minimizes the cumulative loss by jointly learning the optimal feature space partitioning and the corresponding individual partition-region classifiers. They mitigate overtraining issues by using basic linear classifiers at each region while providing a superior modeling power through hierarchical and data adaptive models. The computational complexity of the introduced algorithm scales linearly with the dimensionality of the feature space and the depth of the tree. Their algorithm can be applied to any streaming data without requiring a training phase or a priori information, hence processing data on-the-fly and then discarding it. Therefore, the introduced algorithm is especially suitable for the applications requiring sequential data processing at large scales/high rates. The authors present a comprehensive experimental study in stationary and nonstationary environments. In these experiments, their algorithm is compared with the state-of-the-art methods over the well-known benchmark datasets and shown to be computationally highly superior. The proposed algorithm significantly outperforms the competing methods in the stationary settings and demonstrates remarkable adaptation capabilities to nonstationarity in the presence of drifting concepts and abrupt/sudden concept changes.

Analysis of approaches to feature space partitioning for nonlinear dimensionality reduction

One of the most effective ways to reduce the computational complexity of nonlinear dimensionality reduction is hierarchical partitioning of the space with the subsequent approximation of calculations. In this paper, the efficiency of two approaches to space partitioning, the partitioning of input and output spaces, is analyzed. In addition, a method for nonlinear dimensionality reduction is proposed. It is based on construction of a partitioning tree of the input multidimensional space and an iterative procedure of the gradient descent with the approximation carried out on the nodes of the constructed space partitioning tree. In the method proposed, the relative position of the corrected objects and partitioning tree nodes in both input and output spaces is taken into account in the approximation. The method developed was analyzed based on publicly available datasets.

SIMILARITY CLASSIFIER WITH WEIGHTED ORDERED WEIGHTED AVERAGING OPERATOR

(ProQuest: ... denotes formulae omitted.)1.INTRODUCTIONVast amount of data produced by computers and the internet of things and the increased ability to create automated systems have led to a growing need for effective ways to monitor performance and to detect problems, monitoring and problem detection are often linked to classification as states of performance can be understood as classes. Classification has been used in connection with variety of problems that range from medical diagnostics to fault detection and are embedded in many common appliances such as air conditioners, vacuum cleaners, and industrial equipment, e.g., see (Bagby and Cormier, 1989; Haberl and Claridge, 1987). The use of classifiers is also making its' way into business enterprises, where classifiers are, e.g., used to guide decision making (Dubois and Prade, 1985; Torra and Narukawa, 2007; Yager and Kacprzyk, 1997; Yager, et al., 2011). The main point that separates good and bad classifiers is classification accuracy, because incorrect classification may lead to many unwanted consequences such as the administration of inappropriate drugs, false positive or negative diagnoses, unnecessary actions of control in a management setting, and to bad decisions. One of the key questions in classification is typically the partitioning of the feature space as accurately as possible (Duda and Hart, 1973) to enable as correct as possible decisions based on the classification.Studies have shown that classification accuracy is affected by the kind of aggregation operator that is used in the classifier. In addition to the classical averaging operators there are also more advanced aggregation operators available, such as the Ordered Weighted Averaging (OWA) operator presented in (Yager, 1988) and further extensions of the OWA like the Weighted Ordered Weighted Averaging (WOWA) operator, introduced by Torra (Torra, 1996; 1997; 1998). The WOWA operator was applied to the purpose of interpolation in (Torra, 2000) and in fusing information in (Torra, 2003; Torra and Narukawa, 2007). In addition to the re-ordering step that emphasizes the relative importance of data values, the WOWA operator can also be understood to incorporate the consideration of importance of the data sources used. Two different kinds of aggregations are performed in the WOWA process and it is a combinations and a generalization of the weighted arithmetic mean and the OWA operators, see (Ogryczak and Sliwinski, 2007; Torra and Narukawa, 2007).Several different kinds of classification techniques can be found in literature, in this paper we focus on classifiers that use fuzzy set theoretical methods and that belong to the family of similarity based classifiers, for more information on similarity-based fuzzy classifiers we suggest the interested reader to see, e.g., (Luukka, 2007; Luukka and Kurama, 2013; Luukka and Leppalampi, 2006; Luukka, et al., 2001). Classification accuracy of similarity-based classifiers is also affected by the choice of the aggregation operator used, this is visible from the results presented in (Luukka and Kurama, 2013), where a similarity-based classifier using OWA-based aggregation over-performed the accuracy gained with a previously presented generalized mean-based similarity classifier, see (Luukka and Leppalampi, 2006). In this paper we continue in the same vein with the previous research on the topic of similarity-based classifiers and propose a new similarity-based classifier that uses WOWA in the aggregation of information. We present the construct of the new classifier, study the classification accuracy and compare results with previously presented methods.The rest of this paper is arranged as follows: Section two goes through the preliminaries and introduces the notation and the common definitions used.Section three describes the design of the proposed new classifier and section four is used to present the data sets used and the empirical results obtained. …

Joint Learning of Multiple Regressors for Single Image Super-Resolution

Using a global regression model for single image super-resolution (SISR) generally fails to produce visually pleasant output. The recently developed local learning methods provide a remedy by partitioning the feature space into a number of clusters and learning a simple local model for each cluster. However, in these methods the space partition is conducted separately from local model learning, which results in an abundant number of local models to achieve satisfying performance. To address this problem, we propose a mixture of experts (MoE) method to jointly learn the feature space partition and local regression models. Our MoE consists of two components: gating network learning and local regressors learning. An expectation-maximization (EM) algorithm is adopted to train MoE on a large set of LR/HR patch pairs. Experimental results demonstrate that the proposed method can use much less local models and time to achieve comparable or superior results to state-of-the-art SISR methods, providing a highly practical solution to real applications.

Image interpretation-guided supervised classification using nested segmentation

We present a new binary (two-class) supervised non-parametric classification approach that is based on iterative partitioning of multidimensional feature space into variably-sized and nested hyper-cubes (partitions). The proposed method contains elements of active learning and includes classifier to analyst queries. The spectral transition zone between two thematic classes (i.e., where training labels of different classes overlap in feature space) is targeted through iterative training derivation. Three partition categories are defined: pure, indivisible and unlabeled. Pure partitions contain training labels from only one class, indivisible partitions contain training data from different classes, and unlabeled partitions do not contain training data. A minimum spectral tolerance threshold defines the smallest partition volume to avoid over-fitting. In this way the transition zones between class distributions are minimized, thereby maximizing both the spectral volume of pure partitions in the feature space and the number of pure pixels in the classified image. The classification results are displayed to show each classified pixel's partition category (pure, unlabeled and indivisible). Mapping pixels belonging to unlabeled partitions serves as a query from the classifier to the analyst, targeting spectral regions absent of training data. The classification process is repeated until significant improvement of the classification is no longer realized or when no classification errors and unlabeled pixels are left. Variably-sized partitions lead to intensive training data derivation in the spectral transition zones between the target classes. The methodology is demonstrated for surface water and permanent snow and ice classifications using 30m conterminous United States Landsat 7 Enhanced Thematic Mapper Plus (ETM+) data time series from 2006 to 2010. The surface water result was compared with Shuttle Radar Topography Mission (SRTM) water body and National Land Cover Database (NLCD) open water classes with an overall agreement greater than 99% and Kappa coefficient greater than 0.9 in both of cases. In addition, the surface water result was compared with a classification generated using the same input data and a standard bagged Classification and Regression Tree (CART) classifier. The nested segmentation and CART-generated products had an overall agreement of 99.9 and Kappa coefficient of 0.99.

A Generic Ensemble Approach to Estimate Multidimensional Likelihood in Bayesian Classifier Learning

In Bayesian classifier learning, estimating the joint probability distribution p(x,y) or the likelihood p(x|y) directly from training data is considered to be difficult, especially in large multidimensional data sets. To circumvent this difficulty, existing Bayesian classifiers such as Naive Bayes, BayesNet, and AηDE have focused on estimating simplified surrogates of p(x,y) from different forms of one‐dimensional likelihoods.Contrary to the perceived difficulty in multidimensional likelihood estimation, we present a simple generic ensemble approach to estimate multidimensional likelihood directly from data. The idea is to aggregate pi(x|y) estimated from a random subsample of data . This article presents two ways to estimate multidimensional likelihoods using the proposed generic approach and introduces two new Bayesian classifiers called ENNBayes and MassBayes that estimate pi(x|y) using a nearest‐neighbor density estimation and a probability estimation through feature space partitioning, respectively.Unlike the existing Bayesian classifiers, ENNBayes and MassBayes have constant training time and space complexities and they scale better than existing Bayesian classifiers in very large data sets. Our empirical evaluation shows that ENNBayes and MassBayes yield better predictive accuracy than the existing Bayesian classifiers in benchmark data sets.

A New Ensemble Method with Feature Space Partitioning for High-Dimensional Data Classification

Ensemble data mining methods, also known as classifier combination, are often used to improve the performance of classification. Various classifier combination methods such as bagging, boosting, and random forest have been devised and have received considerable attention in the past. However, data dimensionality increases rapidly day by day. Such a trend poses various challenges as these methods are not suitable to directly apply to high-dimensional datasets. In this paper, we propose an ensemble method for classification of high-dimensional data, with each classifier constructed from a different set of features determined by partitioning of redundant features. In our method, the redundancy of features is considered to divide the original feature space. Then, each generated feature subset is trained by a support vector machine, and the results of each classifier are combined by majority voting. The efficiency and effectiveness of our method are demonstrated through comparisons with other ensemble techniques, and the results show that our method outperforms other methods.