Information Theoretic Learning Research Articles

Kernel density estimation of Tsalli’s entropy with applications in adaptive system training

Information theoretic learning plays a very important role in adaption learning systems. Many non-parametric entropy estimators have been proposed by the researchers. This work explores kernel density estimation based on Tsallis entropy. Firstly, it has been proved that for linearly independent samples and for equal samples, Tsallis-estimator is consistent for the PDF and minimum respectively. Also, it is investigated that Tsallis-estimator is smooth for differentiable, symmetric, and unimodal kernel function. Further, important properties of Tsallis-estimator such as scaling and invariance for both single and joint entropy estimation have been proved. The objective of the work is to understand the mathematics behind the underlying concept.

Computationally efficient robust adaptive filtering algorithm based on improved minimum error entropy criterion with fiducial points

Recently, there has been a strong interest in the minimum error entropy (MEE) criterion derived from information theoretic learning, which is effective in dealing with the multimodal non-Gaussian noise case. However, the kernel function is shift invariant resulting in the MEE criterion being insensitive to the error location. An existing solution is to combine the maximum correntropy (MC) with MEE criteria, leading to the MEE criterion with fiducial points (MEEF). Nevertheless, the algorithms based on the MEEF criterion usually require higher computational complexity. To remedy this problem, an improved MEEF (IMEEF) criterion is devised, aiming to avoid repetitive calculations of the aposteriori error, and an adaptive filtering algorithm based on gradient descent (GD) method is proposed, namely, GD-based IMEEF (IMEEF-GD) algorithm. In addition, we provide the convergence condition in terms of mean sense, along with an analysis of the steady-state and transient behaviors of IMEEF-GD in the mean-square sense. Its computational complexity is also analyzed. Simulation results demonstrate that the computational requirement of our algorithm does not vary significantly with the error sample number and the derived theoretical model is highly consistent with the learning curve. Ultimately, we employ the IMEEF-GD algorithm in tasks such as system identification, wind signal magnitude prediction, temperature prediction, and acoustic echo cancellation (AEC) to validate the effectiveness of the IMEEF-GD algorithm.

Error Loss Networks.

A novel model called error loss network (ELN) is proposed to build an error loss function for supervised learning. The ELN is similar in structure to a radial basis function (RBF) neural network, but its input is an error sample and output is a loss corresponding to that error sample. That means the nonlinear input-output mapper of the ELN creates an error loss function. The proposed ELN provides a unified model for a large class of error loss functions, which includes some information-theoretic learning (ITL) loss functions as special cases. The activation function, weight parameters, and network size of the ELN can be predetermined or learned from the error samples. On this basis, we propose a new machine learning paradigm where the learning process is divided into two stages: first, learning a loss function using an ELN; second, using the learned loss function to continue to perform the learning. Experimental results are presented to demonstrate the desirable performance of the new method.

Learning Korobov Functions by Correntropy and Convolutional Neural Networks.

Combining information-theoretic learning with deep learning has gained significant attention in recent years, as it offers a promising approach to tackle the challenges posed by big data. However, the theoretical understanding of convolutional structures, which are vital to many structured deep learning models, remains incomplete. To partially bridge this gap, this letter aims to develop generalization analysis for deep convolutional neural network (CNN) algorithms using learning theory. Specifically, we focus on investigating robust regression using correntropy-induced loss functions derived from information-theoretic learning. Our analysis demonstrates an explicit convergence rate for deep CNN-based robust regression algorithms when the target function resides in the Korobov space. This study sheds light on the theoretical underpinnings of CNNs and provides a framework for understanding their performance and limitations.

Quantized information-theoretic learning based Laguerre functional linked neural networks for nonlinear active noise control

Traditional functional linked neural networks (FLNNs) impose a significant computational burden due to their input expansion, primarily stemming from the utilization of digital filters. This paper presents a Laguerre FLNNs filter for nonlinear active noise control (NANC) systems. By employing the truncated Laguerre series, the presented filter achieves effective approximation of long primary paths with a reduced filter length. Moreover, we develop adaptive algorithms rooted in information-theoretic learning (ITL) within the framework of the Laguerre-FLNNs NANC model. Using the ITL criterions, a Laguerre filtered-s maximum correntropy criterion (LFsMCC) algorithm is derived and a Laguerre filtered-s quantized minimum error entropy criterion (LFsQMEE) algorithm is proposed by minimizing Renyi’s quadratic entropy. To reduce the computation cost, an online vector quantization method is utilized to improve the LFsQMEE. This technique selectively quantizes the error vectors, reducing them to a smaller subset of samples within the codebook. Moreover, an enhanced LFsQMEE with a fiducial point is introduced. The steady-state performance and the computational complexity are analyzed. Theoretical analysis is validated through simulations and the control performance of the proposed model and algorithms is tested in experiments with both simulated and real paths.

To Compress or Not to Compress-Self-Supervised Learning and Information Theory: A Review.

Deep neural networks excel in supervised learning tasks but are constrained by the need for extensive labeled data. Self-supervised learning emerges as a promising alternative, allowing models to learn without explicit labels. Information theory has shaped deep neural networks, particularly the information bottleneck principle. This principle optimizes the trade-off between compression and preserving relevant information, providing a foundation for efficient network design in supervised contexts. However, its precise role and adaptation in self-supervised learning remain unclear. In this work, we scrutinize various self-supervised learning approaches from an information-theoretic perspective, introducing a unified framework that encapsulates the self-supervised information-theoretic learning problem. This framework includes multiple encoders and decoders, suggesting that all existing work on self-supervised learning can be seen as specific instances. We aim to unify these approaches to understand their underlying principles better and address the main challenge: many works present different frameworks with differing theories that may seem contradictory. By weaving existing research into a cohesive narrative, we delve into contemporary self-supervised methodologies, spotlight potential research areas, and highlight inherent challenges. Moreover, we discuss how to estimate information-theoretic quantities and their associated empirical problems. Overall, this paper provides a comprehensive review of the intersection of information theory, self-supervised learning, and deep neural networks, aiming for a better understanding through our proposed unified approach.

Generalization analysis of deep CNNs under maximum correntropy criterion

Convolutional neural networks (CNNs) have gained immense popularity in recent years, finding their utility in diverse fields such as image recognition, natural language processing, and bio-informatics. Despite the remarkable progress made in deep learning theory, most studies on CNNs, especially in regression tasks, tend to heavily rely on the least squares loss function. However, there are situations where such learning algorithms may not suffice, particularly in the presence of heavy-tailed noises or outliers. This predicament emphasizes the necessity of exploring alternative loss functions that can handle such scenarios more effectively, thereby unleashing the true potential of CNNs. In this paper, we investigate the generalization error of deep CNNs with the rectified linear unit (ReLU) activation function for robust regression problems within an information-theoretic learning framework. Our study demonstrates that when the regression function exhibits an additive ridge structure and the noise possesses a finite pth moment, the empirical risk minimization scheme, generated by the maximum correntropy criterion and deep CNNs, achieves fast convergence rates. Notably, these rates align with the mini-max optimal convergence rates attained by fully connected neural network model with the Huber loss function up to a logarithmic factor. Additionally, we further establish the convergence rates of deep CNNs under the maximum correntropy criterion when the regression function resides in a Sobolev space on the sphere.

Weighted multi-error information entropy based you only look once network for underwater object detection

Underwater object detection is considered as one of the most challenging issues in computer vision. In this paper, a weighted multi-error information entropy based YOLO (You Only Look Once) network is proposed to address underwater illumination noise affecting the detection accuracy. First, underwater illumination is essentially structural and non-uniform, and it is modeled as an independent and piecewise identical distribution, which is a generic noise model to describe the complex underwater illuminating environment and accommodates the traditional Gaussian distribution as a special case. Second, assisted by the proposed illumination noise model, a minimum weighted error entropy criterion, which is an information-theoretic learning method, is introduced into the loss function of YOLO network, and then the network parameters are trained and optimized to improve the detection performance. Furthermore, a multi-error processing strategy is simultaneously used to handle vector errors during information back-propagation in order to accelerate convergence. Experiments on underwater object detection datasets including URPC2018, URPC2019 and Enhanced dataset, show the proposed weighted multi-error information entropy based YOLOv8 network gets mean average precision (MAP) of 88.7%, 91.8% and 96.7% respectively, and average frames per second (FPS) of 116.6. These two evaluation metrics are better than the baseline YOLOv8 and the existing advanced non-YOLO approaches by at least 5.2% and 5.3% respectively. The results verify the effectiveness and superiority of the proposed network for underwater object detection in complex underwater environment.

Event-triggered consensus control based on maximum correntropy criterion for discrete-time multi-agent systems

An event-triggered consensus problem of discrete-time multi-agent systems with impulsive noise under directed switching topology is investigated in this paper, where the control input of each agent is determined by its own state information and the state information of its neighbors that may be corrupted by impulsive noise. Firstly, to reduce the adverse effect of impulsive noise on consensus performance of the discrete-time multi-agent systems, maximum correntropy criterion (MCC) derived from information theoretic learning is introduced to calculate the communication weights among the agents, and an event-triggered strategy is adopted to drive the state variables of each agent to eventually converge to a same value. Secondly, the condition to guarantee consensus of the discrete-time multi-agent systems is derived by graph theory and Lyapunov stability theorem. Finally, the effectiveness of the theory is verified by several numerical simulations.

Probabilistic Point Cloud Modeling via Self-Organizing Gaussian Mixture Models

This letter presents a continuous probabilistic modeling methodology for spatial point cloud data using finite Gaussian Mixture Models (GMMs) where the number of components are adapted based on the scene complexity. Few hierarchical and adaptive methods have been proposed to address the challenge of balancing model fidelity with size. Instead, state-of-the-art mapping approaches require tuning parameters for specific use cases, but do not generalize across diverse environments. To address this gap, we utilize a self-organizing principle from information-theoretic learning to automatically adapt the complexity of the GMM model based on the relevant information in the sensor data. The approach is evaluated against existing point cloud modeling techniques on real-world data with varying degrees of scene complexity.

A Discriminant Information Theoretic Learning Framework for Multi-modal Feature Representation

As sensory and computing technology advances, multi-modal features have been playing a central role in ubiquitously representing patterns and phenomena for effective information analysis and recognition. As a result, multi-modal feature representation is becoming a progressively significant direction of academic research and real applications. Nevertheless, numerous challenges remain ahead, especially in the joint utilization of discriminatory representations and complementary representations from multi-modal features. In this article, a discriminant information theoretic learning (DITL) framework is proposed to address these challenges. By employing this proposed framework, the discrimination and complementation within the given multi-modal features are exploited jointly, resulting in a high-quality feature representation. According to characteristics of the DITL framework, the newly generated feature representation is further optimized, leading to lower computational complexity and improved system performance. To demonstrate the effectiveness and generality of DITL, we conducted experiments on several recognition examples, including both static cases, such as handwritten digit recognition, face recognition, and object recognition, and dynamic cases, such as video-based human emotion recognition and action recognition. The results show that the proposed framework outperforms state-of-the-art algorithms.

Quantized kernel recursive minimum error entropy algorithm

In this paper, we propose a online vector quantization (VQ) method based on the kernel recursive minimum error entropy (KRMEE) algorithm. According to information theoretic learning (ITL), the minimum error entropy criterion (MEE) is robust and can effective resistance to non-Gaussian noise. By combining the kernel recursive least squares (KRLS) algorithm with MEE criterion, KRMEE algorithm has been generated, which has excellent performance in non-Gaussian environments. However, with the size of data increases, the computational complexity will raise. We propose a quantized to solve this problem, the input space of the algorithm is quantized to suppress the linear growth radial basis function (RBF) network in kernel adaptive filtering (KAF). The VQ method is different from novelty criterion (NC), approximate linear dependency (ALD) criterion, and other sparsity methods, the online VQ method need to construct the dictionary, and calculate the distance by Euclidean norm. We propose a novel quantized kernel recursive minimum error entropy (QKRMEE) algorithm by combining VQ method with KRMEE algorithm, and update the solution with a recursive algorithm. In Mackey-Glass time series and a real-world datasets, Monte Carlo simulation experiments show that the proposed algorithm achieves better predictive performance in non-Gaussian noise environment. Meanwhile, the algorithm can restrain the growth of RBF network well, thus reducing the computational complexity and memory consumption effectively.

Robust Matrix Factorization via Minimum Weighted Error Entropy Criterion

Learning the intrinsic low-dimensional subspace from high-dimensional data is a key step for many social systems of artificial intelligence. In practical scenarios, the observed data are usually corrupted by many types of noise, which brings a great challenge for social systems to analyze data. As a commonly utilized subspace learning technique, robust low-rank matrix factorization (LRMF) focuses on recovering the underlying subspaces in a noisy environment. However, most of the existing approaches simply assume that the noise contaminating the data is independent identically distributed (i.i.d.), such as Gaussian and Laplacian noises. This assumption, though greatly simplifies the underlying learning problem, may not hold for more complex non-i.i.d. noise widely existed in social systems. In this work, we suggest a robust LRMF approach to deal with various types of noise in a unified manner. Different from traditional algorithms, noise in our framework is modeled using an independent and piecewise identically distributed (i.p.i.d.) source, which employs a collection of distributions, instead of a single one to characterize the statistical behavior of the underlying noise. Assisted by the generic noise model, we then design a robust LRMF algorithm under the information-theoretic learning (ITL) framework through a new minimization criterion. By adopting the half-quadratic optimization paradigm, we further deliver an optimization strategy for our proposed method. Experimental results on both synthetic and real data are provided to demonstrate the superiority of our proposed scheme.

Composite Antidisturbance Control for Non-Gaussian Stochastic Systems via Information-Theoretic Learning Technique.

In this article, a novel composite hierarchical antidisturbance control (CHADC) algorithm aided by the information-theoretic learning (ITL) technique is developed for non-Gaussian stochastic systems subject to dynamic disturbances. The whole control process consists of some time-domain intervals called batches. Within each batch, a CHADC scheme is applied to the system, where a disturbance observer (DO) is employed to estimate the dynamic disturbance and a composite control strategy integrating feedforward compensation and feedback control is adopted. The information-theoretic measure (entropy or information potential) is employed to quantify the randomness of the controlled system, based on which the gain matrices of DO and feedback controller are updated between two adjacent batches. In this way, the mean-square stability is guaranteed within each batch, and the system performance is improved along with the progress of batches. The proposed algorithm has enhanced disturbance rejection ability and good applicability to non-Gaussian noise environment, which contributes to extending CHADC theory to the general stochastic case. Finally, simulation examples are included to verify the effectiveness of theoretical results.

Generalized minimum error entropy Kalman filter for non-Gaussian noise

Error entropy is a well-known learning criterion in information theoretic learning (ITL), and it has already been applied to a wide range of fields. However, the shape of error entropy cannot be changed freely since its kernel function is the Gaussian kernel function, which causes the error entropy-based algorithm to handle only some specific kinds of noises. Benefiting from the property that the generalized Gaussian kernel function is free to adjust its shape, a novel Kalman-type filter algorithm based on the generalized minimum error entropy (GMEEKF) criterion is derived. Moreover, the mean error behavior, mean square error behavior, and computational complexity of the GMEEKF algorithm are analyzed. Finally, several simulations and experiments are performed to demonstrate the performance of the GMEEKF algorithm in comparison with the existing Kalman-type filter algorithms.

Recursive Hyperparameter-Free Criterion Learning

In the context of adaptive signal processing for non-Gaussian noise scenarios, the paradigm of information theoretic learning (ITL) has emerged useful due to their incorporation of higher order error-statistics, their improved convergence, and for their motivation from the standpoint of statistical mechanics. However, these ITL criteria are well-known to depend on scenario-dependent hyperparameter choices, whose optimal values, in-turn, depend on scenario dependent noise-statistics. This brief proposes hyperparameter free criterion learning using random Fourier features (RFF), which alleviates hyperparameter-dependence, and allows for scenario-independent generalization for underlying noise-distributions. For the proposed approach, detailed convergence analysis is presented and validated via relevant case-studies.

Robust M-estimation-based maximum correntropy Kalman filter

In this paper, a framework that combines an M-estimation and information-theoretic-learning (ITL)-based Kalman filter under impulsive noises is presented. The ITL-based methods make the most of the features of the data itself and can improve robustness by choosing an appropriate kernel bandwidth. However, small kernel bandwidths may lead to divergence. Nonetheless, robust-regression methods can improve the robustness from the statistical perspective and are independent of kernel bandwidth. This motivates us to fuse M-estimation-based weighting methods and the ITL-based Kalman filter. The proposed framework inhibits the divergence trend of ITL-based Kalman filters at low kernel bandwidth and improves the performance at large kernel bandwidth. Additionally, we use the unscented Kalman filtering method to extend the proposed algorithm to the nonlinear case. Monte Carlo simulations demonstrate the robustness and effectiveness of the proposed algorithm.

License Plate Detection via Information Maximization

License plate (LP) detection in the wild remains challenging due to the diversity of environmental conditions. Nevertheless, prior solutions have focused on controlled environments, such as when LP images frequently emerge as from an approximately frontal viewpoint and without scene text which might be mistaken for an LP. However, even for state-of-the-art object detectors, their detection performance is not satisfactory for real-world environments, suffering from various types of degradation. To solve these problems, we propose a novel end-to-end framework for robust LP detection, designed for such challenging settings. Our contribution is threefold: (1) A novel information-theoretic learning that takes advantage of a shared encoder, an LP detector and a scene text detector (excluding LP) simultaneously; (2) Localization refinement for generalizing the bounding box regression network to complement ambiguous detection results; (3) a large-scale, comprehensive dataset, LPST-110K, representing real-world unconstrained scenes including scene text annotations. Computational tests show that the proposed model outperforms other state-of-the-art methods on a variety of challenging datasets.

Weighted Error Entropy-Based Information Theoretic Learning for Robust Subspace Representation.

In most of the existing representation learning frameworks, the noise contaminating the data points is often assumed to be independent and identically distributed (i.i.d.), where the Gaussian distribution is often imposed. This assumption, though greatly simplifies the resulting representation problems, may not hold in many practical scenarios. For example, the noise in face representation is usually attributable to local variation, random occlusion, and unconstrained illumination, which is essentially structural, and hence, does not satisfy the i.i.d. property or the Gaussianity. In this article, we devise a generic noise model, referred to as independent and piecewise identically distributed (i.p.i.d.) model for robust presentation learning, where the statistical behavior of the underlying noise is characterized using a union of distributions. We demonstrate that our proposed i.p.i.d. model can better describe the complex noise encountered in practical scenarios and accommodate the traditional i.i.d. one as a special case. Assisted by the proposed noise model, we then develop a new information-theoretic learning framework for robust subspace representation through a novel minimum weighted error entropy criterion. Thanks to the superior modeling capability of the i.p.i.d. model, our proposed learning method achieves superior robustness against various types of noise. When applying our scheme to the subspace clustering and image recognition problems, we observe significant performance gains over the existing approaches.

Robust Spike-Based Continual Meta-Learning Improved by Restricted Minimum Error Entropy Criterion.

The spiking neural network (SNN) is regarded as a promising candidate to deal with the great challenges presented by current machine learning techniques, including the high energy consumption induced by deep neural networks. However, there is still a great gap between SNNs and the online meta-learning performance of artificial neural networks. Importantly, existing spike-based online meta-learning models do not target the robust learning based on spatio-temporal dynamics and superior machine learning theory. In this invited article, we propose a novel spike-based framework with minimum error entropy, called MeMEE, using the entropy theory to establish the gradient-based online meta-learning scheme in a recurrent SNN architecture. We examine the performance based on various types of tasks, including autonomous navigation and the working memory test. The experimental results show that the proposed MeMEE model can effectively improve the accuracy and the robustness of the spike-based meta-learning performance. More importantly, the proposed MeMEE model emphasizes the application of the modern information theoretic learning approach on the state-of-the-art spike-based learning algorithms. Therefore, in this invited paper, we provide new perspectives for further integration of advanced information theory in machine learning to improve the learning performance of SNNs, which could be of great merit to applied developments with spike-based neuromorphic systems.