Convergence of a Fixed-Point Minimum Error Entropy Algorithm

Yu Zhang,Xi Liu,Badong Chen,Jose Principe,Zejian Yuan

doi:10.3390/e17085549

Yu Zhang, Xi Liu + Show 3 more

Open Access

https://doi.org/10.3390/e17085549

Copy DOI

Journal: Entropy	Publication Date: Aug 3, 2015
Citations: 47	License type: CC BY 4.0

Affiliation: Xi'an Jiaotong University, Zhejiang University

Abstract

The minimum error entropy (MEE) criterion is an important learning criterion in information theoretical learning (ITL). However, the MEE solution cannot be obtained in closed form even for a simple linear regression problem, and one has to search it, usually, in an iterative manner. The fixed-point iteration is an efficient way to solve the MEE solution. In this work, we study a fixed-point MEE algorithm for linear regression, and our focus is mainly on the convergence issue. We provide a sufficient condition (although a little loose) that guarantees the convergence of the fixed-point MEE algorithm. An illustrative example is also presented.

Highlights

In recent years, information theoretic measures, such as entropy and mutual information, have been widely applied in domains of machine learning (so called information theoretic learning (ITL) [1]) and Entropy 2015, 17 signal processing [1,2]
The goal of this paper is to study the convergence of a fixed-point minimum error entropy (MEE) algorithm and provide a sufficient condition that ensures the convergence to a unique solution
We give an illustrative example to verify the derived sufficient condition that guarantees the convergence of the fixed-point MEE algorithm

Summary

Introduction

Information theoretic measures, such as entropy and mutual information, have been widely applied in domains of machine learning (so called information theoretic learning (ITL) [1]) and Entropy 2015, 17 signal processing [1,2]. With a gradient based learning algorithm, one has to select a proper learning rate (or step-size) to ensure the stability and achieve a better tradeoff between misadjustment and convergence speed [4,5,6,7]. Another more promising search algorithm is the fixed-point iterative algorithm, which is step-size free and is often much faster than gradient based methods [11]. For the gradient based MEE algorithms, the convergence problem has already been studied and some theoretical results have been obtained [6,7].

Fixed-Point MEE Algorithm

Convergence of the Fixed-Point MEE

PdX where R MEE

N 2σ 3

Illustrative Example

Conclusion