Transitive Entropy—A Rank Ordered Approach for Natural Sequences

Abstract

Information theoretic entropy measures are calculated from estimates of the probabilities of the constituent symbolic events. In natural sequences, such as those occurring in human language, the probabilistic structure typically follows a rank ordering pattern. Entropy has been used to model language by using large data sets to characterize the underlying source. To model the dynamic characteristics of language, methods such as N-gram entropy are normally used which rely on a very large database from which the statistical samples can be obtained. However it is of interest to apply entropy based methods for classifying natural sequence behavior, such as characterizing changes in language due to diseases such as dementia, using only a small number of samples. There are multiple problems with this approach however: the number of samples required and in addition, we show that degenerative solutions can occur when using current entropy measures which can render them less suitable for classifying disorders in natural language which may result in rank reordering variations of the feature space probability distributions. We propose a new probabilistic measure, termed Transitive Entropy which overcomes this problem. We examine the properties of the proposed entropy measure and demonstrate its effectiveness on successfully classifying patient dementia by application to a probabilistic model of pause length in their speech.