AVERAGE WORD LENGTH AND TEXT REDUNDANCY VARIABILITY: FRENCH TEXTS CASE STUDY

Malvina Marinashvili

doi:10.23856/3849

Abstract

The redundancy and average word length correlation in French texts have been researched. This correlation has been evaluated on the basis of analysis of entropy, redundancy and average word length for literary, scientific, and publicistic texts. It has been revealed that the variability of text redundancy correlates well with the variability of average word length, if calculating the average word length of an individual text we exclude the length of words belonging to the exponential tail of entropy curve. In this regard it is proposed to distinguish between two average word lengths of text: the average length of a word belonging to the exponentially decaying tail of entropy and the average length of a word not belonging to the exponential tail of entropy.

Highlights

Natural language is a complex system with a hierarchical structure, number of set rules and internal connections
We suggest the average word length of a text consists of two lengths and in assessing entropy and text redundancy it is important to take account of not one average word length, but two
We have studied the variability of average word length and redundancy versus maximum entropy of text, which in information theory is understood as information capacity of a message

Summary

Introduction

Natural language is a complex system with a hierarchical structure, number of set rules and internal connections. In great number of linguistic researches (Zipf, 1949; Miller et al, 1958; Mikros et al, 2005; Köhler, 2005; Strauss et al, 2007; Popescu et al, 2013 and others) the word frequency, its length (or their correlation: Zipf’s law) have been investigated and some regularities of words distribution in the texts of different functional styles, genres and various language case study revealed. The relation between these characteristics hasn’t been studied enough yet

Methods

Results

Conclusion