WordCode using WordTrie

Abstract

Computers work with text data by assigning a code for each character, called encoding. Character-encoding techniques emerged in the late 1960s, and a similar type of technique is still used to encode text data. Computers can only understand alphabets, not words. In this article, we develop an approach that enables computers to understand words. We introduce a word-based encoding of text data named WordCode. WordCode encodes the most frequent set of characters (i.e., words) found in Internet directories with a dynamic code combination. Although some dictionary-encoding techniques have been proposed, we still tend to use character encoding, such as Unicode, to encode text data. Dictionary-encoding techniques have not been adopted due to the massive size of the code page and the complexity in accessing the code page. In this article, we introduce a customised trie named WordTrie to store words for faster encoding and decoding. We generate the code combination in such a way that the size of the WordCode for a word is always smaller than the total size of the character coding. Our experimental results from encoding text files from the Gutenberg corpus, Canterbury corpus, large corpus, Calgary corpus and Silesia corpus using WordCode show an up to 19.9% reduction in file size with respect to character-based encoding. This smaller file size means that less storage space is needed and results in faster processing and communication of text data.