СТИСНЕННЯ ПРИРОДНОМОВНИХ ТЕКСТІВ РЕВЕРСНИМИ МУЛЬТИРОЗДІЛЬНИКОВИМИ КОДАМИ

Abstract

We study a class of binary reverse multi-delimiter (RMD) data compression codes in application to natural language text compression. The RMD-codewords start with delimiters, i.e., prefixes of the form that cannot occur in other places of the codeword. The position of the delimiter in an RMD codeword differs from its position in “direct” multi-delimiter (MD) codes, where delimiters are codeword suffixes. RMD and MD codes possess many useful properties, such as unique decodability, completeness, universality, synchronizability, asymptotic densities, and finite automaton acceptability. For RMD-codes, we construct a monotonic mapping from the set of natural numbers to the set of codewords. For original MD-codes, hitherto, this was an open question. The discovered mapping and the byte quantification of a decoding automaton allow us to develop very fast byte-aligned algorithms for decoding and direct Boyer–Moore style search in compressed files. Compared with the known byte (SCDC) and Fibonacci codes, RMD codes demonstrate the best compression ratio on natural language texts (more than four times closer to the entropy bound than that of SCDC). Computer experiments demonstrate that RMD codes can be decoded almost as fast as SCDC and times faster than Fibonacci codes. In natural language text compression, we also practiced the RMD-encoding as a preprocessing tool, which improves the performance of the known modern powerful archivers.