Language discrimination via PPM model

Abstract

In this study, the lossless compression tool employing an adaptive statistical modeling technique called prediction by partial matching (PPM) is used for written language discrimination. PPM can well serve as a cryptographic tool in that, while encoding, as long as the algorithm itself is unknown to the third parties, it rearranges the plaintext in a hard-to-recover form. Furthermore, PPM algorithm yields lossless compression to far better rates (in bits per character - bpc) than that of many other conventional compression tools. Trained version of PPM, which uses training text to gather symbol frequencies, is employed during implementation. Language identification experiment results obtained by applying the PPM model on sample texts from English, French and Turkish corpora are given. The rate of success yielded that the performance of the system is highly dependent on the diversity, as well as the size of the target and training texts. In practice, if the training text itself is kept secret, the system would provide cryptographic security to promising degrees.