An Efficient Compression Scheme for Bangla Natural Text by 16 Bit Unicode Transformation

Abstract

Text compression is an important area of research for natural language processing community. Due to huge number of characters in Unicode, it takes 16 bits to manage the total codes. Each of the language has a block of codes. We describe a technique to compress the 16 bit Unicode Bangla natural text along with English characters by 8-bit encoding. The encoding and decoding schemes consist of two steps. In step1 we replace some frequently used Bangla phrases with the extended ASCII characters and in the second step we assign the ASCII characters by an 8-bit binary value. We create a bit stream by appending the sequence of the binary values that we get after encoding the text. From this bit stream every 16 bit is taken and converted into a Unicode character. It creates an arbitrary sequence of Unicode characters with the values of the encoded bit stream. The scheme guarantees at least 50% compression in the worst case and 63% in the average case. This Unicode character sequence is then stored or transmitted. To decode it, we use the reverse process that was used for encoding. Finally we compare our result with well-known Huffman and LZW algorithm. Experimental result shows the effectiveness of the scheme.