Redundant data high-efficiency compression based on distributed parallel algorithm

Abstract

Redundancy occurs when multiple consumers of the network attempt to access the similar content online. Many state-of-the-art studies have been proposed to remove redundancy from the network and to enhance the network efficiency of Internet applications. Still there is a necessity to explore more data redundancy elimination (DRE) techniques, especially for defense-based networks to enhance the data transmission speed and to compress the irrelevant data optimally for reducing the network latency. In order to improve the optimal storage capacity of redundant data over serial hybrid network cascade database, a high-efficiency redundancy elimination method based on the distributed parallel algorithm is proposed in this paper. The distributed storage structure model for handling redundant data of serial mixed network cascade database is designed and tested in simulated environment. The extraction of features of redundant data is performed by using distributed hybrid feature mining technique. The dimensionality reduction of redundant data is also attained by devising the feature transformation method to remove the unwanted features of data. The two benchmarked techniques have been selected for comparative study and to evaluate the performance of the proposed method. The simulation results show that the proposed DRE method can significantly reduce the redundancy from the network traffic. The bandwidth utilization is improved, the duplicate data are also compressed optimally and it is proved that the proposed DRE method is viable for large networks to eliminate the redundant data.