Performance Analysis of Distributed Processing System using Shard Selection Techniques on Elasticsearch

Abstract

In distributed systems multiple computers interact with each other over a network to perform data processing operations parallel over a network. Distributed systems can be setup depending on the application and its data with different combination of replications and then the partitioning of the data can be performed using sharding technique to enhance the scalability, consistency and fault-tolerance issues. The gaps identified in existing sharding techniques are, it is difficult to determine which particular shard or partition need to be searched to retrieve the relevant document matching to the user query. And also to rank the documents which has highest relevance to the user query. The main goal is to develop a mechanism to handle large scale data processing and searching operations using efficient sharding technique. The proposed paper aims to enhance the performance of distributed processing systems by applying effective shard partitioning and efficient shard selection techniques and perform the comparative study analysis of shard selection techniques considering precision, MAP and cost measures. The results interpret that sharding technique provides efficient mechanism to handle large scale data. Effectiveness and efficiency of shard selection techniques interpret that Rank-S algorithm performance better compared to CORI and Redde algorithms by reducing the overall cost by 28%. Sharding technique is an efficient mechanism to process big data, and provides better scalability and fault-tolerance.