Design and implementation of parallel word occurrence frequency based on message passing interface

Abstract

For large files and files distributed all around the internet, this paper proposed and realized parallel word occurrence frequency counting using C and message passing interface programming. And to solve the specific word occurrence problems like how to extract word precisely, traverse file completely, eliminate redundant words, adapt to any text files, and count the number of letters and word. An improved serial word counting algorithm was designed and implemented. Then the parallel partition strategy based on both balanced file size and balanced task were discussed and a combined partition strategy was designed and implemented using C programming and MapReduce message passing interface (MR-MPI) library. After that the counting of word occurrence frequency for relatively small and very large text file were realized. And last we executed the same task in parallel computing environment. The test showed that the algorithm proposed implements word counting for files of large data set, both in parallel high performance computing and cloud environment with short runtime and high accuracy. Last, some discussion on the parallel implementation of the word counting problem in Google's cloud environment is given out.