Abstract
With the advent of the era of big data, the demand of massive data processing applications is also growing. Currently, MapReduce is the most commonly used data processing programming model. However, in some data processing cases, it has some defects. MapReduce programming based on key/value pairs, matches the output of the Map tasks that will be transported to Reduce task nodes. The data with same key can only be processed by a Reduce node. If the data corresponding to a particular key or several keys accounts for most of all data, the Reduce node task will generate unbalanced load. In view of this defect, this paper proposes a new parallel programming model—Map-Balance-Reduce (MBR) programming model. It runs on our improved Hadoop framework and can effectively process the special data with unbalanced keys. This paper is based on two different scheduling, the processing and self-adaption scheduling. These two scheduling are designed to achieve MBR programming model. The actual testing results show that compared with MapReduce programming model, the MBR programming model under Hadoop can achieve the improvement of 9.7% to 17.6% in efficiency when testing data distributes unevenly. Furthermore, when testing conventional even-distributed data, it will only bring 1.02% time cost.
Published Version
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
