Abstract
In order to shorten the average execution time of ETL tasks in data integration, and utilize the resource of the cluster reasonably, we proposed a multi-objective scheduling optimization algorithm named Moetsa for ETL tasks, which combined the average execution time of tasks and the load balancing of nodes as the optimization objectives. In Moetsa, random forest (RF) model and non-dominated sorting genetic algorithm II (NSGA-II) were employed to improve load balancing for reasonable utilization of resource and reduce task execution time. Specifically, A RF model built to predict the overheads of task execution and a load balancing evaluation function were integrated with NSGA-II to obtain the optimal task scheduling scheme. Our experimental results show that Moetsa method not only effectively improved the resource utilization of the cluster but also shorten the average execution time of tasks.
Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
