Design and Implementation of Componentized Big Data Platform

Abstract

The rapid development of the Internet has produced massive amounts of data, and has promoted the development of various big data components in the big data field, such as Hadoop, Spark and other big data components. However, many big data components face many problems in actual industrial production, make the construction and maintenance cost of the big data platform high. Although there is an automated deployment platform for big data components on the market, it lacks comprehensive management and monitoring functions for the entire big data platform, resulting in users not being able to fully pay attention to all aspects of the big data platform, and the service and resource management is not flexible enough. Therefore, we have designed and implemented an open componentized big data platform, so that users can not only easily install and deploy big data components in the cluster, but also manage the resources and services of big data components conveniently and flexibly. Besides, complete monitoring information also provides the necessary guarantee for the stable running of the big data platform. The open componentized big data platform provides users with one-stop big data services.