Research and implementation of campus network mass log collection platform based on elastic stack

Abstract

With the wide application of cloud computing technology and the rise of container technology, many business architectures have undergone huge changes. In recent years, container technology, microservices and other applications have gradually matured, and many enterprises' business systems have gradually migrated to the microservices model, with many microservices running in their respective business systems. In the production environment, logs play an extremely important role, and the logs in the production environment are required for business system operation monitoring, performance optimization, and troubleshooting. In the traditional business model, most of the services running in the production environment adopt independent localized log storage mode, and the operation and maintenance personnel are often unable to locate them quickly, while the large amount of log storage also puts great pressure on the storage of IT systems, and it is difficult to tap the data value of logs with the independent log mode. This paper proposes an Elastic Stack-based massive log management platform based on the business in our campus network as a prototype, deploys log collection components in various business systems in the campus network, classifies logs with appropriate rules, realizes centralized management and visual output, solves the massive log management problem of the current business systems in the campus network, and helps operation and maintenance and R and D.