Efficient Log Parsing and Visualization System for Large-Scale Logs: A Data-Centric Approach Using Custom Data Structures

Abstract

In the realm of debugging and system optimization, testers frequently deal with large-scale logs from systems such as 5G communication networks. Handling, filtering, and visualizing these logs efficiently are crucial for understanding system performance and identifying issues. This paper presents an efficient log parsing and visualization system that leverages custom data structures to handle logs more effectively, with significant improvements in filtering and querying after initial parsing. The system provides multiple ways to filter logs, including by parent attributes, supports inline scripting to manipulate data tables, and offers various visualization techniques. The tool is optimized for fast filtering and querying, compared to other approaches, and will be available as an open-source project to the broader testing and debugging community.