Evaluating InfluxDB and ClickHouse database technologies for improvements of the ATLAS operational monitoring data archiving

Igor Soloviev,Giuseppe Avolio,Matei-Eugen Vasile

doi:10.1088/1742-6596/1525/1/012027

Abstract

The Trigger and Data Acquisition system of the ATLAS experiment at the Large Hadron Collider at CERN is composed of a large number of distributed hardware and software components which provide the data-taking functionality of the overall system. During data-taking, huge amounts of operational data are created in order to constantly monitor the system. The Persistent Back-End for the ATLAS Information System of TDAQ (P-BEAST) is a system based on a custom-built timeseries database. It is used to archive and retrieve any operational monitoring data for the applications requesting it. P-BEAST stores about 18 TB of highly compacted and compressed raw monitoring data per year. Since P-BEAST’s creation, several promising database technologies for fast access to time-series have become available. InfluxDB and ClickHouse were the most promising candidates for improving the performance and functionality of the current implementation of P-BEAST. This paper presents a short description of main features of both technologies and a description of the tests ran on both database systems. Then, the results of the performance testing performed using a subset of archived ATLAS operational monitoring data are presented and compared.

Highlights

The Trigger and Data Acquisition (TDAQ) system of the ATLAS [1] experiment at the Large Hadron Collider (LHC) [2] at CERN is currently composed of a large number of distributed hardware and software components
The Persistent Back-End for the ATLAS Information System of TDAQ (P-BEAST) [3] is a system based on a custom-built time-series database
Since P-BEAST has been put into production, 4 years ago, several promising database technologies that feature fast access to time-series or column-oriented data become available

Summary

Introduction

The Trigger and Data Acquisition (TDAQ) system of the ATLAS [1] experiment at the Large Hadron Collider (LHC) [2] at CERN is currently composed of a large number of distributed hardware and software components. Testing strategy P-BEAST can store integers, floats, strings and arrays and structures of these data types. The factors directly relevant to the write performance testing described here are concerned with the database systems’ capability of mapping existing P-BEAST data on the data model supported by each database technology: InfluxDB is a time-series database with support for storing integers, floats, strings.

Results

Conclusion