Data Quality Monitoring Software Research Articles

The Data Quality Monitoring Software for the CMS experiment at the LHC: past, present and future

The Data Quality Monitoring software is a central tool in the CMS experiment. It is used in the following key environments: (i) Online, for real-time detector monitoring; (ii) Offline, for the prompt-offline-feedback and final fine-grained data quality analysis and certification; (iii) Validation of all the reconstruction software production releases; (iv) Validation in Monte Carlo productions. Though the basic structure of the Run1 DQM system remains the same for Run2, between the Run1 and Run2 periods, the DQM system underwent substantial upgrades in many areas, not only to adapt to the surrounding infrastructure changes, but also to provide improvements to meet the growing needs of the collaboration with an emphasis on more sophisticated methods for evaluating data quality. We need to cope with the higher-energy and -luminosity proton-proton collision data, as well as the data from various special runs, such as Heavy Ion runs. In this contribution, we will describe the current DQM software, structure and workflow in the different environments. We then discuss the performance and our experiences with the DQM system in Run2. The main technical challenges which we have encountered and the solutions adopted during Run2 will also be discussed, including efficient use of memory in multithreading environments. Finally, we present the prospect of a future DQM upgrade with emphasis on functionality and long-term robustness for LHC Run3.

The Data Quality Monitoring Software for the CMS experiment at the LHC

The Data Quality Monitoring (DQM) Software is a central tool in the CMS experiment. Its flexibility allows for integration in several key environments: Online, for real-time detector monitoring; Offline, for the final, fine-grained data analysis and certification; Release-Validation, to constantly validate the functionality and the performance of the reconstruction software; in Monte Carlo productions. Since the end of data taking at a center of mass energy of 8 TeV, the environment in which the DQM lives has undergone fundamental changes. In turn, the DQM system has made significant upgrades in many areas to respond to not only the changes in infrastructure, but also the growing specialized needs of the collaboration with an emphasis on more sophisticated methods for evaluating data quality, as well as advancing the DQM system to provide quality assessments of various Monte Carlo simulations versus data distributions, monitoring changes in physical effects due to modifications of algorithms or framework, and enabling regression modeling for long-term effects for the CMS detector. The central tool to deliver Data Quality information is an interactive web site for browsing data quality histograms (DQMGUI). In this contribution the usage of the DQM Software in the different environments and its integration in the CMS Reconstruction Software Framework (CMSSW) and in all production workflows are presented, with emphasis on recent developments and improvement in advance of the LHC restart at 13 TeV. The main technical challenges and the adopted solutions to them will be also discussed with emphasis on functionality and long-term robustness.

The CMS data quality monitoring software: experience and future prospects

The Data Quality Monitoring (DQM) Software proved to be a central tool in the CMS experiment. Its flexibility allowed its integration in several environments: Online, for real-time detector monitoring; Offline, for the final, fine-grained Data Certification; Release-Validation, to constantly validate the functionality and the performance of the reconstruction software; in Monte Carlo productions. The central tool to deliver Data Quality information is a web site for browsing data quality histograms (DQM GUI). In this contribution the structure of the DQM framework is described and the usage of the DQM software in the different environments and the performance of the system after the first years of data taking are presented.

The ALICE Data Quality Monitoring: qualitative and quantitative review of three years of operations

ALICE (A Large Ion Collider Experiment) is a detector designed to study the physics of strongly interacting matter produced in heavy-ion collisions at the CERN Large Hadron Collider (LHC). Due to the complexity of ALICE in terms of number of detectors and performance requirements, Data Quality Monitoring (DQM) plays an essential role in providing online feedback on the data being recorded. It intends to provide shifters with precise and complete information to quickly identify problems, and as a consequence to ensure acquisition of high quality data.This paper presents a review of the ALICE DQM system during the first three years of LHC operations from a quantitative and qualitative point of view. We start by presenting the DQM software and tools before moving on to the various analyses carried out. An overview of the produced monitoring quantities is given, presenting the diversity of usage and flexibility of the DQM.Well-prepared shifters and experts, in addition to a precise organisation, were required to ensure smooth and successful operations. The description of the measures taken to ensure both aspects and an account of the DQM shifters' job are followed by a summary of the evolution of the system. We then give a quantitative review of the final setup of the system used during the whole year 2012. We conclude the paper with use cases where the DQM proved to be very valuable, scalable and efficient and with the plans for the coming years.

Online data quality monitoring system at BESIII

The online Data Quality Monitoring (DQM) tool plays an important role in the data recording process of HEP experiments. The BESIII DQM collects data from the online data flow, reconstructs them with offline reconstruction software and automatically analyzes the reconstructed data with user-defined algorithms. The DQM software is a scalable distributed system. The monitored results are gathered and displayed in various formats, which provides the shifter with current run information that can be used to identify problems quickly. This paper gives an overview of the DQM system at BESIII.

A data quality monitoring software framework for the BESIII experiment

Data quality monitoring (DQM) plays an important role in data taking at the BESIII experiments. DQM is used to monitor detector status and data quality. A DQM framework (DQMF) has been developed to make it possible to reuse the BESIII offline reconstruction system in the online environment. In this framework, the DQMF can also simulate a virtual data taking environment, transfer events to the event display, publish histograms to a histogram presenter in a fixed interval, and dump histograms into a ROOT file. The DQMF has been stably running throughout BESIII data taking.

The ALICE data quality monitoring

ALICE is one of the four experiments installed at the CERN Large Hadron Collider (LHC), especially designed for the study of heavy-ion collisions. The online Data Quality Monitoring (DQM) is an important part of the data acquisition (DAQ) software. It involves the online gathering, the analysis by user-defined algorithms and the visualization of monitored data. This paper presents the final design, as well as the latest and coming features, of the ALICE's specific DQM software called AMORE (Automatic MonitoRing Environment). It describes the challenges we faced during its implementation, including the performances issues, and how we tested and handled them, in particular by using a scalable and robust publish-subscribe architecture.We also review the on-going and increasing adoption of this tool amongst the ALICE collaboration and the measures taken to develop, in synergy with their respective teams, efficient monitoring modules for the sub-detectors. The related packaging and release procedure needed by such a distributed framework is also described. We finally overview the wide range of usages people make of this framework, and we review our own experience, before and during the LHC start-up, when monitoring the data quality on both the sub-detectors and the DAQ side in a real-world and challenging environment.

A software framework for Data Quality Monitoring in ATLAS

Data Quality Monitoring (DQM) is an important and integral part of the data taking and data reconstruction of HEP experiments. In an online environment, DQM provides the shift crew with live information beyond basic monitoring. This is used to overcome problems promptly and help avoid taking faulty data. During the off-line reconstruction DQM is used for more complex analysis of physics quantities and its results are used to assess the quality of the reconstructed data. The Data Quality Monitoring software Framework (DQMF) which has been provided for the ATLAS experiment performs analysis of monitoring data through user defined algorithms and relays the summary of the analysis results to the configurable Data Quality output stream. From this stream the results can be stored to a database, displayed on a GUI, or used to make some other relevant actions with respect to the operational environment i.e. sending alarms, stopping the run. This paper describes the implementation of the DQMF and discusses experience from usage and performance of the DQMF during ATLAS commissioning.