HEP Data Research Articles

Exploring XRootD Optimisations Using Advanced Monitoring in the UK

XRootD servers are commonplace to many parts of HEP data management and are a key component to data access and management strategies in both the WLCG and OSG. Deployments of XRootD instances across the UK have demonstrated the versatility and expandability of this data management software. As we become more reliant on these services, there is a requirement to collect low-level metrics to monitor service performance and behaviour. This presentation will share our recent experiences in the collection, monitoring and analysis of such metrics from servers at UK WLCG-Tier2 sites. Building on work presented at VCHEP-2021 we have been collecting XRootD service metrics from various UK sites. In addition to this, we have developed novel technologies for recording and verifying metrics from our grid storage systems at the Edinburgh Tier2. Our custom tooling is integrated with our recently deployed monitoring platform. This work complements ongoing WLCG efforts focussing on capturing and analysing XRootD based traffic flow. Through capturing and analysing additional metrics we’re able to better remotely assess service health and provide insights to assist in debugging. The goal of this is to offer a centralised resource that monitors the performance and health of remote storage services across the UK. Making use of these additional service metrics also allows us a greater insight into the performance of XRootD Proxy File Caches. By applying machine learning techniques to already collected metrics, we aim to determine if gains can potentially be made in optimising the behaviour of these caching systems in production at a UK WLCG-Tier2. The result of this potentially being the building and deployment of custom decision-making libraries for use with XRootDPFC services.

Using a Neural Network to Approximate the Negative Log Likelihood Function

An increasingly frequent challenge faced in HEP data analysis is to characterize the agreement between a prediction that depends on a dozen or more model parameters—such as predictions coming from an effective field theory (EFT) framework—and the observed data. Traditionally, such characterizations take the form of a negative log likelihood (NLL) function, which can only be evaluated numerically. The lack of a closed-form description of the NLL function makes it difficult to convey results of the statistical analysis. Typical results are limited to extracting “best fit” values of the model parameters and 1D intervals or 2D contours extracted from scanning the higher dimensional parameter space. It is desirable to explore these high-dimensional model parameter spaces in more sophisticated ways. One option for overcoming this challenge is to use a neural network to approximate the NLL function. This approach has the advantage of being continuous and differentiable by construction, which are essential properties for an NLL function and may also provide useful handles in exploring the NLL as a function of the model parameters. In this talk, we describe the advantages and limitations of this approach in the context of applying it to a CMS data analysis using the framework of EFT.

An array-oriented Python interface for FastJet

Analysis on HEP data is an iterative process in which the results of one step often inform the next. In an exploratory analysis, it is common to perform one computation on a collection of events, then view the results (often with histograms) to decide what to try next. Awkward Array is a Scikit-HEP Python package that enables data analysis with array-at-a-time operations to implement cuts as slices, combinatorics as composable functions, etc. However, most C++ HEP libraries, such as FastJet, have an imperative, one-particle-at-a-time interface, which would be inefficient in Python and goes against the grain of the array-at-a-time logic of scientific Python. Therefore, we developed fastjet, a pip-installable Python package that provides FastJet C++ binaries, the classic (particle-at-a-time) Python interface, and the new array-oriented interface for use with Awkward Array. The new interface streamlines interoperability with scientific Python software beyond HEP, such as machine learning. In one case, adopting this library along with other array-oriented tools accelerated HEP analysis code by a factor of 20. It was designed to be easily integrated with libraries in the Scikit-HEP ecosystem, including Uproot (file I/O), hist (histogramming), Vector (Lorentz vectors), and Coffea (high-level glue). We discuss the design of the fastjet Python library, integrating the classic interface with the array oriented interface and with the Vector library for Lorentz vector operations. The new interface was developed as open source.

Declarative interfaces for HEP data analysis: FuncADL and ADL/CutLang

Analysis description languages are declarative interfaces for HEP data analysis that allow users to avoid writing event loops, simplify code, and enable performance improvements to be decoupled from analysis development. One example is FuncADL, inspired by functional programming and developed using Python as a host language. FuncADL borrows concepts from database query languages to isolate the interface from the underlying physical and logical schemas. The same query can be used to select data from different sources and formats and with different execution mechanisms. FuncADL is one of the tools being developed by IRIS-HEP for highly scalable physics analysis for the LHC and HL-LHC. FuncADL is demonstrated by implementing example analysis tasks designed by HSF and IRIS-HEP. Another language example is ADL, which expresses the physics content of an analysis in a standard and unambiguous way, independent of computing frameworks. In ADL, analyses are described in human-readable text files composed of blocks with a keyword-expression structure. Two infrastructures are available to render ADL executable: CutLang, a runtime interpreter written in C++; and adl2tnm, a transpiler converting ADL into C++ or Python code. ADL/CutLang are already used in several physics studies and educational projects, and are adapted for use with LHC Open Data.

A GPU-Based Kalman Filter for Track Fitting

Computing centres, including those used to process High-Energy Physics data and simulations, are increasingly providing significant fractions of their computing resources through hardware architectures other than x86 CPUs, with GPUs being a common alternative. GPUs can provide excellent computational performance at a good price point for tasks that can be suitably parallelized. Charged particle (track) reconstruction is a computationally expensive component of HEP data reconstruction, and thus needs to use available resources in an efficient way. In this paper, an implementation of Kalman filter-based track fitting using CUDA and running on GPUs is presented. This utilizes the ACTS (A Common Tracking Software) toolkit; an open source and experiment-independent toolkit for track reconstruction. The implementation details and parallelization approach are described, along with the specific challenges for such an implementation. Detailed performance benchmarking results are discussed, which show encouraging performance gains over a CPU-based implementation for representative configurations. Finally, a perspective on the challenges and future directions for these studies is outlined. These include more complex and realistic scenarios which can be studied, and anticipated developments to software frameworks and standards which may open up possibilities for greater flexibility and improved performance.

Interactive Web-Based Visualization of Multidimensional Physical and Astronomical Data.

In this article, we propose expanding the use of scientific repositories such as Zenodo and HEP data, in particular, to better study multiparametric solutions of physical models. The implementation of interactive web-based visualizations enables quick and convenient reanalysis and comparisons of phenomenological data. To illustrate our point of view, we present some examples and demos for dark matter models, supersymmetry exclusions, and LHC simulations.

Readable and efficient HEP data analysis with bamboo

With the LHC continuing to collect more data and experimental analyses becoming increasingly complex, tools to efficiently develop and execute these analyses are essential. The bamboo framework defines a domain-specific language, embedded in python, that allows to concisely express the analysis logic in a functional style. The implementation based on ROOT’s RDataFrame and cling C++ JIT compiler approaches the performance of dedicated native code. Bamboo is currently being used for several CMS Run 2 analyses that rely on the NanoAOD data format, which will become more common in Run 3 and beyond, and for which many reusable components are included, but it provides many possibilities for customisation, which allow for straightforward adaptation to other formats and workflows.

Recent advances in ADL, CutLang and adl2tnm

This paper presents an overview and features of an Analysis Description Language (ADL) designed for HEP data analysis. ADL is a domainspecific, declarative language that describes the physics content of an analysis in a standard and unambiguous way, independent of any computing frameworks. It also describes infrastructures that render ADL executable, namely CutLang, a direct runtime interpreter (originally also a language), and adl2tnm, a transpiler converting ADL into C++ code. In ADL, analyses are described in humanreadable plain text files, clearly separating object, variable and event selection definitions in blocks, with a syntax that includes mathematical and logical operations, comparison and optimisation operators, reducers, four-vector algebra and commonly used functions. Recent studies demonstrate that adapting the ADL approach has numerous benefits for the experimental and phenomenological HEP communities. These include facilitating the abstraction, design, optimization, visualization, validation, combination, reproduction, interpretation and overall communication of the analysis contents and long term preservation of the analyses beyond the lifetimes of experiments. Here we also discuss some of the current ADL applications in physics studies and future prospects based on static analysis and differentiable programming.

Porting CMS Heterogeneous Pixel Reconstruction to Kokkos

Programming for a diverse set of compute accelerators in addition to the CPU is a challenge. Maintaining separate source code for each architecture would require lots of effort, and development of new algorithms would be daunting if it had to be repeated many times. Fortunately there are several portability technologies on the market such as Alpaka, Kokkos, and SYCL. These technologies aim to improve the developer’s productivity by making it possible to use the same source code for many different architectures. In this paper we use heterogeneous pixel reconstruction code from the CMS experiment at the CERNL LHC as a realistic use case of a GPU-targeting HEP reconstruction software, and report experience from prototyping a portable version of it using Kokkos. The development was done in a standalone program that attempts to model many of the complexities of a HEP data processing framework such as CMSSW. We also compare the achieved event processing throughput to the original CUDA code and a CPU version of it.

Overall quality optimization for DQM stage in High Energy Physics experiments

Data Acquisition (DAQ) and Data Quality Monitoring (DQM) are key parts in the HEP data chain, where the data are processed and analyzed to obtain accurate monitoring quality indicators. Such stages are complex, including an intense processing work-flow and requiring a high degree of interoperability between software and hardware facilities. Data recorded by DAQ sensors and devices are sampled to perform live (and offline) DQM of the status of the detector during data collection providing to the system and scientists the ability to identify problems with extremely low latency, minimizing the amount of data that would otherwise be unsuitable for physical analysis. DQM stage performs a large set of operations (Fast Fourier Transform (FFT), clustering, classification algorithms, Region of Interest, particles tracking, etc.) involving the use of computing resources and time, depending on the number of events of the experiment, sampling data, complexity of the tasks or the quality performance. The objective of our work is to show a proposal with aim of developing a general optimization of the DQM stage considering all these elements. Techniques based on computational intelligence like EA can help improve the performance and therefore achieve an optimization of task scheduling in DQM.

Automated performance tuning of distributed storage system based on deep reinforcement learning

Automated performance tuning is a tricky task for a large scale storage system. Traditional methods highly reply on experience of system administrators and cannot adapt to changes of working load and system configurations. Reinforcement learning is a promising machine learning paradigm which learns an optimized strategy from the trials and errors between agents and environments. Combining with the strong feature learning capability of deep learning, deep reinforcement learning has showed its success in many fields. We implemented a performance parameter tuning engine based on deep reinforcement learning for Lustre file system, a distributed file system widely used in HEP data centres. Three reinforcement learning algorithms: Deep Q-learning, A2C, and PPO are enabled in the tuning engine. Experiments show that, in a small test bed, with IOzone workload, this method can increase the random read throughput by about 30% compared to default settings of Lustre. In the future, it is possible to apply this method to other parameter tuning use cases of data centre operations.

Production Operations Management System (POMS) for Fermilab Experiments

The Production Operations Management System (POMS) software allows production teams and analysis groups across multiple Fermilab experiments to launch, modify and monitor large scale campaigns of related Monte Carlo or data processing jobs, and currently manages the majority of production computing of experiments at Fermilab. POMS provides a web service interface that enables automated job submission on distributed resources according to customers’ requests, as well as subsequent monitoring of those submissions as well as recovery of failed submissions, debugging and record keeping. POMS interfaces with existing HEP data access, processing, movement and monitoring tools at Fermilab, including Jobsub (for job submission), SAM (for data management), and Landscape (for monitoring), and in combination with them handles the creation of intermediate datasets of files for multistage campaigns. POMS visualizes these campaigns with an interactive GUI interface. An important feature of POMS is a one-to-one connection between the GUI campaign visualizer and a text based representation of the complete campaign configuration. An extensible library of template campaign configurations is available. The templates are user modifiable and map cleanly to and from the GUI description of the campaign. This paper will discuss the current usage, history, and some technical details of POMS, as well as planned future extensions, particularly interfacing with Rucio as well as SAM for file movement.

ROOT I/O compression improvements for HEP analysis

We overview recent changes in the ROOT I/O system, enhancing it by improving its performance and interaction with other data analysis ecosystems. Both the newly introduced compression algorithms, the much faster bulk I/O data path, and a few additional techniques have the potential to significantly improve experiment’s software performance. The need for efficient lossless data compression has grown significantly as the amount of HEP data collected, transmitted, and stored has dramatically increased over the last couple of years. While compression reduces storage space and, potentially, I/O bandwidth usage, it should not be applied blindly, because there are significant trade-offs between the increased CPU cost for reading and writing files and the reduces storage space.

Machine Learning-based Anomaly Detection of Ganglia Monitoring Data in HEP Data Center

This paper introduces a generic and scalable anomaly detection framework. Anomaly detection can improve operation and maintenance efficiency and assure experiments can be carried out effectively. The framework facilitates common tasks such as data sample building, retagging and visualization, deviation measurement and performance measurement for machine learning-based anomaly detection methods. The samples we used are sourced from Ganglia monitoring data. There are several anomaly detection methods to handle spatial and temporal anomalies within the framework. Finally, we show the rudimental application of the framework on Lustre distributed file systems in daily operation and maintenance.

МОДИФІКАЦІЯ ГЕНЕТИЧНИХ АЛГОРИТМІВ НА ОСНОВІ МЕТОДУ НЕЦЕНТРОВАНИХ ГОЛОВНИХ КОМПОНЕНТ ТА СТАНДАРТНІ ТЕСТИ

The purpose of this article is to develop the necessary mathematical description of the method of the uncentered principal component analysis for the optimization of the genetic algorithm. A secondary goal is to evaluate the approximations for its application for HEP data analysis and to develop its program implementation for genetic algorithm together with a new operator based on the method of the uncentered principal components (UPCA-operator) and to check its efficiency on the example benchmark tests.

Evolution of the VISPA-project

VISPA (Visual Physics Analysis) is a web-platform that enables users to work on any secure shell (SSH) reachable resource using just their webbrowser. It is used successfully in research and education for HEP data analysis. The emerging JupyterLab is an ideal choice for a comprehensive, browser-based, and extensible work environment and we seek to unify it with the efforts of the VISPA-project. The primary objective is to provide the user with the freedom to access any external resources at their disposal, while maintaining a smooth integration of preconfigured ones including their access permissions. Additionally, specialized HEP tools, such as native format data browsers (ROOT, PXL), are being migrated from VISPA- to JupyterLab-extensions as well. We present these concepts and their implementation progress.

Apache Spark usage and deployment models for scientific computing

This talk is about sharing our recent experiences in providing data analytics platform based on Apache Spark for High Energy Physics, CERN accelerator logging system and infrastructure monitoring. The Hadoop Service has started to expand its user base for researchers who want to perform analysis with big data technologies. Among many frameworks, Apache Spark is currently getting the most traction from various user communities and new ways to deploy Spark such as Apache Mesos or Spark on Kubernetes have started to evolve rapidly. Meanwhile, notebook web applications such as Jupyter offer the ability to perform interactive data analytics and visualizations without the need to install additional software. CERN already provides a web platform, called SWAN (Service for Web-based ANalysis), where users can write and run their analyses in the form of notebooks, seamlessly accessing the data and software they need. The first part of the presentation talks about several recent integrations and optimizations to the Apache Spark computing platform to enable HEP data processing and CERN accelerator logging system analytics. The optimizations and integrations, include, but not limited to, access of kerberized resources, xrootd connector enabling remote access to EOS storage and integration with SWAN for interactive data analysis, thus forming a truly Unified Analytics Platform. The second part of the talk touches upon the evolution of the Apache Spark data analytics platform, particularly sharing the recent work done to run Spark on Kubernetes on the virtualized and container-based infrastructure in Openstack. This deployment model allows for elastic scaling of data analytics workloads enabling efficient, on-demand utilization of resources in private or public clouds.

A Parallelised ROOT for Future HEP Data Processing

In the coming years, HEP data processing will need to exploit parallelism on present and future hardware resources to sustain the bandwidth requirements. As one of the cornerstones of the HEP software ecosystem, ROOT embraced an ambitious parallelisation plan which delivered compelling results. In this contribution the strategy is characterised as well as its evolution in the medium term. The units of the ROOT framework are discussed where task and data parallelism have been introduced, with runtime and scaling measurements. We will give an overview of concurrent operations in ROOT, for instance in the areas of I/O (reading and writing of data), fitting / minimization, and data analysis. This paper introduces the programming model and use cases for explicit and implicit parallelism, where the former is explicit in user code and the latter is implicitly managed by ROOT internally.

Virtual Reality and game engines for interactive data visualization and event displays in HEP, an example from the ATLAS experiment

Interactive 3D data visualization plays a key role in HEP experiments, as it is used in many tasks at different levels of the data chain. Outside HEP, for interactive 3D graphics, the game industry makes heavy use of so-called “game engines”, modern software frameworks offering an extensive set of powerful graphics tools and cross-platform deployment. Recently, a very strong support for Virtual Reality (VR) technology has been added to such engines. In this talk we explore the usage of game engines and VR for HEP data visualization, discussing the needs, the challenges and the issues of using such technologies. We will also make use of ATLASrift, a VR application developed by the ATLAS experiment, to discuss the lessons learned while developing it using the game engine Unreal Engine, and the feedback on the use of Virtual Reality we got from users while using it at many demonstrations and public events.

Building a scalable python distribution for HEP data analysis

There are numerous approaches to building analysis applications across the high-energy physics community. Among them are Python-based, or at least Python-driven, analysis workflows. We aim to ease the adoption of a Python-based analysis toolkit by making it easier for non-expert users to gain access to Python tools for scientific analysis. Experimental software distributions and individual user analysis have quite different requirements. Distributions tend to worry most about stability, usability and reproducibility, while the users usually strive to be fast and nimble. We discuss how we built and now maintain a python distribution for analysis while satisfying requirements both a large software distribution (in our case, that of CMSSW) and user, or laptop, level analysis. We pursued the integration of tools used by the broader data science community as well as HEP developed (e.g., histogrammar, root_numpy) Python packages. We discuss concepts we investigated for package integration and testing, as well as issues we encountered through this process. Distribution and platform support are important topics. We discuss our approach and progress towards a sustainable infrastructure for supporting this Python stack for the CMS user community and for the broader HEP user community.