Abstract
The Message Queue (MQ) architecture is an asynchronous communication scheme that provides an attractive solution for certain scenarios in a distributed computing model. The introduction of MQ as an intermediate component in-between the interacting processes allows to decouple the end-points making the system more flexible and providing high scalability and redundancy. DIRAC is a general-purpose interware software for distributed computing systems, which offers a common interface to a number of heterogeneous providers and guarantees transparent and reliable usage of the resources. The DIRAC platform has been adapted by several scientific projects, including High Energy Physics communities like LHCb, the Linear Collider and Belle2. A Message Queue generic interface has been incorporated into the DIRAC framework to help solving the scalability challenges that must be addressed during LHC Run3, starting in 2021. It allows to use the MQ scheme for a message exchange among the DIRAC components or to communicate with third-party services. Within this contribution we describe the integration of MQ systems with DIRAC and several use cases are shown. Message Queues are foreseen to be used in the pilot logging system, and as a backbone of the DIRAC component logging system and monitoring.
Highlights
We live in a world of large data streams, which are constantly provided by various sources and need to be processed efficiently
High Energy Physics (HEP) communities face similar challenges, since the data produced by the experiments’ detectors and by the Monte Carlo simulation jobs form a significant data stream that must be processed in a coordinated manner [1]
The DIRAC interware was adopted as a computing solution by the HEP experiments like LHCb and Belle2, and by many other projects which use it as a platform to perform advanced GRID operations
Summary
We live in a world of large data streams, which are constantly provided by various sources and need to be processed efficiently This massive amount of data requires the use of all available processing resources together with an efficient computing model, which is scalable and reliable. High Energy Physics (HEP) communities face similar challenges, since the data produced by the experiments’ detectors and by the Monte Carlo simulation jobs form a significant data stream that must be processed in a coordinated manner [1]. For this purpose, several approaches have been proposed, among them the DIRAC framework [2, 3].
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
