Abstract
The OSG has long maintained a central accounting system called Gratia. It uses small probes on each computing and storage resource in order to collect resource usage. The probes report to a central collector which stores the usage in a database. The database is then queried to generate reports. As the OSG aged, the size of the database grew very large. It became too large for the database technology to efficiently query to generate detailed reports. The design of a replacement requires data storage that could be queried efficiently to generate multi-year reports. Additionally, it requires flexibilityto add new attributes to the collected data. In this paper we will describe updates to the GRACC architecture in the last 18 months. GRACC uses modern web technologies that were designed for large data storage, query, and visualization. That includes the open source database Elasticsearch, message broker software RabbitMQ, and Grafana and Kibana as data visualization platforms. It uses multiple agents that perform operations on the data to transform it for easier querying and summarization.
Highlights
In a previous paper [1], we described the Grid Accounting Collector (GRACC) modular architecture
In a previous paper [1], we described the GRACC modular architecture
The GRACC ecosystem consists of 5 main components: probes, data collection, message broker, data sinks and visualization
Summary
In a previous paper [1], we described the GRACC modular architecture. In this paper we will describe the updates to GRACC that have occurred over the last 18 months. The StashCache [2] data federation usage data and the HTCondor TCP transfer statistics are sent to and stored by GRACC. Each of these collections use a common architecture utilizing 5 components: probes, data collection, message broker, data sinks, and visualization and querying tools. Probes query the resources for usage information and convert it into a common format that can be more read by the data collection component. The data collection component receives data from multiple probes, performs some manipulation of the data, and sends the data to a message bus for distribution. Each of the examples shown below follow this common architecture, probes, data collection, message broker, data sinks, and visualization and querying tools
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
