Abstract
With the introduction of federated data access to the workflows of WLCG, it is becoming increasingly important for data centers to understand specific data flows regarding storage element accesses, firewall configurations, as well as the scheduling of batch jobs themselves. As existing batch system monitoring and related system monitoring tools do not support measurements at batch job level, a new tool has been developed and put into operation at the GridKa Tier 1 center for monitoring continuous data streams and characteristics of WLCG jobs and pilots. Long term measurements and data collection are in progress. These measurements already have been proven to be useful analyzing misbehaviors and various issues. Therefore we aim for an automated, realtime approach for anomaly detection. As a requirement, prototypes for standard workflows have to be examined. Based on measurements of several months, different features of HEP jobs are evaluated regarding their effectiveness for data mining approaches to identify these common workflows. The paper will introduce the actual measurement approach and statistics as well as the general concept and first results classifying different HEP job workflows derived from the measurements at GridKa.
Highlights
The increase in opportunistic resource usage [1] and federated storage [2, 3, 4, 5] in different computing models of WLCG communities requires network-centric monitoring techniques
None of the aforementioned approaches tracks the logical connections between a single job and its data transfers
In data centers operating a batch system with commodity hardware, it is essential to know about the logical connections
Summary
- WLCG Monitoring Consolidation and further evolution P Saiz, A Aimar, J Andreeva et al. - The ATLAS Eventlndex: data flow and inclusion of other metadata D Barberis, S E Cárdenas Zárate, A Favareto et al. - Evolution of Database Replication Technologies for WLCG Zbigniew Baranowski, Lorena Lobato Pardavila, Marcin Blaszczyk et al. This content was downloaded from IP address 129.13.72.197 on 12/10/2017 at 12:50
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
