High Performance Computing Cluster Research Articles

Alrecon: computed tomography reconstruction web application based on Solara

Synchrotron X-ray computed tomography is a non-destructive 3D imaging technique that offers the possibility to study the internal microstructure of samples with high spatial and temporal resolution. Given its unmatched image quality and acquisition speed, and the possibility to preserve the specimens, there is an increasing demand for this technique, from scientific users from innumerable disciplines. Computed tomography reconstruction is the computational process by which experimental radiographs are converted to a meaningful 3-dimensional image after the scan. The procedure involves pre-processing steps for image background and artifact correction on raw data, a reconstruction step approximating the inverse Radon-transform, and writing of the reconstructed volume image to disk. Several open-source Python packages exist to help scientists in the process of tomography reconstruction, by offering efficient implementations of reconstruction algorithms exploiting central or graphics processing unit (CPU and GPU, respectively), and by automating significant portions of the data processing pipeline. A further increase in productivity is attained by scheduling and parallelizing demanding reconstructions on high performance computing (HPC) clusters. Nevertheless, visual inspection and interactive selection of optimal reconstruction parameters remain crucial steps that are often performed in close interaction with the end-user of the data. As a result, the reconstruction task involves more than one software. Graphical user interfaces are provided to the user for fast inspection and optimization of reconstructions, while HPC resources are often accessed through scripts and command line interface. We propose Alrecon, a pure Python web application for tomographic reconstruction built using Solara. Alrecon offers users an intuitive and reactive environment for exploring data and customizing reconstruction pipelines. By leveraging upon popular 3D image visualization tools, and by providing a user-friendly interface for reconstruction scheduling on HPC resources, Alrecon guarantees productivity and efficient use of resources for any type of beamline user.

Containerized deep learning in agriculture: Orchestrating GoogleNet with Kubernetes on high performance computing

SummarySmart Farming has become a cornerstone of modern agriculture, offering data‐driven insights and automation that optimize resource utilization and increase crop yields. The use of cutting‐edge technologies in agriculture has given rise to Smart Farming, which has transformed traditional farming practices into efficient, data‐driven operations. This paper explores the synergy between high‐performance computing (HPC) systems, Kubernetes orchestration, GoogleNet architecture, and containerization to redefine the future of farming. At the heart of this transformation lies the GoogleNet architecture, a deep learning powerhouse recognized for its efficiency and accuracy in image recognition tasks. The orchestration capabilities of Kubernetes, a versatile tool for managing containerized workloads efficiently on HPC clusters. Hence, in this work, we investigate the intricacies of deploying GoogleNet‐based deep learning models within containerized environments orchestrated by Kubernetes on HPC infrastructure. It explores resource optimization, scalability, security, and adaptability, all tailored to the unique demands of the agricultural domain to evaluate the effectiveness of the given technique it is compared with the existing techniques namely Hermes, Horus, CYBELE, and RZ‐SHAN. The attained ranges of proposed method of various measures of accuracy, precision, recall, and F1‐score are 98.65%, 97.45%, 97.87%, and 98.12% for the Pilot Wheat Ear dataset. Also, the processing time for the proposed approach is 181.50 and 120.2 m for the Pilot Wheat Ear Dataset and the Pilot soya bean farming dataset. The latency of the proposed approach attains a lower value of 1.5 and 1.1 s pilot soya bean farming dataset and Pilot Wheat Ear dataset. The experimental outcome demonstrates the efficiency of the proposed approaches to improve Smart Farming agriculture.

Swift : a modern highly parallel gravity and smoothed particle hydrodynamics solver for astrophysical and cosmological applications

ABSTRACT Numerical simulations have become one of the key tools used by theorists in all the fields of astrophysics and cosmology. The development of modern tools that target the largest existing computing systems and exploit state-of-the-art numerical methods and algorithms is thus crucial. In this paper, we introduce the fully open-source highly-parallel, versatile, and modular coupled hydrodynamics, gravity, cosmology, and galaxy-formation code Swift. The software package exploits hybrid shared- and distributed-memory task-based parallelism, asynchronous communications, and domain-decomposition algorithms based on balancing the workload, rather than the data, to efficiently exploit modern high-performance computing cluster architectures. Gravity is solved for using a fast-multipole-method, optionally coupled to a particle mesh solver in Fourier space to handle periodic volumes. For gas evolution, multiple modern flavours of Smoothed Particle Hydrodynamics are implemented. Swift also evolves neutrinos using a state-of-the-art particle-based method. Two complementary networks of sub-grid models for galaxy formation as well as extensions to simulate planetary physics are also released as part of the code. An extensive set of output options, including snapshots, light-cones, power spectra, and a coupling to structure finders are also included. We describe the overall code architecture, summarize the consistency and accuracy tests that were performed, and demonstrate the excellent weak-scaling performance of the code using a representative cosmological hydrodynamical problem with ≈300 billion particles. The code is released to the community alongside extensive documentation for both users and developers, a large selection of example test problems, and a suite of tools to aid in the analysis of large simulations run with Swift.

Sustainability of Data Center Digital Twins with Reinforcement Learning

The rapid growth of machine learning (ML) has led to an increased demand for computational power, resulting in larger data centers (DCs) and higher energy consumption. To address this issue and reduce carbon emissions, intelligent design and control of DC components such as IT servers, cabinets, HVAC cooling, flexible load shifting, and battery energy storage are essential. However, the complexity of designing and controlling them in tandem presents a significant challenge. While some individual components like CFD-based design and Reinforcement Learning (RL) based HVAC control have been researched, there's a gap in the holistic design and optimization covering all elements simultaneously. To tackle this, we've developed DCRL-Green, a multi-agent RL environment that empowers the ML community to design data centers and research, develop, and refine RL controllers for carbon footprint reduction in DCs. It is a flexible, modular, scalable, and configurable platform that can handle large High Performance Computing (HPC) clusters. Furthermore, in its default setup, DCRL-Green provides a benchmark for evaluating single as well as multi-agent RL algorithms. It easily allows users to subclass the default implementations and design their own control approaches, encouraging community development for sustainable data centers. Open Source Link: https://github.com/HewlettPackard/dc-rl

Alrecon: computed tomography reconstruction web application based on Solara.

Synchrotron X-ray computed tomography is a non-destructive 3D imaging technique that offers the possibility to study the internal microstructure of samples with high spatial and temporal resolution. Given its unmatched image quality and acquisition speed, and the possibility to preserve the specimens, there is an increasing demand for this technique, from scientific users from innumerable disciplines. Computed tomography reconstruction is the computational process by which experimental radiographs are converted to a meaningful 3-dimensional image after the scan. The procedure involves pre-processing steps for image background and artifact correction on raw data, a reconstruction step approximating the inverse Radon-transform, and writing of the reconstructed volume image to disk. Several open-source Python packages exist to help scientists in the process of tomography reconstruction, by offering efficient implementations of reconstruction algorithms exploiting central or graphics processing unit (CPU and GPU, respectively), and by automating significant portions of the data processing pipeline. A further increase in productivity is attained by scheduling and parallelizing demanding reconstructions on high performance computing (HPC) clusters. Nevertheless, visual inspection and interactive selection of optimal reconstruction parameters remain crucial steps that are often performed in close interaction with the end-user of the data. As a result, the reconstruction task involves more than one software. Graphical user interfaces are provided to the user for fast inspection and optimization of reconstructions, while HPC resources are often accessed through scripts and command line interface. We propose Alrecon, a pure Python web application for tomographic reconstruction built using Solara. Alrecon offers users an intuitive and reactive environment for exploring data and customizing reconstruction pipelines. By leveraging upon popular 3D image visualization tools, and by providing a user-friendly interface for reconstruction scheduling on HPC resources, Alrecon guarantees productivity and efficient use of resources for any type of beamline user.

A New Cloud-Native Tool for Pharmacogenetic Analysis.

The advancement of next-generation sequencing (NGS) technologies provides opportunities for large-scale Pharmacogenetic (PGx) studies and pre-emptive PGx testing to cover a wide range of genotypes present in diverse populations. However, NGS-based PGx testing is limited by the lack of comprehensive computational tools to support genetic data analysis and clinical decisions. Bioinformatics utilities specialized for human genomics and the latest cloud-based technologies were used to develop a bioinformatics pipeline for analyzing the genomic sequence data and reporting PGx genotypes. A database was created and integrated in the pipeline for filtering the actionable PGx variants and clinical interpretations. Strict quality verification procedures were conducted on variant calls with the whole genome sequencing (WGS) dataset of the 1000 Genomes Project (G1K). The accuracy of PGx allele identification was validated using the WGS dataset of the Pharmacogenetics Reference Materials from the Centers for Disease Control and Prevention (CDC). The newly created bioinformatics pipeline, Pgxtools, can analyze genomic sequence data, identify actionable variants in 13 PGx relevant genes, and generate reports annotated with specific interpretations and recommendations based on clinical practice guidelines. Verified with two independent methods, we have found that Pgxtools consistently identifies variants more accurately than the results in the G1K dataset on GRCh37 and GRCh38. Pgxtools provides an integrated workflow for large-scale genomic data analysis and PGx clinical decision support. Implemented with cloud-native technologies, it is highly portable in a wide variety of environments from a single laptop to High-Performance Computing (HPC) clusters and cloud platforms for different production scales and requirements.

Global Navigation Satellite System (GNSS) time series and velocities about a slowly convergent margin processed on high-performance computing (HPC) clusters: products and robustness evaluation

Abstract. The Global Navigation Satellite System (GNSS) is a well-known and fundamental tool for crustal monitoring projects and tectonic studies, thanks to its high coverage and the high quality of the data they provide. In particular, at slowly convergent margins, where deformation rates are of the order of a few millimetres per year, GNSS monitoring proves to be beneficial in detecting the diffuse deformation responsible for tectonic stress accrual. Its strength lies in the high precision achieved by GNSS permanent stations, especially when long-term data and stable structures are available at the stations. North-eastern Italy is a tectonically active region located in the northernmost sector of the Adria microplate, slowly converging with the Eurasia plate, characterized by low deformation rates and moderate seismicity. It greatly benefits from continuous and high-precision geodetic monitoring, since it has been equipped with a permanent GNSS network providing real-time data and daily observations over 2 decades. The Friuli Venezia Giulia Deformation Network (FReDNet) was established in the area in 2002 to monitor crustal deformation and contribute to the regional seismic hazard assessment. This paper describes GNSS time series spanning 2 decades of stations located in north-eastern Italy and surroundings as well as the outgoing velocity field. The documented dataset has been retrieved by processing the GNSS observations with the GAMIT/GLOBK software ver10.71, which allows calculation of high-precision coordinate time series, position and velocity for each GNSS station by taking advantage of the high-performance computing resources of the Italian High-Performance Computing Centre (CINECA) clusters. The GNSS observations (raw and standard RINEX – Receiver INdependent EXchange – formats) and the time series estimated with the same procedure are currently daily continued, collected and stored in the framework of a long-term monitoring project. Instead, velocity solutions are intended for annual updates. The time series and velocity field dataset documented here are available on Zenodo (https://doi.org/10.5281/zenodo.8055800, Tunini et al., 2024).

Modeling Microreactor Requirements for High-Performance Computing

The potential deployment of microreactors as a zero-emission source for critical applications within integrated energy systems such as microgrids has been gaining interest in recent years owing to the microreactors’ dispatchable nature, modular design, small site footprint, and carbon-free generation. A particularly high-value but challenging application with rapidly growing demand is in the deployment of high-performance computing (HPC) clusters within microgrids. In this work, a model of a HPC cluster in an energy-diverse microgrid is developed to determine the requirements of a technology-agnostic microreactor deployed for such a challenging application. The minute-resolution simulations revealed that the cluster’s electrical load fluctuation of up to 4.1 MW/min required a fast and responsive load-following capability. When the load-following capability of the microreactor was perturbed, the required microgrid storage capacity associated with having a 0.1 MW/min dispatchable microreactor decreased by two orders of magnitude as compared with load-following solely by energy storage devices, indicating that load-following capability in microreactors is of great value in such applications. The analysis methods described in this work can be extended to other microgrids, other HPC clusters, or other types of challenging applications, and can help microgrid planners in determining the storage size, output capacity, and ramping capabilities of the storage devices required for a given microgrid configuration.

DIFFERENCE BETWEEN OPENHPC AND HTCONDOR CLUSTER SYSTEMS: IN-DEPTH ANALYSIS

The rapidly developing field of high-performance computing (HPC) requires efficient and scalable solutions to manage extensive computing loads. Although they use different approaches and architectures, OpenHPC and HTCondor are well-known platforms that meet these needs. This article thoroughly analyzes OpenHPC and HTCondor to identify their fundamental differences and work paradigms. OpenHPC is a comprehensive modular structure designed to facilitate the deployment, management, and maintenance of HPC clusters, offering a rich set of pre-integrated HPC software components. Conversely, HTCondor specializes in efficiently planning and managing resource-intensive tasks, using a unique partner selection system for dynamic resource allocation based on job requirements and resource availability. By examining aspects such as system architecture, resource management efficiency, scalability, flexibility, and the user ecosystem, this analysis sheds light on the strengths and weaknesses of each structure. The research aims to provide stakeholders in high-performance computing with the knowledge necessary to make informed decisions regarding the selection and implementation of high-performance computing management systems, ultimately aimed at optimizing the use of computing resources and optimizing research and development workflows.

PARALiA: A Performance Aware Runtime for Auto-tuning Linear Algebra on Heterogeneous Systems

Dense linear algebra operations appear very frequently in high-performance computing (HPC) applications, rendering their performance crucial to achieve optimal scalability. As many modern HPC clusters contain multi-GPU nodes, BLAS operations are frequently offloaded on GPUs, necessitating the use of optimized libraries to ensure good performance. Unfortunately, multi-GPU systems are accompanied by two significant optimization challenges: data transfer bottlenecks as well as problem splitting and scheduling in multiple workers (GPUs) with distinct memories. We demonstrate that the current multi-GPU BLAS methods for tackling these challenges target very specific problem and data characteristics, resulting in serious performance degradation for any slightly deviating workload. Additionally, an even more critical decision is omitted because it cannot be addressed using current scheduler-based approaches: the determination of which devices should be used for a certain routine invocation. To address these issues we propose a model-based approach: using performance estimation to provide problem-specific autotuning during runtime. We integrate this autotuning into an end-to-end BLAS framework named PARALiA. This framework couples autotuning with an optimized task scheduler, leading to near-optimal data distribution and performance-aware resource utilization. We evaluate PARALiA in an HPC testbed with 8 NVIDIA-V100 GPUs, improving the average performance of GEMM by 1.7× and energy efficiency by 2.5× over the state-of-the-art in a large and diverse dataset and demonstrating the adaptability of our performance-aware approach to future heterogeneous systems.

The globus compute dataset: An open function-as-a-service dataset from the edge to the cloud

We present a unique function-as-a-service (FaaS) dataset capturing the use of the Globus Compute (previously funcX) platform. Globus Compute implements a federated model via which users may deploy endpoints on arbitrary remote computers, from the edge to high performance computing (HPC) cluster, and they may then invoke Python functions on those endpoints via a reliable cloud-hosted service. The dataset covers 31 weeks and includes 2121472 task submissions from 252 users executed on 580 remote computing endpoints. It includes 277386 registered functions. We describe the dataset and various observations, some that are similar to other FaaS datasets, for example, that 74% of tasks run for less than 1 s, and some that are unique to Globus Compute, for example, that endpoints are used in different ways and that the majority of functions are related to scientific computing and machine learning. To the best of our knowledge, this dataset represents the first federated FaaS dataset that includes user workloads, distributed computing endpoints, and analysis of registered function bodies. We expect the dataset to be useful for researching FaaS architectures, workload modeling, container warming, and other distributed computing architectures.

Enabling the execution of HPC applications on public clouds with HPC@Cloud toolkit

AbstractThe advent of cloud computing has made access to computing infrastructure available to millions of users that face resource constraints. In the context of high performance computing (HPC), public cloud resources have emerged as a cost‐effective alternative to expensive on‐premises clusters. However, there are several challenges and limitations in adopting this approach. This paper proposes HPC@Cloud , a provider‐agnostic open‐source software toolkit that facilitates the migration, testing, and execution of HPC applications in public clouds. The toolkit takes advantage of various fault tolerance technologies to enable the use of inexpensive transient cloud infrastructure, commonly known as “spot” instances. Also, it features integration with singularity containers, allowing users to run complex applications on virtual HPC clusters in a portable and reproducible way. Finally, it provides a data‐based empirical approach to estimating cloud infrastructure costs for HPC workloads. The results obtained on two public cloud providers (AWS and Vultr) show that: (i) HPC@Cloud can efficiently build virtual HPC clusters on the cloud; (ii) the new adaptive fault tolerance strategy outperforms other existing strategies based on blocking restoration; (iii) the integration of singularity containers into HPC@Cloud improves the portability of HPC applications to public clouds with negligible performance penalty to the applications; (iv) the proposed cost prediction approach can estimate the cost of running the applications on AWS and Vultr with up to 93% accuracy on average.

Scaling the Inference of Digital Pathology Deep Learning Models Using CPU-Based High-Performance Computing

Digital pathology whole-slide images (WSIs) are large-size gigapixel images and image analysis based on deep learning artificial intelligence (AI) technology often involves pixel-wise testing of a trained deep learning neural network (DLNN) on hundreds of WSI images, which is time consuming. We take advantage of High-Performance Computing (HPC) facilities to parallelize this procedure into multiple independent (and hence delightfully parallel) tasks. However, traditional software parallelization techniques and regular file formats can have significant scaling problems on HPC clusters. In this work, a useful computational strategy is designed to localize and extract relevant patches in WSI files and group them in HDF5 files well suited for parallel I/O. HPC's array job facilities are adapted for hierarchical scaling and parallelization of WSI pre-processing and testing of trained algorithms. Applying these techniques to testing a trained DLNN on the CAMELYON datasets with 399 WSIs reduced the theoretical processing time of 18 years on a single CPU or 30 days on a single GPU to less than 45 hours on an HPC cluster of 4,000 CPU cores. The efficiency-accuracy trade-off we demonstrated on this dataset further reinforced the importance of efficient computation techniques, without which the accuracy may be sacrificed. The framework developed here for testing DLNNs does not rely on any specific neural network architecture and HPC cluster setup and can be utilized for any large-scale image processing and big-data analysis.

A software package for efficient patient trajectory analysis applied to analyzing bladder cancer development.

We present the Patient Trajectory Analysis Library (PTRA), a software package for explorative analysis of patient development. PTRA provides the tools for extracting statistically relevant trajectories from the medical event histories of a patient population. These trajectories can additionally be clustered for visual inspection and identifying key events in patient progression. The algorithms of PTRA are based on a statistical method developed previously by Jensen et al, but we contribute several modifications and extensions to enable the implementation of a practical tool. This includes a new clustering strategy, filter mechanisms for controlling analysis to specific cohorts and for controlling trajectory output, a parallel implementation that executes on a single server rather than a high-performance computing (HPC) cluster, etc. PTRA is furthermore open source and the code is organized as a framework so researchers can reuse it to analyze new data sets. We illustrate our tool by discussing trajectories extracted from the TriNetX Dataworks database for analyzing bladder cancer development. We show this experiment uncovers medically sound trajectories for bladder cancer.

Improved deep artificial neural network-powered prediction of extreme mechanical performances of fractal architectures with high hierarchical rank

This work explores the mechanical performances, including anisotropy, wave propagation, and buckling resistance capability of the Menger sponge and Jerusalem fractal structures, with the high rank, using an improved deep artificial neural network (ANN) model inserted by a loop with multiple conditions. Where, the effective mechanical attributes, including Young’s modulus, shear modulus, and Poisson’s ratio of the Menger sponge structure with rank 4 are attained using high-performance computer cluster (HPCC) that are then used to train and also validate the ANN model. The accuracy of the deep ANN model trained by data of fractal structures with and without rank 4 is verified by the numerical model and experiment with the good agreement found or the maximum percent difference being 15.6% where the numerical and experimental data don’t belong to its training samples. Remarkably, the deep ANN model demonstrates an extremely low cost, but very fast compared with the numerical and experimental works. Namely, the improved deep ANN model implemented by a regular computer can accurately predict the mechanical attributes of Menger sponge fractal structures with full ranks in 3 min. While it takes a week for numerical work using HPCC or experiment method to produce the same mechanical properties of the Menger sponge with only rank 4, displaying at least thousands of times slower than the ANN model. The ANN results indicate that the onset of anisotropy of considered fractal structures is from rank 1, and the reversal of anisotropy will take place upon an increase in the rank while such reversal is not seen for the case of lattice structure. The anisotropy reversal could be one of main reasons why the fractal structure tends to have a high rank in nature. In addition, the buckling stress and phase wave velocity are observed to decrease with rising rank, with a nonlinear pattern.

Parallelization and optimization for a pin-by-pin whole core thermal–hydraulic analysis code

With increasingly developing high-performance computing (HPC) technology, pin-by-pin thermal–hydraulic simulations are now feasible. The in-house subchannel analysis code KMC-FBc has been extended to perform pin-by-pin calculations. To expedite the whole-core thermal–hydraulic analysis process, various parallelization and optimization strategies are proposed and implemented at the single-assembly and multi-assembly levels, including acceleration methods for solving large sparse matrix equation systems, memory and file input/output (I/O) optimization, hybrid MPI/OpenMP-based whole-core acceleration method and so on. Parameters for evaluating the parallel performance, such as the running time, speedup, and parallel efficiency, are tested on an HPC cluster. Results show that the maximum acceleration ratio for the whole-core problem with a mesh size in the tens of millions can reach approximately 120, with a parallel efficiency of around 32%, which demonstrates the rationality and effectiveness of the parallel algorithm and optimization methods and lays the foundation for future high-resolution and high-fidelity thermal–hydraulic analysis.

Stochastic analysis of air–sea heat fluxes variability in the North Atlantic in 1979–2022 based on reanalysis data

A dynamic stochastic model based on the Langevin stochastic differential equation is introduced for the reanalysis data of the ERA5 database to model and analyze the behavior of latent and sensible air–sea heat fluxes in the North Atlantic for the period 1979–2022. The point estimates of the random coefficients (the drift vector and the diffusion matrix) of this type of equation for the entire period under consideration are presented. The numerical methods and software tools for statistical analysis of time evolution of the coefficients as well as determination their relationships and the behavior of their maxima, averages and minima at various time intervals (days, months, years), are developed. A strong seasonality for the coefficients of the equation is demonstrated. The spatiotemporal variability of the dynamic and stochastic components of the coefficients of the Langevin equation and their relationship with jet streams of different regions of the North Atlantic is analyzed. The presence of non-trivial positive trends in the drift and diffusion coefficients, especially for the latent fluxes, within the interannual variability is demonstrated. One indicates a quantitative increase in the air–sea interaction on the interannual scale. Numerical estimation was carried out using high-performance computing cluster with software implementation in the Python programming language. The tools for dynamic visualization of various quantities on geographical maps of the region under consideration are also presented.

Estimating hurricane-induced vertical surge and wave loads on elevated coastal buildings based on CFD simulations and ensemble learning

This study focuses on developing Machine Learning models based on CFD simulations to estimate hurricane induced vertical wave and surge forces on elevated coastal buildings caused by regular waves. In particular, this study aims at addressing the existing gap in designing elevated coastal buildings, which stems from lack of a generalizable equation for estimating vertical hurricane wave and surge forces for a variety of building geometries and wave conditions. To this end, based on a validated Computational Fluid Dynamics (CFD) model, six key variables that affect the wave forces on buildings are considered, which include building dimensions (length and width of rectangular plan buildings) and wave characteristics. In addition, the effect of building length in relation to wave length on the resulting vertical forces is explicitly considered, while the effect of building width is considered implicitly. Subsequently, a Latin Hypercube Sampling (LHS) strategy is implemented for generating 256 random samples, each representing a unique building geometry with a rectangular plan and wave condition scenario. A pipeline is then developed for generating CFD models corresponding to each sample, which are then deployed on a high-performance computing (HPC) cluster, creating a dataset of vertical surge and wave forces. Based on the key design variables, an ensemble of Machine Learning (ML) models is developed to predict vertical wave forces. For the purpose of design and risk assessment, different force levels may be of interest. Therefore, different models were trained to predict multiple levels of forces on elevated buildings, including maximum, average, 25th, 75th, and 99th percentiles of peak forces. The developed model for each of these force levels is then investigated to identify the main building geometry and wave conditions features that affect the resulting forces, which can inform the decisions of the designers in the early-stage phases. For further assistance in the early-stage design, based on a subset of identified important features, a simple expression for each force level is provided to narrow down the design space to top performing early design candidates, while the main estimators can be used for the final design of elevated buildings. Results show that based on CFD simulations, highly accurate and generalizable predictive models with error rates of about 4% on the test set can be developed.

Lightweight Integration of a Data Cache for Opportunistic Usage of HPC Resources in HEP Workflows

A data caching setup has been implemented for the High Energy Physics (HEP) computing infrastructure in Freiburg, Germany, as a possible alternative to local long-term storage. Files are automatically cached on disk upon first request by a client, can be accessed from cache for subsequent requests, and are deleted after predefined conditions are met. The required components are provided to a dedicated HEP cluster, and, via virtual research environments, to the opportunistically used High-Performance Computing (HPC) Cluster NEMO (Neuroscience, Elementary Particle Physics, Microsystems Engineering and Materials Science). A typical HEP workflow has been implemented as benchmark test to identify any overhead introduced by the caching setup with respect to direct, non-cached data access, and to compare the performance of cached and non-cached access to several external files sources. The results indicate no significant overhead in the workflow and faster file access with the caching setup, especially for geographically distant file sources. Additionally, the hardware requirements for various numbers of parallel file requests were measured for estimating future requirements.

A Real-Time Multistage False Data Detection Method Based on Deep Learning and Semisupervised Scoring Algorithms

This article proposes a real-time multistage data-driven method to detect false data injected by an intruder. The proposed method is based on an accurate load model constructed by deep neural networks (long short-term memory and gated recurrent unit) where historical demand along with extra weather features are employed to formulate the problem as a sequence of predictions and build an appropriate DNN model. A distinct feature of the proposed method that dramatically improves accuracy is hyperparameter optimization (e.g., dropout rate, batch size, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\beta _{1}$</tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\beta _{2}$</tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\alpha$</tex-math></inline-formula> ) based on cross-validation. Since hyperparameter optimization consumes enormous resources to analyze the different combinations of hyperparameters and causes out-of-memory problems, we have implemented our framework based on a parallel distributed computing technique on high-performance computer clusters. A variety of case studies are examined to find the best model. At the end of this step, the best model is employed as a base reference to a semisupervised scoring algorithm for the purpose of finding sequence of injected false data. To improve the detection efficiency, inter-range, pruning, and minimum score are added to the proposed methodology that suppress high false-positive rates. Several scenarios from a real-world dataset have been studied in this article, and the results demonstrate that the proposed method can be used to detect various types of attacks, including scaling, ramping, professional ramping, and random attacks, with a good performance score (i.e., recall, precision, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$F1$</tex-math></inline-formula> score) for all of them. Furthermore, it can mitigate false data with the forecasted load values.