High Performance Computing Resources Research Articles

TTCF4LAMMPS: A toolkit for simulation of the non-equilibrium behaviour of molecular fluids at experimentally accessible shear rates

We present TTCF4LAMMPS, a toolkit for performing non-equilibrium molecular dynamics (NEMD) simulations to study the fluid behaviour at low shear rates using the LAMMPS software. By combining direct NEMD simulations and the transient-time correlation function (TTCF) technique, we study the fluid response to shear rates spanning 15 orders of magnitude. We present two examples for simple monatomic systems: one consisting of a bulk liquid and another with a liquid layer confined between two solid walls. The small bulk system is suitable for testing on personal computers, while the larger confined system requires high-performance computing (HPC) resources. We demonstrate that the TTCF formalism can successfully detect the system response for arbitrarily weak external fields. We provide a brief mathematical explanation for this feature. Although we showcase the method for simple monatomic systems, TTCF can be readily extended to study more complex molecular fluids. Moreover, in addition to shear flows, the method can be extended to investigate elongational or mixed flows as well as thermal or electric fields. The high computational cost needed for the method is offset by the two following benefits: i) the cost is independent of the magnitude of the external field, and ii) the simulations can be made highly efficient on HPC architectures by exploiting the parallel design of the algorithm. We expect the toolkit to be useful for computational researchers striving to study the nonequilibrium behaviour of fluids under experimentally-accessible conditions. Program summaryProgram title: TTCF4LAMMPSCPC Library link to program files:https://doi.org/10.17632/hh2rkcxbrf.1Developer's repository link:https://github.com/edwardsmith999/TTCF4LAMMPSLicensing provisions: GNU General Public License 3Programming language: Python 3Nature of problem: Measuring the nonequilibrium behaviour of bulk and confined fluids under experimentally accessible strain rates in non-equilibrium molecular dynamics (NEMD) simulations.Solution method: Creating a Python-based code that utilises the transient-time correlation function method and the LAMMPS software to enable the bulk fluid properties (e.g. viscosity) and confined fluid interfacial properties (e.g. shear stress and slip velocity) to be computed at low shear rates with NEMD.

MaPPeRTrac: A Massively Parallel, Portable, and Reproducible Tractography Pipeline.

Large-scale diffusion MRI tractography remains a significant challenge. Users must orchestrate a complex sequence of instructions that requires many software packages with complex dependencies and high computational costs. We developed MaPPeRTrac, an edge-centric tractography pipeline that simplifies and accelerates this process in a wide range of high-performance computing (HPC) environments. It fully automates either probabilistic or deterministic tractography, starting from a subject's magnetic resonance imaging (MRI) data, including structural and diffusion MRI images, to the edge density image (EDI) of their structural connectomes. Dependencies are containerized with Singularity (now called Apptainer) and decoupled from code to enable rapid prototyping and modification. Data derivatives are organized with the Brain Imaging Data Structure (BIDS) to ensure that they are findable, accessible, interoperable, and reusable following FAIR principles. The pipeline takes full advantage of HPC resources using the Parsl parallel programming framework, resulting in the creation of connectome datasets of unprecedented size. MaPPeRTrac is publicly available and tested on commercial and scientific hardware, so it can accelerate brain connectome research for a broader user community. MaPPeRTrac is available at: https://github.com/LLNL/mappertrac .

Molecular simulations of doxorubicin complexed with native and modified cyclodextrins in water

In this study, all-atom molecular dynamics simulations have been performed to investigate from a nanoscopic point of view the interactions in supramolecular complexes formed between doxorubicin and cyclodextrins in an aqueous phase at a temperature of 310 K. All the simulations are on the time scales of microseconds and are carried out on High Performance Computing resources provided by the Greek Supercomputer ARIS. Doxorubicin, which is a well-known anticancer drug belonging to the class of anthracyclines, is simulated with two cyclodextrins of broad utilization in drug delivery, i.e.: γ-cyclodextrin and 2-hydroxypropyl-β-cyclodextrin. In both cases, the complexation process takes place spontaneously during the course of the simulations and a detailed analysis of the formed complexes is conducted in terms of structural, dynamical and thermodynamic properties. In addition, we present a quantitative analysis of how doxorubicin affects, upon complexation, the geometry, the hydrogen bonding network and the hydration of the selected cyclodextrins. Special attention is paid to the thermodynamic description of the simulated systems. More specifically, the standard binding free energy is calculated with an extended version of the linear interaction energy method. The presented in silico approach reveals that doxorubicin forms stable complexes with the chosen cyclodextrin molecules, there is a water solubility enhancement of the drug with respect to its unbound state and also cyclodextrins act protectively to the drug preventing undesired effects, such as the glycosidic degradation. Excellent agreement is found between the presented results and published in silico and experimental findings.

A multi-objective grey-wolf optimization based approach for scheduling on cloud platforms

A cloud computing environment processes user workloads or tasks by exploiting its high performance computational, storage, of reducing and network resources. The virtual machines in the cloud environment are allocated to tasks with the aim of reducing overall execution time. The use of high performance resources incurs monetary costs, as well as high power consumption. The heuristic based approaches implemented for scheduling tasks are unable to cope with the complexity of optimizing multiple parameters. In this paper, we propose a multi-objective grey-wolf optimization based algorithm for scheduling tasks on cloud platforms. The proposed algorithm targets to minimize schedule length (overall execution time), energy consumption, and monetary cost required for executing tasks. For optimization, the algorithm incorporates steps that are performed iteratively for mimicking the behavior of grey wolves attacking their prey. It uses discrete values for positioning wolves for encircling and attacking the prey. The assignment of tasks to virtual machines is performed using the solution found after multi-objective optimization that incorporates weighted sorting for arranging solutions. Our experimentation performed using the CloudSim framework shows that the proposed algorithm outperforms other algorithms with performance improvement ranging from 3.98% to 16.07%, while considering the schedule length, monetary cost, and energy consumption.

HPC resources for CMS offline computing: An integration and scalability challenge for the Submission Infrastructure

HPC resources for CMS offline computing: An integration and scalability challenge for the Submission Infrastructure

Secure Federated Learning Across Heterogeneous Cloud and High-Performance Computing Resources - A Case Study on Federated Fine-tuning of LLaMA 2

Secure Federated Learning Across Heterogeneous Cloud and High-Performance Computing Resources - A Case Study on Federated Fine-tuning of LLaMA 2

Integration of the Barcelona Supercomputing Center for CMS computing: Towards large scale production

The CMS experiment is working to integrate an increasing number of High Performance Computing (HPC) resources into its distributed computing infrastructure. The case of the Barcelona Supercomputing Center (BSC) is particularly challenging as severe network restrictions prevent the use of CMS standard computing solutions. The CIEMAT CMS group has performed significant work in order to overcome these constraints and make BSC resources available to CMS. The developments include adapting the workload management tools, replicating the CMS software repository to BSC storage, providing an alternative access to detector conditions data, and setting up a service to transfer produced output data to a nearby storage facility. In this work, we discuss the current status of this integration activity and present recent developments, such as a front-end service to improve slot usage efficiency and an enhanced transfer service that supports the staging of input data for workflows at BSC. Moreover, significant efforts have been devoted to improving the scalability of the deployed solution, automating its operation, and simplifying the matchmaking of CMS workflows that are suitable for execution at BSC.

ML_INFN project: Status report and future perspectives

The ML_INFN initiative (“Machine Learning at INFN”) is an effort to foster Machine Learning (ML) activities at the Italian National Institute for Nuclear Physics (INFN). In recent years, artificial intelligence inspired activities have flourished bottom-up in many efforts in Physics, both at the experimental and theoretical level. Many researchers have procured desktop-level devices, with consumer-oriented GPUs, and have trained themselves in a variety of ways, from webinars, books, and tutorials. ML_INFN aims to help and systematize such effort, in multiple ways: by offering state-of-the-art hardware for ML, leveraging on the INFN Cloud provisioning solutions and thus sharing more efficiently GPUs and leveling the access to such resources to all INFN researchers, and by organizing and curating Knowledge Bases with productiongrade examples from successful activities already in production. Moreover, training events have been organized for beginners, based on existing INFN ML research and focused on flattening the learning curve. In this contribution, we will update the status of the project reporting in particular on the development of tools to take advantage of High-Performance Computing resources provisioned by CNAF and ReCaS computing centers for interactive support to activities and on the organization of the first in-person advanced-level training event, with a GPU-equipped cloud-based environment provided to each participant.

Machine learning techniques for sequential learning engineering design optimisation

When designing a fusion power plant, many first-of-a-kind components are required. This presents a large potential design space across as many dimensions as the component’s parameters. In addition, multiphysics, multiscale, high-fidelity simulations are required to reliably capture a component’s performance under given boundary conditions. Even with high performance computing (HPC) resources, it is not possible to fully explore a component’s design space. Thus, effective interpolation between data points via machine learning (ML) techniques is essential. With sequential learning engineering optimisation, ML techniques inform the selection of simulation parameters which give the highest expected improvement for the model: balancing exploitation of the current best design with exploration of uncertain areas in the design space. In this paper, the application of an ML-driven design of experiment procedure for the sequential learning engineering design optimisation of a fusion component is shown. A parameterised divertor monoblock is taken as a typical example of a fusion component requiring HPC simulation to model. The component’s geometry is then optimised using Bayesian optimisation, seeking the design which minimises the stress experienced by the component under operational conditions.

Lightweight Integration of a Data Cache for Opportunistic Usage of HPC Resources in HEP Workflows

A data caching setup has been implemented for the High Energy Physics (HEP) computing infrastructure in Freiburg, Germany, as a possible alternative to local long-term storage. Files are automatically cached on disk upon first request by a client, can be accessed from cache for subsequent requests, and are deleted after predefined conditions are met. The required components are provided to a dedicated HEP cluster, and, via virtual research environments, to the opportunistically used High-Performance Computing (HPC) Cluster NEMO (Neuroscience, Elementary Particle Physics, Microsystems Engineering and Materials Science). A typical HEP workflow has been implemented as benchmark test to identify any overhead introduced by the caching setup with respect to direct, non-cached data access, and to compare the performance of cached and non-cached access to several external files sources. The results indicate no significant overhead in the workflow and faster file access with the caching setup, especially for geographically distant file sources. Additionally, the hardware requirements for various numbers of parallel file requests were measured for estimating future requirements.

A Survey of High Performance Computing (HPC) Infrastructure in Thailand

In the last decade, government organizations and private companies in Thailand have invested considerably in computing resources. Research collaborations typically band together based on shared interests and propose projects to compete for funding. Thus, many institutions frequently use similar computing resources. A bird's eye view of high-performance computer infrastructure in Thailand is important in many ways. To the best of our knowledge and ability, we gathered information on government procurements of HPC resources in the past five years, which cover several organizations and ministries. We list the system specifications and tabulate the target application areas for each system. The aggregated number of cores and storage space for HPC in Thailand, commissioned during the past five years, is 54,838 cores and 21 PB, respectively. We also describe the large data transfers using UniNet for HPC applications. The survey results can be used by academics and decision-makers to build research agendas and national development strategies.

Fog-ROCL: A Fog based RSU Optimum Configuration and Localization in VANETs

Intelligent Transportation Systems (ITS) are one of the pillars of smart cities that enable smart traffic and smart mobility. Vehicular Ad-hoc Networks (VANETs) are utilized as platforms for ITS applications. In VANETs, vehicles collect relevant information from sensors and exchange information about road conditions and traffic status with each other and with roadside units (RSUs). Roadside units facilitate reliable communication among the vehicles and perform real-time processing for the sensed data before sending to the cloud. Although cloud computing offers high performance computational and storage resources, it does not conform to the real-time nature of the ITS applications and the massive amount of data exchange and generation rate due to its centralized nature and high communication latency. In this paper we propose a cost-effective strategy to solve the configuration and localization problem of fog-based RSU deployment in VANETs. The proposed strategy is able to assign the computational capacity of each fog node based on the amount of computational demand inside its coverage region. The problem is formulated as a Satisfiability Modulo Theories (SMT) problem. Our proposed strategy is more efficient than other strategies in terms of the total deployment cost and the overall satisfied computational demand.

Numerical tools for burning plasmas

The software stack under development within a European coordinated effort on tools for burning plasma modelling is presented. The project is organised as a Task (TSVV Task 10) under the new E-TASC initiative (Litaudon et al 2022 Plasma Phys. Control. Fusion 64 034005). This is a continued effort within the EUROfusion inheriting from the earlier European coordination projects as well as research projects based at various European laboratories. The ongoing work of the TSVV Tasks is supported by the Advanced Computing Hubs. Major projects requiring the high performance computing (HPC) resources are global gyrokinetic codes and global hybrid particle-magnetohydrodynamics (MHD) codes. Also applications using the integrated modelling tools, such as the Energetic-Particle Workflow, based on the ITER Integrated Modelling & Analysis Suite (IMAS), or the code package for modelling radio-frequency heating and fast-ion generation may require intensive computation and a substantial memory footprint. The continual development of these codes both on the physics side and on the HPC side allows us to tackle frontier problems, such as the interaction of turbulence with MHD-type modes in the presence of fast particles. One of the important mandated outcomes of the E-TASC project is the IMAS-enabling of EUROfusion codes and release of the software stack to the EUROfusion community.

Scalable Data Processing Platform for Earth Observation Data Repositories

Earth observation (EO) satellite data is essential to environmental monitoring. At a national and regional level, the open data cubes harness the power of satellite data by providing application programming interfaces and services to the end-users. The volume and the complexity of satellite observations are increasing, demanding novel approaches for data storing, managing, and processing. High-performance computing (HPC) and cloud platforms may improve Big EO data processing performance. However, it is necessary to consider several vital aspects for efficient and flexible EO data processing, such as the interoperability from cloud-HPC and EO data repositories, automatic provisioning and scaling of cloud-HPC resources, cost-effectiveness, support of new EO data formats and open-source packages, or linkage with data cube platforms. The article proposes a scalable EO data processing platform interoperable from cloud-HPC and EO data repositories. The platform enables linking any data repository supporting web coverage service or SpatioTemporal Asset Catalog Application Programming Interfaces (STAC-API), and any cloud or HPC resource supporting scheduling system API for providing access to the cluster backends.

DeepMiCa: Automatic segmentation and classification of breast MIcroCAlcifications from mammograms

Background and objectiveBreast cancer is the world’s most prevalent form of cancer. The survival rates have increased in the last years mainly due to factors such as screening programs for early detection, new insights on the disease mechanisms as well as personalised treatments. Microcalcifications are the only first detectable sign of breast cancer and diagnosis timing is strongly related to the chances of survival. Nevertheless microcalcifications detection and classification as benign or malignant lesions is still a challenging clinical task and their malignancy can only be proven after a biopsy procedure. We propose DeepMiCa, a fully automated and visually explainable deep-learning based pipeline for the analysis of raw mammograms with microcalcifications. Our aim is to propose a reliable decision support system able to guide the diagnosis and help the clinicians to better inspect borderline difficult cases. MethodsDeepMiCa is composed by three main steps: (1) Preprocessing of the raw scans (2) Automatic patch-based Semantic Segmentation using a UNet based network with a custom loss function appositely designed to deal with extremely small lesions (3) Classification of the detected lesions with a deep transfer-learning approach. Finally, state-of-the-art explainable AI methods are used to produce maps for a visual interpretation of the classification results. Each step of DeepMiCa is designed to address the main limitations of the previous proposed works resulting in a novel automated and accurate pipeline easily customisable to meet radiologists’ needs. ResultsThe proposed segmentation and classification algorithms achieve an area under the ROC curve of 0.95 and 0.89 respectively. Compared to previously proposed works, this method does not require high performance computational resources and provides a visual explanation of the final classification results. ConclusionTo conclude, we designed a novel fully automated pipeline for detection and classification of breast microcalcifications. We believe that the proposed system has the potential to provide a second opinion in the diagnosis process giving the clinicians the opportunity to quickly visualise and inspect relevant imaging characteristics. In the clinical practice the proposed decision support system could help reduce the rate of misclassified lesions and consequently the number of unnecessary biopsies.

Thermo4PFM: Facilitating Phase-field simulations of alloys with thermodynamic driving forces

Phase-field modeling is a popular front-tracking approach used to model solidification. Its time-evolution equations are often coupled to alloy composition and/or thermal diffusion in high-resolution multiphysics approaches. Materials thermodynamic properties tabulated in CALPHAD databases can be used for phase-field modeling to parameterize bulk energies of alloys. In addition, they can be naturally integrated into models such as the Kim-Kim-Suzuki (KKS) model where driving forces depend on the differences between chemical potentials of co-existing phases. In that case, a small system of coupled nonlinear equations needs to be solved at every point in space where the phase-field order parameter is to be updated and evolved in time. We present Thermo4PFM, a solver for the KKS equations for binary and ternary alloys, with two or three phases, and parameterized with CALPHAD models. Thermo4PFM is open source, written in C++, and can take advantage of Graphics Processing Units (GPU) accelerators. Using OpenMP offload capabilities for C++ classes, an excellent performance is demonstrated on GPU using the LLVM compiler. CALPHAD data is read from simple JSON files using an open source parser from the boost library. Program summaryProgram Title: Thermo4PFMCPC Library link to program files:https://doi.org/10.17632/8j3ntp5c7k.1Developer's repository link:https://github.com/ORNL/Thermo4PFMLicensing provisions: BSD 3-clauseProgramming language: C++/OpenMPNature of problem: Accurate modeling of solidification in metallic alloys requires thermodynamic data associated with possible material phases. That data can be used in phase-field modeling (PFM) to evaluate the driving force responsible for phase changes and solidification front motion. Integrating that data into PFM requires the solution of a small system of coupled nonlinear differential equations — the Kim-Kim-Suzuki equations — that needs to be solved at every point of a discretization mesh.Solution method: Thermo4PFM implements a Newton-based solver for the Kim-Kim-Suzuki equations for a few special cases (binary and ternary alloys with two or three phases) for CALPHAD-based models of the bulk energy of the phases. The software is written in C++ and uses the Curiously Recurring Template Pattern and OpenMP offload capabilities to take advantage of GPU accelerators, when available, on modern High Performance Computing resources. It also uses the boost Property Tree library to parse input CALPHAD data.

Replication-Based Dynamic Energy-Aware Resource Provisioning for Scientific Workflows

Distributed computing services in cloud environments are easily accessible to end users. These services are delivered to end users via a subscription-based model. The “infrastructure as a service” (IaaS) cloud model is one of the best cloud environment models for running data- and computing-intensive applications. Real-world scientific applications are the best examples of data and computing intensiveness. For their implementation, scientific workflow applications need high-performance computational resources and a large volume of storage. The workflow tasks are linked based on computational and data interdependence. Considering the high volume and variety of scientific workflows (SWs), the resources of the IaaS cloud model require managing energy efficiently and without failure or loss. Therefore, in order to address the issues of power consumption and task failure for real-world SWs, this research work proposes a replication-based dynamic energy-aware resource provisioning (R-DEAR) strategy for SWs in an IaaS cloud environment. The proposed strategy, R-DEAR, is a resource- and service-provisioning strategy that implements a replication-based fault-tolerant and load-balancing mechanism. The proposed R-DEAR strategy schedules the tasks of a scientific workflow with a replication-based fault-tolerant mechanism. The proposed R-DEAR strategy also manages the power consumption of IaaS cloud resources dynamically through a load-sharing process. Simulation results show that the proposed R-DEAR strategy reduces energy consumption, execution cost, and execution time by 9%, 15%, and 18%, respectively, as compared with the existing state-of-the-art strategy.

Adding multi-core support to the ALICE Grid Middleware

The major upgrade of the ALICE experiment for the LHC Run3 poses unique challenges and opportunities for new software development. In particular, the entirely new data taking and processing software of ALICE relies on process parallelism and large amounts of shared objects in memory. Thus from a single-core single thread workload in the past, the new workloads are exclusively multithreaded. This requires a profound change in the ALICE Grid midleware job handling, from scheduling to execution, and thus the entire middleware has been rewritten during the past 3 years to support the new multithreaded reality.This paper presents the ALICE middlewre development for multi-core job management and the tools used to achieve an efficient and secure environment. In particular, it covers job isolation and scheduling and how they can be implemented in different site configurations, such as sites shared with other experiments or High Performance Computing resources.

Hyperparameter optimization of data-driven AI models on HPC systems

In the European Center of Excellence in Exascale Computing ”Research on AI- and Simulation-Based Engineering at Exascale” (CoE RAISE), researchers develop novel, scalable AI technologies towards Exascale. This work exercises High Performance Computing resources to perform large-scale hyperparameter optimization using distributed training on multiple compute nodes. This is part of RAISE’s work on data-driven use cases which leverages AI- and HPC cross-methods developed within the project. In response to the demand for parallelizable and resource efficient hyperparameter optimization methods, advanced hyperparameter search algorithms are benchmarked and compared. The evaluated algorithms, including Random Search, Hyperband and ASHA, are tested and compared in terms of both accuracy and accuracy per compute resources spent. As an example use case, a graph neural network model known as MLPF, developed for Machine Learned Particle-Flow reconstruction, acts as the base model for optimization. Results show that hyperparameter optimization significantly increased the performance of MLPF and that this would not have been possible without access to large-scale High Performance Computing resources. It is also shown that, in the case of MLPF, the ASHA algorithm in combination with Bayesian optimization gives the largest performance increase per compute resources spent out of the investigated algorithms.

Multi-Jet Event classification with Convolutional neural network at Large Scale

We present an application of Scalable Deep Learning to analyze simulation data of the LHC proton-proton collisions at 13 TeV. We built a Deep Learning model based on the Convolutional Neural Network (CNN) which utilizes detector responses as two-dimensional images reflecting the geometry of the Compact Muon Solenoid (CMS) detector. The model discriminates signal events of the R-parity violating Supersymmetry (RPV SUSY) from the background events with multiple jets due to the inelastic QCD scattering (QCD multi-jets). With the CNN model, we obtained x1.85 efficiency and x1.2 expected significance with respect to the traditional cut-based method. We demonstrated the scalability of the model at a Large Scale with the High-Performance Computing (HPC) resources at the Korea Institute of Science and Technology Information (KISTI) up to 1024 nodes.