Argonne Leadership Computing Facility Research Articles

Physics-inspired spatiotemporal-graph AI ensemble for the detection of higher order wave mode signals of spinning binary black hole mergers

We present a new class of AI models for the detection of quasi-circular, spinning, non-precessing binary black hole mergers whose waveforms include the higher order gravitational wave modes (ℓ,|m|)={(2,2),(2,1),(3,3),(3,2),(4,4)} , and mode mixing effects in the ℓ=3,|m|=2 harmonics. These AI models combine hybrid dilated convolution neural networks to accurately model both short- and long-range temporal sequential information of gravitational waves; and graph neural networks to capture spatial correlations among gravitational wave observatories to consistently describe and identify the presence of a signal in a three detector network encompassing the Advanced LIGO and Virgo detectors. We first trained these spatiotemporal-graph AI models using synthetic noise, using 1.2 million modeled waveforms to densely sample this signal manifold, within 1.7 h using 256 NVIDIA A100 GPUs in the Polaris supercomputer at the Argonne Leadership Computing Facility. This distributed training approach exhibited optimal classification performance, and strong scaling up to 512 NVIDIA A100 GPUs. With these AI ensembles we processed data from a three detector network, and found that an ensemble of 4 AI models achieves state-of-the-art performance for signal detection, and reports two misclassifications for every decade of searched data. We distributed AI inference over 128 GPUs in the Polaris supercomputer and 128 nodes in the Theta supercomputer, and completed the processing of a decade of gravitational wave data from a three detector network within 3.5 h. Finally, we fine-tuned these AI ensembles to process the entire month of February 2020, which is part of the O3b LIGO/Virgo observation run, and found 6 gravitational waves, concurrently identified in Advanced LIGO and Advanced Virgo data, and zero false positives. This analysis was completed in one hour using one NVIDIA A100 GPU.

Comment: AI Accelerates Modeling of New Materials for Carbon Capture

_ Artificial intelligence (AI) is increasingly being employed to assist in the development of materials, including metal-organic frameworks (MOFs), to develop carbon capture technologies. MOFs are modular materials made up of three building blocks: inorganic nodes such as zinc or copper; organic nodes; and organic linkers made up of carbon, oxygen, and other elements. By changing the relative positions and configurations of the building blocks, the potential combinations for creation of unique MOFs are countless. The idea is to create a porous carbon dioxide “trap” to capture carbon from the air. The structure created by the building blocks can be thought of simplistically as a scaffolding with joints (linkers) that functions to absorb carbon. In a recent paper, researchers from the US Department of Energy Argonne National Laboratory, The University of Illinois Chicago (UIC), University of Illinois, Northwestern University, and TotalEnergies described their use of generative AI to identify “good carbon absorbers” among “billions and billions of possibilities.” Although early work on developing MOFs began in the 1990s, the power of AI modeling can generate new models with desired properties such as optimal selectivity and capacity without the laborious, reiterative experimental and computational efforts once required. In a press release, Argonne said, “… The team was able to quickly assemble, building block by building block, over 120,000 new MOF candidates within 30 minutes” on a supercomputer at the Argonne Leadership Computing Facility. “The race for capturing carbon hinges on finding needles in a haystack, and trial and error is too slow. You have billions and billions of possibilities, and then you must narrow down to candidates that are good carbon absorbers,” said paper coauthor Santanu Chaudhuri, professor of civil, materials, and environmental engineering at UIC and director of manufacturing science and engineering at Argonne, in a news release. “With this project, we have taken the first significant step towards closing that gap by using generative AI.” Coauthor Eliu Huerta, an Argonne computational scientist who helped lead the study, said, “The traditional methods have typically involved experimental synthesis and computational modeling with molecular dynamics simulations. But trying to survey the vast MOF landscape in this way is just impractical. “We wanted to add new flavors to the MOFs that we were designing. We needed new ingredients for the AI recipe,” Huerta said. The AI framework, called GHP-MOFassemble, screened the newly built 120,000 MOFs in 40 minutes to identify nearly 79,000 with valid bonds (joints). Those were screened to identify 19,000 with valid chemistry within 205 minutes. From this group, 364 MOFs were identified in 50 minutes with CO2 capacity higher than a selected value. From assembly to selection of MOFs, GHP-MOFassemble completed the analysis within 5 hours and 7 minutes. Powering the AI Future It’s ironic that processing of massive volumes of data on supercomputers for emissions-related research, such as AI simulations, could contribute to the problem to be solved because of the electricity required. Supercomputers use the computing power of multiple interconnected processing cores, which require an immense amount of energy—and depending on the source of the electricity generation, also contribute to emissions. In a November 2023 commentary, International Energy Agency (IEA) analysts estimated that AI uses more energy than other forms of computing. Training a single model uses more electricity than 100 US homes consume in an entire year. In 2022, Google reported that machine learning accounted for about 15% of its total energy use over the prior 3 years. In a January report, IEA estimated that Google could see a tenfold increase in their electricity demand if AI is fully implemented in its search engine. The average electricity demand of a typical search is 0.3 Wh. OpenAI’s ChatGPT requires 2.9 Wh per request. Considering 9 billion searches per day, nearly 10 TWh of additional electricity would be required in a year. Google and other tech companies are shifting their data operations around the world daily or hourly to tap into excess renewable energy production. Google uses its “carbon-intelligent” platform to analyze day-ahead predictions of how much a given grid will be relying on carbon-intensive energy. It then shifts computing globally in favor of regions where more carbon-free electricity is available. As AI advances in modeling complexity with a greater number of parameters, there is a growing emphasis on sustainability and optimizing power consumption, including developing energy-efficient algorithms, hardware, and data center management strategies. For Further Reading A Generative Artificial Intelligence Framework Based on a Molecular Diffusion Model for the Design of Metal-Organic Frameworks for Carbon Capture by H. Park, X. Yan, and R. Zhu, et al. Communications Chemistry. Why AI and Energy Are the New Power Couple by V. Rozite, J. Miller, and S. Oh, International Energy Agency. Electricity 2024. International Energy Agency. We Now Do More Computing Where There’s Cleaner Energy by R. Koningstein, Google Research.

Multiscale Modeling of Heterogeneous Interfaces in All Solid-State Batteries

Ion transport and degradation at the solid-electorlyte/cathode interfaces dictate the performance of all solid-state batteries. For example, it has been routinely observed that in many superionic oxide conductors, Li-ion transport kinetics are orders of magnitude slower at the interfaces, such as grain boundaries or interfaces with common electrode materials, compared to the bulk phases. Upon electrochemical cycling, interdiffusion and/or electrolyte/cathode decomposition at these interfaces can further deteriorate Li-ion transport kinetics. Therefore, the ability to understand, predict and precisely control ion transport kinetics and the degree of interfacial degradation are critical and would have a transformative impact on the development of solid-state battery technologies for practical applications. Here I will address the need of a multiscale and multiphysics modeling framework to understand Li-ion transport mechanisms at complex interfaces and predict the kinetics of Li-ion transport in solid-electrolyte/cathode composite architectures that are relevant for all solid-state Li batteries. I will also emphasize the importance of using a closely coupled theory-experiment approach to reveal the atomic details of the interfaces and unravel the degradation mechanisms of interfaces and its impact on Li-ion transport. I will use the garnet-type solid-state electrotype and lithium cobalt oxide cathode as an example to demonstrate the predictive power of such a multiscale, multiphysics modeling framework towards establishing the correlations between ion transport and local atomic structure, chemical environment, and mesoscopic microstructure features in the cathode/electrolyte composites.This work was sponsored by the Office of Energy Efficiency and Renewable Energy, Vehicle Technologies Office and was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. An award of computer time was provided by the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program. This research used resources of the Argonne Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC02-06CH11357.

End-to-end AI framework for interpretable prediction of molecular and crystal properties

We introduce an end-to-end computational framework that allows for hyperparameter optimization using the DeepHyper library, accelerated model training, and interpretable AI inference. The framework is based on state-of-the-art AI models including CGCNN, PhysNet, SchNet, MPNN, MPNN-transformer, and TorchMD-NET. We employ these AI models along with the benchmark QM9, hMOF, and MD17 datasets to showcase how the models can predict user-specified material properties within modern computing environments. We demonstrate transferable applications in the modeling of small molecules, inorganic crystals and nanoporous metal organic frameworks with a unified, standalone framework. We have deployed and tested this framework in the ThetaGPU supercomputer at the Argonne Leadership Computing Facility, and in the Delta supercomputer at the National Center for Supercomputing Applications to provide researchers with modern tools to conduct accelerated AI-driven discovery in leadership-class computing environments. We release these digital assets as open source scientific software in GitLab, and ready-to-use Jupyter notebooks in Google Colab.

FAIR principles for AI models with a practical application for accelerated high energy diffraction microscopy

A concise and measurable set of FAIR (Findable, Accessible, Interoperable and Reusable) principles for scientific data is transforming the state-of-practice for data management and stewardship, supporting and enabling discovery and innovation. Learning from this initiative, and acknowledging the impact of artificial intelligence (AI) in the practice of science and engineering, we introduce a set of practical, concise, and measurable FAIR principles for AI models. We showcase how to create and share FAIR data and AI models within a unified computational framework combining the following elements: the Advanced Photon Source at Argonne National Laboratory, the Materials Data Facility, the Data and Learning Hub for Science, and funcX, and the Argonne Leadership Computing Facility (ALCF), in particular the ThetaGPU supercomputer and the SambaNova DataScale® system at the ALCF AI Testbed. We describe how this domain-agnostic computational framework may be harnessed to enable autonomous AI-driven discovery.

Parallel Scalability of Three-Level FROSch Preconditioners to 220000 Cores using the Theta Supercomputer

The parallel performance of the three-level fast and robust overlapping Schwarz (FROSch) preconditioners is investigated for linear elasticity. The FROSch framework is part of the Trilinos software library and contains a parallel implementation of different preconditioners with energy minimizing coarse spaces of generalized Dryja--Smith--Widlund type. The three-level extension is constructed by a recursive application of the FROSch preconditioner to the coarse problem. In this paper, the additional steps in the implementation in order to apply the FROSch preconditioner recursively are described in detail. Furthermore, it is shown that no explicit geometric information is needed in the recursive application of the preconditioner. In particular, the rigid body modes, including the rotations, can be interpolated on the coarse level without additional geometric information. Parallel results for a three-dimensional linear elasticity problem obtained on the Theta supercomputer (Argonne Leadership Computing Facility, Argonne, IL) using up to 220 000 cores are discussed and compared to results obtained on the SuperMUC-NG supercomputer (Leibniz Supercomputing Centre, Garching, Germany). Notably, it can be observed that a hierarchical communication operation in FROSch related to the coarse operator starts to dominate the computing time on Theta, which has a dragonfly interconnect, for 100 000 message passing interface (MPI) ranks or more. The same operation, however, scales well and stays within the order of a second in all experiments performed on SuperMUC-NG, which uses a fat tree network. Using hybrid MPI/OpenMP parallelization, the onset of the MPI communication problem on Theta can be delayed. Further analysis of the performance of FROSch on large supercomputers with dragonfly interconnects will be necessary.

Interpretable AI forecasting for numerical relativity waveforms of quasicircular, spinning, nonprecessing binary black hole mergers

We present a deep-learning artificial intelligence model (AI) that is capable of learning and forecasting the late-inspiral, merger and ringdown of numerical relativity waveforms that describe quasicircular, spinning, nonprecessing binary black hole mergers. We used theNRHybSur3dq8 surrogate model to produce train, validation and test sets of $\ensuremath{\ell}=|m|=2$ waveforms that cover the parameter space of binary black hole mergers with mass ratios $q\ensuremath{\le}8$ and individual spins $|{s}_{{1,2}}^{z}|\ensuremath{\le}0.8$. These waveforms cover the time range $t\ensuremath{\in}[\ensuremath{-}5000\text{ }\text{ }\mathrm{M},130\text{ }\text{ }\mathrm{M}]$, where $t=0M$ marks the merger event, defined as the maximum value of the waveform amplitude. We harnessed the ThetaGPU supercomputer at the Argonne Leadership Computing Facility to train our AI model using a training set of 1.5 million waveforms. We used 16 NVIDIA DGX A100 nodes, each consisting of 8 NVIDIA A100 Tensor Core GPUs and 2 AMD Rome CPUs, to fully train our model within 3.5 h. Our findings show that artificial intelligence can accurately forecast the dynamical evolution of numerical relativity waveforms in the time range $t\ensuremath{\in}[\ensuremath{-}100\text{ }\text{ }\mathrm{M},130\text{ }\text{ }\mathrm{M}]$. Sampling a test set of 190,000 waveforms, we find that the average overlap between target and predicted waveforms is $\ensuremath{\gtrsim}99%$ over the entire parameter space under consideration. We also combined scientific visualization and accelerated computing to identify what components of our model take in knowledge from the early and late-time waveform evolution to accurately forecast the latter part of numerical relativity waveforms. This work aims to accelerate the creation of scalable, computationally efficient and interpretable artificial intelligence models for gravitational wave astrophysics.

Accelerating Scientific Applications With SambaNova Reconfigurable Dataflow Architecture

Our exploratory work finds that the SambaNova Reconfigurable Dataflow Architecture (RDA) along with the SambaFlow software stack provides for an attractive system and solution to accelerate AI for science workloads. We have observed the efficacy of using the system with a diverse set of science applications and reasoned their suitability for performance gains over traditional hardware. As the Data-Scale system provides for a very large memory capacity, the system can be used to train models that typically do not fit in a GPU. The architecture also provides for deeper integration with upcoming supercomputers at the Argonne Leadership Computing Facility (ALCF), a US Department of Energy Office of Science user facility, to help advance science insights.

The Outer Rim Simulation: A Path to Many-core Supercomputers

Abstract We describe the Outer Rim cosmological simulation, one of the largest high-resolution N-body simulations performed to date, aimed at promoting science to be carried out with large-scale structure surveys. The simulation covers a volume of (4.225 Gpc)3 and evolves more than one trillion particles. It was executed on Mira, a BlueGene/Q system at the Argonne Leadership Computing Facility. We discuss some of the computational challenges posed by a system such as Mira, a many-core supercomputer, and how the simulation code, Hardware/Hybrid Accelerated Cosmology Code, has been designed to overcome these challenges. We have carried out a large range of analyses on the simulation data and we report on the results as well as the data products that have been generated. The full data set generated by the simulation totals more than 5 PB of data, making curation and handling of the data a large challenge in itself. The simulation results have been used to generate synthetic catalogs for large-scale structure surveys, including Dark Energy Spectroscopic Instrument and eBOSS, as well as experiments on the cosmic microwave background. A detailed catalog for the data challenges of the Large Synoptic Survey Telescope Dark Energy Science Collaboration has been created as well. We publicly release some of the Outer Rim halo catalogs, downsampled particle information, and lightcone data.

HACC Cosmological Simulations: First Data Release

Abstract We describe the first major public data release from cosmological simulations carried out with Argonne’s HACC code. This initial release covers a range of data sets from large gravity-only simulations. The data products include halo information for multiple redshifts, downsampled particles, and lightcone outputs. We provide data from two very large ΛCDM simulations as well as beyond-ΛCDM simulations spanning 11 cosmologies. Our release platform uses Petrel, a research data service, located at the Argonne Leadership Computing Facility. Petrel offers fast data transfer mechanisms and authentication via Globus, enabling simple and efficient access to stored data sets. Easy browsing of the available data products is provided via a web portal that allows the user to navigate simulation products efficiently. The data hub will be extended by adding more types of data products and by enabling computational capabilities to allow direct interactions with simulation results.

Software Stack in a Snapshot

The Argonne Leadership Computing Facility is deploying Singularity to allow HPC resources to adopt “containers”—a technology that has benefitted non-HPC resources like cloud computing servers for a few years now. For HPC users, containerization will allow them to easily migrate their software stack between resources with minimal effort. For HPC facilities themselves, containerization gives users more autonomy to deploy the version of the software that best meets their specific needs.

Use of the ERD for administrative monitoring of Theta

SummaryMonitoring the state of an HPC cluster in a timely and accurate fashion is critical to most system administration functions. For many Cray users, the first step in monitoring is ingestion of log files. Unfortunately, log parsing is an inherently inefficient process, requiring multiple software components to read and write from files on disk. Cray's own utilities use a message bus, the Event Router Daemon (ERD), for a wide variety of purposes. At the Argonne Leadership Computing Facility (ALCF), we have begun to use this message bus for monitoring via a client library written in Go, allowing us to read in structured data directly from Cray's services and, in many instances, bypass log files entirely. In this paper, we will examine the implementation and utilization of this approach on our 4392 node XC40, Theta, as well as the overall benefits and drawbacks to using the ERD for real‐time monitoring.

Performance of the AMS Offline Software at National Energy Research Scientific Computing Centre and Argonne Leadership Computing Facility

The Alpha Magnetic Spectrometer [1] (AMS) is a high energy physics experiment installed and operating on board of the International Space Station (ISS) from May 2011 and expected to last through year 2024 and beyond. More than 50 million of CPU hours has been delivered for AMS Monte Carlo simulations using NERSC and ALCF facilities in 2017. The details of porting of the AMS software to the 2nd Generation Intel Xeon Phi Knights Landing architecture are discussed, including the MPI emulation module to allow the AMS offline software to be run as multiple-node batch jobs. The performance of the AMS simulation software at NERSC Cori (KNL 7250), ALCF Theta (KNL 7230), and Mira (IBM BG/Q) farms is also discussed.

Early Science on Theta

At the Argonne Leadership Computing Facility, the process for standing up a new system is paired with The Early Science Program. This competitive program prepares a select group of applications science teams to use the new machine very readily by providing them staff support and early access to preproduction system hardware and software. This column describes three Theta Early Science projects: two science applications and one code development effort.

Automatic Between-Pulse Analysis of DIII-D Experimental Data Performed Remotely on a Supercomputer at Argonne Leadership Computing Facility

For the first time, an automatically triggered, between-pulse fusion science analysis code was run on-demand at a remotely located supercomputer at Argonne Leadership Computing Facility (ALCF, Lemont, Illinois) in support of in-process experiments being performed at DIII-D (San Diego, California). This represents a new paradigm for combining geographically distant experimental and high-performance computing facilities to provide enhanced data analysis that is quickly available to researchers. Enhanced analysis improves the understanding of the current pulse, translating into a more efficient use of experimental resources and quality of the resultant science. The analysis code used here, called SURFMN, calculates the magnetic structure of the plasma using Fourier transform. Increasing the number of Fourier components provides a more accurate determination of the stochastic boundary layer near the plasma edge by better resolving magnetic islands, but requires 26 min to complete using local DIII-D resources, putting it well outside the useful time range for between-pulse analysis. These islands relate to confinement and edge-localized mode suppression, and may be controlled by adjusting coil currents for the next pulse. ALCF has ensured on-demand execution of SURFMN by providing a reserved queue, a specialized service that launches the code after receiving an automatic trigger, and network access from the worker nodes for data transfer. Runs are executed on 252 cores of ALCF’s Cooley cluster and the data are available locally at DIII-D within 3 min of triggering. The original SURFMN design limits additional improvements with more cores; however, our work shows a path forward where codes that benefit from thousands of processors can run between pulses.

Challenges in scaling NLO generators to leadership computers

Exascale computing resources are roughly a decade away and will be capable of 100 times more computing than current supercomputers. In the last year, Energy Frontier experiments crossed a milestone of 100 million core-hours used at the Argonne Leadership Computing Facility, Oak Ridge Leadership Computing Facility, and NERSC. The Fortran-based leading-order parton generator called Alpgen was successfully scaled to millions of threads to achieve this level of usage on Mira. Sherpa and MadGraph are next-to-leading order generators used heavily by LHC experiments for simulation. Integration times for high-multiplicity or rare processes can take a week or more on standard Grid machines, even using all 16-cores. We will describe our ongoing work to scale the Sherpa generator to thousands of threads on leadership-class machines and reduce run-times to less than a day. This work allows the experiments to leverage large-scale parallel supercomputers for event generation today, freeing tens of millions of grid hours for other work, and paving the way for future applications (simulation, reconstruction) on these and future supercomputers.

Theta: Rapid installation and acceptance of an XC40 KNL system

SummaryIn order to provide a stepping stone from the Argonne Leadership Computing Facility's (ALCF) world class production 10 petaFLOP IBM BlueGene/Q system, Mira, to its next generation 200 petaFLOPS 3rd generation Intel Xeon Phi system, Aurora, ALCF worked with Intel and Cray to acquire an 8.6 petaFLOPS 2nd generation Intel Xeon Phi–based system named Theta. Theta was delivered, installed, integrated, and accepted on an aggressive schedule in just over 3 months. We will detail how we were able to successfully meet the aggressive deadline as well as lessons learned during the process.

Argonne Discovery Yields Self-Healing Diamond-Like Carbon

Large-scale reactive molecular dynamics simulations carried out on the US Department of Energy's IBM Blue Gene/Q Mira supercomputer at the Argonne Leadership Computing Facility, along with experiments conducted by researchers in Argonne's Energy Systems Division, enabled the design of a self-healing anti-wear coating that drastically reduces friction and related degradation in engines and moving machinery. Now, the computational work advanced for this purpose is being used to identify the friction-fighting potential of other catalysts.

Adapting the serial Alpgen parton-interaction generator to simulate LHC collisions on millions of parallel threads

As the LHC moves to higher energies and luminosity, the demand for computing resources increases accordingly and will soon outpace the growth of the Worldwide LHC Computing Grid. To meet this greater demand, event generation Monte Carlo was targeted for adaptation to run on Mira, the supercomputer at the Argonne Leadership Computing Facility. Alpgen is a Monte Carlo event generation application that is used by LHC experiments in the simulation of collisions that take place in the Large Hadron Collider. This paper details the process by which Alpgen was adapted from a single-processor serial-application to a large-scale parallel-application and the performance that was achieved.

Optimising the Termofluids CFD code for petascale simulations

ABSTRACTThis paper presents some recent efforts carried out on the expansion of the scalability of TermoFluids multi-physics Computational Fluid Dynamics (CFD) code, aiming to achieve petascale capacity for a single simulation. We describe different aspects that we have improved in our code in order to efficiently run it on 131,072 CPU-cores. This work has been developed using the BlueGene/Q Mira supercomputer of the Argonne Leadership Computing Facility, where we have obtained feedback at the targeted scale. In summary, this is a practical paper showing our experience at reaching the petascale paradigm for a single simulation with TermoFluids.