National Center For Supercomputing Applications Research Articles

End-to-end AI framework for interpretable prediction of molecular and crystal properties

We introduce an end-to-end computational framework that allows for hyperparameter optimization using the DeepHyper library, accelerated model training, and interpretable AI inference. The framework is based on state-of-the-art AI models including CGCNN, PhysNet, SchNet, MPNN, MPNN-transformer, and TorchMD-NET. We employ these AI models along with the benchmark QM9, hMOF, and MD17 datasets to showcase how the models can predict user-specified material properties within modern computing environments. We demonstrate transferable applications in the modeling of small molecules, inorganic crystals and nanoporous metal organic frameworks with a unified, standalone framework. We have deployed and tested this framework in the ThetaGPU supercomputer at the Argonne Leadership Computing Facility, and in the Delta supercomputer at the National Center for Supercomputing Applications to provide researchers with modern tools to conduct accelerated AI-driven discovery in leadership-class computing environments. We release these digital assets as open source scientific software in GitLab, and ready-to-use Jupyter notebooks in Google Colab.

Robert (Bob) John Novak June 5, 1947 to August 4, 2022

Dr. Robert Novak (“Dr. Bob”), our dear friend and colleague, passed away on August 4, 2022, at the age of 75, in Tampa, Florida. Much can be said about his academic credentials and accomplishments from his long career. He was a respected professor with multiple appointments in several academic departments of medicine, microbiology, and public health. He served as a consultant and advisor for numerous national and international organizations, including the EPA, WHO, Chinese Academy of Sciences, US Fish and Wildlife Service, National Center for Supercomputing Applications, Illinois Governor's Office and Pollution Control Board, Vector Control Division of the Puerto Rico Department of Public Health, Ministry of Health of the US Virgin Islands, US Agency for International Development in Honduras, US Army Medical Virology Institute in Fort Detrick, Maryland, Pan American Health Organization Aedes aegypti biology and control, and Pueblo (Colorado) Board of Water Works. Bob was a world-renowned medical entomologist with his research focused on mosquito biology and control. He was also a member of many professional organizations among which the American Mosquito Control Association (AMCA) and the Society for Vector Ecology (SOVE) stood so dear to him that he seldom missed their annual meetings. For his service and dedication, Bob received several AMCA awards: Presidential Citation (1993), Memorial Lecture Award (1998), and Medal of Honor (2003), while he served as the AMCA President in 1996–1997. He also served as SOVE President in 2007 and received the society's Distinguished Service Award in 2013. While his many notable accomplishments and contributions can be addressed by others, to me and many others, he was known as Dr. Bob, or simply Bob. We all know how much he loved Colorado, second only to how much he loved his wife, Lorraine. His stories about his upbringing on a cattle ranch and how that influenced his decision to enter the field of public health are legendary. The funny thing is that he loved animals but found the connection between them and the bugs that fed on them to be fascinating. This happens to the vast majority of those of us working in medical and livestock entomology, so we all accepted his comments as par for the course, never odd.Bob was a great man. It's been said that you can gauge a person's greatness according to their degree of commitment, and this was certainly true of Bob. A person like Bob is great because he speaks directly to you and lives according to what he says, and when he treats you with affection and respect. A person is great when he does not think about only himself, when he behaves in a kind way, when he provides support, or when he supports others when they most need him. Bob always demonstrated what is most important between two people: friendship, companionship, affection, and respect. Bob was a giant. He was always interested in your life and looked for opportunities for your personal and professional growth, encouraging his students and colleagues to dream and explore areas outside the conventional box. This was especially true when it meant doing things everyone else thought were so farfetched and impossible, such as his project with the satellite imagery for mosquito surveillance and control.Bob never allowed himself to be governed by clichéd behaviors and cared little for how he looked in front of others. He abhorred being manipulated or handling people like a puppeteer. He dealt with most disappointments with grace and deference and shared his opinions clearly and with certainty. Bob was blunt and direct, but he always tried to understand the perspectives of others even when he didn't see things the same way they did. We all have at least one instance when his bluntness was clearly manifested, to be washed off later with a beer. He always placed himself in your position and judged you on what he expected from himself under that circumstance. In his personal life, Bob enjoyed fine and exotic dining. Very seldom can one witness a person enjoy the pleasure of a new dish the way he enjoyed fried crickets and sautéed scorpions at a roadside diner in Cambodia or a bowl of mystery porridge served in a crusty bowl as he did at a gourmet dinner in Eritrea close to the border with Sudan.Bob was a consummate field entomologist who could not be judged through centimeters, meters, or wealth. He was defined by his actions, honesty, and decency, and will be remembered as kind and respectful of the feelings and interests of others.Dr. Bob was a great man whose memory and joy for living will remain vibrant in the hearts of many. For his sensitivity and how he treated others, Dr. Bob was a giant.

2021 VGTC Visualization Service Award—Loretta Auvil, National Center for Supercomputing Applications at the University of Illinois at Urbana Champaign

2021 VGTC Visualization Service Award—Loretta Auvil, National Center for Supercomputing Applications at the University of Illinois at Urbana Champaign

About the Contributors

About the Contributors

Physics-inspired deep learning to characterize the signal manifold of quasi-circular, spinning, non-precessing binary black hole mergers

The spin distribution of binary black hole mergers contains key information concerning the formation channels of these objects, and the astrophysical environments where they form, evolve and coalesce. To quantify the suitability of deep learning to estimate the individual spins, effective spin and mass-ratio of quasi-circular, spinning, non-precessing binary black hole mergers, we introduce a modified version of WaveNet trained with a novel optimization scheme that incorporates general relativistic constraints of the spin properties of astrophysical black holes. The neural network model is trained, validated and tested with 1.5 million ℓ=|m|=2 waveforms generated within the regime of validity of NRHybSur3dq8, i.e., mass-ratios q≤8 and individual black hole spins |s|{1,2}z≤0.8. To reduce time-to-insight, we deployed a distributed training algorithm at the IBM Power9 Hardware-Accelerated Learning cluster at the National Center for Supercomputing Applications to reduce the training stage from 1 month, using a single V100 NVIDIA GPU, to 12.4 hours using 64 V100 NVIDIA GPUs. We have also fully trained this model using 1536 V100 GPUs (256 nodes) in the Summit supercomputer at Oak Ridge National Laboratory, achieving state-of-the-art accuracy within just 1.2 hours. Using this neural network model, we quantify how accurately we can infer the astrophysical parameters of black hole mergers in the absence of noise. We do this by computing the overlap between waveforms in the testing data set and the corresponding signals whose mass-ratio and individual spins are predicted by our neural network. We find that the convergence of high performance computing and physics-inspired optimization algorithms enable an accurate reconstruction of the mass-ratio and individual spins of binary black hole mergers across the parameter space under consideration. This is a significant step towards an informed utilization of physics-inspired deep learning models to reconstruct the spin distribution of binary black hole mergers in realistic detection scenarios.

Resiliency of HPC Interconnects: A Case Study of Interconnect Failures and Recovery in Blue Waters

Availability of the interconnection network in high-performance computing (HPC) systems is fundamental to sustaining the continuous execution of applications at scale. When failures occur, interconnect recovery mechanisms orchestrate complex operations to recover network connectivity between the nodes. As the scale and design complexity of HPC systems increase, so does the system's susceptibility to failures during execution of interconnect-recovery procedures. This study characterizes the recovery procedures of the Gemini interconnect network, the largest Gemini network built by Cray, on Blue Waters, a 13.3 petaflop supercomputer at the National Center for Supercomputing Applications (NCSA). We propose a propagation model that captures interconnect failures and recovery procedures to help understand types of failures and their propagation in both the system and applications during recovery. The measurements show that recovery procedures occur very frequently and that the unsuccessful execution of recovery procedures, when additional failures occur during recovery, causes system-wide outages (SWOs, 28 out of 101) and application failures (3.4 percent of all running applications).

Transferring a petabyte in a day

Extreme-scale simulations and experiments can generate large amounts of data, whose volume can exceed the compute and/or storage capacity at the simulation or experimental facility. With the emergence of ultra-high-speed networks, researchers are considering pipelined approaches in which data are passed to a remote facility for analysis. Here we examine an extreme-scale cosmology simulation that, when run on a large fraction of a leadership computer, generates data at a rate of one petabyte per elapsed day. Writing those data to disk is inefficient and impractical, and in situ analysis poses its own difficulties. Thus we implement a pipeline in which data are generated on one supercomputer and then transferred, as they are generated, to a remote supercomputer for analysis. We use the Swift scripting language to instantiate this pipeline across Argonne National Laboratory and the National Center for Supercomputing Applications, which are connected by a 100 Gb/s network; and we demonstrate that by using the Globus transfer service we can achieve a sustained rate of 93 Gb/s over a 24-hour period, thus attaining our performance goal of one petabyte moved in 24 h. This paper describes the methods used and summarizes the lessons learned in this demonstration.

Publisher Correction: Western US volcanism due to intruding oceanic mantle driven by ancient Farallon slabs

In the version of this Article originally published, data points representing mafic eruptions were missing from Fig. 4b, the corrected version is shown below. Furthermore, the authors omitted to include the following acknowledgements to the provider of the computational resources: “This research is part of the Blue Waters sustained-petascale computing project, which is supported by the National Science Foundation (awards OCI-0725070 and ACI-1238993) and the state of Illinois. Blue Waters is a joint effort of the University of Illinois at Urbana-Champaign and its National Center for Supercomputing Applications. This work is also part of the ‘PRAC Title 4-D Geodynamic Modeling With Data Assimilation: Origin Of Intra-Plate Volcanism In The Pacific Northwest’ PRAC allocation support by the National Science Foundation (award number ACI 1516586). This work also used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1548562.” Figure 4 and the Acknowledgements section have been updated in the online version of the Article.

HPCG and HPGMG benchmark tests on multiple program, multiple data (MPMD) mode on Blue Waters—A Cray XE6/XK7 hybrid system

SummaryThe high‐performance conjugate gradients (HPCG) and high‐performance geometric multi‐grid (HPGMG) benchmarks are alternatives to the traditional LINPACK benchmark (HPL) in measuring the performance of modern HPC platforms. We performed HPCG and HPGMG benchmark tests on a Cray XE6/XK7 hybrid supercomputer, Blue Waters at National Center for Supercomputing Applications (NCSA). The benchmarks were tested on CPU‐based and GPU‐enabled nodes separately, and then we analyzed characteristic parameters that affect their performance. Based on our analyses, we performed HPCG and HPGMG runs in multiple program, multiple data (MPMD) mode in Cray Linux Environment in order to measure their hybrid performance on both CPU‐based and GPU‐enabled nodes. We observed and analyzed several performance issues during those tests. Based on lessons learned from this study, we provide recommendations about how to optimize science applications on modern hybrid HPC platforms.

Impact of a Cyberinfrastructure Enterprise on the Nation's Workforce: Visualizations of a Decade of NCSA's Diaspora

Conceived in terms of diaspora, the impact of CI enterprises can be thought of in terms of workforce development, strengthening leading edge industrial knowledge, and building national capabilities. To investigate how we might think through the diaspora of CI enterprises, we conducted a pilot study of one major CI enterprise: the National Center for Supercomputing Applications (NCSA), housed at the University of Illinois at Urbana-Champaign. We compiled and analyzed data from 425 key employees who departed NCSA in the decade from 2003-2013. Often the best way to make sense of complex data involves visualization, and the purpose of this effort was to experiment with visualizations of this diaspora data in an effort to better understand the impact of NCSA’s diaspora.

Intraorganizational Bargaining in Multi‐Stakeholder Initiatives

Negotiation JournalVolume 31, Issue 4 p. 393-400 New Frontiers in Intraorganizational Bargaining Intraorganizational Bargaining in Multi-Stakeholder Initiatives Joel Cutcher-Gershenfeld, Joel Cutcher-GershenfeldJoel Cutcher-Gershenfeld is a professor at the University of Illinois School of Labor and Employment Relations in Urbana-Champaign, where he was previously dean. He is also a senior research scientist at the National Center for Supercomputing Applications. He also holds a fractional appointment with the University of Sydney. Beginning in January 2016, he will join the faculty at Brandeis University's Heller School for Social Policy and Management in Waltham, Massachusetts. His e-mail address is joelcg@illinois.edu.Search for more papers by this author Joel Cutcher-Gershenfeld, Joel Cutcher-GershenfeldJoel Cutcher-Gershenfeld is a professor at the University of Illinois School of Labor and Employment Relations in Urbana-Champaign, where he was previously dean. He is also a senior research scientist at the National Center for Supercomputing Applications. He also holds a fractional appointment with the University of Sydney. Beginning in January 2016, he will join the faculty at Brandeis University's Heller School for Social Policy and Management in Waltham, Massachusetts. His e-mail address is joelcg@illinois.edu.Search for more papers by this author First published: 23 October 2015 https://doi.org/10.1111/nejo.12115Citations: 2 Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Citing Literature Volume31, Issue4October 2015Pages 393-400 RelatedInformation

(Invited) Molecular Understanding of Oxygen Exchange in Solid Oxide Fuel Cell Cathodes

Materials with the ability to rapidly exchange oxygen with the environment play a critical role in many existing and developing technologies, e.g., solid oxide fuel cells (SOFCs), gas separation membranes, oxygen sensors, chemical looping devices, and memristors. In particular, reducing the working temperature of SOFCs, which is critical to their increased commercialization, is significantly inhibited by the slow oxygen exchange kinetics at the cathode. The oxygen exchange is an essential step in the overall oxygen reduction reaction (ORR), where O2 (gas) combines with 4e- to incorporate as 2O2- into the electrolyte, facilitated by the cathode. Oxygen exchange rates are typically quantified by the oxygen exchange coefficient, K*, which represents the response of the rate of oxygen exchange to a driving force. In our work we are developing techniques to predict K* and provide a molecular scale understanding of processes involved in oxygen surface exchange, with the long-term goal of designing more active oxygen exchange materials. Our work has focused on perovskite materials. A few years ago we identified the distance of the bulk oxygen 2p-band center to the Fermi level as an easily calculated descriptor for oxygen exchange in electronic conducing perovskites.[1] These results suggested that overall oxyphilicity plays a critical role in determining K* and provides a tool for high-throughput ab initio screening of promising materials. We will report on recent predictions of active oxygen exchange systems using these screening approaches. In more recent work we have initiated an effort to model oxygen exchange at the molecular level for the (La,Sr)CoO3-d (LSC) system. Using ab inito energetics and the elementary kinetics formalism developed by Adler, et al.[2] We have modeled the possible reaction steps on the LSC (001) surface. This modeling has yielded valuable insights in the rate limiting steps of oxygen exchange through a perovskite surface, the role of surface termination and Sr segregation on K*,[3, 4] and the absolute limits of surface exchange rates. Acknowledgement We gratefully acknowledge financial support from the U.S. Department of Energy, Office of Basic Energy Sciences, Division of Materials Sciences and Engineering under award number DE-SC0001284, and computing support from NSF National Center for Supercomputing Applications (NCSA - DMR060007).

Reproducible research [From the Editor

One of the concerns of the broader field of computational research is whether the results that are published are reproducible. When the author had a position as a senior research scientist at the National Center for Supercomputing Applications in the late 1990s, he had a conversation with a computational physicist who said that most simulation codes in his field were not made publicly available, and it was his opinion that at least 95% of those codes contained major errors that result in technically incorrect results regularly being published. Because he was a generally balanced and agreeable fellow and had worked for many years at the U.S. Department of Energy national laboratories responsible for the development of most of the foundational computational software used today the author considered his estimate of 95% reasonably accurate. This computational physicist said that the solution to the high occurrence of errors is that all computational codes should be made publicly available so that errors can be found and fixed, and he commented that computational research would progress much more quickly if this simple practice were employed. He later received a major national award based largely on the computational codes that he had developed over his career, all of which he had made publicly available. So the author asks readers: "Should authors in the control field be expected or compelled to make their software public, as a way to reduce errors and to facilitate progress in the field?"

NCSA Mosaic: a global hypermedia system

PurposeThis paper seeks to report on an Internet‐based system for hypermedia information discovery and retrieval and wide‐area distributed asynchronous collaboration designed and built at the National Center for Supercomputing Applications (NCSA).Design/methodology/approachThe NCSA is developing Mosaic clients – user‐friendly information browsers – for the three most popular desktop computing environments of the mid‐1990s: the Unix‐based X Window System, the Apple Macintosh, and Microsoft Windows 3.1. This paper primarily discusses the X client, as it has the most advanced functionality of the three at this time.FindingsThe system, called NCSA Mosaic, integrates cleanly into existing Internet protocols, formats, data sources, and environments, and provides powerful new capabilities for using and sharing information across the Internet.Originality/valueNCSA is making the complete Mosaic system freely available and distributable for all academic and research organizations and purposes.

High-Performance Computing with Accelerators

This issue of CiSE is based on work presented at the US National Science Foundation workshop, Path to Petascale: Adapting Geo/Chem/Astro Applications for Accelerators and Accelerator Clusters, held at the US National Center for Supercomputing Applications (NCSA) in early 2009. The workshop was designed to raise awareness about the emergence of accelerator-based high-performance computing (HPC) resources among computational scientists from the geosciences, computational chemistry, and astronomy and astrophysics communities and to help them get started in using these resources.

Using Domain Based Discovery Methods to Identify Prokaryotic Counterparts to Eukaryotic Protein Ion Channels

Our lab, in collaboration with the laboratory of I. Aravind, used domain based methods to discover the prokaryotic counterparts of Ach receptor channels (Tasneem et al), after whole-protein approaches such as BLAST had failed. Our identification was verified by functional studies and by x-ray crystallization of targets we identified. We have subsequently streamlined and formalized the domain search methods and applied them to a variety of ion channels and other membrane proteins. It has become clear that domain based methods are more powerful than whole-protein approaches, provided one has good domain definitions to start the search. The use of domain-based methods depends on a somewhat different model of evolution from BLAST. In BLAST, the operational model is the substitution of one amino acid for another, with gaps being treated as a particular type of substitution. Domain-based methods deal readily and directly with the phenomenon of large scale reorganization of domains, which is now recognized as an essential process for innovation in evolution. In this presentation we will provide an update on prokaryotic Ach receptor channels, a survey of prokaryotic glutamate receptor channels and their relationship to their eukaryotic counterparts, the discovery of a prokaryotic counterpart to HCN and CNG channels which appears likely to be closely related to their common ancestor, and searches on other families that are under way at the time of preparation of this abstract. We gratefully acknowledge support from NSF grants 0835718 and 0235792, from NIH grants 5PN2EY016570-06 and 5R01NS063405-02 from the Beckman Institute for Advanced Science and Technology, the National Center for Supercomputing Applications, and the Renaissance Computing Institute.

A parallel evolutionary strategy based simulation–optimization approach for solving groundwater source identification problems

Groundwater characterization involves the resolution of unknown system characteristics from observation data, and is often classified as an inverse problem. Inverse problems are difficult to solve due to natural ill-posedness and computational intractability. Here we adopt the use of a simulation–optimization approach that couples a numerical pollutant-transport simulation model with evolutionary search algorithms for solution of the inverse problem. In this approach, the numerical transport model is solved iteratively during the evolutionary search. This process can be computationally intensive since several hundreds to thousands of forward model evaluations are typically required for solution. Given the potential computational intractability of such a simulation–optimization approach, parallel computation is employed to ease and enable the solution of such problems. In this paper, several variations of a groundwater source identification problem is examined in terms of solution quality and computational performance. The computational experiments were performed on the TeraGrid cluster available at the National Center for Supercomputing Applications. The results demonstrate the performance of the parallel simulation–optimization approach in terms of solution quality and computational performance.

Cyberinfrastructure for the analysis of ecological acoustic sensor data: a use case study in grid deployment

The LTER Grid Pilot Study was conducted by the National Center for Supercomputing Applications, the University of New Mexico, and Michigan State University, to design and build a prototype grid for the ecological community. The featured grid application, the Biophony Grid Portal, manages acoustic data from field sensors and allows researchers to conduct real-time digital signal processing analysis on high-performance systems via a web-based portal. Important characteristics addressed during the study include the management, access, and analysis of a large set of field collected acoustic observations from microphone sensors, single signon, and data provenance. During the development phase of this project, new features were added to standard grid middleware software and have already been successfully leveraged by other, unrelated grid projects. This paper provides an overview of the Biophony Grid Portal application and requirements, discusses considerations regarding grid architecture and design, details the technical implementation, and summarizes key experiences and lessons learned that are generally applicable to all developers and administrators in a grid environment.

From Proposal to Production: Lessons Learned Developing the Computational Chemistry Grid Cyberinfrastructure

The Computational Chemistry Grid (CCG) is a three-year, National Middleware Initiative program to develop cyberinfrastructure for the chemistry community. CCG is led by the University of Kentucky and involves collaborating sites at Louisiana State University, Ohio Supercomputing Center, Texas Advanced Computing Center, and the National Center for Supercomputing Applications. This paper discusses experiences developing the CCG cyberinfrastructure in the first year of the project. Special attention is paid to technological issues faced as well as issues raised running the CCG in production. The final section of the paper looks forward to challenges foreseen in the remaining two years.

Diet composition impacts cerebral cortex gene expression profiles of young adult and geriatric dogs

Recent studies suggest that diet may alter gene expression profiles of aging tissues. The objective of this study was to measure gene expression differences of brain tissue of healthy young adult and geriatric dogs consuming two distinct types of dog foods. Six geriatric (11 yr-old at baseline) and 6 weanling (8 wk-old at baseline) female beagles were randomly assigned to one of two diets: 1) Animal-protein based diet (APB) containing 28% protein, 23% fat, and 5% fiber; or 2) Plant-protein based diet (PPB) containing 26% protein, 11% fat, and 15% fiber. Dogs remained on experiment for 12 months. RNA was isolated from cerebral cortex tissue using Trizol (Invitrogen) and hybridized to Affymetrix GeneChip Canine Genome Arrays (Affymetrix) as per the manufacturer’s protocol. Following microarray normalization, data were analyzed using the mixed models procedure of SAS. Transcripts having a P < 0.05 and > 1.5-fold change were considered significantly different among groups. Diet manipulated gene expression level in both geriatrics (17 transcripts) and young adults (21 transcripts). Of particular interest, dogs fed APB had increased expression of genes associated with inflammatory response and acute phase proteins. This research was funded in part by the National Center for Supercomputing Applications and the University of Illinois, under the auspices of the NCSA/UIUC Faculty Fellows Program, and by Pyxis Genomics Inc.