High Performance Computing Resources Research Articles

Accelerating single molecule localization microscopy through parallel processing on a high-performance computing cluster.

SummarySuper‐resolved microscopy techniques have revolutionized the ability to study biological structures below the diffraction limit. Single molecule localization microscopy (SMLM) techniques are widely used because they are relatively straightforward to implement and can be realized at relatively low cost, e.g. compared to laser scanning microscopy techniques. However, while the data analysis can be readily undertaken using open source or other software tools, large SMLM data volumes and the complexity of the algorithms used often lead to long image data processing times that can hinder the iterative optimization of experiments. There is increasing interest in high throughput SMLM, but its further development and application is inhibited by the data processing challenges. We present here a widely applicable approach to accelerating SMLM data processing via a parallelized implementation of ThunderSTORM on a high‐performance computing (HPC) cluster and quantify the speed advantage for a four‐node cluster (with 24 cores and 128 GB RAM per node) compared to a high specification (28 cores, 128 GB RAM, SSD‐enabled) desktop workstation. This data processing speed can be readily scaled by accessing more HPC resources. Our approach is not specific to ThunderSTORM and can be adapted for a wide range of SMLM software.Lay DescriptionOptical microscopy is now able to provide images with a resolution far beyond the diffraction limit thanks to relatively new super‐resolved microscopy (SRM) techniques, which have revolutionized the ability to study biological structures. One approach to SRM is to randomly switch on and off the emission of fluorescent molecules in an otherwise conventional fluorescence microscope. If only a sparse subset of the fluorescent molecules labelling a sample can be switched on at a time, then each emitter will be, on average, spaced further apart than the diffraction‐limited resolution of the conventional microscope and the separate bright spots in the image corresponding to each emitter can be localised to high precision by finding the centre of each feature using a computer program. Thus, a precise map of the emitter positions can be recorded by sequentially mapping the localisation of different subsets of emitters as they are switched on and others switched off.Typically, this approach, described as single molecule localisation microscopy (SMLM), results in large image data sets that can take many minutes to hours to process, depending on the size of the field of view and whether the SMLM analysis employs a computationally‐intensive iterative algorithm. Such a slow workflow makes it difficult to optimise experiments and to analyse large numbers of samples. Faster SMLM experiments would be generally useful and automated high throughput SMLM studies of arrays of samples, such as cells, could be applied to drug discovery and other applications. However, the time required to process the resulting data would be prohibitive on a normal computer.To address this, we have developed a method to run standard SMLM data analysis software tools in parallel on a high‐performance computing cluster (HPC). This can be used to accelerate the analysis of individual SMLM experiments or it can be scaled to analyse high throughput SMLM data by extending it to run on an arbitrary number of HPC processors in parallel. In this paper we outline the design of our parallelised SMLM software for HPC and quantify the speed advantage when implementing it on four HPC nodes compared to a powerful desktop computer.

Training bioinformaticians in High Performance Computing

In the last decade, bioinformatics has become an indispensable branch of modern science research, experiencing an explosion in financial support, developed applications and data collection. The growth of the datasets that are emerging from research laboratories, industry, the health sector, etc., are increasingly raising the levels of demand in computing power and storage. Processing biological data, in the large scales of these datasets, often requires the use of High Performance Computing (HPC) resources, especially when dealing with certain types of omics data, such as genomic and metagenomic data. Such computational resources not only require substantial investments, but they also involve high maintenance costs. More importantly, in order to keep good returns from the investments, specific training needs to be put in place to ensure that wasting is minimized. Furthermore, given that bioinformatics is a highly interdisciplinary field where several other domains intersect (such as biology, chemistry, physics and computer science), researchers from these areas also require bioinformatics-specific training in HPC, in order to fully take advantage of supercomputing centers. In this document, we describe our experience in training researchers from several different disciplines in HPC, as applied to bioinformatics under the framework of the leading European bioinformatics platform ELIXIR, and analyze both the content and outcomes of the course.

Deploying CPU-Intensive Applications on MEC in NFV Systems: The Immersive Video Use Case

Multi-access Edge Computing (MEC) will be a technology pillar of forthcoming 5G networks. Nonetheless, there is a great interest in also deploying MEC solutions in current 4G infrastructures. MEC enables data processing in proximity to end users. Thus, latency can be minimized, high data rates locally achieved, and real-time information about radio link status or consumer geographical position exploited to develop high-value services. To consolidate network elements and edge applications on the same virtualization infrastructure, network operators aim to combine MEC with Network Function Virtualization (NFV). However, MEC in NFV integration is not fully established yet: in fact, various architectural issues are currently open, even at standardization level. This paper describes a novel MEC in an NFV system which successfully combines, at management level, MEC functional blocks with an NFV Orchestrator, and can neutrally support any “over the top” Mobile Edge application with minimal integration effort. A specific ME app combined with an end-user app for the provision of immersive video services is presented. To provide low latency, CPU-intensive services to end users, the proposed architecture exploits High-Performance Computing resources embedded in the edge infrastructure. Experimental results showing the effectiveness of the proposed architecture are reported and discussed.

Determining permeability tensors of porous media: A novel ‘vector kinetic’ numerical approach

New structurally tailored porous materials are nowadays used in many engineering applications due to several attractive properties. Knowledge of pressure losses or flow structures in such complex media and their relationship with geometrical parameters are thus critical for various applications. Precise determination of local flow behavior and macroscale properties of natural media (soils, biomass....) are increasingly needed. It is therefore important to simulate the complex and unsteady flows by reliable numerical methods and to determine intrinsic macroscopic hydraulic properties on porous structures. The availability of low cost, easy-to use high-performance computational resources lead to generalization of pore scale numerical simulations in various fields. The recent development of innovative scheme like Lattice Boltzmann (LBM) to overcome the classical drawback of commercial softwares (Finite Volume, Finite Element) in achieving high accuracy, shows the potential of kinetic based methods for producing efficient and accurate solvers. An alternative vector kinetic method is proposed to solve incompressible Navier–Stokes equations at pore scale and eventually determine permeability tensors of complex porous media. A moment based (vs discrete velocities), non-diffusive, explicit, parallel implementation was implemented and successfully used on several totally different complex geometries. Excellent results at low Reynolds number (Re) were obtained, the method is thus well suited for permeability tensors determination of complex heterogeneous media. The code was validated against classical benchmark as well as experimental and numerical permeability data obtained on different porous media of variable porosity namely (i) foams (virtual model structures and real samples), (ii) sandstone and (iii) wood.

Deep Convolutional Neural Networks for Automated Characterization of Arctic Ice-Wedge Polygons in Very High Spatial Resolution Aerial Imagery

The microtopography associated with ice-wedge polygons governs many aspects of Arctic ecosystem, permafrost, and hydrologic dynamics from local to regional scales owing to the linkages between microtopography and the flow and storage of water, vegetation succession, and permafrost dynamics. Wide-spread ice-wedge degradation is transforming low-centered polygons into high-centered polygons at an alarming rate. Accurate data on spatial distribution of ice-wedge polygons at a pan-Arctic scale are not yet available, despite the availability of sub-meter-scale remote sensing imagery. This is because the necessary spatial detail quickly produces data volumes that hamper both manual and semi-automated mapping approaches across large geographical extents. Accordingly, transforming big imagery into ‘science-ready’ insightful analytics demands novel image-to-assessment pipelines that are fueled by advanced machine learning techniques and high-performance computational resources. In this exploratory study, we tasked a deep-learning driven object instance segmentation method (i.e., the Mask R-CNN) with delineating and classifying ice-wedge polygons in very high spatial resolution aerial orthoimagery. We conducted a systematic experiment to gauge the performances and interoperability of the Mask R-CNN across spatial resolutions (0.15 m to 1 m) and image scene contents (a total of 134 km2) near Nuiqsut, Northern Alaska. The trained Mask R-CNN reported mean average precisions of 0.70 and 0.60 at thresholds of 0.50 and 0.75, respectively. Manual validations showed that approximately 95% of individual ice-wedge polygons were correctly delineated and classified, with an overall classification accuracy of 79%. Our findings show that the Mask R-CNN is a robust method to automatically identify ice-wedge polygons from fine-resolution optical imagery. Overall, this automated imagery-enabled intense mapping approach can provide a foundational framework that may propel future pan-Arctic studies of permafrost thaw, tundra landscape evolution, and the role of high latitudes in the global climate system.

A survey of simulation provenance systems: modeling, capturing, querying, visualization, and advanced utilization

Research and education through computer simulation has been actively conducted in various scientific and engineering fields including computational science engineering. Accordingly, there have been a lot of attentions paid to actively utilize provenance information regarding such computer simulations, particularly conducted on high-performance computing and storage resources. In this manuscript we provide a comprehensive survey of a wide range of existing systems to utilize provenance data produced by simulation. Specifically, we (1) categorize extant provenance research articles into several major themes along with well-motivated criteria, (2) grasp and compare primary functions/features of the existing systems in each category, and (3) then ultimately propose new research directions that have never been pioneered before. In particular, we present a taxonomy of scientific platforms regarding provenance support and holistically tabulate the major functionalities and supporting levels of the studied systems. Finally, we conclude this article with a summary of our contributions.

Strategies of data layout and cache writing for input-output optimization in high performance scientific computing: Applications to the forward electrocardiographic problem.

Input-output (I/O) optimization at the low-level design of data layout on disk drastically impacts the efficiency of high performance computing (HPC) applications. However, such a low-level optimization is in general challenging, especially when using popular scientific file formats designed with an emphasis on portability and flexibility. To reconcile these two aspects, we present a novel low-level data layout for HPC applications, fully independent of the number of dimensions in the dataset. The new data layout improves reading and writing efficiency in large HPC applications using many processors, and in particular during parallel post-processing. Furthermore, its combination with a cached write mode, in order to aggregate multiple writes into larger ones, substantially decreased the writing times of the proposed strategy. When applied to our simulation framework for the forward calculation of the human electrocardiogram, the combined strategy resulted in drastic improvements in I/O performance, of up to 40% in writing and 93–98% in reading for post-processing tasks. Given the generality of the proposed strategies and scientific file formats used, our results may represent significant improvements in I/O performance of HPC applications across multiple disciplines, reducing execution and post-processing times and leading to a more efficient use of HPC resource envelopes.

Building a Science Gateway For Processing and Modeling Sequencing Data Via Apache Airavata.

The amount of DNA sequencing data has been exponentially growing during the past decade due to advances in sequencing technology. Processing and modeling large amounts of sequencing data can be computationally intractable for desktop computing platforms. High performance computing (HPC) resources offer advantages in terms of computing power, and can be a general solution to these problems. Using HPCs directly for computational needs requires skilled users who know their way around HPCs and acquiring such skills take time. Science gateways acts as the middle layer between users and HPCs, providing users with the resources to accomplish compute-intensive tasks without requiring specialized expertise. We developed a web-based computing platform for genome biologists by customizing the PHP Gateway for Airavata (PGA) framework that accesses publicly accessible HPC resources via Apache Airavata. This web computing platform takes advantage of the Extreme Science and Engineering Discovery Environment (XSEDE) which provides the resources for gateway development, including access to CPU, GPU, and storage resources. We used this platform to develop a gateway for the dREG algorithm, an online computing tool for finding functional regions in mammalian genomes using nascent RNA sequencing data. The dREG gateway provides its users a free, powerful and user-friendly GPU computing resource based on XSEDE, circumventing the need of specialized knowledge about installation, configuration, and execution on an HPC for biologists. The dREG gateway is available at: https://dREG.dnasequence.org/.

A run control framework to streamline profiling, porting, and tuning simulation runs and provenance tracking of geoscientific applications

Abstract. Geoscientific modeling is constantly evolving, with next-generation geoscientific models and applications placing large demands on high-performance computing (HPC) resources. These demands are being met by new developments in HPC architectures, software libraries, and infrastructures. In addition to the challenge of new massively parallel HPC systems, reproducibility of simulation and analysis results is of great concern. This is due to the fact that next-generation geoscientific models are based on complex model implementations and profiling, modeling, and data processing workflows. Thus, in order to reduce both the duration and the cost of code migration, aid in the development of new models or model components, while ensuring reproducibility and sustainability over the complete data life cycle, an automated approach to profiling, porting, and provenance tracking is necessary. We propose a run control framework (RCF) integrated with a workflow engine as a best practice approach to automate profiling, porting, provenance tracking, and simulation runs. Our RCF encompasses all stages of the modeling chain: (1) preprocess input, (2) compilation of code (including code instrumentation with performance analysis tools), (3) simulation run, and (4) postprocessing and analysis, to address these issues. Within this RCF, the workflow engine is used to create and manage benchmark or simulation parameter combinations and performs the documentation and data organization for reproducibility. In this study, we outline this approach and highlight the subsequent developments scheduled for implementation born out of the extensive profiling of ParFlow. We show that in using our run control framework, testing, benchmarking, profiling, and running models is less time consuming and more robust than running geoscientific applications in an ad hoc fashion, resulting in more efficient use of HPC resources, more strategic code development, and enhanced data integrity and reproducibility.

Computer-Aided Drug Design in Epigenetics.

Epigenetic dysfunction has been widely implicated in several diseases especially cancers thus highlights the therapeutic potential for chemical interventions in this field. With rapid development of computational methodologies and high-performance computational resources, computer-aided drug design has emerged as a promising strategy to speed up epigenetic drug discovery. Herein, we make a brief overview of major computational methods reported in the literature including druggability prediction, virtual screening, homology modeling, scaffold hopping, pharmacophore modeling, molecular dynamics simulations, quantum chemistry calculation, and 3D quantitative structure activity relationship that have been successfully applied in the design and discovery of epi-drugs and epi-probes. Finally, we discuss about major limitations of current virtual drug design strategies in epigenetics drug discovery and future directions in this field.

On the achievement of high fidelity and scalability for large‐scale diagonalizations in grid‐based DFT simulations

AbstractRecent advance in high performance computing (HPC) resources has opened the possibility to expand the scope of density functional theory (DFT) simulations toward large and complex molecular systems. This work proposes a numerically robust method that enables scalable diagonalizations of large DFT Hamiltonian matrices, particularly with thousands of computing CPUs (cores) that are usual these days in terms of sizes of HPC resources. The well‐known Lanczos method is extensively refactorized to overcome its weakness for evaluation of multiple degenerate eigenpairs that is the substance of DFT simulations, where a multilevel parallelization is adopted for scalable simulations in as many cores as possible. With solid benchmark tests for realistic molecular systems, the fidelity of our method are validated against the locally optimal block preconditioned conjugated gradient (LOBPCG) method that is widely used to simulate electronic structures. Our method may waste computing resources for simulations of molecules whose degeneracy cannot be reasonably estimated. But, compared to LOBPCG method, it is fairly excellent in perspectives of both speed and scalability, and particularly has remarkably less (< 10%) sensitivity of performance to the random nature of initial basis vectors. As a promising candidate for solving electronic structures of highly degenerate systems, the proposed method can make a meaningful contribution to migrating DFT simulations toward extremely large computing environments that normally have more than several tens of thousands of computing cores.

Platform for Automated Real-Time High Performance Analytics on Medical Image Data.

Biomedical data are quickly growing in volume and in variety, providing clinicians an opportunity for better clinical decision support. Here, we demonstrate a robust platform that uses software automation and high performance computing (HPC) resources to achieve real-time analytics of clinical data, specifically magnetic resonance imaging (MRI) data. We used the Agave application programming interface to facilitate communication, data transfer, and job control between an MRI scanner and an off-site HPC resource. In this use case, Agave executed the graphical pipeline tool GRAphical Pipeline Environment (GRAPE) to perform automated, real-time, quantitative analysis of MRI scans. Same-session image processing will open the door for adaptive scanning and real-time quality control, potentially accelerating the discovery of pathologies and minimizing patient callbacks. We envision this platform can be adapted to other medical instruments, HPC resources, and analytics tools.

3D streamers simulation in a pin to plane configuration using massively parallel computing

This paper concerns the 3D simulation of corona discharge using high performance computing (HPC) managed with the message passing interface (MPI) library. In the field of finite volume methods applied on non-adaptive mesh grids and in the case of a specific 3D dynamic benchmark test devoted to streamer studies, the great efficiency of the iterative R&B SOR and BiCGSTAB methods versus the direct MUMPS method was clearly demonstrated in solving the Poisson equation using HPC resources. The optimization of the parallelization and the resulting scalability was undertaken as a function of the HPC architecture for a number of mesh cells ranging from 8 to 512 million and a number of cores ranging from 20 to 1600. The R&B SOR method remains at least about four times faster than the BiCGSTAB method and requires significantly less memory for all tested situations. The R&B SOR method was then implemented in a 3D MPI parallelized code that solves the classical first order model of an atmospheric pressure corona discharge in air. The 3D code capabilities were tested by following the development of one, two and four coplanar streamers generated by initial plasma spots for 6 ns. The preliminary results obtained allowed us to follow in detail the formation of the tree structure of a corona discharge and the effects of the mutual interactions between the streamers in terms of streamer velocity, trajectory and diameter. The computing time for 64 million of mesh cells distributed over 1000 cores using the MPI procedures is about 30 min ns−1, regardless of the number of streamers.

VIDIA: A HUBzero gateway for data analytics education

We describe a scientific gateway collaboration undertaken by members of the State University of New York (SUNY) system. The University at Buffalo’s Center for Computational Research (CCR) partnered with SUNY College at Oneonta to offer a gateway for teaching data analytics. The result, called Virtual Infrastructure for Data Intensive Analysis (VIDIA), hosts open-source tools that have been used by more than 300 students enrolled in 18 SUNY courses. Additional tools enable researchers across the SUNY system to submit larger jobs to CCR’s compute cluster; altogether, 5 SUNY campuses benefit from the resource to date. VIDIA supports data-intensive computation for teaching and research at campuses that lack access to traditional high-performance computing (HPC) resources.

Network slicing to improve multicasting in HPC clusters

In high performance computing (HPC) resources’ extensive experiments are frequently executed. HPC resources (e.g. computing machines and switches) should be able to handle running several experiments in parallel. Typically HPC utilizes parallelization in programs, processing and data. The underlying network is seen as the only non-parallelized HPC component (i.e. no dynamic virtual slicing based on HPC jobs). In this scope we present an approach in this paper to utilize software defined networking (SDN) to parallelize HPC clusters among the different running experiments. We propose to accomplish this through two major components: A passive module (network mapper/remapper) to select for each experiment as soon as it starts the least busy resources in the network, and an SDN-HPC active load balancer to perform more complex and intelligent operations. Active load balancer can logically divide the network based on experiments’ host files. The goal is to reduce traffic to unnecessary hosts or ports. An HPC experiment should multicast, rather than broadcast to only cluster nodes that are used by the experiment. We use virtual tenant network modules in Opendaylight controller to create VLANs based on HPC experiments. In each HPC host, virtual interfaces are created to isolate traffic from the different experiments. The traffic between the different physical hosts that belong to the same experiment can be distinguished based on the VLAN ID assigned to each experiment. We evaluate the new approach using several HPC public benchmarks. Results show a significant enhancement in experiments’ performance especially when HPC cluster experiences running several heavy load experiments simultaneously. Results show also that this multi-casting approach can significantly reduce casting overhead that is caused by using a single cast for all resources in the HPC cluster. In comparison with InfiniBand networks that offer interconnect services with low latency and high bandwidth, HPC services based on SDN can provide two distinguished objectives that may not be possible with InfiniBand: The first objective is the integration of HPC with Ethernet enterprise networks and hence expanding HPC usage to much wider domains. The second objective is the ability to enable users and their applications to customize HPC services with different QoS requirements that fit the different needs of those applications and optimize the usage of HPC clusters.

Multilevel mesh workflows towards CONUS scale watersheds: How small should triangles be to capture stream curvature for hydrological modeling?

Generating quality meshes for hydrological modeling is challenging. This article demonstrates using mesh workflows to incorporate national stream networks into very large dynamic meshes for distributed High Performance Computing (HPC). A multilevel quadtree is used to partition watersheds and merge stream networks ranging from hill-slope to four level-4 Hydrological Unit Code (HUC) scales, generating mesh sizes from hundreds to tens of millions of triangles. By using a mesh workflow, it is demonstrated how users control mesh quality, including triangle sizes, and the removal of small triangles and slivers. Four watersheds are studied, ranging in scale from Shale Hills (hill-slope) to the Chesapeake Bay (177,968 sq. km.) and are used to compare mesh characteristics with user parameters and different elevation sources to generate a spectrum of multilevel quadtree meshes. By using triangles to represent land surface and stream networks, stream curvature is lost and the mesh workflow is used to demonstrate what parameters are important to maintain stream characteristics. To meet HPC constraints, the multilevel quadtree is demonstrated by picking quadtree tiles along a storm path to show how users can combine quadtree tiles at multiple levels to concentrate HPC resources for hydrological science research.

Desktop supercomputer: what can it do?

The paper addresses the issues of solving complex problems that require using supercomputers or multiprocessor clusters available for most researchers nowadays. Efficient distribution of high performance computing resources according to actual application needs has been a major research topic since high-performance computing (HPC) technologies became widely introduced. At the same time, comfortable and transparent access to these resources was a key user requirement. In this paper we discuss approaches to build a virtual private supercomputer available at user’s desktop: a virtual computing environment tailored specifically for a target user with a particular target application. We describe and evaluate possibilities to create the virtual supercomputer based on light-weight virtualization technologies, and analyze the efficiency of our approach compared to traditional methods of HPC resource management.

CMS use of allocation based HPC resources

The higher energy and luminosity from the LHC in Run2 has put increased pressure on CMS computing resources. Extrapolating to even higher luminosities (and thus higher event complexities and trigger rates) in Run3 and beyond, it becomes clear the current model of CMS computing alone will not scale accordingly. High Performance Computing (HPC) facilities, widely used in scientific computing outside of HEP, present a (at least so far) largely untapped computing resource for CMS. Even just being able use a small fraction of HPC facilities computing resources could significantly increase the overall computing available to CMS. Here we describe the CMS strategy for integrating HPC resources into CMS computing, the unique challenges these facilities present, and how we plan on overcoming these challenges. We also present the current status of ongoing CMS efforts at HPC sites such as NERSC (Cori cluster), SDSC (Comet cluster) and TACC (Stampede cluster).

Mapping land cover change over continental Africa using Landsat and Google Earth Engine cloud computing.

Quantifying and monitoring the spatial and temporal dynamics of the global land cover is critical for better understanding many of the Earth’s land surface processes. However, the lack of regularly updated, continental-scale, and high spatial resolution (30 m) land cover data limit our ability to better understand the spatial extent and the temporal dynamics of land surface changes. Despite the free availability of high spatial resolution Landsat satellite data, continental-scale land cover mapping using high resolution Landsat satellite data was not feasible until now due to the need for high-performance computing to store, process, and analyze this large volume of high resolution satellite data. In this study, we present an approach to quantify continental land cover and impervious surface changes over a long period of time (15 years) using high resolution Landsat satellite observations and Google Earth Engine cloud computing platform. The approach applied here to overcome the computational challenges of handling big earth observation data by using cloud computing can help scientists and practitioners who lack high-performance computational resources.

Quantifying airborne dispersal routes of pathogens over continents to safeguard global wheat supply.

Infectious crop diseases spreading over large agricultural areas pose a threat to food security. Aggressive strains of the obligate pathogenic fungus Puccinia graminis f.sp. tritici (Pgt), causing the crop disease wheat stem rust, have been detected in East Africa and the Middle East, where they lead to substantial economic losses and threaten livelihoods of farmers. The majority of commercially grown wheat cultivars worldwide are susceptible to these emerging strains, which pose a risk to global wheat production, because the fungal spores transmitting the disease can be wind-dispersed over regions and even continents 1-11 . Targeted surveillance and control requires knowledge about airborne dispersal of pathogens, but the complex nature of long-distance dispersal poses significant challenges for quantitative research 12-14 . We combine international field surveys, global meteorological data, a Lagrangian dispersion model and high-performance computational resources to simulate a set of disease outbreak scenarios, tracing billions of stochastic trajectories of fungal spores over dynamically changing host and environmental landscapes for more than a decade. This provides the first quantitative assessment of spore transmission frequencies and amounts amongst all wheat producing countries in Southern/East Africa, the Middle East and Central/South Asia. We identify zones of high air-borne connectivity that geographically correspond with previously postulated wheat rust epidemiological zones (characterized by endemic disease and free movement of inoculum) 10,15 , and regions with genetic similarities in related pathogen populations 16,17 . We quantify the circumstances (routes, timing, outbreak sizes) under which virulent pathogen strains such as 'Ug99' 5,6 pose a threat from long-distance dispersal out of East Africa to the large wheat producing areas in Pakistan and India. Long-term mean spore dispersal trends (predominant direction, frequencies, amounts) are summarized for all countries in the domain (Supplementary Data). Our mechanistic modelling framework can be applied to other geographic areas, adapted for other pathogens and used to provide risk assessments in real-time 3 .