High Performance Computing Infrastructure Research Articles

HPC-T-Annotator: an HPC tool for de novo transcriptome assembly annotation

BackgroundThe availability of transcriptomic data for species without a reference genome enables the construction of de novo transcriptome assemblies as alternative reference resources from RNA-Seq data. A transcriptome provides direct information about a species’ protein-coding genes under specific experimental conditions. The de novo assembly process produces a unigenes file in FASTA format, subsequently targeted for the annotation. Homology-based annotation, a method to infer the function of sequences by estimating similarity with other sequences in a reference database, is a computationally demanding procedure.ResultsTo mitigate the computational burden, we introduce HPC-T-Annotator, a tool for de novo transcriptome homology annotation on high performance computing (HPC) infrastructures, designed for straightforward configuration via a Web interface. Once the configuration data are given, the entire parallel computing software for annotation is automatically generated and can be launched on a supercomputer using a simple command line. The output data can then be easily viewed using post-processing utilities in the form of Python notebooks integrated in the proposed software.ConclusionsHPC-T-Annotator expedites homology-based annotation in de novo transcriptome assemblies. Its efficient parallelization strategy on HPC infrastructures significantly reduces computational load and execution times, enabling large-scale transcriptome analysis and comparison projects, while its intuitive graphical interface extends accessibility to users without IT skills.

Containerized deep learning in agriculture: Orchestrating GoogleNet with Kubernetes on high performance computing

SummarySmart Farming has become a cornerstone of modern agriculture, offering data‐driven insights and automation that optimize resource utilization and increase crop yields. The use of cutting‐edge technologies in agriculture has given rise to Smart Farming, which has transformed traditional farming practices into efficient, data‐driven operations. This paper explores the synergy between high‐performance computing (HPC) systems, Kubernetes orchestration, GoogleNet architecture, and containerization to redefine the future of farming. At the heart of this transformation lies the GoogleNet architecture, a deep learning powerhouse recognized for its efficiency and accuracy in image recognition tasks. The orchestration capabilities of Kubernetes, a versatile tool for managing containerized workloads efficiently on HPC clusters. Hence, in this work, we investigate the intricacies of deploying GoogleNet‐based deep learning models within containerized environments orchestrated by Kubernetes on HPC infrastructure. It explores resource optimization, scalability, security, and adaptability, all tailored to the unique demands of the agricultural domain to evaluate the effectiveness of the given technique it is compared with the existing techniques namely Hermes, Horus, CYBELE, and RZ‐SHAN. The attained ranges of proposed method of various measures of accuracy, precision, recall, and F1‐score are 98.65%, 97.45%, 97.87%, and 98.12% for the Pilot Wheat Ear dataset. Also, the processing time for the proposed approach is 181.50 and 120.2 m for the Pilot Wheat Ear Dataset and the Pilot soya bean farming dataset. The latency of the proposed approach attains a lower value of 1.5 and 1.1 s pilot soya bean farming dataset and Pilot Wheat Ear dataset. The experimental outcome demonstrates the efficiency of the proposed approaches to improve Smart Farming agriculture.

EpiDiP/NanoDiP: a versatile unsupervised machine learning edge computing platform for epigenomic tumour diagnostics

DNA methylation analysis based on supervised machine learning algorithms with static reference data, allowing diagnostic tumour typing with unprecedented precision, has quickly become a new standard of care. Whereas genome-wide diagnostic methylation profiling is mostly performed on microarrays, an increasing number of institutions additionally employ nanopore sequencing as a faster alternative. In addition, methylation-specific parallel sequencing can generate methylation and genomic copy number data. Given these diverse approaches to methylation profiling, to date, there is no single tool that allows (1) classification and interpretation of microarray, nanopore and parallel sequencing data, (2) direct control of nanopore sequencers, and (3) the integration of microarray-based methylation reference data. Furthermore, no software capable of entirely running in routine diagnostic laboratory environments lacking high-performance computing and network infrastructure exists. To overcome these shortcomings, we present EpiDiP/NanoDiP as an open-source DNA methylation and copy number profiling suite, which has been benchmarked against an established supervised machine learning approach using in-house routine diagnostics data obtained between 2019 and 2021. Running locally on portable, cost- and energy-saving system-on-chip as well as gpGPU-augmented edge computing devices, NanoDiP works in offline mode, ensuring data privacy. It does not require the rigid training data annotation of supervised approaches. Furthermore, NanoDiP is the core of our public, free-of-charge EpiDiP web service which enables comparative methylation data analysis against an extensive reference data collection. We envision this versatile platform as a useful resource not only for neuropathologists and surgical pathologists but also for the tumour epigenetics research community. In daily diagnostic routine, analysis of native, unfixed biopsies by NanoDiP delivers molecular tumour classification in an intraoperative time frame.

Replay with Feedback: How does the performance of HPC system impact user submission behavior?

High Performance Computing (HPC) is a key infrastructure to solve large scale scientific problems, from weather to quantum simulations. Scheduling jobs in HPC infrastructures is complex due to their scale, the different behaviors of their users, and the multiple objectives, from performance to ecological impact. Schedulers are evaluated on data center simulations, due to the complexity and cost of evaluating them in-situ. One key element for this evaluation is the behavioral model of users. Most studies are limited to replaying past workload of existing data centers. This reduces the realism of performance evaluation in cases where the scheduler and the hardware infrastructure are not exactly the same. Any such change would potentially impact the behavior of the users.In this article we introduce a novel model “Replay with Feedback” accounting for the impact of HPC system performances on user submission behavior in simulations. Instead of keeping the original timestamps of job submissions, we exhibit and use the relationships between each user jobs. We propose an open-source implementation of this model along with an extensive and reproducible set of experiments to assess the impact of the scheduler and infrastructure changes. We also provide new metrics adapted to the flexibility of user submission behaviors. Results show that using this model, we advance towards more realistic simulations of schedulers in HPC systems.

Distributed Multi-GPU Ab Initio Density Matrix Renormalization Group Algorithm with Applications to the P-Cluster of Nitrogenase.

The presence of many degenerate d/f orbitals makes polynuclear transition-metal compounds, such as iron-sulfur clusters in nitrogenase, challenging for state-of-the-art quantum chemistry methods. To address this challenge, we present the first distributed multi-graphics processing unit (GPU) ab initio density matrix renormalization group (DMRG) algorithm suitable for modern high-performance computing (HPC) infrastructures. The central idea is to parallelize the most computationally intensive part─the multiplication of O(K2) operators with a trial wave function, where K is the number of spatial orbitals, by combining operator parallelism for distributing the workload with a batched algorithm for performing contractions on GPU. With this new implementation, we are able to reach an unprecedentedly large bond dimension D = 14,000 on 48 GPUs (NVIDIA A100 80 GB SXM) for an active space model (114 electrons in 73 active orbitals) of the P-cluster, which is nearly 3 times larger than the bond dimensions reported in previous DMRG calculations for the same system using only central processing units (CPUs).

Performing highly parallelized and reproducible GWAS analysis on biobank-scale data.

Genome-wide association studies (GWAS) are transforming genetic research and enable the detection of novel genotype-phenotype relationships. In the last two decades, over 60 000 genetic associations across thousands of traits have been discovered using a GWAS approach. Due to increasing sample sizes, researchers are increasingly faced with computational challenges. A reproducible, modular and extensible pipeline with a focus on parallelization is essential to simplify data analysis and to allow researchers to devote their time to other essential tasks. Here we present nf-gwas, a Nextflow pipeline to run biobank-scale GWAS analysis. The pipeline automatically performs numerous pre- and post-processing steps, integrates regression modeling from the REGENIE package and supports single-variant, gene-based and interaction testing. It includes an extensive reporting functionality that allows to inspect thousands of phenotypes and navigate interactive Manhattan plots directly in the web browser. The pipeline is tested using the unit-style testing framework nf-test, a crucial requirement in clinical and pharmaceutical settings. Furthermore, we validated the pipeline against published GWAS datasets and benchmarked the pipeline on high-performance computing and cloud infrastructures to provide cost estimations to end users. nf-gwas is a highly parallelized, scalable and well-tested Nextflow pipeline to perform GWAS analysis in a reproducible manner.

Design and implementation of a scalable high-performance computing (HPC) cluster for omics data analysis: achievements, challenges and recommendations in LMICs.

The advent of high-throughput technologies, including cutting-edge sequencing devices, has revolutionized biomedical data generation and processing. Nevertheless, big data applications require novel hardware and software for parallel computing and management to handle the ever-growing data size and analysis complexity. On-premise, high-performance computing (HPC) is increasingly used in biomedical research for big data stewardship. In this work, we present Tunisia's first high-performance computational infrastructure for omics research. We highlight measurements and recommendations that may help institutions in other low- and middle-income countries that are eager to implement local HPC in facilities for bioinformatics research and omics data analyses.

Probing Reactivity with External Forces: The Case of Nitroacetamides in Water.

Many computational methods have been applied to interpret and predict changes in reactivity by slight modifications of a given molecular scaffold. We describe a novel and simple method based on approximate density-functional theory of valence electrons that can be applied within a large high-performance computational infrastructure to probe such changes using a statistical sample of molecular configurations, including the solvent. All the used computational tools are fully open-source. Following our previous application, we are able to explain the high acidity of C-H bond at α position in nitro compounds when the amide linkage an ammonium group is inserted into the α substituent.

Health data governance for research use in Alberta.

Alberta has rich clinical and health services data held under the custodianship of Alberta Health and Alberta Health Services (AHS), which is not only used for clinical and administrative purposes but also disease surveillance and epidemiological research. Alberta is the largest province in Canada with a single payer centralised health system, AHS, and a consolidated data and analytics team supporting researchers across the province. This paper describes Alberta's data custodians, data governance mechanisms, and streamlined processes followed for research data access. AHS has created a centralised data repository from multiple sources, including practitioner claims data, hospital discharge data, and medications dispensed, available for research use through the provincial Data and Research Services (DRS) team. The DRS team is integrated within AHS to support researchers across the province with their data extraction and linkage requests. Furthermore, streamlined processes have been established, including: 1) ethics approval from a research ethics board, 2) any necessary operational approvals from AHS, and 3) a tripartite legal agreement dictating terms and conditions for data use, disclosure, and retention. This allows researchers to gain timely access to data. To meet the evolving and ever-expanding big-data needs, the University of Calgary, in partnership with AHS, has built high-performance computing (HPC) infrastructure to facilitate storage and processing of large datasets. When releasing data to researchers, the analytics team ensures that Alberta's Health Information Act's guiding principles are followed. The principal investigator also ensures data retention and disposition are according to the plan specified in ethics and per the terms set out by funding agencies. Even though there are disparities and variations in the data protection laws across the different provinces in Canada, the streamlined processes for research data access in Alberta are highly efficient.

Unsupervised Mixture Models on the Edge for Smart Energy Consumption Segmentation with Feature Saliency.

Smart meter datasets have recently transitioned from monthly intervals to one-second granularity, yielding invaluable insights for diverse metering functions. Clustering analysis, a fundamental data mining technique, is extensively applied to discern unique energy consumption patterns. However, the advent of high-resolution smart meter data brings forth formidable challenges, including non-Gaussian data distributions, unknown cluster counts, and varying feature importance within high-dimensional spaces. This article introduces an innovative learning framework integrating the expectation-maximization algorithm with the minimum message length criterion. This unified approach enables concurrent feature and model selection, finely tuned for the proposed bounded asymmetric generalized Gaussian mixture model with feature saliency. Our experiments aim to replicate an efficient smart meter data analysis scenario by incorporating three distinct feature extraction methods. We rigorously validate the clustering efficacy of our proposed algorithm against several state-of-the-art approaches, employing diverse performance metrics across synthetic and real smart meter datasets. The clusters that we identify effectively highlight variations in residential energy consumption, furnishing utility companies with actionable insights for targeted demand reduction efforts. Moreover, we demonstrate our method's robustness and real-world applicability by harnessing Concordia's High-Performance Computing infrastructure. This facilitates efficient energy pattern characterization, particularly within smart meter environments involving edge cloud computing. Finally, we emphasize that our proposed mixture model outperforms three other models in this paper's comparative study. We achieve superior performance compared to the non-bounded variant of the proposed mixture model by an average percentage improvement of 7.828%.

A Survey of High Performance Computing (HPC) Infrastructure in Thailand

In the last decade, government organizations and private companies in Thailand have invested considerably in computing resources. Research collaborations typically band together based on shared interests and propose projects to compete for funding. Thus, many institutions frequently use similar computing resources. A bird's eye view of high-performance computer infrastructure in Thailand is important in many ways. To the best of our knowledge and ability, we gathered information on government procurements of HPC resources in the past five years, which cover several organizations and ministries. We list the system specifications and tabulate the target application areas for each system. The aggregated number of cores and storage space for HPC in Thailand, commissioned during the past five years, is 54,838 cores and 21 PB, respectively. We also describe the large data transfers using UniNet for HPC applications. The survey results can be used by academics and decision-makers to build research agendas and national development strategies.

Edge computing at sea: high-throughput classification of in-situ plankton imagery for adaptive sampling

The small sizes of most marine plankton necessitate that plankton sampling occur on fine spatial scales, yet our questions often span large spatial areas. Underwater imaging can provide a solution to this sampling conundrum but collects large quantities of data that require an automated approach to image analysis. Machine learning for plankton classification, and high-performance computing (HPC) infrastructure, are critical to rapid image processing; however, these assets, especially HPC infrastructure, are only available post-cruise leading to an ‘after-the-fact’ view of plankton community structure. To be responsive to the often-ephemeral nature of oceanographic features and species assemblages in highly dynamic current systems, real-time data are key for adaptive oceanographic sampling. Here we used the new In-situ Ichthyoplankton Imaging System-3 (ISIIS-3) in the Northern California Current (NCC) in conjunction with an edge server to classify imaged plankton in real-time into 170 classes. This capability together with data visualization in a heavy.ai dashboard makes adaptive real-time decision-making and sampling at sea possible. Dual ISIIS-Deep-focus Particle Imager (DPI) cameras sample 180 L s-1, leading to >10 GB of video per min. Imaged organisms are in the size range of 250 µm to 15 cm and include abundant crustaceans, fragile taxa (e.g., hydromedusae, salps), faster swimmers (e.g., krill), and rarer taxa (e.g., larval fishes). A deep learning pipeline deployed on the edge server used multithreaded CPU-based segmentation and GPU-based classification to process the imagery. AVI videos contain 50 sec of data and can contain between 23,000 - 225,000 particle and plankton segments. Processing one AVI through segmentation and classification takes on average 3.75 mins, depending on biological productivity. A heavyDB database monitors for newly processed data and is linked to a heavy.ai dashboard for interactive data visualization. We describe several examples where imaging, AI, and data visualization enable adaptive sampling that can have a transformative effect on oceanography. We envision AI-enabled adaptive sampling to have a high impact on our ability to resolve biological responses to important oceanographic features in the NCC, such as oxygen minimum zones, or harmful algal bloom thin layers, which affect the health of the ecosystem, fisheries, and local communities.

Emerging Trends of Computational Chemistry and Molecular Modeling in Froth Flotation: A Review.

Froth flotation is the most versatile process in mineral beneficiation, extensively used to concentrate a wide range of minerals. This process comprises mixtures of more or less liberated minerals, water, air, and various chemical reagents, involving a series of intermingled multiphase physical and chemical phenomena in the aqueous environment. Today's main challenge facing the froth flotation process is to gain atomic-level insights into the properties of its inherent phenomena governing the process performance. While it is often challenging to determine these phenomena via trial-and-error experimentations, molecular modeling approaches not only elicit a deeper understanding of froth flotation but can also assist experimental studies in saving time and budget. Thanks to the rapid development of computer science and advances in high-performance computing (HPC) infrastructures, theoretical/computational chemistry has now matured enough to successfully and gainfully apply to tackle the challenges of complex systems. In mineral processing, however, advanced applications of computational chemistry are increasingly gaining ground and demonstrating merit in addressing these challenges. Accordingly, this contribution aims to encourage mineral scientists, especially those interested in rational reagent design, to become familiarized with the necessary concepts of molecular modeling and to apply similar strategies when studying and tailoring properties at the molecular level. This review also strives to deliver the state-of-the-art integration and application of molecular modeling in froth flotation studies to assist either active researchers in this field to disclose new directions for future research or newcomers to the field to initiate innovative works.

A Sustainability Model for Continental Hybridization of High Performance Computing (HPC) in Africa

The potential for High-performance computing (HPC) opportunities in the areas of health, environment, energy, climate change, business and livelihood are immense in Africa. However, the lack of sufficient HPC facilities and capacity skills deficiencies continue to curtail its growth. This paper maps the status of HPC facilities in Africa. Thereafter based on a review of various business and funding models, it suggests a hybridization model as a way for building sustainable HPC infrastructures across the continent to provide support for teaching and learning, research and education. The main contribution of this paper is the anchoring of hybridization as a sustainable business model for improving the access and establishment of HPC infrastructure in Africa.

MYH3-associated non-syndromic palatoschisis (cleft palate, CP) in Limousine cattle.

Palatoschisis (also called cleft palate, CP) is an opening or cleft in the upper lip, the roof of the palate, or both, and is a common defect in both humans and domestic animals (Mossey & Modell, 2012; Mulvihill et al., 1980). It can also occur in cattle as an isolated (non-syndromic) birth defect (OMIA 000197-9913), owing to de novo mutations (Jacinto et al., 2021; OMIA 002483-9913) or associated with familial recessive syndromes (Agerholm et al., 2016; OMIA 000070-9913). A 2-years-old Italian-bred Limousine heifer was referred for a non-syndromic form of palatoschisis involving both soft and hard palate, rhinitis and retarded growth (Figure S1). We hypothesized a genetic etiology for this congenital anomaly and performed whole-genome sequencing. Further details of the methodology are described in Appendix S1. Two homozygous private protein-changing variants were located in a putative candidate gene for the observed phenotype, myosin heavy chain 3 (MYH3) (Table S1). Inherited forms of MYH3-related human syndromes, including cleft palate, are known (OMIM 160720). The homozygous variants at chr19:29609623A>G and at chr19:29609604TGTCAGCTCAAG>T represent a missense variant (NM_001101835.1:c.2864T>C; NP_001095305.1:p.Ile955Thr) and frameshift deletion (NM_001101835.1:c.2872_2882delCTTGAGCTGAC; NP_001095305.1:p.Leu958_Leu962delinsThrGlyGlnGlyTer), respectively, in exon 24 of bovine MYH3 (Figure S2). The isoleucine-to-threonine substitution affects an evolutionarily highly conserved amino acid in the myosin tail domain, and the frameshift deletion was predicted to cause a premature stop codon. Sanger sequencing confirmed the presence of both homozygous variants in MYH3 in the case (Figure S2). Among the additional 28 private homozygous coding variants, 23 were probably common variants, while the remaining five were classified as variants of uncertain significance (Appendix S1). A recent publication from France (Vaiman et al., 2022) finally proved the assumption of causality of MYH3 variants. To estimate the prevalence of MYH3 carriers within the Swiss Limousine population, we genotyped a population sample of purebred animals, including the bulls currently used for insemination (Table S2). The frequency of the variant allele in 1167 genotyped cattle was 1.5%. We have successfully identified the most likely cause for a rare autosomal recessive disorder with only one case. Our clinicopathologic and genomic findings for the MYH3-related recessively inherited form of non-syndromic cleft palate are consistent with those recently reported by Vaiman et al. (2022) (OMIA002590-9913). Interestingly, the 2-year-old Limousine female in our study appears to be the oldest known animal with MYH3-associated non-syndromic palatoschisis. According to Vaiman et al. (2022), life expectancy is short, with most animals euthanized in the first month of life. In addition, the variant allele frequency was found to be lower in the current Swiss population compared with the French Limousine (2.4%). Whether the MYH3 missense or the loss-of-function variant, or both, is actually causal remains open. The affected gene, MYH3, encodes the myosin heavy chain 3 protein, which is known to be responsible for muscle contraction. In addition, MYH3 is expressed in a variety of human and mouse fetal tissues (Chong et al., 2015; Zieba et al., 2017). In particular, in the mouse, Myh3 has been shown to be moderately expressed in head tissues during embryonic development (MGI: 1339709). There are several factors that can lead to the development of palatoschisis, such as impaired function and position of the tongue and surrounding muscles, which can disrupt the embryonic development of the palate (Rot & Kablar, 2013). Therefore, considering the rarity, the in silico effect prediction, the role of skeletal muscle in embryonic palate formation, the known function of MYH3 and the report of Vaiman et al. (2022), the identified variants can be considered as the cause. Future studies evaluating the functional consequences will be valuable in understanding the biological impact of the MYH3 variants during craniofacial development. We thank the Interfaculty Bioinformatics Unit of the University of Bern for providing high-performance computational infrastructure. We acknowledge Isabella Aebi-Huber for expert technical assistance. We are grateful to IFN Schönow GmbH, Germany for the genotyping of the Swiss Limousine population control cohort. Open access funding provided by Universitat Bern. This study was financially supported by the Arbeitsgemeinschaft Schweizerischer Rinderzüchter, Zollikofen, Switzerland and the Federal Office for Agriculture, Bern, Switzerland. The authors declare no conflict of interest. The whole genome sequencing data are available under the study accession no. PRJEB18113 at the European Nucleotide Archive (www.ebi.ac.uk/ena; SAMEA111531537). Appendix S1. Figure S1. Figure S2. Table S1. Table S2. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.

MYH3-associated non-syndromic palatoschisis (cleft palate,CP) in Limousine cattle.

MYH3-associated non-syndromic palatoschisis (cleft palate,CP) in Limousine cattle.

Approach to Urban Geospatial Monitoring Combining Sensor Web and High-performance Computing Infrastructure

Approach to Urban Geospatial Monitoring Combining Sensor Web and High-performance Computing Infrastructure

Large AI Spaces and Russia’s Strategy in the Context of the “Sanctions War”

The article is devoted to the study of the role of technological development in the field of artificial intelligence (AI) in the international political processes, the formation of large policy spaces in the field of AI, as well as the development of Russia in this area. The author provides an analysis of national AI strategies, examines the priority spheres and directions of development of these technologies for different countries, the availability of resources for the goals set, and the specifics of national AI policies of the leading states. The article compares the data on the volume of funding for AI developments, patent activities, and the existing infrastructure for high-performance computing in the countries, that occupy leading positions in the field of AI. The paper provides a brief overview of the key scientific and technological areas that are crucial for the creation of next-generation AI technologies: quantum computing and neuromorphic technologies within the framework of existing major national projects on the study of the brain. The author devotes a special place to the study of international political aspects of technological development in the field of AI in the work. From the perspective of the impact of AI policy on integration processes, the conclusion is made about the formation of two spaces: the first unites the OECD countries with the unconditional leadership of the USA and the EU in the field of research, development, infrastructure, resources, and international standards. The second, restrained space, relies on the technological and financial power of China, where are starting to get involved countries, including Russia, that are facing with narrowing range of opportunities for cooperation with Western countries. The author concludes that the confrontation between the two AI spaces is influenced by a more general trend of decoupling of the economies of the USA and China, which in turn contributes to technological isolation trends at the global level. A special place in the article is devoted to the analysis of the development of AI technologies in Russia; it considers official documents, leading players, and Russia’s positions at the international level. The possible directions of Russia’s cooperation with the world leading players are considered, which allows to develop AI technologies and at the same time maintain technological sovereignty in this area.

Exploring Parameter and Hyper-Parameter Spaces of Neuroscience Models on High Performance Computers With Learning to Learn.

Neuroscience models commonly have a high number of degrees of freedom and only specific regions within the parameter space are able to produce dynamics of interest. This makes the development of tools and strategies to efficiently find these regions of high importance to advance brain research. Exploring the high dimensional parameter space using numerical simulations has been a frequently used technique in the last years in many areas of computational neuroscience. Today, high performance computing (HPC) can provide a powerful infrastructure to speed up explorations and increase our general understanding of the behavior of the model in reasonable times. Learning to learn (L2L) is a well-known concept in machine learning (ML) and a specific method for acquiring constraints to improve learning performance. This concept can be decomposed into a two loop optimization process where the target of optimization can consist of any program such as an artificial neural network, a spiking network, a single cell model, or a whole brain simulation. In this work, we present L2L as an easy to use and flexible framework to perform parameter and hyper-parameter space exploration of neuroscience models on HPC infrastructure. Learning to learn is an implementation of the L2L concept written in Python. This open-source software allows several instances of an optimization target to be executed with different parameters in an embarrassingly parallel fashion on HPC. L2L provides a set of built-in optimizer algorithms, which make adaptive and efficient exploration of parameter spaces possible. Different from other optimization toolboxes, L2L provides maximum flexibility for the way the optimization target can be executed. In this paper, we show a variety of examples of neuroscience models being optimized within the L2L framework to execute different types of tasks. The tasks used to illustrate the concept go from reproducing empirical data to learning how to solve a problem in a dynamic environment. We particularly focus on simulations with models ranging from the single cell to the whole brain and using a variety of simulation engines like NEST, Arbor, TVB, OpenAIGym, and NetLogo.

Harvesting the Aggregate Computing Power of Commodity Computers for Supercomputing Applications

Distributed supercomputing is becoming common in different companies and academia. Most of the parallel computing researchers focused on harnessing the power of commodity processors and even internet computers to aggregate their computation powers to solve computationally complex problems. Using flexible commodity cluster computers for supercomputing workloads over a dedicated supercomputer and expensive high-performance computing (HPC) infrastructure is cost-effective. Its scalable nature can make it better employed to the available organizational resources, which can benefit researchers who aim to conduct numerous repetitive calculations on small to large volumes of data to obtain valid results in a reasonable time. In this paper, we design and implement an HPC-based supercomputing facility from commodity computers at an organizational level to provide two separate implementations for cluster-based supercomputing using Hadoop and Spark-based HPC clusters, primarily for data-intensive jobs and Torque-based clusters for Multiple Instruction Multiple Data (MIMD) workloads. The performance of these clusters is measured through extensive experimentation. With the implementation of the message passing interface, the performance of the Spark and Torque clusters is increased by 16.6% for repetitive applications and by 73.68% for computation-intensive applications with a speedup of 1.79 and 2.47 respectively on the HPDA cluster. We conclude that the specific application or job could be chosen to run based on the computation parameters on the implemented clusters.