High Performance Computing Centers Research Articles

Analysis model of energy consumption variables for data processing in high-performance computing systems

One of the main challenges in the efficient operation of a high-performance computing (HPC) center is the energy consumption generated by the operation of the data center where the HPC equipment is housed, mainly because this consumption is reflected in very high accounts payable, and this may affect the level of service offered to users. The study of the different factors and elements that can make energy consumption more efficient in these data centers provides an opportunity to focus these resources on elements that favor the use of HPC. The design variables provided by manufacturers to manage HPC systems and monitoring systems provide an accurate view of the behavior of these variables according to how they are used. HPC architectures are configured in a very particular way for each HPC data center, creating particular scenarios of operation and performance in each implementation. Various proposals and technologies have been developed for the analysis of the energy consumption of a data center, and the processing elements include a series of indicators and technologies that manufacturers have developed to determine the energy efficiency. This article seeks to identify this series of processing and performance variables, which affect the energy consumption of HPC equipment, for the implemented computing architectures based on the analysis of performance models to obtain a general over-view of their effect on energy consumption in a case study to identify the behaviors of particular job assignment factors and provide an analysis of the energy consumption under particular conditions.

The Advanced Computing Hub at BSC: improving fusion codes following modern software engineering standards

Several dedicated High-Performance Computing (HPC) centers provide essential expertise and support in developing a suitable portfolio of EUROfusion standard codes. Barcelona Supercomputing Center (BSC) is one of these HPC hubs involved in this complex task. Several fusion codes were selected, installed and analyzed to meet the developers’ requirements, ranging from portability to GPU, improving the performance, getting better data management, extending the capacity of coupling with other codes, etc. In this paper, we will describe the work developed by BSC and some of the tasks carried out in this project. We will explain briefly how the project is faced and the work required to create good quality codes, i.e. robust and trustable software capable of running efficiently in modern HPC systems.

Vitamin B1 Supplementation Ameliorates Obesogenic Effects of a High Fat Diet in Male Mice

Dietary deficiencies in vitamins such as thiamine (Vitamin B1) are common even in developed countries and can have a range of deleterious health effects. This trend in Vitamin B1 deficiency coincides with increased fat consumption in the American diet, particularly of soybean oil, the most widely consumed cooking oil in the U.S. which is also present in many processed foods. Here, we hypothesized that the concurrent malnutrition caused by a soybean oil-based high fat diet (SO-HFD) and a physiological deficiency in thiamine can exacerbate conditions such as obesity, diabetes, and fatty liver that are associated with consumption of HFDs. To test this, C57BL/6N male mice (N=8-10 per group) were fed either a low-fat chow or an SO-HFD, with or without supplementation with 640 mg/kg of the thiamine analog thiamine tetrahydrofurfuryl disulfide (TTFD), for 18 weeks. Our results showed that the SO-HFD supplemented with TTFD leads to significantly decreased weight gain and delayed the onset of glucose intolerance compared to SO-HFD without added TTFD (p<0.05). In contrast, TTFD supplementation did not have any effect on body weight or glucose intolerance in the context of a low-fat diet. Metabolomic analysis revealed that the SO-HFD decreased absolute thiamine levels both in the blood and liver tissue compared to the low fat control diet (p<0.05); this decrease was reversed by TTFD supplementation suggesting that some of the negative effects of the SO-HFD may be due to its impact on the Vitamin B1 pathway. Untargeted metabolomic analysis identified several other metabolites — such as amino acids, organic acids, methylamine osmolytes and those involved in nucleotide metabolism — that were also affected by TTFD supplementation, but in a beneficial fashion. Transcriptomic analysis of the liver and intestines revealed alterations in thiamine absorption and metabolic pathways by the SO-HFD; effects of TTFD on hepatic gene expression and fatty liver will be presented. TTFD supplementation given after the onset of SO-HFD-induced obesity was not able to reverse the phenotype, suggesting that the deleterious effects of the SO-HFD-induced Vitamin B1 deficiency may be permanent. In conclusion, our findings indicate that consumption of an SO-enriched diet, similar in fat content to what is consumed by most Americans, can result in a Vitamin B1 deficiency. Although TTFD supplementation may mitigate some of the adverse metabolic effects of this diet, this intervention must be done at an early stage. Funding: The Thiamine Advocacy Foundation Research Grant (FMS, PHD, PD); NIH R01DK127082 (FMS)Acknowledgements: UCR Genomics Core Facility, UCR Metabolomics Core Facility, UCR High Performance Computing Center (HPCC). This is the full abstract presented at the American Physiology Summit 2024 meeting and is only available in HTML format. There are no additional versions or additional content available for this abstract. Physiology was not involved in the peer review process.

HPC resources for CMS offline computing: An integration and scalability challenge for the Submission Infrastructure

HPC resources for CMS offline computing: An integration and scalability challenge for the Submission Infrastructure

Optical switching will innovate intra data center networks [Invited Tutorial

Reflecting the recent slow-down in Moore’s law and the proliferation of artificial intelligence/machine learning workloads, the performance and energy consumption of networks are becoming barriers in high-performance computing (HPC) and data centers. Optical switches are expected to break these barriers, and indeed their introduction has recently commenced in data centers. This paper discusses how optical switching technologies can innovate future intra data center networks. Hyperscale data centers are much bigger in scale, and network requirements are slightly different from those of HPC. This paper focus on data center networks, since the impact of optical technologies will be more significant in data centers than in HPC. In addition to the scale issue, important metrics to be considered for network design are traffic characteristics and latency, both of which are highlighted in this paper. For hybrid (electrical packet and optical circuit) switching networks, the target latency for the optical circuit switch network (connection setup/teardown time) is shown to be around 10 µs, and the needed technologies are clarified and verified by experiments. The optical switch can simplify the present multi-tier switching network above tier-1 switches into a single tier configuration, which is possible with the development of efficient large port count optical switches. Among the different switching architectures, combining the different dimensions of space and wavelength is shown to be one of the best solutions. Fast switching needs fast device response time. Si photonics devices using Mach–Zehnder interferometers or ring-resonator-based switches and tunable filters are the most promising candidates; they offer cost-effective mass-production and fast operation and so are excellent candidates for the optical switches envisaged. Another critical technology to maximize the benefits of optical switches is a simple and low-latency control mechanism. Different approaches have been suggested as summarized in this work. Among them, harnessing optical switch parallelism is a unique technique that matches recent advances in electrical switch chips. A fast control network is realized by using a fully decentralized and asynchronous control mechanism. A hyperscale data center offers a wide variety of services, and no one system fits all needs. Optimization of parameters is an important task for maximizing the impact of optical switching in different kinds of data centers.

The Roles of the National Center for High-performance Computing in Projects Involving Health-Care Big Data

<p>生物檢體數位化後將有機會永久保存生物檢體中的資訊與成就資料再利用之最大效益，然而數位化後的生醫巨量資料將仰賴大型的資訊基礎設施使資料服務為之不墜。近年來隨著生醫科技的進展，無論是基因定序或人工智慧技術的應用下，資料的規模，或者是計算規模，與背後的資訊設備需求，早已不是個別研究人員的實驗室資源所可以承受。再者隨著對個人資料保護的要求、注重資訊安全，及尊重資料當事人對自身資料權利表達機會，乃至於資料使用者對於資料分析的方便性與功能完整性的要求，讓生醫資料的運用與處理更仰賴於各項先進資訊技術集結，透過資訊專業建置出符合各界期待的生醫平台，讓生醫資料發揮其最大的效益。</p> <p> </p><p>The digitalization of biosamples offers an opportunity to permanently retain the information contained in biosamples and maximize data reutilization. However, the digitalized biomedical big data require large-scale information infrastructure to sustain relevant services. With the advancement of biomedical technologies, applications involving gene sequencing and artificial intelligence have exceeded a level that individual researchers or laboratories can afford in terms of data volume, computational capacity, and hardware requirements. The demands for personal information protection, information security, right of information self-determination, and the convenience and functional completeness of data analysis have also made biomedical data application and processing more reliant on advanced information technologies. A biomedical platform that meets the expectations of all stakeholders should thus be established through advanced information technologies to extract the maximum potential from biomedical data.</p> <p> </p>

Genome-wide association study meta-analysis supports association between MUC1 and ectopic pregnancy.

Can we identify genetic variants associated with ectopic pregnancy by undertaking the first genome-wide association study (GWAS) leveraging two large-scale biobank initiatives? We identified two novel genome-wide significant associations with ectopic pregnancy, highlighting MUC1 (mucin 1) as the most plausible affected gene. Ectopic pregnancy is an important cause of maternal morbidity and mortality worldwide. Despite being a common early pregnancy complication, the genetic predisposition to this condition remains understudied and no large scale genetic studies have been performed so far. A GWAS meta-analysis including 7070 women with ectopic pregnancy and 248 810 controls from Estonian Biobank and the FinnGen study. We identified ectopic pregnancy cases from national registers by ICD (International Classification of Disease) codes (ICD-10 O00), and all remaining women were considered controls. We carried out standard GWAS meta-analysis and additionally annotated GWAS signals, analysed co-localization with quantitative trait loci, estimated genetic correlations and identified associated phenotypes to characterize the genetic signals, as well as to analyse the genetic and phenotypic relationships with the condition. We identified two genome-wide significant loci on chromosomes 1 (rs4971091, P = 5.32×10-9) and 10 (rs11598956, P = 2.41×10-8) potentially associated with ectopic pregnancy. Follow-up analyses propose MUC1, which codes for an epithelial glycoprotein with an important role in barrier function, as the most likely candidate gene for the association on chromosome 1. We also characterize the phenotypic and genetic correlations with other phenotypes, identifying a genetic correlation with smoking and diseases of the (genito)urinary and gastrointestinal system, and phenotypic correlations with various reproductive health diagnoses, reflecting the previously known epidemiological associations. The GWAS meta-analysis summary statistics are available from the GWAS Catalogue (GCST90272883). The main limitation is that the findings are based on European-based ancestry populations, with limited data on other populations, and we only captured maternal genomes. Additionally, further larger meta-analysis or independent studies are needed to validate these findings. This study encourages the use of large-scale genetic datasets to unravel genetic factors linked to ectopic pregnancy, which is difficult to study in experimental settings. Increased sample size might bring additional genetic factors associating with ectopic pregnancy and inform its heritability. Altogether, our results provide more insight into the biology of ectopic pregnancy and, accordingly, the biological processes governing embryo implantation. N.P.G. was supported by MATER Marie Sklodowska-Curie which received funding from the European Union's Horizon 2020 research and innovation program under grant agreement No. 813707. This study was funded by European Union through the European Regional Development Fund Project No. 2014-2020.4.01.15-0012 GENTRANSMED. Computations were performed in the High-Performance Computing Center of University of Tartu. The authors declare no competing interests.

Advanced numerical techniques for predicting frequency response functions

Numerical methods have been widely used to predict Frequency Response Functions (FRF) of structures. The FRF can be used to characterize structural dynamics and support sound and vibration control. Generally, frequency sweep is conducted in the FRF calculations to identify damping and resonant frequencies of structures under excitations. However, as numerical models are getting complex and larger, the demands for computational resources (e.g., CPU time, memory, disk space) are greatly increased. Lately, high performance computing (HPC) at the DoD HPC center has been used with advanced numerical techniques to reduce the overall computational time. Those advanced numerical techniques include Krylov subspace and Galerkin Projection (KGP) and Pade Approximation. Significantly CPU time reduction has been demonstrated for relatively smaller FE models. This paper will further discuss adaptive KGP by automating the frequency sweep process via calculated relative residues, then combine with Finite Element Tearing and Interconnecting (FETI) to solve for FRF of a large FEM model in the order of 60M DOF. A flat plate with viscoelastic layer will serve as an example to demonstrate accuracy and efficiency of AKGP with FETI. The benefits of using HPC with advanced numerical techniques to solve for large FEM models are clearly displayed.

Variable Neighborhood Search for precedence-constrained tasks optimization on heterogeneous systems

Supercomputing power is one of the fundamental pillars of the digital society, which depends on the accurate scheduling of parallel applications in High-Performance Computing (HPC) centers to minimize computing times. However, precedence-constraint task scheduling is a well-known NP-Hard optimization problem, and no optimal polynomial-time algorithm exists to solve it. Therefore, as accurate as possible to the optimal values, heuristic algorithms are of relevant interest. Our new scheduling proposal name is EFT-GVNS, which stands for Earliest Finish Time - General Variable Neighborhood Search. EFT-GVNS uses a Composite Local Search (CLS), making our proposal more efficient than traditional GVNS. EFT-GVNS accuracy against four high-performance algorithms from the state-of-the-art (EDA, EFT-ILS, GRASP-CPA, MPQGA) and one reference algorithm in the literature (HEFT) is studied. Experimental results over four real-world applications (Fpppp, LIGO, Robot, Sparse) and 14 synthetic instances from the literature show that EFT-GVNS outperforms in terms of the median achieved results, with a global improvement of 37.6%, 27.4%, 17.8%, 6.1%, 2.2% to HEFT, EDA, EFT-ILS, GRASP-CPA, and MPQGA, respectively. EFT-GVNS achieves all the 14 optimal values of the synthetic benchmark.

Namibia's first high performance computer

High performance computing (HPC) refers to the practice of aggregating computing power of several computing nodes in a way that delivers much higher performance than one could achieve by a typical desktop computer in order to solve large problems in business, science, or engineering. The University of Namibia has so far received two HPC racks from the Centre for High Performance Computing in South Africa, of which one is operational. The primary use of the rack was foreseen to be human capacity development and awareness in HPC and to form part of Namibia’s readiness in participating in the Square Kilometre Array (SKA) and the African Very Long Baseline Interferometry Network (AVN) projects, but is now also being used for research in multi-wavelength astronomy and beyond. This is one of the first HPC services set up and operated by an entirely African team. We perform tests to benchmark the computational power and data transfer capabilities of the system and find that each node, on average, has a peak performance power of 82.4±1.1 GFLOPS. We also summarise all the projects that have enlisted the HPC facility.

Efficient and Reliable Data Management for Biomedical Applications.

This chapter discusses the challenges and requirements of modern Research Data Management (RDM), particularly for biomedical applications in the context of high-performance computing (HPC). The FAIR data principles (Findable, Accessible, Interoperable, Reusable) are of special importance. Data formats, publication platforms, annotation schemata, automated data management and staging, the data infrastructure in HPC centers, file transfer and staging methods in HPC, and the EUDAT components are discussed. Tools and approaches for automated data movement and replication in cross-center workflows are explained, as well as the development of ontologies for structuring and quality-checking of metadata in computational biomedicine. The CompBioMed project is used as a real-world example of implementing these principles and tools in practice. The LEXIS project has built a workflow-execution and data management platform that follows the paradigm of HPC-Cloud convergence for demanding Big Data applications. It is used for orchestrating workflows with YORC, utilizing the data documentation initiative (DDI) and distributed computing resources (DCI). The platform is accessed by a user-friendly LEXIS portal for workflow and data management, making HPC and Cloud Computing significantly more accessible. Checkpointing, duplicate runs, and spare images of the data are used to create resilient workflows. The CompBioMed project is completing the implementation of such a workflow, using data replication and brokering, which will enable urgent computing on exascale platforms.

Cloud benchmarking and performance analysis of an HPC application in Amazon EC2

Cloud computing platforms have been continuously evolving. Features such as the Elastic Fabric Adapter (EFA) in the Amazon Web Services (AWS) platform have brought yet another revolution in the High Performance Computing (HPC) world, further accelerating the convergence of HPC and cloud computing. Other public clouds also support similar features further fueling this change. In this paper, we show how and why the performance of a large-scale computational fluid dynamics (CFD) HPC application on AWS competes very closely with the one on Beskow—a Cray XC40 supercomputer at the PDC Center for High-Performance Computing - in terms of cost-efficiency with strong scaling up to 2304 processes. We perform an extensive set of micro and macro benchmarks in both environments and conduct a comparative analysis. Until as recently as 2020 these benchmarks have notoriously yielded unsatisfactory results for the cloud platforms compared with on-premise infrastructures. Our aim is to access the HPC capabilities of the cloud, and in general to demonstrate how researchers can scale and evaluate the performance of their application in the cloud.

The N-shaped partition method: A novel parallel implementation of the Crank Nicolson algorithm

We develop an algorithm to solve tridiagonal systems of linear equations, which appear in implicit finite-difference schemes of partial differential equations (PDEs), being the time-dependent Schrödinger equation (TDSE) an ideal candidate to benefit from it. Our N-shaped partition method optimizes the implementation of the numerical calculation on parallel architectures, without memory size constraints. Specifically, we discuss the realization of our method on graphics processing units (GPUs) and the Message Passing Interface (MPI). In GPU implementations, our scheme is particularly advantageous for systems whose size exceeds the global memory of a single processor. Moreover, because of its lack of memory constraints and the generality of the algorithm, it is well-suited for mixed architectures, typically available in large high performance computing (HPC) centers. We also provide an analytical estimation of the optimal parameters to implement our algorithm, and test numerically the suitability of our formula in a GPU implementation. Our method will be helpful to tackle problems which require large spatial grids for which ab-initio studies might be otherwise prohibitive both because of large shared-memory requirements and computation times.

Overcoming Challenges to Continuous Integration in HPC

Continuous integration (CI) has become a ubiquitous practice in modern software development, with major code hosting services offering free automation on popular platforms. CI offers major benefits, as it enables detecting bugs in code prior to committing changes. While high-performance computing (HPC) research relies heavily on software, HPC machines are not considered "common" platforms. This presents several challenges that hinder the adoption of CI in HPC environments, making it difficult to maintain bug-free HPC projects, and resulting in adverse effects on the research community. In this article, we explore the challenges that impede HPC CI, such as hardware diversity, security, isolation, administrative policies, and non-standard authentication, environments, and job submission mechanisms. We propose several solutions that could enhance the quality of HPC software and the experience of developers. Implementing these solutions would require significant changes at HPC centers, but if these changes are made, it would ultimately enable faster and better science.

Segment Linking: A Highly Parallelizable Track Reconstruction Algorithm for HL-LHC

The High Luminosity upgrade of the Large Hadron Collider (HL-LHC) will produce particle collisions with up to 200 simultaneous proton-proton interactions. These unprecedented conditions will create a combinatorial complexity for charged-particle track reconstruction that demands a computational cost that is expected to surpass the projected computing budget using conventional CPUs. Motivated by this and taking into account the prevalence of heterogeneous computing in cutting-edge High Performance Computing centers, we propose an efficient, fast and highly parallelizable bottom-up approach to track reconstruction for the HL-LHC, along with an associated implementation on GPUs, in the context of the Phase 2 CMS outer tracker. Our algorithm, called Segment Linking (or Line Segment Tracking), takes advantage of localized track stub creation, combining individual stubs to progressively form higher level objects that are subject to kinematical and geometrical requirements compatible with genuine physics tracks. The local nature of the algorithm makes it ideal for parallelization under the Single Instruction, Multiple Data paradigm, as hundreds of objects can be built simultaneously. The computing and physics performance of the algorithm has been tested on an NVIDIA Tesla V100 GPU, already yielding efficiency and timing measurements that are on par with the latest, multi-CPU versions of existing CMS tracking algorithms.

European HPC cloud infrastructure for managing SARS-CoV-2 data in compliance with GDPR

Abstract The Connecting European SARS-CoV-2 Cohorts to Increase Common and Effective Response to SARS-CoV-2 Pandemic (ORCHESTRA) consortium, led by University of Verona (Italy), brings together key European academic experts and research institutions in infectious diseases, data management and High Performance Computing (HPC) from 26 organizations (extending to 37 partners) from 15 countries. The project aims to create a new pan-European cohort built on existing and new large-scale population cohorts in European and non-European countries to significantly impact on the responsiveness to SARS-CoV-2. The integration and analysis of the very heterogeneous characteristics of SARS-CoV-2 data coming from many different sources such as EHR, retrospective and prospective patient registries, and related ‘omics’ data (incl. genomics, proteomics and transcriptomics) can benefit of data analytics enabled by HPC, where both high compute performance and fast storage capabilities are immensely important. During the first year of the project, a dedicated HPC cloud infrastructure have been designed and partially deployed to fulfill the functional requirements for data management ensuring healthcare data confidentiality/privacy, integrity and security in compliance with the European GDPR regulations. The result is an infrastructure for Data Management composed by three main layers: National Data Providers; National Hubs (one for each HPC center involved: CINECA - Italy, CINES - France and HLRS - Germany), to centralize data at national level and to support data storage, sharing and analysis on data ingested from the National Data Providers; ORCHESTRA Data Portal: the pan-European portal for sharing aggregated data and results. Currently data collection is on going; at the end of the project, clinical centers are expected to have enrolled more than 10.000 patients with about 50.000 samples for the prospective studies. Key messages • The SARS-CoV-2 crisis made evident the need to manage and analyse very heterogeneous health data coming from many different resources across different countries. • The HPC cloud infrastructure released for the Orchestra project can act as a model to manage future public health threats.

Accelerating HPC With Quantum Computing: It Is a Software Challenge Too

With quantum computing (QC) maturing, high-performance computing (HPC) centers are already preparing to host early-phase production versions of such systems. Unlike their experimental predecessors in physics laboratories, with a very small and dedicated user community, this next generation of systems needs to serve a wider user community and must work in concert with existing HPC systems and software stacks. This article describes our vision for an integrated ecosystem that combines existing HPC and evolving quantum software stacks into a single system to enable a common and continuous user experience. This integration comes with several major challenges as quantum systems pose significantly different requirements including increased need for compilation at run time, long optimization times, statistical evaluations of results, and the need to work with few centralized resources. To overcome these challenges, new scheduling approaches on the HPC side and new programming approaches on the QC side are required.

Burner: Recipe Automatic Generation for HPC Container Based on Domain Knowledge Graph

As one of the emerging cloud computing technologies, containers are widely used in academia and industry. The cloud computing built by the container in the high performance computing (HPC) center can provide high-quality services to users at the edge. Singularity Definition File and Dockerfile (we refer to such files as recipes) have attracted wide attention due to their encapsulation of the application running environment in a container. However, creating a recipe requires extensive domain knowledge, which is error-prone and time-consuming. Accordingly, more than 34% of Dockerfiles in Github cannot successfully build container images. The crucial points about recipe creation include selecting the entities (base images and packages) and determining their relationships (correct installation order for transitive dependencies). Since the relationships between entities can be expressed accurately and efficiently by the knowledge graph, we introduce knowledge graph to generate high-quality recipes automatically. This paper proposes an automatic recipe generation system named Burner, enabling users with no professional computer background to generate the recipes. We first develop a toolset including a recipe parser and an entity-relationship miner. Our two-phase recipe parsing method can perform abstract syntax tree (AST) parsing more deeply on the recipe file to achieve entity extraction; the parsing success rate (PSR) of the two-phase parsing method is 10.1% higher than the one-phase parsing. Then, we build a knowledge base containing 2,832 entities and 62,614 entity relationships, meeting the needs of typical HPC applications. In the test of image build, the singularity image build success rate reaches 80%. Compared with the ItemCF recommendation method, our recommendation method TB-TFIDF achieves a performance improvement by up to 50.86%.

Performance trade-offs in reconfigurable networks for HPC

Designing efficient interconnects to support high-bandwidth and low-latency communication is critical toward realizing high performance computing (HPC) and data center (DC) systems in the exascale era. At extreme computing scales, providing the requisite bandwidth through overprovisioning becomes impractical. These challenges have motivated studies exploring reconfigurable network architectures that can adapt to traffic patterns at runtime using optical circuit switching. Despite the plethora of proposed architectures, surprisingly little is known about the relative performances and trade-offs among different reconfigurable network designs. We aim to bridge this gap by tackling two key issues in reconfigurable network design. First, we study how cost, power consumption, network performance, and scalability vary based on optical circuit switch (OCS) placement in the physical topology. Specifically, we consider two classes of reconfigurable architectures: one that places OCSs between top-of-rack (ToR) switches—ToR-reconfigurable networks (TRNs)—and one that places OCSs between pods of racks—pod-reconfigurable networks (PRNs). Second, we tackle the effects of reconfiguration frequency on network performance. Our results, based on network simulations driven by real HPC and DC workloads, show that while TRNs are optimized for low fan-out communication patterns, they are less suited for carrying high fan-out workloads. PRNs exhibit better overall trade-off, capable of performing comparably to a fully non-blocking fat tree for low fan-out workloads, and significantly outperform TRNs for high fan-out communication patterns.

A Study of Failure Recovery and Logging of High-Performance Parallel File Systems

Large-scale parallel file systems (PFSs) play an essential role in high-performance computing (HPC). However, despite their importance, their reliability is much less studied or understood compared with that of local storage systems or cloud storage systems. Recent failure incidents at real HPC centers have exposed the latent defects in PFS clusters as well as the urgent need for a systematic analysis. To address the challenge, we perform a study of the failure recovery and logging mechanisms of PFSs in this article. First, to trigger the failure recovery and logging operations of the target PFS, we introduce a black-box fault injection tool called PFault , which is transparent to PFSs and easy to deploy in practice. PFault emulates the failure state of individual storage nodes in the PFS based on a set of pre-defined fault models and enables examining the PFS behavior under fault systematically. Next, we apply PFault to study two widely used PFSs: Lustre and BeeGFS. Our analysis reveals the unique failure recovery and logging patterns of the target PFSs and identifies multiple cases where the PFSs are imperfect in terms of failure handling. For example, Lustre includes a recovery component called LFSCK to detect and fix PFS-level inconsistencies, but we find that LFSCK itself may hang or trigger kernel panics when scanning a corrupted Lustre. Even after the recovery attempt of LFSCK, the subsequent workloads applied to Lustre may still behave abnormally (e.g., hang or report I/O errors). Similar issues have also been observed in BeeGFS and its recovery component BeeGFS-FSCK. We analyze the root causes of the abnormal symptoms observed in depth, which has led to a new patch set to be merged into the coming Lustre release. In addition, we characterize the extensive logs generated in the experiments in detail and identify the unique patterns and limitations of PFSs in terms of failure logging. We hope this study and the resulting tool and dataset can facilitate follow-up research in the communities and help improve PFSs for reliable high-performance computing.