Barcelona Supercomputing Center Research Articles

The Advanced Computing Hub at BSC: Improving fusion codes following modern software engineering standards

Abstract Several dedicated High-Performance Computing (HPC) centers provide essential expertise and support in developing a suitable portfolio of EUROfusion standard codes. Barcelona Supercomputing Center (BSC) is one of these HPC hubs involved in this complex task. Several fusion codes were selected, installed and analyzed to meet the developers’ requirements, ranging from portability to GPU, improving the performance, getting better data management, extending the capacity of coupling with other codes, etc. In this paper, we will describe the work developed by BSC and some of the tasks carried out in this project. We will explain briefly how the project is faced and the work required to create good quality codes, i.e. robust and trustable software capable of running efficiently in modern HPC systems.

Integration of the Barcelona Supercomputing Center for CMS computing: Towards large scale production

The CMS experiment is working to integrate an increasing number of High Performance Computing (HPC) resources into its distributed computing infrastructure. The case of the Barcelona Supercomputing Center (BSC) is particularly challenging as severe network restrictions prevent the use of CMS standard computing solutions. The CIEMAT CMS group has performed significant work in order to overcome these constraints and make BSC resources available to CMS. The developments include adapting the workload management tools, replicating the CMS software repository to BSC storage, providing an alternative access to detector conditions data, and setting up a service to transfer produced output data to a nearby storage facility. In this work, we discuss the current status of this integration activity and present recent developments, such as a front-end service to improve slot usage efficiency and an enhanced transfer service that supports the staging of input data for workflows at BSC. Moreover, significant efforts have been devoted to improving the scalability of the deployed solution, automating its operation, and simplifying the matchmaking of CMS workflows that are suitable for execution at BSC.

Raising awareness of sex and gender bias in artificial intelligence and health.

Historically, biomedical research has been led by and focused on men. The recent introduction of Artificial Intelligence (AI) in this area has further proven this practice to be discriminatory for other sexes and genders, more noticeably for women. To move towards a fair AI development, it is essential to include sex and gender diversity both in research practices and in the workplace. In this context, the Bioinfo4women (B4W) program of the Barcelona Supercomputing Center (i) promotes the participation of women scientists by improving their visibility, (ii) fosters international collaborations between institutions and programs and (iii) advances research on sex and gender bias in AI and health. In this article, we discuss methodology and results of a series of conferences, titled “Sex and Gender Bias in Artificial Intelligence and Health, organized by B4W and La Caixa Foundation from March to June 2021 in Barcelona, Spain. The series consisted of nine hybrid events, composed of keynote sessions and seminars open to the general audience, and two working groups with invited experts from different professional backgrounds (academic fields such as biology, engineering, and sociology, as well as NGOs, journalists, lawyers, policymakers, industry). Based on this awareness-raising action, we distilled key recommendations to facilitate the inclusion of sex and gender perspective into public policies, educational programs, industry, and biomedical research, among other sectors, and help overcome sex and gender biases in AI and health.

High Performance Computing PP-Distance Algorithms to Generate X-ray Spectra from 3D Models.

X-ray crystallography is a powerful method that has significantly contributed to our understanding of the biological function of proteins and other molecules. This method relies on the production of crystals that, however, are usually a bottleneck in the process. For some molecules, no crystallization has been achieved or insufficient crystals were obtained. Some other systems do not crystallize at all, such as nanoparticles which, because of their dimensions, cannot be treated by the usual crystallographic methods. To solve this, whole pair distribution function has been proposed to bridge the gap between Bragg and Debye scattering theories. To execute a fitting, the spectra of several different constructs, composed of millions of particles each, should be computed using a particle–pair or particle–particle (pp) distance algorithm. Using this computation as a test bench for current field-programmable gate array (FPGA) technology, we evaluate how the parallel computation capability of FPGAs can be exploited to reduce the computation time. We present two different solutions to the problem using two state-of-the-art FPGA technologies. In the first one, the main C program uses OmpSs (a high-level programming model developed at the Barcelona Supercomputing Center, that enables task offload to different high-performance computing devices) for task invocation, and kernels are built with OpenCL using reduced data sizes to save transmission time. The second approach uses task and data parallelism to operate on data locally and update data globally in a decoupled task. Benchmarks have been evaluated over an Intel D5005 Programmable Acceleration Card, computing a model of 2 million particles in 81.57 s – 24.5 billion atom pairs per second (bapps)– and over a ZU102 in 115.31 s. In our last test, over an up-to-date Alveo U200 board, the computation lasted for 34.68 s (57.67 bapps). In this study, we analyze the results in relation to the classic terms of speed-up and efficiency and give hints for future improvements focused on reducing the global job time.

Validating NEUTRO, a deterministic finite element neutron transport solver for fusion applications, with literature tests, experimental benchmarks and other neutronic codes

Neutron reactions on fusion reactor materials are key phenomena to be understood to enable fusion as a feasible energy source for future reactors. We present significant improvements and validations of NEUTRO, a deterministic neutron transport code dedicated to solving the Boltzmann transport equation. The code is integrated as a module in the Alya system developed by the Barcelona Supercomputing Center and uses the discrete ordinates method over an angular portion, multigroup for energy discretization and the finite element method over unstructured meshes to treat special complex domains. Material anisotropy of the scattering medium is introduced into the scattering kernel using real base expressions for spherical harmonics. In order to build the total cross-section and the respective group matrix for the elastic cross-section, we use the NJOY code. We test the solver using different geometries and materials with varying levels of scattering properties. We compare our results on classic tests, benchmarks obtained from a Nuclear Energy Agency database and test cases using other neutron transport codes.

The MONARCH high-resolution reanalysis of desert dust aerosol over Northern Africa, the Middle East and Europe (2007–2016)

Abstract. One of the challenges in studying desert dust aerosol along with its numerous interactions and impacts is the paucity of direct in situ measurements, particularly in the areas most affected by dust storms. Satellites typically provide column-integrated aerosol measurements, but observationally constrained continuous 3D dust fields are needed to assess dust variability, climate effects and impacts upon a variety of socio-economic sectors. Here, we present a high-resolution regional reanalysis data set of desert dust aerosols that covers Northern Africa, the Middle East and Europe along with the Mediterranean Sea and parts of central Asia and the Atlantic and Indian oceans between 2007 and 2016. The horizontal resolution is 0.1∘ latitude × 0.1∘ longitude in a rotated grid, and the temporal resolution is 3 h. The reanalysis was produced using local ensemble transform Kalman filter (LETKF) data assimilation in the Multiscale Online Nonhydrostatic AtmospheRe CHemistry model (MONARCH) developed at the Barcelona Supercomputing Center (BSC). The assimilated data are coarse-mode dust optical depth retrieved from the Moderate Resolution Imaging Spectroradiometer (MODIS) Deep Blue Level 2 products. The reanalysis data set consists of upper-air variables (dust mass concentrations and the extinction coefficient), surface variables (dust deposition and solar irradiance fields among them) and total column variables (e.g. dust optical depth and load). Some dust variables, such as concentrations and wet and dry deposition, are expressed for a binned size distribution that ranges from 0.2 to 20 µm in particle diameter. Both analysis and first-guess (analysis-initialized simulation) fields are available for the variables that are diagnosed from the state vector. A set of ensemble statistics is archived for each output variable, namely the ensemble mean, standard deviation, maximum and median. The spatial and temporal distribution of the dust fields follows well-known dust cycle features controlled by seasonal changes in meteorology and vegetation cover. The analysis is statistically closer to the assimilated retrievals than the first guess, which proves the consistency of the data assimilation method. Independent evaluation using Aerosol Robotic Network (AERONET) dust-filtered optical depth retrievals indicates that the reanalysis data set is highly accurate (mean bias = −0.05, RMSE = 0.12 and r = 0.81 when compared to retrievals from the spectral de-convolution algorithm on a 3-hourly basis). Verification statistics are broadly homogeneous in space and time with regional differences that can be partly attributed to model limitations (e.g. poor representation of small-scale emission processes), the presence of aerosols other than dust in the observations used in the evaluation and differences in the number of observations among seasons. Such a reliable high-resolution historical record of atmospheric desert dust will allow a better quantification of dust impacts upon key sectors of society and economy, including health, solar energy production and transportation. The reanalysis data set (Di Tomaso et al., 2021) is distributed via Thematic Real-time Environmental Distributed Data Services (THREDDS) at BSC and is freely available at http://hdl.handle.net/21.12146/c6d4a608-5de3-47f6-a004-67cb1d498d98 (last access: 10 June 2022).

Microscale simulations of extreme events in complex terrain driven by mesoscalar budget components

In this work, we apply a downscaling strategy to analyze extreme weather events that may impact wind farm operation. The coupling applies mesoscale momentum budget components (tendencies) from the WRF model as forcing terms to the governing microscale equations. Our study focuses on flow over complex terrain during specific days to reproduce extreme weather events that produced wind turbine damage. The interaction of the meso- and micro-scale features are relevant in the simulation of extreme conditions. The simulation results are compared with observations from nacelle anemometers of the wind turbines in two different wind farms by analyzing time series and wind profiles.The microscale code Alya, developed at the Barcelona Supercomputing Center (BSC), is closed with URANS and LES closures to solve the momentum and energy equations. Both closures use the same mesoscalar to microscalar coupling methodology and are used in this work to simulate the wind flow. We present the implementation of the mesoscalar coupling to the microscale solver when using URANS and LES closures.We show that the coupling via tendencies has excellent potential for understanding transient events under extreme weather conditions in very complex terrain. The wind industry can use such simulations to enhance forensic analysis in cases of wind turbine accidents or any other event that may impact turbine operation, such as high turbulence phenomena. We test the ability of the meso- to microscale coupling model to reproduce extreme events with regard to quantities of interest in wind energy.Simulation results using URANS and LES closures agree reasonably well with observations. In some scenarios, the LES provides results that are closer to measurements. LES models have the advantage of providing wind gusts. We compare the accuracy and performance (CPU-time) of the URANS vs. LES approaches.

PyCOMPSs as an Instrument for Translational Computer Science

With the advent of distributed computing, the need for frameworks that facilitate its programming and management has also appeared. These tools have typically been used to support the research on application areas that require them. This poses good initial conditions for translational computer science (TCS), although this does not always occur. This article describes our experience with the PyCOMPSs project, a programming model for distributed computing. While it is a research instrument for our team, it has also been applied in multiple real use cases under the umbrella of European Funded projects, or as part of internal projects between various departments at the Barcelona Supercomputing Center. This article illustrates how the authors have engaged in TCS as an underlying research methodology, collecting experiences from three European projects.

Computational modelling of an aerosol extraction device for use in COVID-19 surgical tracheotomy

In view of the ongoing COVID-19 pandemic and its effects on global health, understanding and accurately modelling the propagation of human biological aerosols has become crucial. Worldwide, health professionals have been one of the most affected demographics, representing approximately 20% of all cases in Spain, 10% in Italy and 4% in China and US. Methods to contain and remove potentially infected aerosols during Aerosol Generating Procedures (AGPs) near source offer advantages in reducing the contamination of protective clothing and the surrounding theatre equipment and space. In this work we describe the application of computational fluid dynamics in assessing the performance of a prototype extraction hood as a means to contain a high speed aerosol jet. Whilst the particular prototype device is intended to be used during tracheotomies, which are increasingly common in the wake of COVID-19, the underlying physics can be adapted to design similar machines for other AGPs. Computational modelling aspect of this study was largely carried out by Barcelona Supercomputing Center using the high performance computational mechanics code Alya. Based on the high fidelity LES coupled with Lagrangian frameworks the results demonstrate high containment efficiency of generated particles is feasible with achievable air extraction rates.

The ethical use of high-performance computing and artificial intelligence: fighting COVID-19 at Barcelona Supercomputing Center

The COVID-19 pandemic has created an extraordinary medical, economic and humanitarian emergency. Artificial intelligence, in combination with other digital technologies, is being used as a tool to support the fight against the viral pandemic that has affected the entire world since the beginning of 2020. Barcelona Supercomputing Center collaborates in the battle against the coronavirus in different areas: the application of bioinformatics for the research on the virus and its possible treatments, the use of artificial intelligence, natural language processing and big data techniques to analyse the spread and impact of the pandemic, and the use of the MareNostrum 4 supercomputer to enable massive analysis on COVID-19 data. Many of these activities have included the use of personal and sensitive data of citizens, which, even during a pandemic, should be treated and handled with care. In this work we discuss our approach based on an ethical, transparent and fair use of this information, an approach aligned with the guidelines proposed by the European Union.

Validations of the radiation transport module NEUTRO: A deterministic solver for the neutron transport equation

We present significant improvements and validations of a deterministic neutron transport code (NEUTRO) dedicated to solving the Boltzmann Transport Equation. The code is integrated as a module in the Alya software package developed by the Barcelona Supercomputing Center which uses the Discrete Ordinates Method on angular coordinates, multi-group for energy discretization and FEM on unstructured meshes to treat special complex domains. The anisotropy of the scattering medium is introduced into the scattering kernel using real base expressions for spherical harmonics. In order to build the total cross-section and the respective group matrix for the elastic cross-section, we use the NJOY code. We test the solver using different geometries, orders of integration for the angular discretization and number of energy groups. Finally, we compare our results against benchmarks obtained from an NEA database that reported measurements of leakage spectra of several materials.

Implementation of a high-accuracy phase unwrapping algorithm using parallel-hybrid programming approach for displacement sensing using self-mixing interferometry

Phase unwrapping is an integral part of multiple algorithms with diverse applications. Detailed phase unwrapping is also necessary for achieving high-accuracy metric sensing using laser feedback-based self-mixing interferometry (SMI). Among SMI specific phase unwrapping approaches, a technique called Improved Phase Unwrapping Method (IPUM) provides the highest accuracy. However, due to its complex, sequential, and compute-intensive nature, this method requires a high-performance computing architecture, capable of scalable parallel processing so that such a high-accuracy algorithm can be used for high-bandwidth sensing applications. In this work, the existing sequential IPUM C program is parallelized by using hybrid OpenMP/MPI (Open Multi-Processing/Message Passing Interface) parallel programming models and tested on Barcelona Supercomputing Center Nord-III Supercomputer. The computational performance of the proposed parallel-hybrid IPUM algorithm is compared with existing IPUM sequential code by executing multi-core and uni-core processor architecture, respectively. While comparing the performance of sequential IPUM with the parallel-hybrid IPUM algorithm on 16 nodes of Nord-III supercomputer, the results show that the parallel-hybrid algorithm gets 345.9x times performance improvement as compared to IPUM’s standard, sequential implementation on a single node system. The results show that the parallel-hybrid version of IPUM gives a scalable performance for different target velocities and a different number of processing cores.

Exploitation of the MareNostrum 4 HPC using ARC-CE

The resources of the HPC centers are a potential aid to meet the future challenges of HL-LHC [1] in terms of computational requirements. Spanish HPC centers have recently been used to implement all necessary edge services to integrate resources into the LHC experiment workflow management system. In this article, we describe the integration of ATLAS with the extension plan to other LHC experiments. We chose to configure a dedicated ARC-CE [2] and interact with the HPC login and transfer nodes using ssh commands. The repository that includes a partial copy of the ATLAS experiment software on CVMFS is packaged in a singularity image to overcome network isolation for HPC nodes and reduce software requirements. ATLAS provided the initial container, and the authors adapted it to the specific HPC environment. This article shows the Spanish contribution to the simulation of experiments after the Spanish Ministry of Science agreement and the Barcelona Supercomputing Center (BSC), the center that operates MareNostrum 4. Finally, we discuss some challenges to take advantage of the next generation of HPC machines with heterogeneous architecture combining CPU and GPU.

Exploitation of network-segregated CPU resources in CMS

CMS is tackling the exploitation of CPU resources at HPC centers where compute nodes do not have network connectivity to the Internet. Pilot agents and payload jobs need to interact with external services from the compute nodes: access to the application software (CernVM-FS) and conditions data (Frontier), management of input and output data files (data management services), and job management (HTCondor). Finding an alternative route to these services is challenging. Seamless integration in the CMS production system without causing any operational overhead is a key goal. The case of the Barcelona Supercomputing Center (BSC), in Spain, is particularly challenging, due to its especially restrictive network setup. We describe in this paper the solutions developed within CMS to overcome these restrictions, and integrate this resource in production. Singularity containers with application software releases are built and pre-placed in the HPC facility shared file system, together with conditions data files. HTCondor has been extended to relay communications between running pilot jobs and HTCondor daemons through the HPC shared file system. This operation mode also allows piping input and output data files through the HPC file system. Results, issues encountered during the integration process, and remaining concerns are discussed.

Refined electron-spin transport model for single-element ferromagnetic systems: Application to nickel nanocontacts

This work was supported by the Generalitat Valenciana through Grant No. PROMETEO2017/139. C.S. gratefully acknowledges financial support from the Dean Fellowship of the Weizmann Institute of Science and Generalitat Valenciana (Grant No. CDEIGENT2018/028). O.T. appreciates the support of the Harold Perlman family, and acknowledges funding by a research grant from Dana and Yossie Hollander, the Israel Science Foundation (Grant No. 1089/15), the Minerva Foundation (Grant No. 120865), and The Ministry of Science and Technology of Israel (Grant No. 3-16244). J.J.P. acknowledges financial support from Spanish MINECO through Grants No. FIS2016-80434-P and No. PID2019-109539GB-C43, the Fundacion Ramon Areces, the Maria de Maeztu Program for Units of Excellence in R&D (Grant No. CEX2018-000805-M), the Comunidad Autonoma de Madrid through the Nanomag COST-CM Program (Grant No. S2018/NMT-4321), the European Union Seventh Framework Programme under Grant Agreement No. 604391 Graphene Flagship, the Centro de Computacion Cientifica of the Universidad Autonoma de Madrid and the computer resources at MareNostrum and the technical support provided by the Barcelona Supercomputing Center (Grant No. FI-2019-2-0007). The SLD and DFT calculations in this paper were performed on the high-performance computing facilities of the University of Alicante and the University of South Africa.

Educating and engaging new communities of practice with high performance computing through the integration of teaching and research.

The identification of strategies by which to increase the representation of women and increase diversity in STEM fields (science, technology, engineering and mathematics), including medicine, has been a pressing matter for global agencies including the European Commission, UNESCO and numerous international scientific societies. In my role as UCL training lead for CompBioMed, a European Commission Horizon 2020-funded Centre of Excellence in Computational Biomedicine (compbiomed.eu), and as Head of Teaching for Molecular Biosciences at UCL from 2010 to 2019, I have integrated research and teaching to lead the development of high-performance computing (HPC)-based education targeting medical students and undergraduate students studying biosciences in a way that is explicitly integrated into the existing university curriculum as a credit-bearing module. One version of the credit-bearing module has been specifically designed for medical students in their pre-clinical years of study and one of the unique features of the course is the integration of clinical and computational aspects, with students obtaining and processing clinical samples and then interrogating the results computationally using code that was ported to HPC at CompBioMed's HPC Facility core partners (EPCC (UK), SURFsara (The Netherlands) and the Barcelona Supercomputing Centre (Spain)). Another version of the credit-bearing module has, over the course of this project, evolved into a replacement for the third year research project course for undergraduate biochemistry, biotechnology and molecular biology students, providing students with the opportunity to design and complete an entire specialist research project from the formulation of experimental hypotheses to the investigation of these hypotheses in a way that involves the integration of experimental and HPC-based computational methodologies. Since 2017-2018, these UCL modules have been successfully delivered to over 350 students-a cohort with a demographic of greater than 50% female. CompBioMed's experience with these two university modules has enabled us to distil our methodology into an educational template that can be delivered at other universities in Europe and worldwide. This educational approach to training enables new communities of practice to effectively engage with HPC and reveals a means by which to improve the underrepresentation of women in supercomputing.

Cats and llamas could offer a path to coronavirus therapies

Computation and experimentation also yield possible therapeutic compounds for COVID-19 [ABSTRACT FROM AUTHOR] Copyright of Physics Today is the property of American Institute of Physics and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission However, users may print, download, or email articles for individual use This abstract may be abridged No warranty is given about the accuracy of the copy Users should refer to the original published version of the material for the full abstract (Copyright applies to all Abstracts )

Improving maritime traffic emission estimations on missing data with CRBMs

Maritime traffic emissions are a major concern to governments as they heavily impact the Air Quality in coastal cities. Ships use the Automatic Identification System (AIS) to continuously report position and speed among other features, and therefore this data is suitable to be used to estimate emissions, if it is combined with engine data. However, important ship features are often inaccurate or missing. State-of-the-art complex systems, like CALIOPE at the Barcelona Supercomputing Center, are used to model Air Quality. These systems can benefit from AIS based emission models as they are very precise in positioning the pollution. Unfortunately, these models are sensitive to missing or corrupted data, and therefore they need data curation techniques to significantly improve the estimation accuracy. In this work, we propose a methodology for treating ship data using Conditional Restricted Boltzmann Machines (CRBMs) plus machine learning methods to improve the quality of data passed to emission models that can also be applied to other GPS and time-series problems. Results show that we can improve the default methods proposed to cover missing data. In our results, we observed that using our method the models boosted their accuracy to detect otherwise undetectable emissions. In particular, we used a real data-set of AIS data, provided by the Spanish Port Authority, to estimate that thanks to our method, the model was able to detect 45% of additional emissions, representing 152 tonnes of pollutants per week in Barcelona and propose new features that may enhance emission modeling.

Freezing time emulating new and faster devices with virtual machines

Recent proposals of emerging data storage devices make it necessary to reevaluate all levels of the storage hierarchy to optimize the software stack performance. However, these new devices are not always widely available and therefore early experiments may be impossible. Emulators aim at mimicking as close as possible the behavior of a component, nonetheless, emulating new and fast storage devices is a challenging task due to time perception. In this work, we propose an approach to emulate storage devices using virtual machines (VMs) allowing the evaluation of a new device within a real system. We use a technique called freezing time, which pauses a VM to manipulate its clock and hide the real I/O completion time. Our approach is implemented at the hypervisor level and it is transparent to the guest operating system or application. We evaluate the technique under a real system using regular magnetic disks to emulate faster storage devices. Our method presented a latency error of 6.5% compared to a real device. Moreover, decoupled experiment between two laboratories, at the Barcelona Super Computing Center (BSC) in Spain, and the Center of Computer Science and Free Software (C3SL) in Brazil, demonstrated that our approach is reproducible and promising to allow the virtual evaluation of next-gen storage devices.

Exploiting network restricted compute resources with HTCondor: a CMS experiment experience

In view of the increasing computing needs for the HL-LHC era, the LHC experiments are exploring new ways to access, integrate and use non-Grid compute resources. Accessing and making efficient use of Cloud and High Performance Computing (HPC) resources present a diversity of challenges for the CMS experiment. In particular, network limitations at the compute nodes in HPC centers prevent CMS pilot jobs to connect to its central HTCondor pool in order to receive payload jobs to be executed. To cope with this limitation, new features have been developed in both HTCondor and the CMS resource acquisition and workload management infrastructure. In this novel approach, a bridge node is set up outside the HPC center and the communications between HTCondor daemons are relayed through a shared file system. This conforms the basis of the CMS strategy to enable the exploitation of the Barcelona Supercomputing Center (BSC) resources, the main Spanish HPC site. CMS payloads are claimed by HTCondor condor_startd daemons running at the nearby PIC Tier-1 center and routed to BSC compute nodes through the bridge. This fully enables the connectivity of CMS HTCondor-based central infrastructure to BSC resources via the PIC HTCondor pool. Other challenges include building custom singularity images with CMS software releases, bringing conditions data to payload jobs, and custom data handling between BSC and PIC. This report describes the initial technical prototype, its deployment and tests, and future steps. A key aspect of the technique described in this contribution is that it could be universally employed in similar network-restrictive HPC environments elsewhere.