High-performance Data Analytics Research Articles

Contemporary High-Performance Computing for Big Data Applications

High-performance computing (HPC) involves leveraging parallel data processing to enhance computer performance and handle difficult tasks. HPC meets these aims by pooling computing capacity, enabling efficient, reliable, and prompt execution of even complex programs according to user demands and expectations. The rapid growth of HPDA in many sectors has led to the extension of the HPC market into new territory. HPC as well as Big Data systems differ not just in terms of technology but also in ecosystems. Extensive research in this sector has led to the emergence of various Big Data analytics models in recent years. As Big Data analytics spreads across several fields, new challenges about the usefulness of analytical paradigms also emerge. This article discusses the key analytical models, as well as the difficulties and challenges associated with high-performance data analytics. This research work aims to identify the factors influencing the integration of HPC with big data, including present and future trends. The study also proposes an architecture for big data with HPC convergence based on design principles.

Survey on Resource Management Solutions to Speed up Processing Small Files in Hadoop Cluster

High performance data analytics is a computing paradigm involving optimal placement of data, analytics and other computational resources such that superior performance is achieved with lesser resource consumption. Resource allocation and scheduling are the two major functionalities to be addressed in Hadoop clusters to satisfy the service level agreements of users for High performance data analytics applications. Though many solutions have been proposed for optimal resource allocation and scheduling, those schemes are designed for large Hadoop files. Recently with Internet of Things (IoT) convergence with big data, there is need to process large volumes of small files whose size is lower than block size of Hadoop. This creates huge storage overhead and exhausts Hadoop clusters computational resources. This survey analyzes the existing works on resource allocation and scheduling in Hadoop clusters and their suitability for small files. The aim is to identify the problems in existing resource allocation and scheduling approaches while handling small files. Based on the problems identified, prospective solution architecture is proposed.

HPC+ in the medical field: Overview and current examples.

To say data is revolutionising the medical sector would be a vast understatement. The amount of medical data available today is unprecedented and has the potential to enable to date unseen forms of healthcare. To process this huge amount of data, an equally huge amount of computing power is required, which cannot be provided by regular desktop computers. These areas can be (and already are) supported by High-Performance-Computing (HPC), High-Performance Data Analytics (HPDA), and AI (together "HPC+"). This overview article aims to show state-of-the-art examples of studies supported by the National Competence Centres (NCCs) in HPC+ within the EuroCC project, employing HPC, HPDA and AI for medical applications. The included studies on different applications of HPC in the medical sector were sourced from the National Competence Centres in HPC and compiled into an overview article. Methods include the application of HPC+ for medical image processing, high-performance medical and pharmaceutical data analytics, an application for pediatric dosimetry, and a cloud-based HPC platform to support systemic pulmonary shunting procedures. This article showcases state-of-the-art applications and large-scale data analytics in the medical sector employing HPC+ within surgery, medical image processing in diagnostics, nutritional support of patients in hospitals, treating congenital heart diseases in children, and within basic research. HPC+ support scientific fields from research to industrial applications in the medical area, enabling researchers to run faster and more complex calculations, simulations and data analyses for the direct benefit of patients, doctors, clinicians and as an accelerator for medical research.

DDS: integrating data analytics transformations in task-based workflows.

High-performance data analytics (HPDA) is a current trend in e-science research that aims to integrate traditional HPC with recent data analytic frameworks. Most of the work done in this field has focused on improving data analytic frameworks by implementing their engines on top of HPC technologies such as Message Passing Interface. However, there is a lack of integration from an application development perspective. HPC workflows have their own parallel programming models, while data analytic (DA) algorithms are mainly implemented using data transformations and executed with frameworks like Spark. Task-based programming models (TBPMs) are a very efficient approach for implementing HPC workflows. Data analytic transformations can also be decomposed as a set of tasks and implemented with a task-based programming model. In this paper, we present a methodology to develop HPDA applications on top of TBPMs that allow developers to combine HPC workflows and data analytic transformations seamlessly. A prototype of this approach has been implemented on top of the PyCOMPSs task-based programming model to validate two aspects: HPDA applications can be seamlessly developed and have better performance than Spark. We compare our results using different programs. Finally, we conclude with the idea of integrating DA into HPC applications and evaluation of our method against Spark.

USE OF ARTIFICIAL INTELLIGENCE FOR THE GENERATION OF MEDIA CONTENT

Artificial intelligence (AI) is a powerful catalyst that enables media organizations to optimize and improve their operations. The aim of the paper is to emphasize the importance and potential of AI during high-performance data analytics and media content generation. The focus is on responsible and ethically acceptable use of AI, which imposes the need to continuously build trust, through numerous challenges, during the management of potential risks. Standardization and mandatory publication of AI product data can pave the way for increased transparency and trust building. Otherwise, the power of AI, which has a huge potential to contribute to a better quality of life, could be transformed into a suicide tool.

On the feasibility and usefulness of high-performance computing in probabilistic volcanic hazard assessment: An application to tephra hazard from Campi Flegrei

For active volcanoes, knowledge about probabilities of eruption and impacted areas becomes valuable information for decision-makers to develop short- and long-term emergency plans, for which probabilistic volcanic hazard assessment (PVHA) is needed. High-resolution or spatially extended PVHA requires extreme-scale high-performance computing systems. Within the framework of ChEESE (Center of Excellence for Exascale in Solid Earth; www.cheese-coe.eu), an effort was made to generate exascale-suitable codes and workflows to collect and process in some hours the large amount of data that a quality PVHA requires. To this end, we created an optimized HPC-based workflow coined PVHA_HPC-WF to develop PVHA for a volcano. This tool uses the Bayesian event tree methodology to calculate eruption probabilities, vent-opening location(s), and eruptive source parameters (ESPs) based on volcano history, monitoring system data, and meteorological conditions. Then, the tool interacts with the chosen hazard model, performing a simulation for each ESP set or volcanic scenario (VS). Finally, the resulting information is processed by proof-of-concept-subjected high-performance data analytics (HPDA) scripts, producing the hazard maps which describe the probability over time of exceeding critical thresholds at each location in the investigated geographical domain. Although PVHA_HPC-WF can be adapted to other hazards, we focus here on tephra (i.e., lapilli and ash) transport and deposition. As an application, we performed PVHA for Campi Flegrei (CF), Italy, an active volcano located in one of the most densely inhabited areas in Europe and under busy air traffic routes. CF is currently in unrest, classified as being in an attention level by the Italian Civil Protection. We consider an approximate 2,000 × 2,000 × 40 km computational domain with 2 km grid resolution in the horizontal and 40 vertical levels, centered in CF. To explore the natural variability and uncertainty of the eruptive conditions, we consider a large number of VSs allowing us to include those of low probability but high impact, and simulations of tephra dispersal are performed for each of them using the FALL3D model. Results show the potential of HPC to timely execute a vast range of simulations of complex numerical models in large high-resolution computational domains and analyze great volumes of data to obtain quality hazard maps.

MISPR: an open-source package for high-throughput multiscale molecular simulations

Computational tools provide a unique opportunity to study and design optimal materials by enhancing our ability to comprehend the connections between their atomistic structure and functional properties. However, designing materials with tailored functionalities is complicated due to the necessity to integrate various computational-chemistry software (not necessarily compatible with one another), the heterogeneous nature of the generated data, and the need to explore vast chemical and parameter spaces. The latter is especially important to avoid bias in scattered data points-based models and derive statistical trends only accessible by systematic datasets. Here, we introduce a robust high-throughput multi-scale computational infrastructure coined MISPR (Materials Informatics for Structure–Property Relationships) that seamlessly integrates classical molecular dynamics (MD) simulations with density functional theory (DFT). By enabling high-performance data analytics and coupling between different methods and scales, MISPR addresses critical challenges arising from the needs of automated workflow management and data provenance recording. The major features of MISPR include automated DFT and MD simulations, error handling, derivation of molecular and ensemble properties, and creation of output databases that organize results from individual calculations to enable reproducibility and transparency. In this work, we describe fully automated DFT workflows implemented in MISPR to compute various properties such as nuclear magnetic resonance chemical shift, binding energy, bond dissociation energy, and redox potential with support for multiple methods such as electron transfer and proton-coupled electron transfer reactions. The infrastructure also enables the characterization of large-scale ensemble properties by providing MD workflows that calculate a wide range of structural and dynamical properties in liquid solutions. MISPR employs the methodologies of materials informatics to facilitate understanding and prediction of phenomenological structure–property relationships, which are crucial to designing novel optimal materials for numerous scientific applications and engineering technologies.

On the HPC/HPDA/AI Competences in Bulgaria

EU member states and the European commission realized the importance of High Performance Computing (HPC) and its applications in High- Performance Data Analytics (HPDA) and Artificial Intelligence (AI) and provided substantial funding for buying hardware and developing the national competences in the area. The EuroCC project (https://www.eurocc-access.eu/) supported the establishment of National Competence Centres in HPC/HPDA/AI, which serve as a focus point for access to expertise and know-how in the area and foster the European collaboration in the area. Bulgaria tries to maintain a leading position in the area of HPC in the region and thus it is important to collect and analyse information about the available competences in the field, the demand from the research community and the private sector, and to map ways for future development. The paper gives an overview on the current competences in HPC, HPDA and AI in Bulgaria and provides information about the plans for future development.

DDS: integrating data analytics transformations in task-based workflows

High-performance data analytics (HPDA) is a current trend in e-science research that aims to integrate traditional HPC with recent data analytic frameworks. Most of the work done in this field has focused on improving data analytic frameworks by implementing their engines on top of HPC technologies such as Message Passing Interface. However, there is a lack of integration from an application development perspective. HPC workflows have their own parallel programming models, while data analytic (DA) algorithms are mainly implemented using data transformations and executed with frameworks like Spark. Task-based programming models (TBPMs) are a very efficient approach for implementing HPC workflows. Data analytic transformations can also be decomposed as a set of tasks and implemented with a task-based programming model.In this paper, we present a methodology to develop HPDA applications on top of TBPMs that allow developers to combine HPC workflows and data analytic transformations seamlessly. A prototype of this approach has been implemented on top of the PyCOMPSs task-based programming model to validate two aspects: HPDA applications can be seamlessly developed and have better performance than Spark. We compare our results using different programs. Finally, we conclude with the idea of integrating DA into HPC applications and evaluation of our method against Spark.

TOURISM 4.0 TOOLS FOR FACILITATION OF SUSTAINABLE TOURISM DATA COMPARISON (USING A CASE STUDY FROM VYLKOVE AND SFÂNTU GHEORGHE IN THE DANUBE DELTA)

Topicality. Tourism is recognized as one of the most important commercial activities today. In 2019 it generated 10% of the total employment and represented a share of 10.4% of global GDP. Following the lead of cities, a growing number of towns and villages in rural areas are seeking to benefit from tourism, based on their local cultural and natural assets. This trend can be observed in the transboundary Danube Delta Biosphere Reserve of Romania and Ukraine where many formerly remote settlements have started to host increasing numbers of visitors. However, tourism can also impose a number of negative economic, social and environmental impacts on the destination and its region that could undermine the very attractiveness of the destinations concerned. Aims and tasks. A particular problem for authorities charged with developing regional rural tourism strategies and regulations is to have a standard method for assessing and comparing the visitor capacity and sustainability of different destinations. Emerging Tourism 4.0 technologies such as High-Performance Data Analytics (HPDA) can improve the efficiency and effectiveness of strategic planning and environmental sustainability. The general objective of the research reported here was to compare the current situation of two similar settlements in the Danube Delta Biosphere Reserve: Vylkove in Ukraine and Sfântu Gheorghe in Romania. In collaboration with the municipal councils, local businesses and civic organisations, we applied the Tourism Impact Model (TIM) developed by the Slovene company Arctur to data collected using the same methodology between August 2020 and May 2021. Overall, 295 questions were addressed. The data could be quantitative (amount of electricity or water used per day), or more subjective expert opinion (whether and when the settlement suffered from traffic congestion or satisfaction of residents with levels of incoming tourists). The data were also quality controlled and labelled according to their accuracy, type (digital or analogue) and frequency of collection. Research results. With respect to Vylkove and Sfântu Gheorghe, in both cases the tourism data collected were not sufficient to produce a full analysis for an overall DCC assessment. However, it was possible to examine the data for each of the main TIM pillars individually (Environment, Economy, Social / Cultural and Collaboration) and make a comparison between the destinations. From these data, it is clear that while both destinations are generally managing tourism poorly (as Sleepers), Sfântu Gheorghe has a better overall condition than Vylkove across the four pillars since it scores higher in the Champion character. Vylkove suffers from having a large amount of missing data, probably because the town council derives little direct benefit from tourism (it is a receiver of impacts but most of the revenue goes to external parties) so it does not collect information. On the other hand, compared with Sfântu Gheorghe, it does not misuse or exploit its environment. Both destinations have a good level of collaboration between the tourism sector and the local community. Conclusion. The results from the first application of the Tourism Impact Model in Vylkove and Sfântu Gheorghe show that both destinations have considerable potential for improving their performance and the sustainability of their tourism offers. At present, the lead is mostly taken by private initiative and investment, with little involvement of the local authorities. In particular, environmental, social and economic data are largely unavailable for planning sustainable tourism development, gaining benefits or mitigating impacts. These aspects merit further action and research as a matter of urgency.

IoT data analytics architecture for smart healthcare using RFID and WSN

The importance of big data analytics has become apparent with the increasing volume of data on the Internet. The amount of data will increase even more with the widespread use of Internet of Things (IoT). One of the most important application areas of the IoT is healthcare. This study introduces new real-time data analytics architecture for an IoT-based smart healthcare system, which consists of a wireless sensor network and a radio-frequency identification technology in a vertical domain. The proposed platform also includes high-performance data analytics tools, such as Kafka, Spark, MongoDB, and NodeJS, in a horizontal domain. To investigate the performance of the system developed, a diagnosis of Wolff–Parkinson–White syndrome by logistic regression is discussed. The results show that the proposed IoT data analytics system can successfully process health data in real-time with an accuracy rate of 95% and it can handle large volumes of data. The developed system also communicates with a riverbed modeler using Transmission Control Protocol (TCP) to model any IoT-enabling technology. Therefore, the proposed architecture can be used as a time-saving experimental environment for any IoT-based system.

PRELIMINARY APPLICATION OF TOURISM 4.0 DATA ANALYTICS IN ODESSA CITY REVEALS CHALLENGES AND OPPORTUNITIES FOR SUSTAINABLE TOURISM DEVELOPMENT

Topicality. Modern tourism is recognized as one of the most important commercial activities globally. In 2019, it generated 10% of the total employment and represented a share of 10.4% of global GDP. The tourism sector in Ukraine as a whole saw slow development after independence in 1991. However, the introduction of a visa-free regime in 2005 boosted the country’s global image and visitor numbers. In 2013, Ukraine was visited by over 26 million tourists, primarily from Eastern Europe, but also from Western Europe, USA, Israel and Canada. However, tourism can impose a number of negative economic, social and environmental impacts on the destination and its region.Aims and tasks. Unfortunately, there remains a persistent gap between the tourism sector (both regulators and operators) and the appreciation and use of emerging technologies such as those applied in Tourism 4.0 that can improve its economic efficiency and environmental sustainability. The general objective of the research reported here was to test the current level and effectiveness of Tourism 4.0 technologies (or more specifically High-Performance Data Analytics - HPDA) under the conditions of a large urban coastal destination with a highly diverse economy that is not solely dependent on tourism. Data collection for the TIM took place between August 2020 and May 2021, in collaboration with the Odessa City Council Department for Culture and Tourism. Overall, 295 questions were addressed. The data could be quantitative (amount of electricity or water used per day), or more subjective expert opinion (whether and when the city suffers from traffic congestion or satisfaction of residents with levels of incoming tourists). The data was also quality controlled and labelled according to its accuracy, type (digital or analogue) and frequency of collection. The target baseline year was 2019, with data from 2018 and 2017 obtained where available for trend analysis. In addition, expert estimations of values for 2020 were also made in order to forecast future demands. Data sources fell into three broad categories: (i) government agencies at regional or national level, including the State Statistics Service of Ukraine and Ministry of Justice; (ii) private enterprises; and (3) civil society organisations.Research results. Among the results, it was found that power consumption did not significantly increase during the summer as a result of increased visitors. In addition, the revenue directly generated from a tourist tax levied on registered accommodation providers in 2019 amounted to Euro 393,100, which was only 0.11% of the city’s total budget of Euro 344,947,580. The amount contributed indirectly from other service providers (catering, entertainment, retail, transport etc) through employment and profit taxes was unknown, as was the amount lost in the informal economy.Conclusion. Through a better understanding of current patterns of tourist visits, visitor demographics and revenues, infrastructure use, resource consumption and stakeholder collaboration, the study aimed to spur innovative touristic services and policies tailored to the local challenges and opportunities. While useful insights were obtained from the TIM analysis, it proved impossible at the present time to create a robust overall model of the tourism sector in the city owing to numerous issues concerning the availability and quality of the data needed.

JURECA: Data Centric and Booster Modules implementing the Modular Supercomputing Architecture at Jülich Supercomputing Centre

JURECA is a Pre-Exascale Modular Supercomputer operated by Jülich Supercomputing Centre at Forschungszentrum Jülich. The system combines a flexible Data Centric (DC) module, based on the Atos BullSequana XH2000 with a selection of best-of-its-kind components, and a scalability-focused Booster module, delivered by Intel and Dell Technologies based on the Xeon Phi many-core processor. With its novel architecture, it supports a wide variety of high-performance computing and data analytics workloads.

CultivData: Application of IoT to the Cultivation of Agricultural Data

CultivData proposes the convergence of technologies, such as IoT, big data, HPC, open data and artificial intelligence, to apply HPDA (High Performance Data Analytics) to the cultivation of agricultural data and improve the efficiency and effectiveness of farms. An information system has been developed as an IT platform for the cultivation of open data to extract knowledge and to support the decision making of stakeholders in the agricultural sector, so that it is possible to improve product quality and farm productivity. The resulting system integrates access to data provided by IoT devices that sensorize farms and public and open data sources (Open Data). The platform was designed to make precision agriculture a reality and to be useful not only to farmers, but also to agricultural decision-makers who plan species and crops based on data such as available water; expected weather; prices and market demands, and so forth. In addition, the platform provides to agricultural producers access to historical climate data; climate forecasts to anticipate times of drought or disasters; pest situations or monitoring of their plantations with sensorization and orthophotographs.

Application of HPC and big data in post-pandemic times

We lay out the ramifications of the 2020 pandemic for all people in geosciences, especially the young, and argue for significant changes on training and career development. We focus primarily on its devastating impact in USA and compare with that in other countries especially China. We review the potential effect for the next four years or so on the aspirations of an academic career versus more realistic career goals. We urge people in mid-career about the need to reassess previous goals. We stress the need for students or researchers to acquire technical skills in high-performance computing (HPC), data analytics, artificial intelligence, and/or visualization along with a broad set of technical skills in applied computer science and mathematics. We give advice about hot prospects in several areas that have great potential for advancement in the coming decade, such as visualization, deep learning, quantum computing and information, and cloud computing, all of which lie within the aegis of HPC. Our forecast is that the pandemic will significantly reshape the job landscape and career paths for both young and established researchers and we discuss bluntly the dire situation facing junior people in geosciences in the aftermath of the pandemic around the world until 2024.

ARENA: Asynchronous Reconfigurable Accelerator Ring to Enable Data-Centric Parallel Computing

The next generation HPC and data centers are likely to be reconfigurable and data-centric due to the trend of hardware specialization and the emergence of data-driven applications. In this article, we propose ARENA - an asynchronous reconfigurable accelerator ring architecture as a potential scenario on how the future HPC and data centers will be like. Despite using the coarse-grained reconfigurable arrays (CGRAs) as the substrate platform, our key contribution is not only the CGRA-cluster design itself, but also the ensemble of a new architecture and programming model that enables asynchronous tasking across a cluster of reconfigurable nodes, so as to bring specialized computation to the data rather than the reverse. We presume distributed data storage without asserting any prior knowledge on the data distribution. Hardware specialization occurs at runtime when a task finds the majority of data it requires are available at the present node. In other words, we dynamically generate specialized CGRA accelerators where the data reside. The asynchronous tasking for bringing computation to data is achieved by circulating the task token, which describes the dataflow graphs to be executed for a task, among the CGRA cluster connected by a fast ring network. Evaluations on a set of HPC and data-driven applications across different domains show that ARENA can provide better parallel scalability with reduced data movement (53.9 percent). Compared with contemporary compute-centric parallel models, ARENA can bring on average 4.37× speedup. The synthesized CGRAs and their task-dispatchers only occupy 2.93mm$^2$2<; inline-graphic xlink:href="tan-ieq1-3081074.gif"/> chip area under 45nm process technology and can run at 800MHz with on average 759.8mW power consumption. ARENA also supports the concurrent execution of multi-applications, offering ideal architectural support for future high-performance parallel computing and data analytics systems.

A Scalable Machine Learning Pipeline for Paddy Rice Classification Using Multi-Temporal Sentinel Data

The demand for rice production in Asia is expected to increase by 70% in the next 30 years, which makes evident the need for a balanced productivity and effective food security management at a national and continental level. Consequently, the timely and accurate mapping of paddy rice extent and its productivity assessment is of utmost significance. In turn, this requires continuous area monitoring and large scale mapping, at the parcel level, through the processing of big satellite data of high spatial resolution. This work designs and implements a paddy rice mapping pipeline in South Korea that is based on a time-series of Sentinel-1 and Sentinel-2 data for the year of 2018. There are two challenges that we address; the first one is the ability of our model to manage big satellite data and scale for a nationwide application. The second one is the algorithm’s capacity to cope with scarce labeled data to train supervised machine learning algorithms. Specifically, we implement an approach that combines unsupervised and supervised learning. First, we generate pseudo-labels for rice classification from a single site (Seosan-Dangjin) by using a dynamic k-means clustering approach. The pseudo-labels are then used to train a Random Forest (RF) classifier that is fine-tuned to generalize in two other sites (Haenam and Cheorwon). The optimized model was then tested against 40 labeled plots, evenly distributed across the country. The paddy rice mapping pipeline is scalable as it has been deployed in a High Performance Data Analytics (HPDA) environment using distributed implementations for both k-means and RF classifiers. When tested across the country, our model provided an overall accuracy of 96.69% and a kappa coefficient 0.87. Even more, the accurate paddy rice area mapping was returned early in the year (late July), which is key for timely decision-making. Finally, the performance of the generalized paddy rice classification model, when applied in the sites of Haenam and Cheorwon, was compared to the performance of two equivalent models that were trained with locally sampled labels. The results were comparable and highlighted the success of the model’s generalization and its applicability to other regions.

Towards HPC and Big Data Analytics Convergence: Design and Experimental Evaluation of a HPDA Framework for eScience at Scale

Over the last two decades, scientific discovery has increasingly been driven by the large availability of data from a multitude of sources, including high-resolution simulations, observations and instruments, as well as an enormous network of sensors and edge components. In such a dynamic and growing landscape where data continue to expand, advances in Science have become intertwined with the capacity of analysis tools to effectively handle and extract valuable information from this ocean of data. In view of the exascale era of supercomputers that is rapidly approaching, it is of the utmost importance to design novel solutions that can take full advantage of the upcoming computing infrastructures. The convergence of High Performance Computing (HPC) and data-intensive analytics is key to delivering scalable High Performance Data Analytics (HPDA) solutions for scientific and engineering applications. The aim of this paper is threefold: reviewing some of the most relevant challenges towards HPDA at scale, presenting a HPDA-enabled version of the Ophidia framework and validating the scalability of the proposed framework through an experimental performance evaluation carried out in the context of the Centre of Excellence in Simulation of Weather and Climate in Europe (ESiWACE). The experimental results show that the proposed solution is capable of scaling over several thousand cores and hundreds of cluster nodes. The proposed work is a contribution in support of scientific large-scale applications along the wider convergence path of HPC and Big Data followed by the scientific research community.

A Hybrid Learning Approach to Prognostics and Health Management Applied to Military Ground Vehicles Using Time-Series and Maintenance Event Data

Attempts to leverage operational time-series data in Condition Based Maintenance (CBM) approaches to optimize the life cycle management and Reliability, Availability, and Maintainability (RAM) of military vehicles have encountered several obstacles over decades of data collection. These obstacles have beset similar approaches on civilian ground vehicles, as well as on aircraft and other complex systems. Analysis of operational data is critical because it represents a continuous recording of the state of the system. Applying rudimentary data analytics to operational data can provide insights like fuel usage patterns or observed reliability of one vehicle or even a fleet. Monitoring trends and analyzing patterns in this data over time, however, can provide insight into the health of a vehicle, a complex system, or a fleet, predicting mean time to failure or compiling logistic or life cycle needs. Such High-Performance Data Analytics (HPDA) on operational time-series datasets has been historically difficult due to the large amount of data gathered from vehicle sensors, the lack of association between clusters observed in the data and failures or unscheduled maintenance events, and the deficiency of unsupervised learning techniques for time-series data. We present an HPDA environment and a method of discovering patterns in vehicle operational data that determines models for predicting the likelihood of imminent failure, referred to as Parameter-Based Indicators (PBIs). Our method is a data-driven approach that uses both time-series and relational maintenance data. This hybrid approach combines both supervised and unsupervised machine learning and data analytic techniques to correlate labeled, relational maintenance event data with unlabeled operational time-series data utilizing the DoD High Performance Computing (HPC) capabilities at the U.S. Army Engineer Research and Development Center. In leveraging both time-series and relational data, we demonstrate a means of fast, purely data-driven model creation that is more broadly applicable and requires less a priori information than physics informed, data-driven models. By blending these approaches, this system will be able to relate some lifecycle management goals through the workflow to generate specific PBIs that will predict failures or highlight appropriate areas of concern in individual or collective vehicle histories.

AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance

The ever-growing availability of computing power and the sustained development of advanced computational methods have contributed much to recent scientific progress. These developments present new challenges driven by the sheer amount of calculations and data to manage. Next-generation exascale supercomputers will harden these challenges, such that automated and scalable solutions become crucial. In recent years, we have been developing AiiDA (aiida.net), a robust open-source high-throughput infrastructure addressing the challenges arising from the needs of automated workflow management and data provenance recording. Here, we introduce developments and capabilities required to reach sustained performance, with AiiDA supporting throughputs of tens of thousands processes/hour, while automatically preserving and storing the full data provenance in a relational database making it queryable and traversable, thus enabling high-performance data analytics. AiiDA’s workflow language provides advanced automation, error handling features and a flexible plugin model to allow interfacing with external simulation software. The associated plugin registry enables seamless sharing of extensions, empowering a vibrant user community dedicated to making simulations more robust, user-friendly and reproducible.