Scalable Data Analysis Research Articles

Improving PHIRI performance and scalability: working within EGI-ACE

Abstract EGI-ACE is a 30-month H2020 project (Jan 2021 - June 2023) with a mission to empower researchers from all disciplines to collaborate in data- and compute-intensive Open Science, enabled by free-at-point-of-use services that are delivered through the European Open Science Cloud (EOSC). EGI-ACE delivers the EOSC Compute Platform (ECP), a federated system of compute and storage infrastructure extended with platform services to support diverse types of data processing and data analytics cases. The ECP currently includes High Throughput Compute (HTC) and Cloud Compute facilities, and will broaden its scope with High Performance Compute services later in 2022. The platform layer of the ECP provides assistance for single sign-on, transfer and federation of distributed data, interactive computing, management of large numbers of jobs, orchestration of compute clusters, AI and machine learning tasks. There are over 25 thematic services in EOSC that build on the ECP, and deliver scalable data analysis for different domains, from astrophysics, through life sciences, environmental sciences, to humanities. PHIRI participates in EGI-ACE as one of the ‘Early Adopters’ of the ECP. Under the EGI-ACE workplan PHIRI will explore reproducible population health workflows with the use of cloud computing, single-sign-on, Jupyter Notebooks and Binder services of the ECP. The tests will enable PHIRI to scale out existing data analysis notebooks to big capacity machines, to reproduce simulations and models across users, and to overall validate the technological and sustainability approaches of EGI-ACE. PHIRI will also advise the project on best ways to introduce secure processing capabilities within the ECP services.

An Adaptive Elastic Multi-model Big Data Analysis and Information Extraction System

With the diverse applications to industry and domain-specific context, multi-source information extraction on semi-structured and unstructured data, as well as across data models, is becoming more common. However, multi-model information extraction often requires the deployment of multiple data model management, storage, and analysis subsystems on the cloud, many subsystems are not high-resource utilization at the same time, and the resource waste phenomenon is often serious. Therefore, an adaptive scalable multi-model big data analysis and information extraction system is designed and implemented in this paper, which can support data maintenance and cross-model query of relational, graph, document, key and other data models, and can provide efficient cross-model information extraction. On this basis, we can achieve the system resource allocation on demand and fast scaling mechanism, according to the real-time requirements of multi-model big data analysis, and dynamic adjustment of each subsystem resource allocation. Therefore, our solution not only guarantees multi-model query and information extraction performance and quality of service, but also significantly reduces the total consumption of system resources and cost.

Sensitivity estimation for dark matter subhalos in synthetic Gaia DR2 using deep learning

The abundance of dark matter subhalos orbiting a host galaxy is a generic prediction of the cosmological framework, and is a promising way to constrain the nature of dark matter. In this paper, we investigate the use of machine learning-based tools to quantify the magnitude of phase-space perturbations caused by the passage of dark matter subhalos. A simple binary classifier and an anomaly detection model are proposed to estimate if stars or star particles close to dark matter subhalos are statistically detectable in simulations. The simulated datasets are three Milky Way-like galaxies and nine synthetic Gaia DR2 surveys derived from these. Firstly, we find that the anomaly detection algorithm, trained on a simulated galaxy with full 6D kinematic observables and applied on another galaxy, is nontrivially sensitive to the dark matter subhalo population. On the other hand, the classification-based approach is not sufficiently sensitive due to the extremely low statistics of signal stars for supervised training. Finally, the sensitivity of both algorithms in the Gaia-like surveys is negligible. The enormous size of the Gaia dataset motivates the further development of scalable and accurate data analysis methods that could be used to select potential regions of interest for dark matter searches to ultimately constrain the Milky Way’s subhalo mass function, as well as simulations where to study the sensitivity of such methods under different signal hypotheses.

A new strategy for efficient retrospective data analyses for designer benzodiazepines in large LC-HRMS datasets

To develop a retrospective data analysis (RDA) workflow tailored to benzodiazepines and use this to identify designer and uncommon benzodiazepines (DBZD) in blood samples from driving-under-the-influence-of-drugs (DUID) cases analyzed from 2014 to 2020. The RDA workflow was developed from a training set of hits in ultra-high performance liquid chromatography–quadrupole time-of-flight–mass spectrometry (UHPLC-QTOF-MS) data files, corresponding to 13 common benzodiazepines that also had been analyzed with a complementary UHPLC- tandem mass spectrometry (MS/MS) method. Raw data from all UHPLC-QTOF-MS data files were available from a structured query language (SQL) data archive. Quantitative results in the training set were used as the true condition to evaluate whether a hit in the UHPLC-QTOF-MS data file was true or false positive. The training set was used to evaluate and set filters for the RDA. The RDA was finally used to screen for 47 DBZD in 13514 UHPLC-QTOF-MS data files from DUID cases analysed from 2014 to 2020, with filters on retention time window, count level, and mass error. Filters for retention time and mass filters was narrower and the count limit lower than what is used for our in-house drug screening method. Additional filters for diagnostic fragment ions, isotopic pattern, and residual precursor ion in high-energy spectra were evaluated, but not used in the final RDA. Extracting hits from 13514 LC-HRMS data files with simple count, mass, and retention time filters for 47 compounds was executed in less than one minute. Sixteen DBZD were detected in the UHPLC-QTOF-MS data files. 47 identifications had been confirmed by complementary methods when the cases were open (confirmed positive finding), and 43 targets were not reported, when the cases were open (tentative positive finding). Metabolites were not considered targets, once their parent drug was detected. The most common tentative and confirmed findings were etizolam ( n = 26), phenazepam ( n = 13), lorazepam ( n = 9), and flualprazolam ( n = 8). This method efficiently found DBZD in previously acquired UHPLC-QTOF-MS data files, with only 9 false positive hits. Retention time is a required filter in the RDA workflow and therefore relies on availability of the standard or a clean seizure. Alternatively, a workflow relying on diagnostic fragment ions could allow the search for emerging DBZD. Using predicted or online sourced analytical data, for ranking of hits, was not necessary considering the low number of false positive hits. The sensitivity and selectivity can be attributed to the distinguishable chemical nature of benzodiazepines. The RDA workflow can easily be adjusted to other groups of NPS, but would require a new training set and other filter settings. The three most frequently detected DBZD in this study have all been placed under international control. The prevalence of detected DBZD from 2014-2020 remains very low in DUID samples in Eastern Denmark when comparing with conventional benzodiazepines. The overall framework presented in this study to tailor RDA workflows to specific groups of compounds from historic data of similar compounds We developed an efficient and scalable retrospective data analysis workflow for DBZD that can easily be tailored and optimized for other groups of NPS.

An efficient and scalable data analysis solution for automated electrophysiology platforms.

Ion channels are drug targets for neurologic, cardiac, and immunologic diseases. Many disease-associated mutations and drugs modulate voltage-gated ion channel activation and inactivation, suggesting that characterizing state-dependent effects of test compounds at an early stage of drug development can be of great benefit. Historically, the effects of compounds on ion channel biophysical properties and voltage-dependent activation/inactivation could only be assessed by using low-throughput, manual patch clamp recording techniques. In recent years, automated patch clamp (APC) platforms have drastically increased in throughput. In contrast to their broad utilization in compound screening, APC platforms have rarely been used for mechanism of action studies, in large part due to the lack of sophisticated, scalable analysis methods for processing the large amount of data generated by APC platforms. In the current study, we developed a highly efficient and scalable software workflow to overcome this challenge. This method, to our knowledge the first of its kind, enables automated curve fitting and complex analysis of compound effects. Using voltage-gated sodium channels as an example, we were able to immediately assess the effects of test compounds on a spectrum of biophysical properties, including peak current, voltage-dependent steady state activation/inactivation, and time constants of activation and fast inactivation. Overall, this automated data analysis method provides a novel solution for in-depth analysis of large-scale APC data, and thus will significantly impact ion channel research and drug discovery.

A New Strategy for Efficient Retrospective Data Analyses for Designer Benzodiazepines in Large LC-HRMS Datasets.

The expanding and dynamic market of new psychoactive substances (NPSs) poses challenges for laboratories worldwide. The retrospective data analysis (RDA) of previously analyzed samples for new targets can be used to investigate analytes missed in the first data analysis. However, RDA has historically been unsuitable for routine evaluation because reprocessing and reevaluating large numbers of forensic samples are highly work- and time-consuming. In this project, we developed an efficient and scalable retrospective data analysis workflow that can easily be tailored and optimized for groups of NPSs. The objectives of the study were to establish a retrospective data analysis workflow for benzodiazepines in whole blood samples and apply it on previously analyzed driving-under-the-influence-of-drugs (DUID) cases. The RDA workflow was based on a training set of hits in ultrahigh-performance liquid chromatography–quadrupole time-of-flight–mass spectrometry (UHPLC-QTOF-MS) data files, corresponding to common benzodiazepines that also had been analyzed with a complementary UHPLC–tandem mass spectrometry (MS/MS) method. Quantitative results in the training set were used as the true condition to evaluate whether a hit in the UHPLC-QTOF-MS data file was true or false positive. The training set was used to evaluate and set filters. The RDA was used to extract information from 47 DBZDs in 13,514 UHPLC-QTOF-MS data files from DUID cases analyzed from 2014 to 2020, with filters on the retention time window, count level, and mass error. Sixteen designer and uncommon benzodiazepines (DBZDs) were detected, where 47 identifications had been confirmed by using complementary methods when the case was open (confirmed positive finding), and 43 targets were not reported when the case was open (tentative positive finding). The most common tentative and confirmed findings were etizolam (n = 26), phenazepam (n = 13), lorazepam (n = 9), and flualprazolam (n = 8). This method efficiently found DBZDs in previously acquired UHPLC-QTOF-MS data files, with only nine false-positive hits. When the standard of an emerging DBZD becomes available, all previously acquired DUID data files can be screened in less than 1 min. Being able to perform a fast and accurate retrospective data analysis across previously acquired data files is a major technological advancement in monitoring NPS abuse.

Toward Reference Architectures: A Cloud-Agnostic Data Analytics Platform Empowering Autonomous Systems

This work introduces a scalable, cloud-agnostic and fault-tolerant data analytics platform for state-of-the-art autonomous systems that is built from open-source, reusable building blocks. As the baseline for further new reference architectures, it represents an architecture blueprint for processing, enriching and analyzing various feeds of structured and non-structured input data from advanced Internet-of-Things (IoT) based use cases. The platform builds on industry best practices, leverages on solid open-source components in a reusable fashion, and is based on our experience gathered from numerous IoT and Big Data research projects. The platform is currently used in the framework of the National Laboratory for Autonomous Systems in Hungary (abbreviated as ARNL). The platform is demonstrated through selected use cases from ARNL including the areas of smart/autonomous production systems (collaborative robotic assembly) and autonomous vehicles (mobile robots with smart vehicle control). Finally, we validate the platform through the evaluation of its streaming ingestion capabilities.

A Survey on Big Data Processing Frameworks for Mobility Analytics

In the current era of big spatial data, the vast amount of produced mobility data (by sensors, GPS-equipped devices, surveillance networks, radars, etc.) poses new challenges related to mobility analytics. A cornerstone facilitator for performing mobility analytics at scale is the availability of big data processing frameworks and techniques tailored for spatial and spatio-temporal data. Motivated by this pressing need, in this paper, we provide a survey of big data processing frameworks for mobility analytics. Particular focus is put on the underlying techniques; indexing, partitioning, query processing are essential for enabling efficient and scalable data management. In this way, this report serves as a useful guide of state-of-the-art methods and modern techniques for scalable mobility data management and analytics.

Toward reproducible, scalable, and robust data analysis across multiplex tissue imaging platforms

SummaryThe emergence of megascale single-cell multiplex tissue imaging (MTI) datasets necessitates reproducible, scalable, and robust tools for cell phenotyping and spatial analysis. We developed open-source, graphics processing unit (GPU)-accelerated tools for intensity normalization, phenotyping, and microenvironment characterization. We deploy the toolkit on a human breast cancer (BC) tissue microarray stained by cyclic immunofluorescence and present the first cross-validation of breast cancer cell phenotypes derived by using two different MTI platforms. Finally, we demonstrate an integrative phenotypic and spatial analysis revealing BC subtype-specific features.

Supercomputing in Python With Legate

Legate is a recently developed software system for constructing scalable simulation and data analysis programs using convenient, familiar notation. It demonstrates how coupling Python, with a runtime system originally designed for high-performance computing, can enable the creation of libraries that mimic the familiar interface of NumPy and Pandas for execution on both desktops and supercomputers.

ROVEC: Runtime Optimization of Vectorized Expression Evaluation for Column Store

Due to the increasing demand for scalable and interactive data analytics, column stores have become the de-facto choice in many analytical databases. As a common and fundamental operation in column stores, expression evaluation has a remarkable effect on many queries. To speed up expression evaluation, vectorized techniques such as Single-Instruction-Multiple-Data (SIMD) instructions are widely used. However, there are few works concerning dedicated optimizations for SIMD-based expression evaluation for column stores. In this paper, we propose a runtime optimization framework named ROVEC that enables effective optimizations for SIMD-based expression evaluation. The key idea is to optimize logical expression at execution time, by leveraging lightweight compression and fine-grained statistics associated with the compressed data. ROVEC removes unnecessary type casting and finds the tightest type during evaluation, which maximizes the concurrent operands in SIMD instructions. ROVEC can be applied to many expression-evaluation-intensive operators (e.g., table scan and theta join) for different data types (e.g., numeric, time and string). To validate the effectiveness of ROVEC, we integrate it into a columnar database PolarDB-C. Our evaluation results show that ROVEC improves up to 120% (60% on average) throughput of table scan and up to 50% (30% on average) latency of theta join.

Cloud Computing in Remote Sensing: Big Data Remote Sensing Knowledge Discovery and Information Analysis

With the rapid development of remote sensing technology, our ability to obtain remote sensing data has been improved to an unprecedented level. We have entered an era of big data. Remote sensing data clear showing the characteristics of Big Data such as hyper spectral, high spatial resolution, and high time resolution, thus, resulting in a significant increase in the volume, variety, velocity and veracity of data. This paper proposes a feature supporting, salable, and efficient data cube for time-series analysis application, and used the spatial feature data and remote sensing data for comparative study of the water cover and vegetation change.The spatial-feature remote sensing data cube (SRSDC) is described in this paper. It is a data cube whose goal is to provide a spatial-feature-supported, efficient, and scalable multidimensional data analysis system to handle large-scale RS data. It provides a high-level architectural overview of the SRSDC.The SRSDC offers spatial feature repositories for storing and managing vector feature data, as well as feature translation for converting spatial feature information to query operations.The paper describes the design and implementation of a feature data cube and distributed execution engine in the SRSDC. It uses the long time-series remote sensing production process and analysis as examples to evaluate the performance of a feature data cube and distributed execution engine. Big data has become a strategic highland in the knowledge economy as a new strategic resource for humans. The core knowledge discov-ery methods include supervised learning methods data analysis supervised learning, unsupervised learning methods data analysis unsupervised learning, and their combinations and variants.

Research on intelligent construction of Hydropower Enterprises

Hydropower is the largest clean and low-carbon renewable energy, which can generate electricity with comprehensive utilization functions such as flood control, water supply, shipping and irrigation, and has significant social, ecological and economic benefits. However, at present, the progress of intelligent construction of hydropower enterprises is relatively slow, and the theoretical and strategic research is still relatively weak, resulting in low management efficiency and low degree of production visualization. How to carry out intelligent enterprise construction for traditional hydropower enterprises, improve the management level of enterprises and better adapt to the competitive environment of power market is a major research topic for hydropower managers. This paper mainly expounds the necessity, strategic framework and overall construction scheme of intelligent construction of hydropower enterprises. Through the application of new technologies such as cloud computing, big data and Internet of things, an efficient and scalable unified data management and analysis platform is established to provide guarantee for safe, economic, efficient and intelligent operation of hydropower enterprises and provide basis for command and decision-making.

Open Source Framework for Enabling HPC and Cloud Geoprocessing Services

Geoprocessing is a set of tools that can be used to efficiently address several pressing chal-lenges for the global economy ranging from agricultural productivity, the design of transport networks, to the prediction of climate change and natural disasters. This paper describes an Open Source Framework developed, within three European projects, for Ena-bling High-Performance Computing (HPC) and Cloud geoprocessing services applied to agricultural challenges. The main goals of the European Union projects EUXDAT (EUro-pean e-infrastructure for eXtreme Data Analytics in sustainable developmenT), CYBELE (fostering precision agriculture and livestock farming through secure access to large-scale HPC-enabled virtual industrial experimentation environment empowering scalable big data analytics), and EOPEN (opEn interOperable Platform for unified access and analysis of Earth observatioN data) are to enable the use of large HPC systems, as well as big data management, user-friendly access and visualization of results. In addition, these projects focus on the development of software frameworks, and fuse Earth-observation data, such as Copernicus data, with non-Earth-observation data, such as weather, environmental and social media information. In this paper, we describe the agroclimatic-zones pilot used to validate the framework. Finally, performance metrics collected during the execution (up to 182 times speedup with 256 MPI processes) of the pilot are presented.

SQL for data scientists

The SQL query language is the 'lingua franca' of transactional databases, and is essential for scalable data analytics. Learning SQL requires practical exercises with databases, including those parts of SQL with side-effects such as DDL statements, triggers, and stored procedures. Teaching this effectively to a non-technical audience is a challenge, especially in times of COVID-19 without face-to-face classes. The Grok Learning platform allows to design self-paced online tutorials with auto-graded exercises - but it was originally built for teaching programming languages. In this demo, we show how we extended the Grok platform to teach SQL for Data Scientists with comprehensive online learning. Grok supports a rich user interface with interactive examples where students can explore and experiment with each example query. This is ideal for learning declarative querying. Each query is executed in its own sandbox on a freshly initialised database instance which allows to teach all parts of SQL including DDL statements, stored procedures, triggers and UDFs. At the same time, the platform scales to thousands of concurrent users, while maintaining interactive response times.

A Novel Automatic Way of Finding Direct and Linear Sequential Inherent Relationships in the Data

Finding linear sequential relationships (LSRs) in the data and applying them for obtaining fruitful results is an essential task in many modern day to day useful real and simulation based applications. In many previous research applications many research people usually assumed that there exists certain relationships in the data and then they have tried to bring forth some useful results after processing the selected datasets using one more data mining, machine leaning, and big data techniques. Take it for granted assumptions on the data may not be true in all the cases and in all the applications. The purpose of the present study is to bring out some automatic, smart, simple, scalable, fruitful and useful data analysis techniques after analyzing the datasets in the hand and at the same time assumptions on the data are not considered just like the general fashion of take it for granted option. The proposed model is particularly useful and applicable for finding the drug to disease relationships.

Introduction to the Special Issue on Role of Scalable Computing and Data Analytics in Evolution of Internet of Things

The evolution of Internet of Things has given way to a Smart World where there is an improved integration of devices, systems and processes in humans through all pervasive connectivity. Anytime, anywhere connection and transaction is the motto of the Internet of things which brings comfort to the users and sweeps the problem of physical boundary out of the way. Once it has come into the purview of developers, new areas have been identified and new applications have been introduced. Small wearables which can track your health to big automated vehicles which can move from one place to another self navigating without human intervention are the order of the day. This has also brought into existence a new technology called cloud, since with IoT comes a large number of devices connected to the internet continuously pumping data into the cloud for storage and processing. Another area benefited from the evolution of IoT is the wireless and wired connectivity through a wide range of connectivity standards.
 As with any technology, it has also created a lot of concerns regarding the security, privacy and ethics.
 
 Data protection issues created by new technologies are a threat which has been recognized by developers, public and also the governing body long back. The complexity of the system arises because of the various sensors and technologies which clearly tell the pattern of the activities of the individual as well an organization making us threat prone. Moreover, the volume of the data in the cloud makes it too difficult to recognize the privacy requirement of the data or to segregate open data from private data. Data analytics is another technology which supposedly increases the opportunity of increasing business by studying this private data collected from IoT and exploring ways to monetize them. It also helps the individual by recognizing their priorities and narrowing their search. But the data collected are real world data and aggregation of this data in the cloud is an open invitation to the hackers to study about the behaviors of the individuals.
 
 The special issues of Scalable Computing has attract related to the Role of Scalable Computing and Data Analytics in Evolution of Internet of Things has attracted 28 submissions from which were selected 12.
 

A dockerized framework for hierarchical frequency-based document clustering on cloud computing infrastructures

Scalable big data analysis frameworks are of paramount importance in the modern web society, which is characterized by a huge number of resources, including electronic text documents. Document clustering is an important field in text mining and is commonly used for document organization, browsing, summarization and classification. Hierarchical clustering methods construct a hierarchy structure that, combined with the produced clusters, can be useful in managing documents, thus making the browsing and navigation process easier and quicker, and providing only relevant information to the users’ queries by leveraging the structure relationships. Nevertheless, the high computational cost and memory usage of baseline hierarchical clustering algorithms render them inappropriate for the vast number of documents that must be handled daily. In this paper, we propose a new scalable hierarchical clustering framework, which uses the frequency of the topics in the documents to overcome these limitations. Our work consists of a binary tree construction algorithm that creates a hierarchy of the documents using three metrics (Identity, Entropy, Bin Similarity), and a branch breaking algorithm which composes the final clusters by applying thresholds to each branch of the tree. The clustering algorithm is followed by a meta-clustering module which makes use of graph theory to obtain insights in the leaf clusters’ connections. The feature vectors representing each document derive from topic modeling. At the implementation level, the clustering method has been dockerized in order to facilitate its deployment on cloud computing infrastructures. Finally, the proposed framework is evaluated on several datasets of varying size and content, achieving significant reduction in both memory consumption and computational time over existing hierarchical clustering algorithms. The experiments also include performance testing on cloud resources using different setups and the results are promising.

Scalable Data Analysis in Proteomics and Metabolomics Using BioContainers and Workflows Engines.

The recent improvements in mass spectrometry instruments and new analytical methods are increasing the intersection between proteomics and big data science. In addition, bioinformatics analysis is becoming increasingly complex and convoluted, involving multiple algorithms and tools. A wide variety of methods and software tools have been developed for computational proteomics and metabolomics during recent years, and this trend is likely to continue. However, most of the computational proteomics and metabolomics tools are designed as single-tiered software application where the analytics tasks cannot be distributed, limiting the scalability and reproducibility of the data analysis. In this paper the key steps of metabolomics and proteomics data processing, including the main tools and software used to perform the data analysis, are summarized. The combination of software containers with workflows environments for large-scale metabolomics and proteomics analysis is discussed. Finally, a new approach for reproducible and large-scale data analysis based on BioContainers and two of the most popular workflow environments, Galaxy and Nextflow, is introduced to the proteomics and metabolomics communities.

CYBELE – Fostering precision agriculture & livestock farming through secure access to large-scale HPC enabled virtual industrial experimentation environments fostering scalable big data analytics

According to McKinsey & Company, about a third of food produced is lost or wasted every year, amounting to a $940 billion economic hit. Inefficiencies in planting, harvesting, water use, reduced animal contributions, as well as uncertainty about weather, pests, consumer demand and other intangibles contribute to the loss. Precision Agriculture (PA) and Precision Livestock Farming (PLF) come to assist in optimizing agricultural and livestock production and minimizing the wastes and costs aforementioned. PA is a technology-enabled, data-driven approach to farming management that observes, measures, and analyzes the needs of individual fields and crops. PLF is also a technology-enabled, data-driven approach to livestock production management, which exploits technology to quantitatively measure the behavior, health and performance of animals. Big data delivered by a plethora of data sources related to these domains, has a multitude of payoffs including precision monitoring of fertilizer and fungicide levels to optimize crop yields, risk mitigation that results from monitoring when temperature and humidity levels reach dangerous levels for crops, increasing livestock production while minimizing the environmental footprint of livestock farming, ensuring high levels of welfare and health for animals, and more. By adding analytics to these sensor and image data, opportunities also exist to further optimize PA and PLF by having continuous data on how a field or the livestock is responding to a protocol. For these domains, two main challenges exist: 1) to exploit this multitude of data facilitating dedicated improvements in performance, and 2) to make available advanced infrastructure so as to harness the power of this information in order to benefit from the new insights, practices and products, efficiently time-wise, lowering responsiveness down to seconds so as to cater for time-critical decisions. The current paper aims to introduce CYBELE, a platform aspiring to safeguard that the stakeholders involved in the agri-food value chain (research community, SMEs, entrepreneurs, etc.) have integrated, unmediated access to a vast amount of very large scale datasets of diverse types and coming from a variety of sources, and that they are capable of actually generating value and extracting insights out of these data, by providing secure and unmediated access to large-scale High Performance Computing (HPC) infrastructures supporting advanced data discovery, processing, combination and visualization services, solving computationally-intensive challenges modelled as mathematical algorithms requiring very high computing power and capability.