Data Management Paradigm Research Articles

Adaptive and Robust Query Execution for Lakehouses at Scale

Many organizations have embraced the "Lakehouse" data management paradigm, which involves constructing structured data warehouses on top of open, unstructured data lakes. This approach stands in stark contrast to traditional, closed, relational databases and introduces challenges for performance and stability of distributed query processors. Firstly, in large-scale, open Lakehouses with uncurated data, high ingestion rates, external tables, or deeply nested schemas, it is often costly or wasteful to maintain perfect and up-to-date table and column statistics. Secondly, inherently imperfect cardinality estimates with conjunctive predicates, joins and user-defined functions can lead to bad query plans. Thirdly, for the sheer magnitude of data involved, strictly relying on static query plan decisions can result in performance and stability issues such as excessive data movement, substantial disk spillage, or high memory pressure. To address these challenges, this paper presents our design, implementation, evaluation and practice of the Adaptive Query Execution (AQE) framework, which exploits natural execution pipeline breakers in query plans to collect accurate statistics and re-optimize them at runtime for both performance and robustness. In the TPC-DS benchmark, the technique demonstrates up to 25× per query speedup. At Databricks, AQE has been successfully deployed in production for multiple years. It powers billions of queries and ETL jobs to process exabytes of data per day, through key enterprise products such as Databricks Runtime, Databricks SQL, and Delta Live Tables.

A CFD simulation platform for surface finishing processes in advanced manufacturing

Products created by additive manufacturing often have surface imperfections that require post-processing operations to remove extraneous material in order to meet design specifications. The usage of computational fluid dynamics (CFD) simulations to predict material removal rates of components, allows practitioners to optimize the setup and usage of post-processing equipment. However, those without in-depth knowledge of CFD or the related specialized software, require an easy-to-use and cost-effective application to manage the computational workflow. The two specific surface finishing applications investigated here, are, abrasive flow machining (AFM) and robotic stream finishing (RSF). In order to satisfy user requirements, a modular, threaded, fault-tolerant and object-oriented project management application, written with the Python programming language and PyQt6 framework, has been developed to conduct surface finishing-related CFD simulations using OpenFOAM®. The advantages of the proposed software are: 1) the modern PyQt6 framework is used to develop a cross-platform and user-friendly application which employs the model-view class architectural paradigm for data management and its display, 2) step-by-step interactive project workflows have been tailored specifically for AFM and RSF simulations, 3) the developed steady-state viscoelastic flow solver for AFM and continuum-based steady-state dense granular flow solver for RSF, offer advantages over those provided by OpenFOAM® and 4) simulation results have been corroborated by experimental data to assess the improved accuracy of material removal prediction of the current software when compared to other commercial applications.

More is better? The role of strategic data management in a lean manufacturing process

Purpose Lean manufacturing has been pivotal in emphasizing the reduction of cycle times, minimizing manufacturing costs and diminishing inventories. This research endeavors to formulate a lean data management paradigm, through the design and execution of a strategic edge-cloud data governance approach. This study aims to discern anomalous or unforeseen patterns within data sets, enabling an efficacious examination of product shortcomings within manufacturing processes, while concurrently minimizing the redundancy associated with the storage, access and processing of nonvalue-adding data. Design/methodology/approach Adopting a lean data management framework within both edge and cloud computing contexts, this study ensures the preservation of significant time series sequences, while ascertaining the optimal quantity of normal time series data to retain. The efficacy of detected anomalous patterns, both at the edge and in the cloud, is assessed. A comparative analysis between traditional data management practices and the strategic edge-cloud data governance approach facilitates an exploration into the equilibrium between anomaly detection and space conservation in cloud environments for aggregated data analysis. Findings Evaluation of the proposed framework through a real-world inspection case study revealed its capability to navigate alternative strategies for harmonizing anomaly detection with data storage efficiency in cloud-based analysis. Contrary to the conventional belief that retaining comprehensive data in the cloud maximizes anomaly detection rates, our findings suggest that a strategic edge-cloud data governance model, which retains a specific subset of normal data, can achieve comparable or superior accuracy with less normal data relative to traditional methods. This approach further demonstrates enhanced space efficiency and mitigates various forms of waste, including temporal delays, storage of noncontributory normal data, costs associated with the analysis of such data and excess data transmission. Originality/value By treating inspected normal data as nonvalue-added, this study probes the intricacies of maintaining an optimal balance of such data in light of anomaly detection performance from aggregated data sets. Our proposed methodology augments existing research by integrating a strategic edge-cloud data governance model within a lean data analytical framework, thereby ensuring the retention of adequate data for effective anomaly detection.

Decentralized file storage using Blockchain

Amidst the exponential growth in data volumes and growing apprehensions regarding centralized control, decentralized cloud storage systems have emerged as a beacon of promise. This study presents a comprehensive examination of these systems, delving into their structural intricacies, benefits, and hurdles. The essence of decentralized cloud storage lies in dispersing data across a network of nodes, obviating the necessity for a central governing entity. Through the utilization of blockchain technology and cryptographic methods, information is encrypted, fragmented, and dispersed among myriad nodes, ensuring both redundancy and heightened security. This framework effectively mitigates the inherent risks of data loss or tampering associated with centralized infrastructures, thus bolstering data integrity and confidentiality. In summary, decentralized cloud storage systems signify a revolutionary shift in data management paradigms, furnishing a resilient, scalable, and secure alternative to centralized storage models. Ongoing research and development endeavours are imperative to surmount extant challenges and fully harness the transformative potential of this groundbreaking technology. Keywords— - Blockchain, Data Security, IPFS, Encryption, File Storage, Decentralized storage, Metamask, Pinata.

Unleashing real-time analytics: A comparative study of in-memory computing vs. traditional disk-based systems

The article presents a comprehensive study evaluating the performance differences between in-memory computing (IMC) and traditional disk-based database systems, specifically focusing on Redis and PostgreSQL. Given the escalating demands for real-time data analytics across various sectors, the research delves into the comparative efficiency of these two data management paradigms in processing large datasets. Utilizing a synthetic dataset of 23.6 million records, we orchestrated a series of data manipulation tasks, including aggregation, table joins, and filtering operations, to simulate real-world data analytics scenarios. The experiment, conducted on a high-performance computing setup, revealed that Redis significantly outperformed PostgreSQL in all tested operations, showcasing the inherent advantages of IMC in terms of speed and efficiency. Data aggregation tasks saw Redis completing the process up to ten times faster than PostgreSQL. Similarly, table joining, and data filtering tasks were executed more swiftly on Redis, emphasizing IMC's potential to facilitate instantaneous data analytics. These findings underscore the pivotal role of IMC technologies like Redis in empowering organizations to harness real-time insights from big data, a critical capability in today's fast-paced business environment. The study further discusses the implications of adopting IMC over traditional systems, considering aspects such as cost, integration challenges, and the importance of skill development for IT teams. Concluding with strategic recommendations, the article advocates for a nuanced approach to incorporating IMC technologies, highlighting their transformative potential while acknowledging the need for balanced investment and operational planning.

Biomedical knowledge graph construction of Sus scrofa and its application in anti-PRRSV traditional Chinese medicine discovery

As a new data management paradigm, knowledge graphs can integrate multiple data sources and achieve quick responses, reasoning and better predictions in drug discovery. Characterized by powerful contagion and a high rate of morbidity and mortality, porcine reproductive and respiratory syndrome (PRRS) is a common infectious disease in the global swine industry that causes economically great losses. Traditional Chinese medicine (TCM) has advantages in low adverse effects and a relatively affordable cost of application, and TCM is therefore conceived as a possibility to treat PRRS under the current circumstance that there is a lack of safe and effective approaches. Here, we constructed a knowledge graph containing common biomedical data from humans and Sus Scrofa as well as information from thousands of TCMs. Subsequently, we validated the effectiveness of the Sus Scrofa knowledge graph by the t-SNE algorithm and selected the optimal model (i.e., transR) from six typical models, namely, transE, transR, DistMult, ComplEx, RESCAL and RotatE, according to five indicators, namely, MRR, MR, HITS@1, HITS@3 and HITS@10. Based on embedding vectors trained by the optimal model, anti-PRRSV TCMs were predicted by two paths, namely, VHC-Herb and VHPC-Herb, and potential anti-PRRSV TCMs were identified by retrieving the HERB database according to the pharmacological properties corresponding to symptoms of PRRS. Ultimately, Dan Shen's (Salvia miltiorrhiza Bunge) capacity to resist PRRSV infection was validated by a cell experiment in which the inhibition rate of PRRSV exceeded 90% when the concentrations of Dan Shen extract were 0.004, 0.008, 0.016 and 0.032 mg/mL. In summary, this is the first report on the Sus Scrofa knowledge graph including TCM information, and our study reflects the important application values of deep learning on graphs in the swine industry as well as providing accessible TCM resources for PRRS.

Elevating Android Privacy: A Blockchain-Powered Paradigm for Secure Data Management

Elevating Android Privacy: A Blockchain-Powered Paradigm for Secure Data Management

China Data Cube (CDC) for Big Earth Observation Data: Practices and Lessons Learned

In the face of tight natural resources and complex as well as volatile environments, and in order to meet the pressure brought by population growth, we need to overcome a series of challenges. As a new data management paradigm, the Earth Observation Data Cube simplifies the way that users manage and use earth observation data, and provides an analysis-ready form to access big spatiotemporal data, so as to realize the greater potential of earth observation data. Based on the Open Data Cube (ODC) framework, combined with analysis-ready data (ARD) generation technology, the design and implementation of CDC_DLTool, extending the support for data loading and the processing of international and Chinese imagery data covering China, this study eventually constructs the China Data Cube (CDC) framework. In the framework of this CDC grid, this study carried out case studies of water change monitoring based on international satellite imagery data of Landsat 8 in addition to vegetation change monitoring based on Chinese satellite imagery data of GF-1. The experimental results show that, compared with traditional scene-based data organization, the minimum management unit of this framework is a pixel, which makes the unified organization and management of multisource heterogeneous satellite imagery data more convenient and faster.

GF-1 Satellite Imagery Data Service and Application Based on Open Data Cube

With the application of big data in Earth observation, satellite imagery data are gradually becoming important means of observation for monitoring changes in vegetation, water bodies, and urbanization. Therefore, new satellite imagery data organization and management paradigms are urgently needed to fully mine the useful information from these data and provide new ways to better quantify and serve the sustainable development of resources and the environment. In this paper, a framework for processing and analyzing Chinese GF-1 satellite imagery data was developed using the latest technologies such as Open Data Cube (ODC) grids, Analysis Ready Data (ARD) generation, and space subdivision, which extended the data loading and processing capacities of the ODC grids for Chinese satellite imagery data. Using the proposed framework, we conducted a case study to investigate the spatial and temporal changes in vegetation and water mapping with GF-1 data collected from 2014 to 2021 covering the Miyun Reservoir, Beijing, China. The experimental results showed that the proposed framework had significantly improved temporal and spatial efficiency compared with the traditional scene-based data management approach, thus demonstrating the advantages and potential of the ODC grids as a new data management paradigm.

Blockchain for Doping Control Applications in Sports: A Conceptual Approach

Doping is a well-known problem in competitive sports. Along the years, several cases have come to public, evidencing corrupt practices from within the sports environment. To guarantee fair play and prevent public health issues, anti-doping organizations and sports authorities are expected to cooperate in the fight against doping. To achieve this mission, doping-related data must be produced, stored, accessed, and shared in a secure, tamperproof, and privacy-preserving manner. This paper investigates the processes and tools established by the World Anti-Doping Agency for the global harmonization of doping control activities. From this investigation, it is possible to conclude that there is an inherent trust problem, in part due to a centralized data management paradigm and to the lack of fully digitalized processes. Therefore, this paper presents two main contributions: the concept of a multiorganizational decentralized data governance model and a blockchain-based design for one of the most sensitive data-sharing processes within the anti-doping ecosystem. Throughout this article, it is shown that the adoption of a permissioned blockchain can benefit the whole anti-doping community, creating more reliable processes for handling data, where privacy and security are enhanced.

Дослідження принципів, моделей та методів парадигми менеджменту наукових даних FAIR для аналізу метаданих BIG DATA

In this research work we considere the basic principles, models and methods of the FAIR (Findable, Accessible, Interoperable, Reusable) scientific data management paradigm as a separate case of Big Data. This paradigm is focused on the reuse of scientific research results.Basic principles of FAIR apply to three types of entities: data (or any digital object), metadata (information about this digital object) and infrastructure. Information on various projects, initiatives and communities working on solving the problems of scientific data and their metadata integration is examined. We analyze how the properties of FAIR data contribute to the unification and integration of the scientific infrastructure in the paradigm of open science, which is based on free access to research results and open data.

Innovation policy mix: mapping and measurement

The “policy mix” concept has gained popularity among science, technology and innovation policy communities over the past two decades in a context of growing policy complexity and need for policy evidence. Pressing societal challenges are also prompting governments to rethink policy making in order to better align public intervention across policy domains and leverage the transformative potential of system innovations. Governments faced multiple obstacles in implementing a policy mix approach in policy making and evaluation. Based on a comparative analysis of international STI policy repositories, a conceptual framework is proposed, as well as structuring principles and operational guidelines for mapping the composition of a policy mix, identifying interactions among components and translating the mapping into measurement. In that view, a range of new policy mix metrics is introduced. Finally, the discussion focuses on the need for moving towards a new data management paradigm and enlarging the measurement mix.

The Case for a Linked Data Research Engine for Legal Scholars

This contribution explores the application of data science and artificial intelligence to legal research, more specifically an element that has not received much attention: the research infrastructure required to make such analysis possible. In recent years, EU law has become increasingly digitised and published in online databases such as EUR-Lex and HUDOC. However, the main barrier inhibiting legal scholars from analysing this information is lack of training in data analytics. Legal analytics software can mitigate this problem to an extent. However, current systems are dominated by the commercial sector. In addition, most systems focus on search of legal information but do not facilitate advanced visualisation and analytics. Finally, free to use systems that do provide such features are either too complex to use for general legal scholars, or are not rich enough in their analytics tools. In this paper, we motivate the case for building a software platform that addresses these limitations. Such software can provide a powerful platform for visualising and exploring connections and correlations in EU case law, helping to unravel the “DNA” behind EU legal systems. It will also serve to train researchers and students in schools and universities to analyse legal information using state-of-the-art methods in data science, without requiring technical proficiency in the underlying methods. We also suggest that the software should be powered by a data infrastructure and management paradigm following the seminal FAIR (Findable, Accessible, Interoperable and Reusable) principles.

A Blockchain-Based Trusted Data Management Scheme in Edge Computing

With rapid development of computing technologies, large amount of data are gathered from edge terminals or Internet of Things (IoT) devices, however data trust and security in edge computing environment are very important issues to be considered, especially when the gathered data are fraud or dishonest, or the data are misused or spread without any authorization, which may lead to serious problems. In this article, a blockchain-based trusted data management scheme (called BlockTDM) in edge computing is proposed to solve the above problems, in which we proposed a flexible and configurable blockchain architecture that includes mutual authentication protocol, flexible consensus, smart contract, block and transaction data management, blockchain nodes management, and deployment. The BlockTDM scheme can support matrix-based multichannel data segment and isolation for sensitive or privacy data protection, and moreover, we have designed user-defined sensitive data encryption before the transaction payload stores in blockchain system, and have implemented conditional access and decryption query of the protected blockchain data and transactions through smart contract. Finally, we have evaluated the proposed BlockTDM scheme security, availability, and efficiency with large amount of experiments. Analysis and evaluations manifest that the proposed BlockTDM scheme provides a general, flexible, and configurable blockchain-based paradigm for trusted data management with tamper-resistance, which is suitable for edge computing with high-level security and creditability.

The Internet of People: A human and data-centric paradigm for the Next Generation Internet

The cyber-physical convergence, the fast expansion of the Internet at its edge, and tighter interactions between human users and their personal mobile devices push towards a data-centric Internet where the human user becomes more central than ever. We argue that this will profoundly impact primarily on the way data should be handled in the Next Generation Internet. It will require a radical change of the Internet data-management paradigm, from the current platform-centric to a human-centric model. In this paper we present a new paradigm for Internet data management that we name Internet of People (IoP) because it embeds human behavior models in its algorithms. To this end, IoP algorithms exploit quantitative models of the humans’ individual and social behavior, from sociology, anthropology, psychology, economics, physics. IoP is not a replacement of the current Internet networking infrastructure, but it exploits legacy Internet services as (reliable) primitives to achieve end-to-end connectivity on a global-scale. In this opinion paper, we first discuss the key features of the IoP paradigm along with the underlying research issues and challenges. Then, we present emerging data-management paradigms that are anticipating IoP.

Message-oriented middleware for smart grids

In order to increase the efficiency in the use of energy resources, the electrical grid is slowly evolving into a smart(er) grid that allows users' production and storage of energy, automatic and remote control of appliances, energy exchange between users, and in general optimizations over how the energy is managed and consumed. One of the main innovations of the smart grid is its organization over an energy plane that involves the actual exchange of energy, and a data plane that regards the Information and Communication Technology (ICT) infrastructure used for the management of the grid's data.In the particular case of the data plane, the exchange of large quantities of data can be facilitated by a middleware based on a messaging bus. Existing messaging buses follow different data management paradigms (e.g.: request/response, publish/subscribe, data-oriented messaging) and thus satisfy smart grids' communication requirements at different extents.This work contributes to the state of the art by identifying, in existing standards and architectures, common requirements that impact in the messaging system of a data plane for the smart grid. The paper analyzes existing messaging bus paradigms that can be used as a basis for the ICT infrastructure of a smart grid and discusses how these can satisfy smart grids' requirements.

Dividing secrets to secure data outsourcing

Data outsourcing or database as a service is a new paradigm for data management. The third party service provider hosts databases as a service. These parties provide efficient and cheap data management by obviating the need to purchase expensive hardware and software, deal with software upgrades and hire professionals for administrative and maintenance tasks. However, due to recent governmental legislations, competition among companies and database thefts, companies cannot use database service providers directly. They need secure and privacy preserving data management techniques to be able to use them in practice. Since data is remotely stored in a privacy preserving manner, there are efficiency related problems such as poor query response time. We propose a new framework that provides efficient and scalable query response times by reducing the computation and communication costs. Furthermore, the proposed technique uses several service providers to guarantee the availability of the services while detecting the dishonest or faulty service providers without introducing additional overhead on the query response time. The evaluations demonstrate that our data outsourcing framework is scalable and practical.

On The Potential Integration of an Ontology-Based Data Access Approach in NoSQL Stores

NoSQL stores are emerging as an efficient alternative to relational database management systems in the context of big data. Many actors in this domain consider that to gain a wider adoption, several extensions have to be integrated. Some of them focus on the ways of proposing more schemas, supporting adapted declarative query languages and providing integrity constraints in order to control data consistency and enhance data quality. The authors consider that these issues can be dealt with in the context of Ontology Based Data Access (OBDA). OBDA is a new data management paradigm that exploits the semantic knowledge represented in ontologies when querying data stored in a database. They provide a proof of concept of OBDA's ability to tackle these three issues in a social application related to the medical domain.

Novel CBIR system based on Ripplet Transform using interactive Neuro-Fuzzy technique

Content Based Image Retrieval (CBIR) system is an emerging research area in effective digital data management and retrieval paradigm. In this article, a novel CBIR system based on a new Multiscale Geometric Analysis (MGA)-tool, called Ripplet Transform Type-I (RT) is presented. To improve the retrieval result and to reduce the computational complexity, the proposed scheme utilizes a Neural Network (NN) based classifier for image pre-classification, similarity matching using Manhattan distance measure and relevance feedback mechanism (RFM) using fuzzy entropy based feature evaluation technique. Extensive experiments were carried out to evaluate the effectiveness of the proposed technique. The performance of the proposed CBIR system is evaluated using a 2 £ 5-fold cross validation followed by a statistical analysis. The experimental results suggest that the proposed system based on RT, performs better than many existing CBIR schemes based on other transforms, and the difference is statistically significant.

BioMart: driving a paradigm change in biological data management

BioMart: driving a paradigm change in biological data management