Distributed Database Systems Research Articles

Too-Short-Arc Association and Clustering Method for Space-Based Optical Observations

Space-based optical observation is assuming an increasingly significant role in space surveillance and tracking. However, it often yields tracks that are too short, which are known as too short arcs (TSAs). A single TSA lacks sufficient information for reliable initial orbit determination of a resident space object. Therefore, it is typically necessary to perform tracklet association during catalog building and maintenance. This study presents a novel tracklet association method that improves upon existing techniques by combining admissible regions and nonlinear orbital uncertainty propagation. Proceeding by constructing TSA association matrices and TSA clustering matrices, the bond energy algorithm, traditionally employed in the design of distributed database systems, is applied to cluster TSAs based on the association results between two arbitrary TSAs. Furthermore, the proposed splitting algorithm reduces the computational load associated with clustering multiple tracklets and effectively handles erroneous association results. To assess the effectiveness of this approach, it was tested in a space-based optical-survey scenario, accounting for practical observation uncertainties. The simulation results demonstrate that the method achieves a high level of accuracy in both TSA association and clustering.

TDSQL: Tencent Distributed Database System

Distributed databases have become indispensable in contemporary computing and data processing, owing to their pivotal role in ensuring high availability and scalability. They effectively cater to the requirements of data management and high-concurrency access. However, developing a distributed database system that is well-suited for diverse application scenarios, particularly for large-scale applications, presents several challenges. These challenges include ensuring data consistency and achieving high levels of performance. This paper presents TDSQL, a distributed database system that prioritizes core design principles of distributed systems, including high availability, strong consistency, and scalability. In particular, TDSQL has achieved high performance through over a decade of practical experience and optimization in various modules, such as the kernel, synchronous replication, and transaction processing, in large-scale application scenarios. By conducting the TPC-C benchmark test, TDSQL demonstrated outstanding performance, achieving a throughput of 814 million tpmC across 1650 database nodes, with a jitter rate of less than 0.2%. This jitter rate is an order of magnitude lower than the standard required, showcasing the system's stability and reliability. During the 8-hour TPC-C standard stress test, TDSQL successfully completed over 860 billion transactions and processed 40 trillion order details, with zero forced rollbacks and zero data inconsistency.

QOTUM: The Query Optimizer for Distributed Database in Cloud Environment

Abstract: In distributed and cloud database system, large amount of resources are used by complex queries that increase in payment overheads of cloud users. These resource needs can be minimised by evaluating common join operations and caching their results so that they could be applied to the execution of additional queries. In this research, the Query Optimization Technique Using Materialization (QOTUM) system is designed and developed that uses the mechanism of materialized views during query execution at distributed database in cloud environment. Extensive experiments are conducted with the help of standard benchmarks datasets and query workloads in cloud environment. The performance of QOTUM system is studied and evaluated against conventional SQL system based on various performance parameters.

Energy consumption estimation and profiling for queries in distributed database systems based on a bottom-up comprehensive energy model

Quantifying energy consumption of database operations is the foundation of building energy-efficient database systems. Existing approaches only focused on stand-alone database servers, while we are interested in modeling energy consumption of database operations in the distributed environments. In this study, we aim at providing an accurate energy consumption model for queries executed in distributed database systems, profiling energy consumption characteristics of both the individual queries and the system as a whole, to guide the design of green database systems and to reveal opportunities for energy-efficient computing. As the execution of a distributed query is a combination of a set of subqueries that decomposed from it, we start from building energy models for individual subqueries by extracting basic operations that can effectively reflect their energy consumption. Then we use a bottom-up measuring and modeling method as the basis to provide a comprehensive energy estimation model for the entire distributed query. To validate the accuracy of the model, we use queries with a variety complexity that generated from three standard benchmarks (TPC-H, SSB, and Sysbench) on real distributed databases. Extensive experimental results show that our solution can achieve a high average accuracy of 94.63% for distributed queries. More importantly, based on these results we further explored energy consumption patterns for distributed queries and presented several important implications. And finally, a significant role of distributed joins has been discovered in saving both idle and dynamic energy cost of the system. We hope that taking advantage of our observations can help readers who wish to substantially improve energy efficiency for distributed database systems.

IDify - Distributed Database System for Digital Identification

”IDify” Distributed database system for digital identification is a big data handling system to centralize the important services provided by the government to make the facilities efficient and reliable. It has integrated services such as digital authentication, E-Banking, Personal Information Management, and Government Surveys to provide users with a centralized and digitized platform to make their lives easier. In this system there is a distributed database platform to carry out all the data storing and data processing tasks to facilitate a large user count and to avoid the drawbacks of monolithic systems. The database system has been designed by integrating several modern database technologies to make the system efficient and coherent as the facilities provided by the system are extremely mission-critical. To make the system low latent on data ingestion and data reading tasks a special library is also implemented in collaboration with modern query processing engines. It is proposed to utilize the thread system of a processor to implement parallelism to enhance the performance of data read/write tasks. Apart from that a fault-tolerant framework has been implemented to free the system from any erroneous scenario taking place.

Thorough Data Pruning for Join Query in Database System

The improvement of robustness and efficiency for multi-way equijoin query is challenging, no-matter for centralized database systems or distributed database systems. Due to lots of unnecessary data existing during query processing, these two metrics will be seriously reduced. If we can thoroughly prune unnecessary data in advance, the robustness and efficiency will be highly improved. However, the pruning power of current strategies, such as predicate push-down and algebraic equivalence, is limited. We present deepDP, a powerful, generalized, and efficient strategy for data pruning. deepDP builds multiple independent pruning spaces by generating longest transitive closures and applies appropriate data pruning strategy for each pruning space. For thoroughly pruning unnecessary data, deepDP employs <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\alpha \cdot \beta$</tex-math></inline-formula> pruning strategy to clean each pruning space based on a newly designed statistic information-Hollow Range and re-shuffles the elements in all pruned spaces for maximizing robustness and efficiency benefits meanwhile minimizing the invasion. We implement deepDP in PostgreSQL but are not limited to it, and evaluate deepDP on TPC-H, JOB, and our synthesis benchmark–DHR. The experiment results show that compared to traditional data pruning strategy, deepDP can improve multi-way equijoin query on efficiency by 3.5x.

Data Synchronization for Distributed Heterogeneous Database

Heterogeneous distributed database concurrent data is one of the important topics of heterogeneous databases that have important meaning. This paper discusses the mechanism of optimizing data synchronization for distributed heterogeneous database and trying to propose real solutions. Database synchronization is the exchange of records in databases on the condition that they are different. When modifications are made to a specific database, they will be applied to the database that has been synchronized in different environments and servers horizontally. The synchronization is either a one-way or two-way application and can be incidental, i.e. it is synchronous to the local network and asynchronous to the Internet due to its lagging technology. Heterogeneous distributed database systems consisting of a row storage database and a column storage database will face many difficulties, due to the inconsistent storage type such as: synchronization speed mismatch in data synchronization compared to a homogeneously distributed database system. Heterogeneous database management systems work on exchanging information for a package of various formats, accessing modification information in a heterogeneous database, maintaining the independence of the proposed system, and ensuring a good application of the proposed system in order to reach data synchronization between several packages of databases.

Design of storage benchmark kit framework for supporting the file storage retrieval

An open-source software framework called the storage benchmark kit (SBK) is used to store the system benchmarking performance framework. The SBK is designed to perform any storage client or device using any data type as a payload. SBK simultaneously helps number of readers as well as writes to the storage system of large amounts of data as well as allows end-to-end latency benchmarking for multiple writers and readers. The SBK uses standardized performance measures for comparing and evaluating various storage systems and their combinations. Distributed file systems, distributed database systems, single or local node databases, systems of object storage, platforms of distributed streaming and messaging, and systems of key-value storage are the storage solutions supported by SBK. The SBK supports various storage systems like XFS, Kafka streaming storage systems, and Hadoop distributed file system (HDFS) performance benchmarking. The experimental results show that a proposed method achieves execution time of 65.530 s, 40.826 s and 30.351 s for the 100k, 500k and 1000k files respectively which ensures better improvement than the existing methods such as simple data interface and distributed data protection system.

Web Design for Emotion Tracking in Individuals with Major Depressive Disorder

Major Depressive Disorder (MDD), classified as an affective disorder, has been observing an escalating incidence annually. Despite the growing focus on MDD, there remains a paucity of research dedicated to emotion-tracking software tailored for MDD patients. While a plethora of emotion-tracking applications exists, none can concurrently document emotions and facilitate psychological interventions. This manuscript delineates a web application that amalgamates emotion documentation, analytical procedures, and psychological counseling tailored for MDD patients. The discourse encompasses three pivotal facets: the implications and intricacies of UI design, the functional blueprint, and the actualization of functionalities. The UI design is elucidated from both visual and interactive dimensions, and the functional architecture is broached from the perspectives of both patients and clinicians. Employing the React framework, the application adopts the MVVM architectural pattern, integrates a distributed database system, and endorses interfaces for advanced algorithmic extensions, proffering a streamlined online modality for emotional alleviation and therapeutic interventions for MDD patients.

BASIS DATA TERDISTRIBUSI

In the context of a distributed database, data is stored on various computers that are interconnected through communication media such as high-speed buses or telephone lines. A distributed database system consists of a set of sites, where each site has the ability to execute transactions that access data at one or more sites. A distributed database is a collection of logical data that is physically dispersed in a computer network, which is independent of current or future application programs. Fragmentation is a phenomenon in which storage space is inefficiently utilized, resulting in reduced storage capacity, while the term also refers to the condition of the deserted storage itself. Replication, on the other hand, is a technique for copying and distributing data and database objects from one database to another, as well as maintaining data consistency through synchronization between databases.

Query Optimization in Distributed Database Based on Improved Artificial Bee Colony Algorithm

Query optimization is one of the key factors affecting the performance of database systems that aim to enact the query execution plan with minimum cost. Particularly in distributed database systems, due to the multiple copies of the data that are stored in different data nodes, resulting in the dramatic increase in the feasible query execution plans for a query statement. Because of the increasing volume of stored data, the cluster size of distributed databases also increases, resulting in poor performance of current query optimization algorithms. In this case, a dynamic perturbation-based artificial bee colony algorithm is proposed to solve the query optimization problem in distributed database systems. The improved artificial bee colony algorithm improves the global search capability by combining the selection, crossover, and mutation operators of the genetic algorithm to overcome the problem of falling into the local optimal solution easily. At the same time, the dynamic perturbation factor is introduced so that the algorithm parameters can be dynamically varied along with the process of iteration as well as the convergence degree of the whole population to improve the convergence efficiency of the algorithm. Finally, comparative experiments conducted to assess the average execution cost of Top-k query plans generated by the algorithms and the convergence speed of algorithms under the conditions of query statements in six different dimension sets. The results demonstrate that the Top-k query plans generated by the proposed method have a lower execution cost and a faster convergence speed, which can effectively improve the query efficiency. However, this method requires more execution time.

A Secure and Optimization Based Clustering for Vertical and Horizontal Fragmentation in Distributed Database Management System

Distributed database systems are increasingly important due to the massive data output, and their effectiveness is largely based on their design. Two key processes, fragmentation and allocation, are used to improve the efficiency and efficacy of these systems. Effective data fragmentation requires both horizontal and vertical categorization of tuples. Advanced optimization techniques are used for both fragmentations, such as the Enhanced Arithmetic Optimization (EAO) algorithm with Opposition-based Learning (OBL) and Levy Flight Distributer (LFD) for vertical fragmentation and the hybrid Aquila Optimizer (AO) with Artificial Rabbit Optimization (ARO) algorithm for horizontal fragmentation. The fragmented data is securely transmitted using the Fully Homomorphic Encryption (FHE) algorithm. The implementation is executed using the Python language, and the performance of the proposed algorithms is evaluated using different performance parameters. The execution time analysis shows that the proposed EAOA algorithm consumes 5.5 seconds for vertical fragmentation, while the hybrid AARO algorithm takes 5.9 seconds for horizontal fragmentation. The vertical fragmentation is found to be better than the horizontal one in DDBMS.

Ensuring secure interoperation of access control in a multidomain environment

Interoperation can combine multiple resources and domains, thus it has been widely used in many practical industrial applications, such as distributed database systems. However, the merger of local access control policies in such systems may lead to security violations with regard to access control. For instance, a person can potentially have access (indirectly) to another one's file or data in the interoperation to which s/he should be denied access in the individual system. Therefore, it is critical to deal with such issues in a multidomain environment. Nevertheless, a real-world interoperation contains a large number of entities and access. This imposes a challenge to find the maximum secure interoperation in terms of direct data sharing among individual systems. To overcome this difficulty, we propose an integer linear programming-based approach which can find the maximum secure interoperation in a computationally efficient way. Experimental results are given to demonstrate the efficacy of our approach.

On hierarchical clustering-based approach for RDDBS design

Distributed database system (DDBS) design is still an open challenge even after decades of research, especially in a dynamic network setting. Hence, to meet the demands of high-speed data gathering and for the management and preservation of huge systems, it is important to construct a distributed database for real-time data storage. Incidentally, some fragmentation schemes, such as horizontal, vertical, and hybrid, are widely used for DDBS design. At the same time, data allocation could not be done without first physically fragmenting the data because the fragmentation process is the foundation of the DDBS design. Extensive research have been conducted to develop effective solutions for DDBS design problems. But the great majority of them barely consider the RDDBS's initial design. Therefore, this work aims at proposing a clustering-based horizontal fragmentation and allocation technique to handle both the early and late stages of the DDBS design. To ensure that each operation flows into the next without any increase in complexity, fragmentation and allocation are done simultaneously. With this approach, the main goals are to minimize communication expenses, response time, and irrelevant data access. Most importantly, it has been observed that the proposed approach may effectively expand RDDBS performance by simultaneously fragmenting and assigning various relations. Through simulations and experiments on synthetic and real databases, we demonstrate the viability of our strategy and how it considerably lowers communication costs for typical access patterns at both the early and late stages of design.

WITHDRAWN: Efficient fragmentation and allocation on clustering in distributed environment (FACE)

Distributed data processing systems require a reliable storage structure that ensures consistent, high-scale, and scalable data access for all users. Data fragmentation and subsequent allocation are important aspects of a flexible distributed database system. Existing techniques may result in significant data loss while performing data allocation and fragmentation. To overcome these issues a novel fragmentation and Allocation on Clustering in Distributed Environment (FACE) is proposed for reliable storage with scalable data access for all users. Initially, the user query is given to distributed database system (DDBS), then the query is fed to the query parser. The query parser parses and converts a textual query into a query object. After parsing the query data, it is passed to the query optimizer, which optimizes the query to access the data more efficiently. In the next step, the query is fragmented using dual (horizontal and vertical) fuzzy C means clustering. In Dual fuzzy C means clustering, the vertical fragmentation focuses on clustering attribute values that are frequently accessed together, and horizontal fragmentation focuses on grouping records that meet data selection criteria. After fragmentation, the query is allocated using sequential rule mining, in which queries occur in sequential order. The allocated query is optimized using the Efficient Fragment Security (EFS) algorithm to normalize the parameters to achieve better results. The performance metrics of the proposed techniques are calculated based on Final Prediction Error (FPE), and Execution time. The proposed FACE achieves a lower execution time of the 1.56 s. The result shows that the proposed FACE decreases the overall execution time lower than the 4.70 s, 4.46 s, 3.3 Sec, and 3.14 s in MAPA, DMA, BBE, and RATS-HM respectively.

Introduction and analysis of bitcoin-based blockchain technology principle

Blockchain technology is a technology that inherently solves trust issues. It has the character-istics of decentralization, distributed storage, tamper resistance, security, and transparency. It ensures reliable communication between nodes that do not trust each other through consensus mechanisms, smart contracts, and other means. Blockchain stores each transaction data on each transaction node to make the data public and transparent, and generates the data into a blockchain.. As a relatively new distributed database system, blockchain has expanded to many other fields since the initial digital currency, but its development is seriously constrained by problems such as large storage overhead and low query efficiency. In order to find a suita-ble optimization method, select the representative blockchain system Bitcoin, analyze its data structure, data storage and data query processing mechanism, discuss the problems existing in the two functions of storage and query, summarize the existing relevant optimization methods, and look forward to the main research problems of the blockchain system represented by Bitcoin in the future.

Modified and Improved Algorithm for Finding a Median Path with a Specific Length (ℓ) for a Tree Network

The median path problem (min-sum criterion) is a common problem in graph theory and tree networks. This problem is open to study because its applications are growing and extending in different fields, such as providing insight for decision-makers when selecting the optimal location for non-emergency services, including railroad lines, highways, pipelines, and transit routes. Also, the min-sum criterion can deal with several networks in different applications. The location problem has traditionally been concerned with the optimal location of a single-point facility at either a vertex or along an edge in a network. Recently, numerous investigators have investigated this classic problem and have studied the location of many facilities, such as paths, trees, and cycles. The concept of the median, which measures the centrality of a vertex in a graph, is extended to the paths in a graph. In this paper, we consider the problem of locating path-shaped facilities on a tree network. A new modified and improved algorithm for finding a median single path facility of a specified length in a tree network is proposed. The median criterion for optimality considers the sum of the distances from all vertices of the tree to the path facility. This problem under the median criterion is called the ℓ-core problem. The distance between any two vertices in the tree is equal to the length of the unique path connecting them. This location problem usually has applications in distributed database systems, pipelines, the design of public transportation routes, and communication networks.

A Bibliometric Analysis of the Evolution of State-of-the-Art Blockchain Technology (BCT) in the Agrifood Sector from 2014 to 2022.

In recent years, BCT has garnered significant attention from researchers worldwide. The technology in question is a distributed database system characterised by its decentralised nature and lack of reliability. BCT has been widely adopted by numerous governments and scholars across various sectors for a number of years. Blockchain technology also involves highly innovative and advanced concepts. Given the increasing interest among scholars in the academic community regarding the agrifood supply chain, the objective of this study was to investigate BCT and its potential for application in the fields of food and agriculture. This research paper presents a bibliometric analysis of articles on the utilisation of BCT in the fields of food and agriculture. This study discusses scholarly articles that have been published in esteemed academic journals and conferences. Through our bibliometric analysis, we aimed to discern the recurring trends and themes within the research on BCT in relation to agrifood systems. Furthermore, this study examines a diverse array of research domains, prominent scholarly publications, leading publishing platforms, prominent funding institutions, and the prospective trajectory of future research. This study also presents the prominent patterns and themes within this field through an analysis of the most influential scholarly articles, authors, countries, and keywords found in the existing literature. Hence, this research employed various analytical techniques, including analyzing the co-occurrence of author keywords, bibliographic coupling analysis, network view map analysis, and co-citation analysis. This study holds promise as a valuable learning resource for aspiring researchers seeking to acquire compelling and pertinent information about research outcomes from studies on the utilisation of BCT in the field of smart agriculture.

Comparative Studies on Intelligent Swarming Network (iSWAN) Geno-Generative Algorithm and Top-K Query Processing Algorithm

This paper proposed an enhanced Top-k query processing in a real time distributed database system. The system employs a Particle Swarm Optimizer (PSO) based Geno-Generative iSWAN Model technique that enhances and allows multi-task concurrent query processing in a real time co-simulation data acquisition platform and as part of refinement to an existing Top-k query processing Technique. In this paper, the proposed system is compared for efficiency with the Top-K Query Algorithm, which is emerging as an alternative to more conventional technique for real time query processing in distributed databases. Dynamic simulations were performed with a real time small testbed comprising of physical and non-physical devices to test and evaluate the performance and efficiency of the two systems. Considering the estimated and expected temperatures, the result of simulation study proves that the Intelligent Swarming Network (iSWAN) Geno-Generative Model is more preferred over Top-K Query Algorithm due to its 70% accuracy over the Top-K Model, which reported a lower accuracy level of 40%.

Scheduling distributed multiway spatial join queries: optimization models and algorithms

Multiway spatial joins are a commonly occurring and fundamental type of query for spatial data processing. This article presents models and algorithms to schedule this type of query in distributed database systems while attempting to strike a balance between makespan and communication costs. We propose three algorithms based on combinatorial optimization methods: the well-known linear relaxation technique of rounding a solution generated by linear programming (LP), a more sophisticated Lagrangian Relaxation method (LR), as well as a greedy heuristic (GR) for baseline comparison. Our evaluation shows that a schedule built using GR consumes, on average, 22% more processing and communication resources than a more elaborate schedule constructed via the LR method, when scheduling a query for 64 machines. The schedule provided by LR is also, on average, an order of magnitude closer to the optimal schedule for a query compared to GR. We show that scheduling Gigabyte-size multiway queries before execution can reduce its processing time by an order of magnitude compared to state-of-the-art frameworks for spatial data processing that do not have this capability, and can significantly reduce the amount of shuffled data in the network.