Traditional Relational Database Management Systems Research Articles

Using Time-Series Databases for Energy Data Infrastructures

The management of energy market data, such as load, production, forecasts, and prices, is critical for energy market participants, who develop in-house energy data infrastructure services to aggregate data from many sources to support their business operations. Energy data management frequently involves time sensitive operations, including rapid data ingestion, real-time querying, filling in gaps from missing or delayed data, and updating large volumes of timestamped and loosely structured data, all of which demand high processing power. Traditional relational database management systems (RDBMSs) often struggle with these operations, whereas time series databases (TSDBs) appear to be a more efficient solution, providing enhanced scalability, reliability, real-time data availability and superior performance. This paper examines the advantages of TSDBs over RDBMS for energy data management, demonstrating that TSDBs can either replace or complement RDBMSs. We present quantitative improvements in digestion, integration, architecture, and performance, demonstrating that operations such as importing and querying time-series energy data, along with the overall system’s efficiency, can be significantly improved, achieving up to 100 times faster operations compared to relational databases, all without requiring extensive modifications to the existing information system’s architecture.

NoSQL Technologies in Big Data Ecosystems: a Comprehensive Review of Architectural Paradigms and Performance Metrics

The exponential growth of data in volume, velocity, and variety has challenged traditional relational database management systems, leading to the emergence and widespread adoption of NoSQL and other innovative database technologies. This article provides a comprehensive review of NoSQL databases and their role in modern Big Data infrastructure stacks. Through a systematic analysis of current literature and industry case studies, we explore the architectural paradigms, performance metrics, and use cases of various NoSQL database types, including document-based, key-value, column-family, and graph databases. Our findings indicate that NoSQL solutions offer significant advantages in scalability, flexibility, and real-time processing capabilities, particularly for unstructured and semi-structured data. However, challenges persist in areas such as data consistency, security, and interoperability with existing systems. We also examine emerging trends, including NewSQL, time-series databases, and the integration of artificial intelligence in database management. This article contributes to the understanding of how organizations can leverage NoSQL technologies to optimize their Big Data infrastructure, highlighting both the opportunities and considerations for implementation. Our conclusions underscore the transformative impact of NoSQL on data management practices and provide directions for future research in this rapidly evolving field.

Integrating NoSQL, Hilbert Curve, and R*-Tree to Efficiently Manage Mobile LiDAR Point Cloud Data

The widespread use of Light Detection and Ranging (LiDAR) technology has led to a surge in three-dimensional point cloud data; although, it also poses challenges in terms of data storage and indexing. Efficient storage and management of LiDAR data are prerequisites for data processing and analysis for various LiDAR-based scientific applications. Traditional relational database management systems and centralized file storage struggle to meet the storage, scaling, and specific query requirements of massive point cloud data. However, NoSQL databases, known for their scalability, speed, and cost-effectiveness, provide a viable solution. In this study, a 3D point cloud indexing strategy for mobile LiDAR point cloud data that integrates Hilbert curves, R*-trees, and B+-trees was proposed to support MongoDB-based point cloud storage and querying from the following aspects: (1) partitioning the point cloud using an adaptive space partitioning strategy to improve the I/O efficiency and ensure data locality; (2) encoding partitions using Hilbert curves to construct global indices; (3) constructing local indexes (R*-trees) for each point cloud partition so that MongoDB can natively support indexing of point cloud data; and (4) a MongoDB-oriented storage structure design based on a hierarchical indexing structure. We evaluated the efficacy of chunked point cloud data storage with MongoDB for spatial querying and found that the proposed storage strategy provides higher data encoding, index construction and retrieval speeds, and more scalable storage structures to support efficient point cloud spatial query processing compared to many mainstream point cloud indexing strategies and database systems.

TIME OPTIMIZATION OF PROCESS OF DATA CONSISTENCY IN NOSQL

It is emphasized that the performance of a distributed information system depends on many factors, including the size and structure of the database, the hardware used, the number of replicas and their geographical location. A conclusion was made about the perspective of the concept of non-relational NoSQL databases and it was concluded that it was proposed for efficient storage and provision of quick access to large volumes of information, the so-called Big Data, which cannot be achieved using traditional relational database management systems. The essence of the concept of “NoSQL database” is revealed and the currently existing difficulties in using large systems are named. In NoSQL databases, multiple replication (copying) of a record is used to ensure high fault tolerance. But there is a significant drawback for them, because these systems do not support the mode of conducting transactions and blocking, so there is a problem of data reconciliation. It is emphasized that NoSQL databases mainly use two methods of placing and updating the time of the data consistency process in NoSQL: according to the principle of “master-slave” and “ring”. A model of strong data reconciliation in NoSQL is described and an example of time optimization for the healthcare system is considered. It has been established that even with large values of the intensity of receiving requests for reading λ, the waiting time does not exceed 3 ms. However, when performing analytical queries, this time can increase, so with large N and high intensity of incoming read requests, this time can reach 7 ms.

RDBMS to distributed in-memory data grid: A paradigm shift for improved performance

As per available Cisco VNI report [1], global IP traffic will grow more than three times and global network device connections will be more than 28.5 billion. Availability of high-speed network and connectivity will lead to more online business opportunities to enterprises and in turn will generate large data. Sometimes these large datasets need to be processed in near real time to do some critical analysis. Traditional databases are not designed to do real time data analysis and have challenges in terms of performance and throughput. This paper describes various widely used data storage tools and frameworks currently available and how to achieve the best performance while processing large datasets using in memory data grid. It briefly covers the evolution of in-memory data grid from traditional relational database management systems. It also provides a high-level feature comparison across available in-memory database solutions and how they provide the best of solution for processing real time data beyond Hadoop referencing use cases like fraud detection and genetic data processing. It also emphasizes on leveraging power of GPU alongside CPU to achieve high performance and throughput while processing large data with multiple iterations.

Performance evaluation of various deployment scenarios of the 3-replicated Cassandra NoSQL cluster on AWS

A concept of distributed replicated NoSQL data storages Cassandra-like, HBase, MongoDB has been proposed to effectively manage Big Data set whose volume, velocity and variability are difficult to deal with by using the traditional Relational Database Management Systems. Tradeoffs between consistency, availability, partition tolerance and latency is intrinsic to such systems. Although relations between these properties have been previously identified by the well-known CAP and PACELC theorems in qualitative terms, it is still necessary to quantify how different consistency settings, deployment patterns and other properties affect system performance.This experience report analysis performance of the Cassandra NoSQL database cluster and studies the tradeoff between data consistency guaranties and performance in distributed data storages. The primary focus is on investigating the quantitative interplay between Cassandra response time, throughput and its consistency settings considering different single- and multi-region deployment scenarios. The study uses the YCSB benchmarking framework and reports the results of the read and write performance tests of the three-replicated Cassandra cluster deployed in the Amazon AWS. In this paper, we also put forward a notation which can be used to formally describe distributed deployment of Cassandra cluster and its nodes relative to each other and to a client application. We present quantitative results showing how different consistency settings and deployment patterns affect Cassandra performance under different workloads. In particular, our experiments show that strong consistency costs up to 22 % of performance in case of the centralized Cassandra cluster deployment and can cause a 600 % increase in the read/write requests if Cassandra replicas and its clients are globally distributed across different AWS Regions.

Impact of Data Compression on the Performance of Column-oriented Data Stores

Compression of data in traditional relational database management systems significantly improves the system performance by decreasing the size of the data that results in less data transfer time within the communication environment and higher efficiency in I/O operations. The column-oriented database management systems should perform even better since each attribute is stored in a separate column, so that its sequential values are stored and accessed sequentially on the disk. That further increases the compression efficiency as the entire column is compressed/decompressed at once. The aim of this research is to determine if data compression could improve the performance of HBase, running on a small-sized Hadoop cluster, consisted of one name node and nine data nodes. Test scenario includes performing Insert and Select queries on multiple records with and without data compression. Four data compression algorithms are tested since they are natively supported by HBase - SNAPPY, LZO, LZ4 and GZ. Results show that data compression in HBase highly improves system performance in terms of storage saving. It shrinks data 5 to 10 times (depending on the algorithm) without any noticeable additional CPU load. That allows smaller but significantly faster SSD disks to be used as cluster’s primary data storage. Furthermore, the substantial decrease in the network traffic is an additional benefit with major impact on big data processing.

EXPLORING TIMEOUT AS A PERFORMANCE AND AVAILABILITY FACTOR OF DISTRIBUTED REPLICATED DATABASE SYSTEMS

A concept of distributed replicated data storages like Cassandra, HBase, MongoDB has been proposed to effectively manage the Big Data sets whose volume, velocity, and variability are difficult to deal with by using the traditional Relational Database Management Systems. Trade-offs between consistency, availability, partition tolerance, and latency are intrinsic to such systems. Although relations between these properties have been previously identified by the well-known CAP theorem in qualitative terms, it is still necessary to quantify how different consistency and timeout settings affect system latency. The paper reports results of Cassandra's performance evaluation using the YCSB benchmark and experimentally demonstrates how to read latency depends on the consistency settings and the current database workload. These results clearly show that stronger data consistency increases system latency, which is in line with the qualitative implication of the CAP theorem. Moreover, Cassandra latency and its variation considerably depend on the system workload. The distributed nature of such a system does not always guarantee that the client receives a response from the database within a finite time. If this happens, it causes so-called timing failures when the response is received too late or is not received at all. In the paper, we also consider the role of the application timeout which is the fundamental part of all distributed fault tolerance mechanisms working over the Internet and used as the main error detection mechanism here. The role of the application timeout as the main determinant in the interplay between system availability and responsiveness is also examined in the paper. It is quantitatively shown how different timeout settings could affect system availability and the average servicing and waiting time. Although many modern distributed systems including Cassandra use static timeouts it was shown that the most promising approach is to set timeouts dynamically at run time to balance performance, availability and improve the efficiency of the fault-tolerance mechanisms.

Big Data and Business Analytics: Trends, Platforms, Success Factors and Applications

Big data and business analytics are trends that are positively impacting the business world. Past researches show that data generated in the modern world is huge and growing exponentially. These include structured and unstructured data that flood organizations daily. Unstructured data constitute the majority of the world’s digital data and these include text files, web, and social media posts, emails, images, audio, movies, etc. The unstructured data cannot be managed in the traditional relational database management system (RDBMS). Therefore, data proliferation requires a rethinking of techniques for capturing, storing, and processing the data. This is the role big data has come to play. This paper, therefore, is aimed at increasing the attention of organizations and researchers to various applications and benefits of big data technology. The paper reviews and discusses, the recent trends, opportunities and pitfalls of big data and how it has enabled organizations to create successful business strategies and remain competitive, based on available literature. Furthermore, the review presents the various applications of big data and business analytics, data sources generated in these applications and their key characteristics. Finally, the review not only outlines the challenges for successful implementation of big data projects but also highlights the current open research directions of big data analytics that require further consideration. The reviewed areas of big data suggest that good management and manipulation of the large data sets using the techniques and tools of big data can deliver actionable insights that create business values.

A Study on Join Operations in MongoDB Preserving Collections Data Models for Future Internet Applications

Presently, we are observing an explosion of data that need to be stored and processed over the Internet, and characterized by large volume, velocity and variety. For this reason, software developers have begun to look at NoSQL solutions for data storage. However, operations that are trivial in traditional Relational DataBase Management Systems (DBMSs) can become very complex in NoSQL DBMSs. This is the case of the join operation to establish a connection between two or more DB structures, whose construct is not explicitly available in many NoSQL databases. As a consequence, the data model has to be changed or a set of operations have to be performed to address particular queries on data. Thus, open questions are: how do NoSQL solutions work when they have to perform join operations on data that are not natively supported? What is the quality of NoSQL solutions in such cases? In this paper, we deal with such issues specifically considering one of the major NoSQL document oriented DB available on the market: MongoDB. In particular, we discuss an approach to perform join operations at application layer in MongoDB that allows us to preserve data models. We analyse performance of the proposes approach discussing the introduced overhead in comparison with SQL-like DBs.

Consistency Models of NoSQL Databases

Internet has become so widespread that most popular websites are accessed by hundreds of millions of people on a daily basis. Monolithic architectures, which were frequently used in the past, were mostly composed of traditional relational database management systems, but quickly have become incapable of sustaining high data traffic very common these days. Meanwhile, NoSQL databases have emerged to provide some missing properties in relational databases like the schema-less design, horizontal scaling, and eventual consistency. This paper analyzes and compares the consistency model implementation on five popular NoSQL databases: Redis, Cassandra, MongoDB, Neo4j, and OrientDB. All of which offer at least eventual consistency, and some have the option of supporting strong consistency. However, imposing strong consistency will result in less availability when subject to network partition events.

AIDA

With the tremendous growth in data science and machine learning, it has become increasingly clear that traditional relational database management systems (RDBMS) are lacking appropriate support for the programming paradigms required by such applications, whose developers prefer tools that perform the computation outside the database system. While the database community has attempted to integrate some of these tools in the RDBMS, this has not swayed the trend as existing solutions are often not convenient for the incremental, iterative development approach used in these fields. In this paper, we propose AIDA - an abstraction for advanced in-database analytics. AIDA emulates the syntax and semantics of popular data science packages but transparently executes the required transformations and computations inside the RDBMS. In particular, AIDA works with a regular Python interpreter as a client to connect to the database. Furthermore, it supports the seamless use of both relational and linear algebra operations using a unified abstraction. AIDA relies on the RDBMS engine to efficiently execute relational operations and on an embedded Python interpreter and NumPy to perform linear algebra operations. Data reformatting is done transparently and avoids data copy whenever possible. AIDA does not require changes to statistical packages or the RDBMS facilitating portability.

A Distributed Storage Model for Healthcare Big Data Designed on HBase.

With the explosive growth of healthcare data, traditional relational database management systems (RDBMS) are limited in scalability, storage of unstructured data, concurrency and cost. Thus we proposed a snowflake model based on HBase with a multi-table structure and three kinds of index tables for efficient assess of large-scale health records. A guideline for designing of index tables was proposed which covers the demands of most healthcare data processing applications. A benchmark test was carried out with six types of queries on a large dataset comprising 750 million records to compare the performance of the proposed model against the traditional tall-table model on HBase. We found that the snowflake model was more efficient than the tall-table model. The adoption of index tables could greatly improve the query speed and provided real-time queries for two models. In general, snowflake model could be used for managing large-scale healthcare data as an advantageous alternative.

Estimating the effect of network element events in a wireless network

Network events, like outages, are costly events for communication service providers (CSPs) not only because they represent lost revenue but also because of adverse effects suffered by the CSP’s customers. Quantifying the effect of negative events on certain key performance indicators allows the CSP to measure the network resources impacted, to provide data for a more robust revenue assurance process, and to assign appropriate severity to the events. These additional insights may help optimize the resource allocation, ticketing, and troubleshooting response times. This paper presents a novel heuristic algorithm that takes advantage of the daily patterns observed in most key performance indicators of a wireless network and the stability observed in the differences between the original time series and the lagged version. The proposed algorithm uses those differences and the previous actual values to make accurate predictions of time-series traffic volume data that represent the estimated effect of a wireless network event. The performance of the algorithm is compared with that of the state-of-the-art autoregressive, integrated, moving average (ARIMA) model and the results are reported. The proposed algorithm has reduced standard deviation in error percentage by 4.8 percentage points, has no negative bias, and executes 97% faster than the ARIMA model. The algorithm provides an accurate methodology for online or batch network event impact estimation that could potentially be implemented in traditional relational database management systems (SQL) or Big Data environments.

Modeling temporal aspects of sensor data for MongoDB NoSQL database

Proliferation of structural, semi-structural and no-structural data, has challenged the scalability, flexibility and processability of the traditional relational database management systems (RDBMS). The next generation systems demand horizontal scaling by distributing data over autonomously addable nodes to a running system. For schema flexibility, they also want to process and store different data formats along the sequence factor in the data. NoSQL approaches are solutions to these, hence big data solutions are vital nowadays. But in monitoring scenarios sensors transmit the data continuously over certain intervals of time and temporal factor is the main property of the data. Therefore the key research aspect is to investigate schema flexibility and temporal data integration aspects together. We need to know that: what data modelling should we adopt for a data driven real-time scenario; that we could store the data effectively and evolve the schema accordingly during data integration in NoSQL environments without losing big data advantages. In this paper we explain a middleware based schema model to support the temporal oriented storage of real-time data of ANT+ sensors as hierarchical documents. We explain how to adopt a schema for the data integration by using an algorithm based approach for flexible evolution of the model for a document oriented database, i.e, MongoDB. The proposed model is logical, compact for storage and evolves seamlessly upon new data integration.

Quantitative Analysis of Consistency in NoSQL Key-Value Stores

The promise of high scalability and availability has prompted many companies to replace traditional relational database management systems (RDBMS) with NoSQL key-value stores. This comes at the cost of relaxed consistency guarantees: key-value stores only guarantee eventual consistency in principle. In practice, however, many key-value stores seem to offer stronger consistency. Quantifying how well consistency properties are met is a non-trivial problem. We address this problem by formally modeling key-value stores as probabilistic systems and quantitatively analyzing their consistency properties by both statistical model checking and implementation evaluation. We present for the first time a formal probabilistic model of Apache Cassandra, a popular NoSQL key-value store, and quantify how much Cassandra achieves various consistency guarantees under various conditions. To validate our model, we evaluate multiple consistency properties using two methods and compare them against each other. The two methods are: (1) an implementation-based evaluation of the source code; and (2) a statistical model checking analysis of our probabilistic model.

Object-NoSQL Database Mappers: a benchmark study on the performance overhead

In recent years, the hegemony of traditional relational database management systems (RDBMSs) has declined in favour of non-relational databases (NoSQL). These database technologies are better adapted to meet the requirements of large-scale (web) infrastructures handling Big Data by providing elastic and horizontal scalability. Each NoSQL technology however is suited for specific use cases and data models. As a consequence, NoSQL adopters are faced with tremendous heterogeneity in terms of data models, database capabilities and application programming interfaces (APIs). Opting for a specific NoSQL database poses the immediate problem of vendor or technology lock-in. A solution has been proposed in the shape of Object-NoSQL Database Mappers (ONDMs), which provide a uniform abstraction interface for different NoSQL technologies.Such ONDMs however come at a cost of increased performance overhead, which may have a significant economic impact, especially in large distributed setups involving massive volumes of data.In this paper, we present a benchmark study quantifying and comparing the performance overhead introduced by Object-NoSQL Database Mappers, for create, read, update and search operations. Our benchmarks involve five of the most promising and industry-ready ONDMs: Impetus Kundera, Apache Gora, EclipseLink, DataNucleus and Hibernate OGM, and are executed both on a single node and a 9-node cluster setup.Our main findings are summarised as follows: (i) the introduced overhead is substantial for database operations in-memory, however on-disk operations and high network latency result in a negligible overhead, (ii) we found fundamental mismatches between standardised ONDM APIs and the technical capabilities of the NoSQL database, (iii) search performance overhead increases linearly with the number of results, (iv) DataNucleus and Hibernate OGM’s search overhead is exceptionally high in comparison to the other ONDMs.

Big Data Management

Organizations and governments leverage on the potentials in data to plan and compete globally. Data from various sources are continually mined, stored in databases and utilized in a manner that improves processes, products and ensure steady profitability. The traditional relational database management systems are unable to cope with these new forms of data. The velocity, volume and variety in which data are generated qualify them as “Big Data”. This scales up the storage and processing needs of client organizations, allowing them to focus on their core areas of expertise. This paper investigates the relevance of aggregating big data and its efficient management in a cloud computing environment. It adopts the prototyping design and a case study approach, based on Couchbase Server, a non-structured query language document database and cloud computing platform. The accruable benefit of this work is organizational cost-effectiveness, timeliness in decision making and higher profitability.

An Efficient Distributed Data Processing Method for Smart Environment

Background/Objectives: In current times, huge volume of data at a very high velocity gets generated through social media and various sensors in embedded systems that are connected to the Internet which causes a very Big data problem. These challenging Big data’s need to be processed and stored by traditional Relational Database Management Systems (RDBMS). Due to this reason the need for new software solutions has emerged for managing the Big data in an efficient, scalable and smart way. Methods/Statistical Analysis: In this study, an approach to combine the concept of batch processing and stream processing to an end where we can query the data set which also supports Adhoc Querying with less latency, that can be run on any Large scale Machine Learning Algorithms for recognizing any interest pattern in the streaming data set was employed. The functionalities of Hadoop ecosystem’s tool HIVE can also be used to produce the results to Adhoc queries, User Defined Functions (UDF) similar to writing a SQL Stored Procedures in the Spark System. An interface with SerDes which is Serialization and De-serialization that helps us to talk to the standard stream where we can exactly query the dataset are employed. Findings: : By proposing a new software solution AllJoyn Lambda, in which AllJoyn is integrated in the lambda architecture and the prototype implementation of the architecture is done using Apache Hadoop Yarn over Apache Spark Streaming are presented . This study illuminates the high velocity streaming data set on a database without losing any data from the streaming domain, to support Adhoc Querying from the data set and to provide a mechanism for fast data processing and analytics using Large Scale Machine Learning. This paper highlights the analysis of large scale dataset processing, handling challenges, and its comprehensive systematic review. Applications/Improvements: From this study, we conclude that, building a smart environment by using the big data setup platform improves and enhances the results for the smart environment.

Sustainable system design for gridded, spatio-temporal, agroecosystem forecasting models

Decision support systems able to capitalize on publicly available high resolution datasets have become increasingly valuable to agroecosystem, hydrologic and urban system stakeholders. In this paper we address the common agroecosystem modeling problem of weather-based risk forecasting. We compare storage system designs for an expandable crop disease forecasting system that relies on multiple gridded weather forecast inputs to artificial neural network disease risk models. A traditional relational database management system (PostgreSQL), a NoSQL database system (MongoDB) and a scientific file format version (netCDF) of a single crop disease risk modeling system in one region of the country, for potato late blight in the US Great Lakes region, were designed and compared for speed. To test expandability, another crop disease risk modeling system, for modeling risk of economically significant deoxynivalenol (eDON) accumulation due to Fusarium head blight of barley in the northern US Great Plains, was also created in the three formats. Speeds for the three types of systems were fairly similar. Expandability, which is becoming highly desirable in agroecosystem model design, differed based on designer priorities.