Time Series Databases Research Articles

Using Time-Series Databases for Energy Data Infrastructures

The management of energy market data, such as load, production, forecasts, and prices, is critical for energy market participants, who develop in-house energy data infrastructure services to aggregate data from many sources to support their business operations. Energy data management frequently involves time sensitive operations, including rapid data ingestion, real-time querying, filling in gaps from missing or delayed data, and updating large volumes of timestamped and loosely structured data, all of which demand high processing power. Traditional relational database management systems (RDBMSs) often struggle with these operations, whereas time series databases (TSDBs) appear to be a more efficient solution, providing enhanced scalability, reliability, real-time data availability and superior performance. This paper examines the advantages of TSDBs over RDBMS for energy data management, demonstrating that TSDBs can either replace or complement RDBMSs. We present quantitative improvements in digestion, integration, architecture, and performance, demonstrating that operations such as importing and querying time-series energy data, along with the overall system’s efficiency, can be significantly improved, achieving up to 100 times faster operations compared to relational databases, all without requiring extensive modifications to the existing information system’s architecture.

NoSQL Technologies in Big Data Ecosystems: a Comprehensive Review of Architectural Paradigms and Performance Metrics

The exponential growth of data in volume, velocity, and variety has challenged traditional relational database management systems, leading to the emergence and widespread adoption of NoSQL and other innovative database technologies. This article provides a comprehensive review of NoSQL databases and their role in modern Big Data infrastructure stacks. Through a systematic analysis of current literature and industry case studies, we explore the architectural paradigms, performance metrics, and use cases of various NoSQL database types, including document-based, key-value, column-family, and graph databases. Our findings indicate that NoSQL solutions offer significant advantages in scalability, flexibility, and real-time processing capabilities, particularly for unstructured and semi-structured data. However, challenges persist in areas such as data consistency, security, and interoperability with existing systems. We also examine emerging trends, including NewSQL, time-series databases, and the integration of artificial intelligence in database management. This article contributes to the understanding of how organizations can leverage NoSQL technologies to optimize their Big Data infrastructure, highlighting both the opportunities and considerations for implementation. Our conclusions underscore the transformative impact of NoSQL on data management practices and provide directions for future research in this rapidly evolving field.

Developing an internet of things system for hydrogen leak detection at hydrogen refueling stations integrating LoRa and global positioning system

Developing a hydrogen leak detection system that can quickly detect even the smallest amount of hydrogen is extremely crucial in the hydrogen industry. Currently, hydrogen refueling stations use a fixed hydrogen leak detection system. This study develops a mobile hydrogen gas leak detection system for hydrogen refueling stations using s, global positioning system, and Internet of Things (IoT) equipment, including LoRa and Wi-Fi. A digital twin is implemented by linking hydrogen gas-sensing data at a location that is quickly required or suspected using Google Maps. A time-series database is used to store the hydrogen gas-sensing data, and the transfer delay of hydrogen gas-sensing data between the hydrogen equipment was measured. Therefore, we developed a prototype system that transmits, stores, and analyzes hydrogen gas-sensing data in real time using IoT equipment for hydrogen refueling stations.

Research on Flexible Data Model of MES System based on Time-series Database

In the process of advancing towards digitization and intelligence in manufacturing, Manufacturing Execution Systems (MES) have gradually become the core of production processes. The design of the data model is crucial as it affects the scalability, maintainability, and stability of the system. This study focuses on addressing issues in MES systems such as high investment costs, imperfect data models, lack of flexibility, and inefficient data flow. It proposes a novel flexible data model based on time-series databases.This paper introduces the definition and characteristics of time-series databases, analyzes the shortcomings of existing MES system data models, and proposes a novel flexible MES data model based on time-series databases. Finally, the practicality and efficiency of the new data model are validated through experiments.

EAST plant data storage system based on IoTDB time series database

EAST (Experimental Advanced Superconducting Tokamak) has been in operation since 2006. With increasing operation time, the volume of plant data stored in MySQL has steadily grown to several billion rows. The current storage architecture centered around the relational database has shown poor performance when facing massive time series data, so it is crucial to adopt TSDB (Time Series Database) for storing plant data. In initial testing, IoTDB demonstrated a performance advantage of at least 2 times over other TSDBs in terms of both write throughput and large-scale queries for plant data management. However, as the inflexible underlying infrastructure of EAST, the plant database cannot be easily modified directly. To remedy this problem, we propose a MySQL-IoTDB Hierarchical Mechanism (MIHM). Specifically, we utilized a MySQL master–slave to IoTDB cluster design to seamlessly transfer the performance burden from relational database to TSDB IoTDB. Extensive tests on the EAST plant data demonstrate that MIHM-based plant data storage system has increased the write throughput by 20 times and the large-scale data querying speed by 100 times compared to previous systems.

Tobacco retailer density and its association with birth outcomes in the USA: 2000–2016

IntroductionSignificant progress has been made in reducing maternal exposure to tobacco smoke and subsequent adverse birth outcomes, however, reductions may require strategies that reduce the availability of tobacco retailers. In...

Mining Weighted Patterns from Time Series Databases Based on Sliding Window

Data mining has traditionally relied heavily on sliding window-based challenges, which has sparked a variety of studies. For each new window in time series mining, current literature mandates the rebuilding of the underlying structure, Suffix Tree - A trie-based structure representing all the suffixes of a string. However, reconstruction struggles when the window is wide or when sliding happens frequently. As a result, we provide a new technique Dynamic Tree-Based Approach to handle Sliding Windows (DTSW) in time series in this study that dynamically changes the representative suffix tree structure rather than reconstructing it after every alteration or sliding. In addition, we also put forth a different approach to the issue of extracting weighted periodic patterns from time series. To prevent testing pointless patterns, existing studies mostly rely on the weight of the database's highest-weighted item. However, these methods continue to examine numerous patterns. These methods still examine numerous patterns to see whether they can be candidates. Our proposed measure Maximum Possible Weighted Support (MPWS) accelerates the candidate generation process by removing numerous unnecessary patterns in advance. The novelty of MPWS is it considers the maximum weighted average over the maximum weighted item extension by enforcing more constraints. The usefulness of our two techniques in handling sliding windows and trimming redundant candidate patterns is demonstrated by experimental results using a variety of real-world datasets. Our experiments state that our dynamic handling technique significantly improves runtime than the reconstruction in a dynamic sliding window-based environment with simultaneous insertion and deletion actions and MPWS reduces the number of tested patterns resulting in lesser mining time in weighted time series pattern mining. DUJASE Vol. 8 (1) 13-25, 2023 (January)

TSDSystem: a framework to collect, archive and share time series data at volcanological observatories

This paper presents a framework designed to collect, archive, and share time series data coming from sensor networks at Istituto Nazionale di Geofisica e Vulcanologia, Osservatorio Etneo (Italy), which we have developed and called Time Series Database management System (TSDSystem). The framework proposes a flexible database model for the standardization of sensor networks data and implements an optimized technology for storage and retrieval of acquired time series data. It is designed for the implementation of multiparametric databases and then suitable for development in volcanological observatories worldwide. The proposed framework provides a web service to perform writing and reading data via a standard web communication protocol, which easily enables interaction with other instruments or automatic systems. All results provided by the TSDSystem web service are represented using common data formats in the context of online services. In particular, the station networks metadata representation follows a schema inspired by the International Federation of Digital Seismograph Networks, widely known in seismology. A web GUI (graphical user interface) is provided to test and document the web service. Additionally, basic built-in web applications are supplied with the web GUI to perform joint and synchronized time series data visualization as well as representation of stations on a geographical map. The web GUI also offers administration tools for data access policy management, creation of monitoring dashboards and data publication through web pages. The framework implements an authorization system that can be used to restrict both writing or reading operations. The TSDSystem can also be a useful tool for engineering surveillance systems. The implementing code of the framework is available with an open source license on a public repository together with a user manual.

LCGCT: A light curve generator in customisable-time-bin based on time-series database

In the era of time-domain astronomy, scientists often need to generate light curves with varying time-bin. However, an increase in time resolution typically leads to a substantial increase in data transmission. To enhance the data processing efficiency in time-domain astronomy, we propose a novel time-series data model for storing time-series observation data, and we construct the LCGCT, a tool designed to produce light curves with customisable time bins. To validate our approach, we utilise the 7-year MAXI/GSC (Gas Slit Camera of the Monitor of All-sky X-ray Image) X-ray source catalogue, incorporating its 24-h binned light curves for a comparative analysis with our approach. The results obtained confirm the accuracy and effectiveness of our proposed approach. Subsequently, we compare the storage capacity and query performance of LCGCT with a PostgreSQL-based implementation, and results show that LCGCT conserves 75% of the storage space and achieves three times the query speed. Owing to its noteworthy storage and query performance, our proposed time-series data model-based LCGCT can be used in time-domain astronomical projects with high time resolution.

Tracking states and transitions in semiarid rangelands: A spatiotemporal archetypal analysis of productivity dynamics using wavelets

Climate change poses challenges in classifying ecosystem dynamics, as they are influenced by shifting dynamics resulting from changes in climate forces and meteorological variables, including temperature and water availability. To address this, our study presents a novel approach using Continuous Wavelet Transform (CWT) and power spectrum analysis to classify vegetation dynamics, considering the time-dependent variability of ecosystem frequencies. We applied our method to centred and standardized MODIS NDVI time series for the period 2000–2021, using an experimental field station in northern Patagonia as a case study. By performing a continuous wavelet transform on the data for each pixel, we obtained instantaneous power spectra, capturing variability across different dates and pixels. These spectrums were then consolidated into a comprehensive database, and subsequently classified using archetypal analysis. We identified a convex combination of archetypal spectrums that best represented the entire power spectrum database. Mapping the resulting archetypes and their weights in both space and time allowed us to explore pixels' variations in archetype weights in relation to factors such as time, topography, and climate. In addition, to examine the potential relationship between the NDVI time series and climate drivers, we computed the Average Cross-Wavelet Power Spectrum (ACWPS) to different climatic indices. Three archetypes were sufficient to explain the majority of power spectrum variability in the study area. These archetypes exhibited distinctive characteristics: 1) medium-frequency variability (2–4 years), 2) low-frequency variability (>4 years), and 3) an annual (i.e. seasonal) cycle with low-frequency variability. Spatially, the first two archetypes were predominantly observed in highland steppes, while the third archetype prevailed in lowland areas associated with meadows. At the beginning of the studied period, Archetypes 1 and 3 dominated, but after the Puyehue-Cordón Caulle Volcanic Complex eruption in 2011 their prominence diminished, and Archetype 2 became more prevalent in the whole study area. Finally, all three NDVI series representative of archetypes showed a relative peak at approximately four years, which could be linked to the Indian Ocean Dipole variability. These results highlight an abrupt shift in the system's behaviour, primarily related to changes in variability distribution rather than mean values. This disturbance-induced transition aligns with the theory of state and transitions in ecological system dynamics. We propose that the states in this model are not fixed but represent alternative dynamic behaviours, akin to different types of limit cycles. Consequently, employing a wavelet analysis-based classification method provides a robust means of studying and understanding such variability and transitions, thereby offering clarity and comprehension of ecosystem states. Notably, this methodology proves particularly effective for large databases of detailed time series.

Artificial intelligence based succored power prediction in stand alone PV system

ABSTRACT Accurate forecasting of solar power generation is crucial for effective management of renewable energy resources. This literature review emphasises the effectiveness of Artificial Neural Networks (ANNs) in predicting solar energy potential. While traditional ANNs with single hidden layers and predefined neuron configurations have shown success in various applications, they are not suitable for predicting nonlinear time-series databases in the context of this research. The limitations of traditional ANNs in predicting solar power potential necessitate the exploration of alternative approaches. Configuring the structure of traditional ANNs for solar power prediction is a time-consuming process that often requires manual or trial-and-error methods. To address this issue, optimisation techniques are incorporated into the ANN model. Various traditional methods have been explored in this research, but none has achieved satisfactory performance. Drawing from experiences with traditional techniques, a novel Evolution of Cub to Predator (ECP) technique is proposed. The investigation results demonstrate that the incorporation of optimisation techniques yields superior performance compared to traditional ANNs. The proposed ECP technique achieves an impressive prediction accuracy of 97.2%, surpassing existing optimisation techniques. These findings highlight the potential of the ECP technique for accurate solar power prediction and its contribution to optimising renewable energy resource management.

ФУНКЦІОНАЛЬНІ МОЖЛИВОСТІ СИСТЕМ КЕРУВАННЯ БАЗАМИ ДАНИХ ЧАСОВИХ РЯДІВ

This work provides a comprehensive overview of the capabilities of time series databases, with a special emphasis on InfluxDB –a leading time series database known for its efficiency and support for complex analyses through the Flux query language, including data aggregation, filtering, trend detection, and forecasting. The practical application of InfluxDB is demonstrated through various examples, showcasing its versatility in performing complex data operations, such as data filtering, identifying significant changes in data, and conducting stochastic analysis. These functionalities highlight the potential of InfluxDB across a broad range of applications, from IoT telemetry to financial analysis, underscoring the critical role of time series databases in modern data management and analysis strategies.

Time Series Representation for Visualization in Apache IoTDB

When analyzing time series, often interactively, the analysts frequently demand to visualize instantly large-scale data stored in databases. M4 visualization selects the first, last, bottom and top data points in each pixel column to ensure pixel-perfectness of the two-color line chart visualization. While M4 already shows its preciseness of encasing time series in different scales into a fixed size of pixels, how to efficiently support M4 representation in a time series native database is still absent. It is worth noting that, to enable fast writes, the commodity time series database systems, such as Apache IoTDB or InfluxDB, employ LSM-Tree based storage. That is, a time series is segmented and stored in a number of chunks, with possibly out-of-order arrivals, i.e., disordered on timestamps. To implement M4, a natural idea is to merge online the chunks as a whole series, with costly merge sort on timestamps, and then perform M4 representation as in relational databases. In this study, we propose a novel chunk merge free approach called M4-LSM to accelerate M4 representation and visualization. In particular, we utilize the metadata of chunks to prune and avoid the costly merging of any chunk. Moreover, intra-chunk indexing and pruning are enabled for efficiently accessing the representation points, referring to the special properties of time series. Remarkably, the time series database native operator M4-LSM has been implemented in Apache IoTDB, an open-source time series database, and deployed in companies across various industries. In the experiments over real-world datasets, the proposed M4-LSM operator demonstrates high efficiency without sacrificing preciseness.

Determining Exact Quantiles with Randomized Summaries

Quantiles are fundamental statistics in various data science tasks, but costly to compute, e.g., by loading the entire data in memory for ranking. With limited memory space, prevalent in end devices or databases with heavy loads, it needs to scan the data in multiple passes. The idea is to gradually shrink the range of the queried quantile till it is small enough to fit in memory for ranking the result. Existing methods use deterministic sketches to determine the exact range of quantile, known as deterministic filter, which could be inefficient in range shrinking. In this study, we propose to shrink the ranges more aggressively, using randomized summaries such as KLL sketch. That is, with a high probability the quantile lies in a smaller range, namely probabilistic filter, determined by the randomized sketch. Specifically, we estimate the expected passes for determining the exact quantiles with probabilistic filters, and select a proper probability that can minimize the expected passes. Analyses show that our exact quantile determination method can terminate in P passes with 1-δ confidence, storing O(N 1/P logP-1/2P (1/δ)) items, close to the lower bound Ømega(N1/P) for a fixed δ. The approach has been deployed as a function in an LSM-tree based time-series database Apache IoTDB. Remarkably, the randomized sketches can be pre-computed for the immutable SSTables in LSM-tree. Moreover, multiple quantile queries could share the data passes for probabilistic filters in range estimation. Extensive experiments on real and synthetic datasets demonstrate the superiority of our proposal compared to the existing methods with deterministic filters. On average, our method takes 0.48 fewer passes and 18% of the time compared with the state-of-the-art deterministic sketch (GK sketch).

An Efficient NoSQL-Based Storage Schema for Large-Scale Time Series Data

In IoT (internet of things), most data from the connected devices change with time and have sampling intervals, which are called time-series data. It is challenging to design a time series storage model that can write massive time-series data in a short time and can query and analyze the persistent time-series data for a long time. This paper constructs the RHTSDB (Redis-HBase Time Series Database) storage model based on Redis and HBase. RHTSDB uses the memory database Redis (Remote Dictionary Server) to cache massive time-series data, providing efficient data storage and query functions. HBase is used in RHTSDB for long-term storage of time-series data to realize their persistence. The paper designs a cold and hot separation mechanism for time-series data, where the infrequently accessed cold data are stored in HBase, and the frequently accessed and latest data are stored in Redis. Experiments verify that RHTSDB has apparent advantages over Apache IoTDB and HBase in data intake and query efficiency.

Pricing of Climate Risks in the Capital Market of South Africa

ABSTRACT: Climate risk represents an increasingly vital issue to countries, companies, and institutional investors, making it a reality but not a distant threat to humanity. Considering the effects of climate risks on firms' financial indices and financing options, the study investigates whether climate risk is priced by the capital markets of South Africa. The study used reported carbon emissions as a measure of climate risk of 81 listed companies in the Johannesburg Stock Exchange from 2011 to 2020 to examine whether climate risk is considered and priced by the South Africa capital market. Data was sourced from DataStream database- a global financial and macroeconomic time-series database providing data on equities, stock market indices, currencies, company fundamentals, fixed income securities, and key economic indicators. We used the two-step system Generalized Method of Moments estimation technique that copes with endogeneity concerns to corroborate the effects of climate risk on cost of capital and capital structure. We find that climate risk is priced in both cost of debt capital and cost of equity capital. Specifically, we find that an increase in a firm's exposure to climate risk increases the cost associated with issuing debt and equity capital. We also find that climate risk exposure decreases the debt-equity ratio. Additionally, the study showed that firm size, leverage ratio, capitalization, profitability, and turnover affect both cost of capital and capital structure of listed firms. The study concludes that climate risk is priced in cost of financing in the capital market of South Africa. The study recommends that firms should invest in installing eco-friendly machinery that aligns with changing market expectations in order to reduce their carbon emissions. The study therefore highlights the need for companies to proactively assess and manage climate risks, incorporate climate considerations into their strategic decision-making, and enhance their resilience to climate-related challenges.

Weighted Fuzzy Clustering for Time Series With Trend-Based Information Granulation.

The highly dimensional characteristic of time series brings many challenges on direct mining time series, such as high cost in time and space. Granular computing provides a potential strategy for representing and dealing with time series at a higher level of abstraction. In this study, we propose an information granulation-based weighted fuzzy C -means (wFCM) method to realize time-series clustering, which could avoid high dimensionality processing and provide a concise and visible granular prototype for each cluster. In this method, each time series is first transformed into a series of information granules with trend following the principle of justifiable granularity. The formed granular time series can well capture the main features lying in the original time series and help realize dimensionality reduction. Then, the wFCM method is developed to complete time-series clustering in the granular space. Here, the dynamic time warping (DTW) is extended to capture the similarity for trend-based granular time series. Furthermore, the weighted DTW barycenter averaging is introduced to derive prototypes presented in a granular format, capturing the level, the fluctuation, and the changing trend, which are meaningful and understandable clustering results. The experiments conducted on real-world datasets coming from the UCR time-series database and Chinese stocks are presented to illustrate the effectiveness and practicality of the designed time-series clustering model.

Linked data to streaming data sensors sources for direct access

Goal: Industrial operations are complex, and data sensors assure safety and reliable information for production improvements. Multiple stakeholders can take advantage of data acquisition for post-analysis and process control. Providing users and systems with friendly access to operating data is fundamental to the digital transition in the industry 4.0 scenario. Linking data and systems over ontologies and Industry Foundation Classes will boost supply chain performance in many layers. This paper presents the concept of valid data points over Uniform Resource Identifiers for sensor time-series into triples stores via Application Programming Interfaces. Design/Methodology/Approach: A streaming data source approach to integrating industrial sensor data and sharing it via Uniform Resource Identifiers is developed and tested using Node-Red with multiple data connection types, such as the Industry Foundation Classes and open-source time series databases. Results: The detailed proof of concept presented valid the feasibility of sharing sensor data via Uniform Resource Identifiers. The findings provide a backbone of a system able to interop Message Queuing Telemetry Transport data, Resource Description Framework datasets and Industry Foundation Classes schema. Limitations of the investigation: The system envisaged was tested using simulated data. However, it is expected to have similar results from real data use. Nevertheless, more research will be needed to implement more features, such as three-dimensional object integration. Practical implications: The solution designed and tested presented can be used in practice for companies that desire to expand via linked data shareability and interoperability. Also, researchers can advance the solution for specific features, such as creating an open-source data query and manipulation language. Originality/Value: This paper examines future deployments of systems-to-systems interoperability targeting user-friendly data shareability. It is meant to be useful for industrial and academic developments.

A statistical characteristics preserving watermarking scheme for time series databases

Database watermarking is one of the most effective methods to protect the copyright of databases. However, traditional database watermarking has a potential drawback: watermark embedding will change the distribution of data, which may affect the use and analysis of databases. Considering that most analyses are based on the statistical characteristics of the target database, keeping the consistency of the statistical characteristics is the key to ensuring analyzability. Since statistical characteristics analysis is performed in groups, compared with traditional relational databases, time series databases (TSDBs) have obvious time-grouping characteristics and are more valuable for analysis. Therefore, this paper proposes a robust watermarking algorithm for time series databases, effectively ensuring the consistency of statistical characteristics. Based on the time-group characteristics of TSDBs, we propose a three-step watermarking method, which is based on linear regression, error compensation, and watermark verification, named RCV. According to the properties of the linear regression model and error compensation, the proposed watermark method generates a series of data that have the same statistical characteristics. Then, the verification mechanism is performed to validate the generated data until it conveys the target watermark message. Compared with the existing methods, our method achieves superior robustness and preserves constant statistical properties better.r.

Selecting a Monitoring Technology for a Control System of Distributed Oil Production Facilities

The article proposes the structure of a SCADA system for monitoring and control of oil production facilities that are distributed over a large area. The main emphasis is on the selection of technology that will enable effective monitoring of the equipment of each oil well. Factors such as reliability, ease of use, availability of protection against third-party interference, as well as availability and accessibility of an open-source software code were taken into account. After reviewing the most common software platforms, a system based on Prometheus and Grafana was selected. It is a combination of the Prometheus time series database server and the Grafana information visualization and analysis system. The important factors that determined the choice of this platform were the availability of the open source code and a large library of ready-made templates for displaying the well parameters in real time. An example of the created visualization window of the dynamometer card of the well, built on the basis of the experimentally recorded data, is presented.