Time Series Databases Research Articles

Finding K Most Significant Motifs in Big Time Series Data

Abstract An efficient discovery algorithm of frequently occurring patterns, called motifs, in a time series would be useful as a tool for summarizing and visualizing big time series databases. In this paper, we propose an efficient approximate algorithm, called DiscMotifs, to discover the K most significant (KMS) motifs from time series. First, the proposed algorithm transforms the time series into a SAX representation and then the algorithm divides the SAX representation into subsequences. Next, these subsequences are linearized by projecting them into a one-dimensional space based on their distances form a randomly selected reference point, or a subsequence. By utilizing the linear ordering of subsequences, DiscMotifs efficiently discovers the KMS motifs. DiscMotifs algorithm requires a storage space linear to the number of subsequences. We demonstrate the feasibility of this approach on several synthetic and real application datasets.

ATLAS Operational Monitoring Data Archival and Visualization

The Information Service (IS) is an integral part of the Trigger and Data Acquisition (TDAQ) system of the ATLAS experiment at the Large Hadron Collider (LHC) at CERN. The IS allows online publication of operational monitoring data, and it is used by all sub-systems and sub-detectors of the experiment to constantly monitor their hardware and software components including more than 25000 applications running on more than 3000 computers. The Persistent Back-End for the ATLAS Information System (PBEAST) service stores all raw operational monitoring data for the lifetime of the experiment and provides programming and graphical interfaces to access them including Grafana dashboards and notebooks based on the CERN SWAN platform. During the ATLAS data taking sessions (for the full LHC Run 2 period) PBEAST acquired data at an average information update rate of 200 kHz and stored 20 TB of highly compacted and compressed data per year. This paper reports how over six years PBEAST became an essential piece of the experiment operations including details of the challenging requirements, the failures and successes of the various attempted implementations, the new types of monitoring data and the results of the time-series database technology evaluations for the improvements towards LHC Run 3.

Evaluation of a new visualization and analytics solution for slow control data for large scale experiments

Large experiments in high energy physics require efficient and scalable monitoring solutions to digest data of the detector control system. Plotting multiple graphs in the slow control system and extracting historical data for long time periods are resource intensive tasks. The proposed solution leverages the new virtualization, data analytics and visualization technologies such as InfluxDB time-series database for faster access large scale data, Grafana to visualize time-series data and an OpenShift container platform to automate build, deployment, and management of application. The monitoring service runs separately from the control system thus reduces a workload on the control system computing resources. As an example, a test version of the new monitoring was applied to the ATLAS Tile Calorimeter using the CERN Cloud Process as a Service platform. Many dashboards in Grafana have been created to monitor and analyse behaviour of the High Voltage distribution system. They visualize not only values measured by the control system, but also run information and analytics data (difference, deviation, etc.). The new monitoring with a feature-rich visualization, filtering possibilities and analytics tools allows to extend detector control and monitoring capabilities and can help experts working on large scale experiments.

Generation of Trajectory and Contact Force Based on Time-Series Database of Writing

Generation of Trajectory and Contact Force Based on Time-Series Database of Writing

Time is money: Dynamic-model-based time series data-mining for correlation analysis of commodity sales

The correlation analysis of commodity sales is very important in cross-marketing. A means of undertaking dynamic-model-based time series data-mining was proposed to analyze the sales correlations among different commodities. A dynamic model comprises some distance models in different observation windows for a time series database that is transformed from a commodities transaction database. There are sales correlations in two time series at different times, and this may produce valuable rules and knowledge for those who wish to practice cross-marketing and earn greater profits. It means that observation time points denoting the time at which the sales correlation occurs constitute important information. The dynamic model that leverages the techniques inherent in time series data-mining can uncover the kinds of commodities that have similar sales trends and how those sales trends change within a particular time period, which indicates that the “right” commodities can be commended to the “right” customers at the “right” time. Moreover, some of the time periods used to pinpoint similar sales patterns can be used to retrieve much more valuable information, which can in turn be used to increase the sales of the correlated commodities and improve market share and profits. Analysis results of retail commodities datasets indicate that the proposed method takes into consideration the time factor, and can uncover interesting sales patterns by which to improve cross-marketing quality. Moreover, the algorithm can be regarded as an intelligent component of the recommendation and marketing systems so that human–computer interaction system can make intelligent decision.

Present State-of-The-Art of Association Rule Mining Algorithms

A Data mining is the method of extracting useful information from various repositories such as Relational Database, Transaction database, spatial database, Temporal and Time-series database, Data Warehouses, World Wide Web. Various functionalities of Data mining include Characterization and Discrimination, Classification and prediction, Association Rule Mining, Cluster analysis, Evolutionary analysis. Association Rule mining is one of the most important techniques of Data Mining, that aims at extracting interesting relationships within the data. In this paper we study various Association Rule mining algorithms, also compare them by using synthetic data sets, and we provide the results obtained from the experimental analysis.

Suitability Of Influxdb Database For Iot Applications

Large amounts of data are generated every moment by connected objects creating Internet of Things (IoT). IoT isn’t about things; it’s about the data those things create and collect. Organizations rely on this data to provide better user experiences, to make smarter business decisions, and ultimately fuel their growth. However, none of this is possible without a reliable database that is able to handle the massive amounts of data generated by IoT devices. Relational databases are known for being flexible, easy to work with, and mature but they aren’t particularly known for is scale, which prompted the creation of NoSQL databases. Another thing to note is that IoT data is time-series in nature. In this paper we are discussed and compare about top five time-series database like InfluxDB, Kdb+, Graphite, Prometheus and RRDtool.

Forecasting across time series databases using recurrent neural networks on groups of similar series: A clustering approach

With the advent of Big Data, nowadays in many applications databases containing large quantities of similar time series are available. Forecasting time series in these domains with traditional univariate forecasting procedures leaves great potentials for producing accurate forecasts untapped. Recurrent neural networks (RNNs), and in particular Long Short Term Memory (LSTM) networks, have proven recently that they are able to outperform state-of-the-art univariate time series forecasting methods in this context, when trained across all available time series. However, if the time series database is heterogeneous, accuracy may degenerate, so that on the way towards fully automatic forecasting methods in this space, a notion of similarity between the time series needs to be built into the methods. To this end, we present a prediction model that can be used with different types of RNN models on subgroups of similar time series, which are identified by time series clustering techniques. We assess our proposed methodology using LSTM networks, a widely popular RNN variant, together with various clustering algorithms, such as kMeans, DBScan, Partition Around Medoids (PAM), and Snob. Our method achieves competitive results on benchmarking datasets under competition evaluation procedures. In particular, in terms of mean sMAPE accuracy it consistently outperforms the baseline LSTM model, and outperforms all other methods on the CIF2016 forecasting competition dataset.

A Search Method to Support Temporal Transcriptome Analysis.

Recently, there has been an increasing interest in mining time series databases with a focus on data representation. We propose a hybrid statistical and temporal logic model to yield search pattern that can be used for pattern matching and database search to identify novel and common patterns in temporal expression experiments. The method accounts for various challenges that can be found in publicly available gene expression databases.

Dual-PISA: An index for aggregation operations on time series data

Aggregation operations play an essential role in time series database management. As the number of data increases, it is difficult for current solutions, such as summary table and MapReduce-based methods to respond to such queries with low latency. Other approaches, such as segment tree-based methods, have a poor insertion performance when the data size exceeds the available memory. This paper proposes a Persistent Index for Segmented Aggregations (PISA), which has fast insertion performance and low latency for aggregation queries. PISA uses a forest to overcome the performance disadvantage of insertion in traditional segment trees. By defining two kinds of tags, namely code number and serial number, we propose an algorithm to accelerate queries by avoiding unnecessary reading data on disk. Additionally, we extend it to Dual-PISA to tolerate a range of unordered data, which is very important in the real world. Dual-PISA is stored on disk and is hugely memory-efficient — only takes a few hundred bytes of memory for billions of data points. Dual-PISA can be easily implemented on both traditional databases and NoSQL systems. It handles aggregation queries within milliseconds on a commodity server, for a time range that contains tens of billions of data points.

Hydrological stream data pipeline framework based on IoTDB

With the increasing amount of hydrological data in Chuhe river basin, the traditional relational database has been unable to meet the needs of users, which not only makes it difficult to achieve low latency and high throughput in the real-time transmission of hydrological data, but also causes the phenomenon of long time or even system crash when querying large amount of annual water-level data. To solve this problem, this paper proposes a stream data pipeline framework based on timeseries databases IoTDB and Kafka, which can provide services for hydrological early warning and anomaly detection researchers. Based on the hydrological sensor data of Chuhe river, the processing scenarios of sensor stream data are set and compared with other NoSQL (HBase, MongoDB, RiakTS and Redis) in different scenarios. The performance and workload of different NoSQL in this data pipeline are tested. Finally, it is docked with Flink real-time stream data processing platform and compared with other data pipelines. The experimental results show that the stream data pipeline composed of IoTDB, Kafka and Flink is outstanding in data acquisition, transmission, incremental query and data analysis.

Polymorphic accident inversion method in on-line monitoring system of power transmission and transformation equipment

Most of the equipment on-line monitoring system does not have the function of accident inversion or weak function. Even if it has the function of inversion, it only records the real-time steady-state information of a specific type within a period of time before and after the time of the accident, which cannot meet the actual needs of customers. This paper proposes a polymorphic accident inversion method based on time series database, equipment on-line monitoring of the site of all collected data by time sequence database for high-speed storage, reappear in the inversion environment equipment failure accident system before and after all of the conditions and for the whole process of continuous play, for the analysis of status evaluation for power grid provides a simulation environment of inversion and effective data to support. Based on the design idea of polymorphism, the user accident inversion requirements under different scenarios are satisfied.

The Forces of Imperialism: Internalist and Global Explanations of the Anglo-European Empires, 1750–1960

ABSTRACTThe authors explain the imperial expansion of the Anglo-European states from the 18th century to the 20th century using an original pooled time series database on rates of colonization by the Western powers. They employ a negative binomial regression that tests different theories, from classical internalist theories that emphasize domestic or national-level factors, to more global theories that explain formal imperialism by reference to the wider international context in which states operate. Testing these different theories, the authors find that states are most likely to engage in formal imperialism when (a) the world-system is at its most economically competitive, (b) when those societies are more militarized, and (c) when they have more power capabilities relative to other states.

The Density of Convenience Retail and the Type of Traffic—Commuting, Local and Retail

We use data from the National Establishment Time Series (NETS) database to examine gas station density in metropolitan areas within the State of Wisconsin over the 2010–2012 period. Our results suggest that while gas station density is affected by overall traffic volume, it is more important to know what type of traffic is passing through the market area. Specifically, after controlling for total volume having more retail increases the density of gas stations in an area. Consistent with gasoline purchases being part of a shopping trip, destination retail (department and home improvement stores) has the largest impact on gasoline station density. The benefits of retail agglomerations are evident.

Krishna Sudarsana—A Z-Space Interest Measure for Mining Similarity Profiled Temporal Association Patterns

Similarity profiled association mining from time stamped transaction databases is an important topic of research relatively less addressed in the field of temporal data mining. Mining temporal patterns from these time series databases requires choosing and applying similarity measure for similarity computations and subsequently pruning temporal patterns. This research proposes a novel z-space based interest measure named as Krishna Sudarsana for time-stamped transaction databases by extending interest measure Srihass proposed in previous research. Krishna Sudarsana is designed by using the product based fuzzy Gaussian membership function and performs similarity computations in z-space to determine the similarity degree between any two temporal patterns. The interest measure is designed by considering z-values between z = 0 and z = 3.09. Applying the Krishna Sudarsana requires moving the threshold value given by user to a different transformation space (z-space) which is a defined as a function of standard deviation. In addition to proposing interest measure, new expressions for standard deviation and equivalent z-space threshold are derived for similarity computations. For experimental evaluation, we considered Naive, Sequential and Spamine algorithms that applies Euclidean distance function and compared performance of these three approaches to Z-Spamine algorithm that uses Krishna Sudarsana by choosing various test cases. Experiment results proved the performance of the proposed approach is better to Sequential approach that uses snapshot database scan strategy and Spamine approach that uses lattice based database scan strategy.

Similarity join on time series by utilizing a dynamic segmentation index

Similarity join on time series databases is an essential operation for data analysis applications. Due to the curse of dimensionality, it is not suitable to use traditional index techniques, such as R-tree and kd-tree. In the paper, a dynamic segment index (i.e., DSTree) is utilized to reduce the huge comparison cost on the similarity join on time series databases. However, the DSTree is designed for similarity search and only supports bound estimations between a time series and a batch of time series in a DSTree node. To make the DSTree suitable for the similarity join on time series databases, it is necessary to have tight bounds for the nodes to achieve a better pruning power, where the biggest challenge is that the DSTree nodes may have different segmentations. To solve the problem aforementioned, a segmentation alignment and synopsis evaluation method is proposed to support the estimation of DSTree nodes to significantly reduce the time cost by pruning unnecessary comparisons. Moreover, to make our approach I/O efficient, a caching strategy is proposed by taking advantage of both graph partitioning and the locality of the DSTree index. The efficiency and effectiveness of the proposed approaches are verified by experiments on real-life datasets.

EdgeDB: An Efficient Time-Series Database for Edge Computing

Massive time-series data streams from high-sampling-frequency sensors in Internet of Things (IoT) can overwhelm the networks connecting the sensors to centralized clouds. Thus, edge computing servers have to be introduced to locally store and analyze growing time-series data. Unfortunately, conventional time-series databases exhibit low efficiency on edge nodes with limited resources for both computation and storage. In this paper, we propose a highly efficient time-series database, called EdgeDB, to fully utilize the capacity of the edge nodes. EdgeDB effectively improves the performances of both inserting and retrieving data from ingest streams by efficiently merging multiple streams and optimizing the storage data structure concurrently. EdgeDB first compactly organizes multiple online streams into a tablet within a time window and embeds predefined aggregate query results together. EdgeDB adopts Time Partitioned Elastic Index (TPEI) to build indexing on all tablets, enhancing the time-range query performance while reducing the memory usage by optimizing the indexing storage. EdgeDB further develops Time Merged Tree (TMTree) to combine a set of tablets into a large one, significantly boosting the write throughput and potentially strengthening the performance of inter-tablet query. Extensive experiments based on real-world datasets show that, compared with the state-of-the-art time-series database BTrDB, EdgeDB achieves performance improvements of up to 2.2× in insert throughput, 3.6× in write throughput, and 67% in query latency with lower memory consumption.

Data Capsule: Representation of Heterogeneous Data in Cloud-Edge Computing

Nowadays, robots (including non-humanoid ones, like self-driving cars) are part of the most promising technologies, and they rise various computing requirements. Some of those requirements came from the fact that the robots are involved in highly dynamic environments and have to execute complex decision algorithms in real-time, while other requirements ask for batch processing of big data compatible datasets. In this paper, we propose a cloud architecture for optimizing data processing using a cloud-edge infrastructure. Besides the computational architecture, we develop a mathematical model for each type of entity in our proposal and a formal description of a data capsule, which represents a generic and flexible representation for unstructured units of data in time series databases. The architecture includes multiple processing platforms. We evaluate the proposed model in edge-cloud computing platforms designed for robots that run machine learning tasks.

LiveBox: A Self-Adaptive Forensic-Ready Service for Drones

Unmanned Aerial Vehicles (UAVs), or drones, are increasingly expected to operate in spaces populated by humans while avoiding injury to people or damaging property. However, incidents and accidents can, and increasingly do, happen. Traditional investigations of aircraft incidents require on-board flight data recorders (FDRs); however, these physical FDRs only work if the drone can be recovered. A further complication is that physical FDRs are too heavy to mount on light drones, hence not suitable for forensic digital investigations of drone flights. In this paper, we propose a self-adaptive software architecture, LiveBox, to make drones both forensic-ready and regulation compliant. We studied the feasibility of using distributed technologies for implementing the LiveBox reference architecture. In particular, we found that updates and queries of drone flight data and constraints can be treated as transactions using decentralised ledger technology (DLT), rather than a generic time-series database, to satisfy forensic tamper-proof requirements. However, DLTs such as Ethereum, have limits on throughput (i.e. transactions-per-second), making it harder to achieve regulation-compliance at runtime. To overcome this limitation, we present a self-adaptive reporting algorithm to dynamically reduce the precision of flight data without sacrificing the accuracy of runtime verification. Using a real-life scenario of drone delivery, we show that our proposed algorithm achieves a 46% reduction in bandwidth without losing accuracy in satisfying both tamper-proof and regulation-compliant requirements.

Energy renovation rates in the Netherlands – comparing long and short term prediction methods

Energy renovation rates in the Netherlands – comparing long and short term prediction methods