Time Series Databases Research Articles

Long‐term occurrence probabilities of intense geomagnetic storm events

A quantitative assessment of the occurrence probability of intense geomagnetic storms (peak Dst < −100 nT) has been investigated by analyzing the Dst index time series database from 1957 to 2001. The main purpose was to derive two parameters, the probable intensity ST and the occurrence frequency λt, that can act as proxies for long‐term space weather quantities. The intensity ST represents the expected maximum storm level with an occurrence rate of 1/T (a−1, where a is years) and has been derived from the probability density function (PDF) of extreme (∣Dst∣ > 280 nT) storms. The mathematical tool to determine this type of PDF is the extreme value modeling, which exhibits more accurate statistics for extreme behavior. Our results estimate S60 ≈ 589, indicating that the March 1989 storm (the event with the largest ∣Dst∣ in the database) corresponds to an event expected to occur only once every 60 a. The other parameter λt gives the average occurrence rate of storm events. We have tested the null hypothesis that the storm occurrence pattern can be modeled as a Poisson process represented by λt, where different λt exist for the active and quiet periods of the solar cycle. Ordinary χ2 tests of goodness of fit can not reject this hypothesis, except within the periods that include extremely frequent occurrences. The rate λt is approximately 2.3 (0.7) per 3 months in the active (quiet) period. A future practical application of this work is that the resultant Poisson probability will enable us to calculate the expected damage due to storms, which represent potential risks in space activities.

National Establishment Time-Series Database©: Audio of Presentation

For more than thirty-two million establishments, the National Establishment Time-Series (NETS) Database© provides time-series information on mobility patterns, employment and sales growth performance, job creation and destruction, business startups and failures, changes in primary markets, corporate affiliations, and historical Dun and Bradstreet ratings, to name a few.

National Establishment Time-Series Database©: Presentation

For more than thirty-two million establishments, the National Establishment Time-Series (NETS) Database© provides time-series information on mobility patterns, employment and sales growth performance, job creation and destruction, business startups and failures, changes in primary markets, corporate affiliations, and historical Dun and Bradstreet ratings, to name a few.

Perception-based approach to time series data mining

Time series data mining (TSDM) techniques permit exploring large amounts of time series data in search of consistent patterns and/or interesting relationships between variables. TSDM is becoming increasingly important as a knowledge management tool where it is expected to reveal knowledge structures that can guide decision making in conditions of limited certainty. Human decision making in problems related with analysis of time series databases is usually based on perceptions like “end of the day”, “high temperature”, “quickly increasing”, “possible”, etc. Though many effective algorithms of TSDM have been developed, the integration of TSDM algorithms with human decision making procedures is still an open problem. In this paper, we consider architecture of perception-based decision making system in time series databases domains integrating perception-based TSDM, computing with words and perceptions, and expert knowledge. The new tasks which should be solved by the perception-based TSDM methods to enable their integration in such systems are discussed. These tasks include: precisiation of perceptions, shape pattern identification, and pattern retranslation. We show how different methods developed so far in TSDM for manipulation of perception-based information can be used for development of a fuzzy perception-based TSDM approach. This approach is grounded in computing with words and perceptions permitting to formalize human perception-based inference mechanisms. The discussion is illustrated by examples from economics, finance, meteorology, medicine, etc.

National Establishment Time-Series Database©: Data Overview

For more than thirty-two million establishments, the National Establishment Time-Series (NETS) Database© provides time-series information on mobility patterns, employment and sales growth performance, job creation and destruction, business startups and failures, changes in primary markets, corporate affiliations, and historical Dun and Bradstreet ratings, to name a few. Download the slides at: http://papers.ssrn.com/paper=1027941 Download the audio file at: http://papers.ssrn.com/paper=1028868

National Establishment Time-Series (Nets) Database: Coffee Analysis

For more than thirty-two million establishments, the National Establishment Time-Series (NETS) Database provides time-series information on mobility patterns, employment and sales growth performance, job creation and destruction, business startups and failures, changes in primary markets, corporate affiliations, and historical Dun and Bradstreet ratings, to name a few. Developed through a joint venture with Dun and Bradstreet, it covers more than 250 fields and seventeen years of historical information thoroughly documenting economic activity from 1989 through 2006. This unique business establishment data-base is the only commercially-available historical database of its kind on the market. This file shows a sample analysis of the coffee industry.

National Establishment Time-Series Database©: Sample File 2006

For more than thirty-two million establishments, the National Establishment Time-Series (NETS) Database provides time-series information on mobility patterns, employment and sales growth performance, job creation and destruction, business startups and failures, changes in primary markets, corporate affiliations, and historical Dun and Bradstreet ratings, to name a few. Developed through a joint venture with Dun and Bradstreet, it covers more than 250 fields and seventeen years of historical information thoroughly documenting economic activity from 1989 through 2006. This unique business establishment data-base is the only commercially-available historical database of its kind on the market. This file shows a sample of company-level information available.

A compact multi-resolution index for variable length queries in time series databases

We study the problem of searching similar patterns in time series data for variable length queries. Recently, a multi-resolution indexing technique (MRI) was proposed in (Kahveci and Singh, in proceedings of the international conference on data engineering, pp. 273–282, 2001; Kahveci and Singh, IEEE Trans Knowl Data Eng 16(4):418–433, 2004) to address this problem, which uses compression as an additional step to reduce the index size. In this paper, we propose an alternative technique, called compact MRI (CMRI), which uses adaptive piecewise constant approximation (APCA) representation as dimensionality reduction technique, and which occupies much less space without requiring compression. We implemented both MRI and CMRI, and conducted extensive experiments to evaluate and compare their performance on real stock data as well as synthetic. Our results indicate that CMRI provides a much better precision ranging from 0.75 to 0.89 on real data, and from 0.80 to 0.95 on synthetic data, while for MRI, these ranges are from 0.16 to 0.34, and from 0.03 to 0.65, respectively. Compared to sequential scan, we found that CMRI is 4–30 times faster and the number of disk I/Os it required is close to minimal. In terms of storage utilization, CMRI occupies 1% of the memory occupied by MRI. These results and analysis show CMRI to be an efficient and scalable indexing technique for large time series databases.

Modelling UK Black Smoke and Sulphur Dioxide Concentrations, 1955–2001 to Estimate Lifecourse air Pollution Exposures

ISEE-590 Objective: The UK has a long-running set of national air pollution data with over 3100 monitoring stations in existence at various points from 1955. This project aims to construct a detailed time series database of modeled concentrations of Black Smoke (BS) and SO2 for UK that can be applied to existing British cohort studies. This offers a unique opportunity to investigate health effects of lifecourse air pollution exposures. Materials and Methods: Different methods for modeling concentration surfaces are in development including Kriging, regression mapping, and dispersion modeling. The initial steps reported here are Ordinary Kriging of monitored annual BS concentrations for sample years (1962, 1971, 1981, and 1991), including data from stations with concentrations available for ≥75% of days in the year. Supplementary datasets for possible inclusion in regression and dispersion modeling have also been identified. Results: Kriging models were developed using 75% of the data with remaining data reserved for validation. Models worked well for 1962 and 1971 (r2=0.6 and 0.6, RMSE 49.3 and 18.8) but were less good for later years (r2=0.5 and 0.4, RMSE 7.8 and 6.6) when monitoring station numbers fell from around 1000 to less than 200. Highest concentrations were consistently observed in northern England (Liverpool, Manchester, and Tyneside), with lowest values in Scotland and the South-West. BS concentrations declined markedly over time with 90th percentile values falling from 270 μg/m3 in 1962 to 25 μg/m3 in 1991. Additional historical datasets identified for use in other modeling techniques included population density, land use, emissions inventories, gas networks, smoke control areas (relating to domestic fuel use), and the railway network. Conclusions: Kriging provides reasonable baseline maps against which more sophisticated modeling techniques can be compared. A range of datasets including air pollution concentrations are available to aid modeling of historical exposures to air pollution in UK.

Using multiple indexes for efficient subsequence matching in time-series databases

A time- series database is a set of data sequences, each of which is a list of changing values of an object in a given period of time. Subsequence matching is an operation that searches for such data subsequences whose changing patterns are similar to a query sequence in a time-series database. This paper addresses a performance issue of time-series subsequence matching. First, we quantitatively examine the performance degradation caused by the window size effect, and then show that the performance of subsequence matching with a single index is not satisfactory in real applications. We claim that index interpolation is a fairly effective tool to solve this problem. Index interpolation performs subsequence matching by selecting the most appropriate one from multiple indexes built on windows of their distinct sizes. For index interpolation, we need to decide the sizes of windows for multiple indexes to be built. In this paper, we solve the problem of selecting optimal window sizes from the perspective of physical database design. Given a set of pairs 〈 length, frequency〉 of query sequences to be performed in a target application and a set of window sizes for building multiple indexes, we devise a formula that estimates the overall cost of all the subsequence matchings performed in a target application. By using this formula, we propose an algorithm that determines the optimal window sizes for maximizing the performance of entire subsequence matchings. We formally prove the optimality as well as the effectiveness of the algorithm. Finally, we show the superiority of our approach by performing extensive experiments with a real-life stock data set and a large volume of synthetic data sets.

Performance bottleneck of subsequence matching in time-series databases: Observation, solution, and performance evaluation

Subsequence matching is an operation that finds subsequences whose changing patterns are similar to a given query sequence from time-series databases. This paper identifies a performance bottleneck in subsequence matching, and then proposes an effective method that substantially improves the performance of entire subsequence matching by resolving the performance bottleneck. First, we analyze the disk access and CPU processing times required during the index searching and post-processing steps of subsequence matching through preliminary experiments. Based on these results, we show that the post-processing step is a main performance bottleneck in subsequence matching. Then, we argue that the optimization of the post-processing step is a crucial issue overlooked in previous approaches. In order to resolve the performance bottleneck, we propose a simple yet highly effective method for expediting the post-processing step. By rearranging the order of candidate subsequences to be compared with a query sequence, our method completely eliminates the redundancies of disk accesses and CPU processing that occur in the post-processing step. Our method is fairly efficient, and does not incur any false dismissal. We quantitatively demonstrate the superiority of our method through extensive experimentation. The results show that our method produces a significantly faster post-processing step; When using a data set of real-world stock sequences, our method was 43.36–96.75 times faster than previous methods, and when using data sets of large numbers of synthetic sequences, our method was 12.48–26.95 times faster than previous methods. Also, the results show that our method reduces the weight of the post-processing step over entire subsequence matching from more than 97% to less than 67%. This implies that our method successfully resolves the performance bottleneck in subsequence matching. As a result, our method provides excellent performance in entire subsequence matching. Compared with previous methods, our method is 16.17–32.64 times faster when using a data set of real-world stock sequences and 8.64–14.29 times faster when using data sets of large numbers of synthetic sequences.

Pattern recognition in time series database

Today, there are more and more time series data that coexist with other data. These data exist in useful and understandable patterns. Data management of time series data must take into account an i...

Efficient moving average transform-based subsequence matching algorithms in time-series databases

Moving average transform is very useful in finding the trend of time-series data by reducing the effect of noise, and has been used in many areas such as econometrics. Previous subsequence matching methods with moving average transform, however, are problematic in that, since they must build multiple indexes in supporting transform of arbitrary order, they incur index overhead both in storage space and in update maintenance. To solve this problem, we propose a single-index approach to subsequence matching that supports moving average transform of arbitrary order in time-series databases. Using the single-index approach, we can reduce both the storage space and the index maintenance overhead. In explaining the single-index approach, we first introduce the notion of poly-order moving average transform by generalizing the original definition of moving average transform. We then formally prove the correctness of poly-order transform-based subsequence matching. We also propose two subsequence matching methods based on poly-order transform that efficiently support moving average transform of arbitrary order. Experimental results for real stock data show that, compared with the sequential scan, our methods improve average performance significantly, by a factor of 22.6–33.6. Also, compared with cases in which an index is built for every moving average order, our methods reduce storage space and maintenance effort significantly while incurring only marginal performance degradation. Our approach entails the additional advantage of being generalized to support many other transforms in addition to moving average transform. Therefore, we believe that our approach will be widely used in many transform-based subsequence matching methods.

Query-sensitive embeddings

A common problem in many types of databases is retrieving the most similar matches to a query object. Finding these matches in a large database can be too slow to be practical, especially in domains where objects are compared using computationally expensive similarity (or distance) measures. Embedding methods can significantly speed-up retrieval by mapping objects into a vector space, where distances can be measured rapidly using a Minkowski metric. In this article we present a novel way to improve embedding quality. In particular, we propose to construct embeddings that use a query-sensitive distance measure for the target space of the embedding. This distance measure is used to compare those vectors that the query and database objects are mapped to. The term “query-sensitive” means that the distance measure changes, depending on the current query object. We demonstrate theoretically that using a query-sensitive distance measure increases the modeling power of embeddings and allows them to capture more of the structure of the original space. We also demonstrate experimentally that query-sensitive embeddings can significantly improve retrieval performance. In experiments with an image database of handwritten digits and a time-series database, the proposed method outperforms existing state-of-the-art non-Euclidean indexing methods, meaning that it provides significantly better tradeoffs between efficiency and retrieval accuracy.

Correlating burst events on streaming stock market data

We address the problem of monitoring and identification of correlated burst patterns in multi-stream time series databases. We follow a two-step methodology: first we identify the burst sections in our data and subsequently we store them for easy retrieval in an efficient in-memory index. The burst detection scheme imposes a variable threshold on the examined data and takes advantage of the skewed distribution that is typically encountered in many applications. The detected bursts are compacted into burst intervals and stored in an interval index. The index facilitates the identification of correlated bursts by performing very efficient overlap operations on the stored burst regions. We present the merits of the proposed indexing scheme through a thorough analysis of its complexity. We also manifest the real-time response of our burst indexing technique, and demonstrate the usefulness of the approach for correlating surprising volume trading events using historical stock data of the NY stock exchange. While the focus of this work is on financial data, the proposed methods and data-structures can find applications for anomaly or novelty detection in telecommunication, network traffic and medical data.

General Hierarchical Model (GHM) to measure similarity of time series

Similarity query is a frequent subroutine in time series database to find the similar time series of the given one. In this process, similarity measure plays a very important part. The previous methods do not consider the relation between point correspondences and the importance (role) of the points on the content of time series during measuring similarity, resulting in their low accuracies in many real applications. In the paper, we propose a General Hierarchical Model (GHM), which determines the point correspondences by the hierarchies of points. It partitions the points of time series into different hierarchies, and then the points are restricted to be compared with the ones in the same hierarchy. The practical methods can be implemented based on the model with any real requirements, e.g. FFT Hierarchical Measures (FHM) given in this paper. And the hierarchical filtering methods of GHM are provided for range and k -NN queries respectively. Finally, two common data sets were used in k -NN query and clustering experiments to test the effectiveness of our approach and others. The time performance comparisons of all the tested methods were performed using the synthetic data set with various sizes. The experimental results show the superiority of our approach over the competitors. And we also give the experimental powers of the filtering methods proposed in the queries.

Efficient Discovery of Unusual Patterns in Time Series

The problem of finding a specified pattern in a time series database (i.e., query by content) has received much attention and is now a relatively mature field. In contrast, the important problem of enumerating all surprising or interesting patterns has received far less attention. This problem requires a meaningful definition of and an efficient search technique. All previous attempts at finding surprising patterns in time series use a very limited notion of surprise, and/or do not scale to massive datasets. To overcome these limitations we propose a novel technique that defines a pattern surprising if the frequency of its occurrence differs substantially from that expected by chance, given some previously seen data. This notion has the advantage of not requiring the user to explicitly define what is a surprising pattern, which may be hard, or perhaps impossible, to elicit from a domain expert. Instead, the user gives the algorithm a collection of previously observed normal data. Our algorithm uses a suffix tree to efficiently encode the frequency of all observed patterns and allows a Markov model to predict the expected frequency of previously unobserved patterns. Once the suffix tree has been constructed, a measure of surprise for all the patterns in a new database can be determined in time and space linear in the size of the database. We demonstrate the utility of our approach with an extensive experimental evaluation.

BitTableFI: An efficient mining frequent itemsets algorithm

Mining frequent itemsets in transaction databases, time-series databases and many other kinds of databases is an important task and has been studied popularly in data mining research. The problem of mining frequent itemsets can be solved by constructing a candidate set of itemsets first, and then, identifying those itemsets that meet the frequent itemset requirement within this candidate set. Most of the previous research mainly focuses on pruning to reduce the candidate itemsets amounts and the times of scanning databases. However, many algorithms adopt an Apriori-like candidate itemsets generation and support count approach that is the most time-wasted process. To address this issue, the paper proposes an effective algorithm named as BitTableFI. In the algorithm, a special data structure BitTable is used horizontally and vertically to compress database for quick candidate itemsets generation and support count, respectively. The algorithm can also be used in many Apriori-like algorithms to improve the performance. Experiments with both synthetic and real databases show that BitTableFI outperforms Apriori and CBAR which uses ClusterTable for quick support count.

Pattern recognition in time series database: A case study on financial database

Today, there are more and more time series data that coexist with other data. These data exist in useful and understandable patterns. Data management of time series data must take into account an integrated approach. However, many researches face numeric data attributes. Therefore, the need for time series data mining tool has become extremely important. The purpose of this paper is to provide a novel pattern in mining architecture with mixed attributes that uses a systematic approach in the financial database information mining. Time series pattern mining (TSPM) architecture combines the extended visualization-induced self-organizing map algorithm and the extended Naive Bayesian algorithm. This mining architecture can simulate human intelligence and discover patterns automatically. The TSPM approach also demonstrates good returns in pattern research.

The Stepwise Dimensionality Increasing (SDI) Index for High-Dimensional Data

Similarity search is a powerful paradigm for image and multimedia databases, time series databases, and DNA and protein sequence databases. Objects are represented by high-dimensional feature vectors based on color, texture, and shape, in the case of images, for example object similarity is usually implemented via k-nearest neighbor (k-NN) queries, determined by the distance of the endpoints of the feature vectors. The cost of processing k-NN queries via a sequential scan increases with the number of objects and the number of dimensions. Multi-dimensional indexing structures can be used to improve the efficiency of k-NN query processing, but lose their effectiveness as the dimensionality increases. The curse of dimensionality manifests itself in the form of increased overlap among the nodes of the index, so that a high fraction of index pages are touched in processing k-NN queries. The increased dimensionality results in a reduced fanout and an increased index height. We propose a stepwise dimensionality increasing (SDI)-tree index, which aims at reducing the number of disk accesses and CPU processing cost. The index is built using feature vectors transformed via principal component analysis. Dimensions are retained in non-increasing order of their variance according to a parameter p, which specifies the incremental fraction of variance at each level of the index. The optimal value for p is determined experimentally. Experiments on three datasets have shown that SDI-trees access fewer disk pages and incur less CPU time than SR-trees, VAMSR-trees, vector approximation (VA)-Files and the recently proposed iDistance method. In CPU time SDI outperforms the sequential scan and OMNI methods.