Abstract
In the last 5 years there have been a large number of new time series classification algorithms proposed in the literature. These algorithms have been evaluated on subsets of the 47 data sets in the University of California, Riverside time series classification archive. The archive has recently been expanded to 85 data sets, over half of which have been donated by researchers at the University of East Anglia. Aspects of previous evaluations have made comparisons between algorithms difficult. For example, several different programming languages have been used, experiments involved a single train/test split and some used normalised data whilst others did not. The relaunch of the archive provides a timely opportunity to thoroughly evaluate algorithms on a larger number of datasets. We have implemented 18 recently proposed algorithms in a common Java framework and compared them against two standard benchmark classifiers (and each other) by performing 100 resampling experiments on each of the 85 datasets. We use these results to test several hypotheses relating to whether the algorithms are significantly more accurate than the benchmarks and each other. Our results indicate that only nine of these algorithms are significantly more accurate than both benchmarks and that one classifier, the collective of transformation ensembles, is significantly more accurate than all of the others. All of our experiments and results are reproducible: we release all of our code, results and experimental details and we hope these experiments form the basis for more robust testing of new algorithms in the future.
Highlights
Time series classification (TSC) problems are differentiated from traditional classification problems because the attributes are ordered
To help understand the variation of techniques by grouping, we look at the relative performance of algorithms without Collection of transformation ensembles (COTE) and DTW features (DTWF)
Our results indicate that COTE is, on average, clearly superior to other published techniques
Summary
Time series classification (TSC) problems are differentiated from traditional classification problems because the attributes are ordered. Most experiments involve evaluation on over forty data sets, often with sophisticated significance testing and most authors release source code. There is the tendency to not provide code that performs model selection, which can lead to suspicions that parameters were selected to minimize test error, biasing the results To address these problems we have implemented 18 different TSC algorithms in Java, integrated with the WEKA toolkit (Hall et al 2009). Of those 9 significantly better than both benchmarks, by far the best classifier is COTE (Bagnall et al 2015), an algorithm proposed by a subset of the current authors It is on average over 8% more accurate than either benchmark.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have