High-dimensional Data Streams Research Articles

Real-time image recognition using weighted spatial pyramid networks

The latest-generation earth observation instruments on airborne and satellite platforms are currently producing an almost continuous high-dimensional data stream. This exponentially growing data poses a new challenge for real-time image processing and recognition. Making full and effective use of the spectral information and spatial structure information of high-resolution remote sensing image is the key to the processing and recognition of high-resolution remote sensing data. In this paper, the adaptive multipoint moment estimation (AMME) stochastic optimization algorithm is proposed for the first time by using the finite lower-order moments and adding the estimating points. This algorithm not only reduces the probability of local optimum in the learning process, but also improves the convergence rate of the convolutional neural network (Lee Cun et al. in Advances in neural information processing systems, 1990). Second, according to the remote sensing image with characteristics of complex background and small sensitive targets, and by automatic discovery, locating small targets, and giving high weights, we proposed a feature extraction method named weighted pooling to further improve the performance of real-time image recognition. We combine the AMME and weighted pooling with the spatial pyramid representation (Harada et al. in Comput Vis Pattern Recognit 1617–1624, 2011) algorithm to form a new, multiscale, and multilevel real-time image recognition model and name it weighted spatial pyramid networks (WspNet). At the end, we use the MNIST, ImageNet, and natural disasters under remote sensing data sets to test WspNet. Compared with other real-time image recognition models, WspNet achieve a new state of the art in terms of convergence rate and image feature extraction compared with conventional stochastic gradient descent method [like AdaGrad, AdaDelta and Adam (Zeiler in Comput Sci, 2012; Kingma and Ba in Comput Sci, 2014; Duchi et al. in J Mach Learn Res 12(7):2121–2159, 2011] and pooling method [like max-pooling, avg-pooling and stochastic-pooling (Zeiler and Fergus in stochastic-pooling for regularization of deep convolutional neural networks, 2013)].

On-line control of false discovery rates for multiple datastreams

Although some false discovery rate control procedures have been proposed in the continual surveillance of high dimensional datastreams, most of them ignore the sequential feature over the time domain and dependence information among the stream observations. This inspires us to exploit the sequential feature by restricting the ongoing streams at each time point to be a dynamic set, which is determined by previous complex controlling procedures. Based on the exponentially weighted moving average (EWMA) scheme, we develop a dynamic multiple testing procedure for high dimensional datastreams with the control of false discovery rates (FDR). The FDR is shown to be controlled pointwise under the condition that the average of correlations of the stream observations decreases to zero at a polynomial rate. Numerical results illustrate that the proposed method is able to deliver satisfactory control performance.

Generating High-Dimensional Datastreams for Change Detection

Abstract A popular testbed for change-detection algorithms consists in detecting changes that have been synthetically injected in real-world datastreams. Unfortunately, most of experimental practices in the literature lead to injecting changes whose magnitude is unknown and can not be controlled. As a consequence, results are difficult to interpret, reproduce, and compare with. Most importantly, controlling the change magnitude is a primary requirement to investigate the change-detection performance when data dimension scales, which is an issue to be typically addressed in big data scenarios. Here we present a best practice to inject changes in multivariate/high-dimensional datastreams: “Controlling Change Magnitude” (CCM) is a rigorous method to generate datastreams affected by a change having a desired magnitude at a known location. In CCM, changes are introduced by directly applying a roto-translation to the data, and the change magnitude is measured by the symmetric Kullback–Leibler divergence between the pre- and post-change data distributions. The roto-translation parameters yielding the desired change magnitude are identified by two iterative algorithms whose convergence is here proven. Our experiments show that CCM can effectively control the change magnitude in real-world datastreams, while traditional experimental practices might not be appropriate for assessing the performance of change-detection algorithms in high-dimensional data.

Subspace Learning From Bits

Networked sensing, where the goal is to perform complex inference using a large number of inexpensive and decentralized sensors, has become an increasingly attractive research topic due to its applications in wireless sensor networks and internet-of-things. To reduce the communication, sensing, and storage complexity, this paper proposes a simple sensing and estimation framework to faithfully recover the principal subspace of high-dimensional data streams using a collection of binary measurements from distributed sensors, without transmitting the whole data. The binary measurements are designed to indicate comparison outcomes of aggregated energy projections of the data samples over pairs of randomly selected directions. When the covariance matrix is a low-rank matrix, we propose a spectral estimator that recovers the principal subspace of the covariance matrix as the subspace spanned by the top eigenvectors of a properly designed surrogate matrix, which is provably accurate as soon as the number of binary measurements is sufficiently large. An adaptive rank selection strategy based on soft thresholding is also presented. Furthermore, we propose a tailored spectral estimator when the covariance matrix is additionally Toeplitz, and show that the reliable estimation can be obtained from a substantially smaller number of binary measurements. Our results hold even when a constant fraction of the binary measurements is randomly flipped. Finally, we develop a low-complexity online algorithm to track the principal subspace when new measurements arrive sequentially. Numerical experiments are provided to validate the proposed approach.

Landmark diffusion maps (L-dMaps): Accelerated manifold learning out-of-sample extension

Diffusion maps are a nonlinear manifold learning technique based on harmonic analysis of a diffusion process over the data. Out-of-sample extensions with computational complexity $\mathcal{O}(N)$, where $N$ is the number of points comprising the manifold, frustrate applications to online learning applications requiring rapid embedding of high-dimensional data streams. We propose landmark diffusion maps (L-dMaps) to reduce the complexity to $\mathcal{O}(M)$, where $M \ll N$ is the number of landmark points selected using pruned spanning trees or k-medoids. Offering $(N/M)$ speedups in out-of-sample extension, L-dMaps enables the application of diffusion maps to high-volume and/or high-velocity streaming data. We illustrate our approach on three datasets: the Swiss roll, molecular simulations of a C$_{24}$H$_{50}$ polymer chain, and biomolecular simulations of alanine dipeptide. We demonstrate up to 50-fold speedups in out-of-sample extension for the molecular systems with less than 4% errors in manifold reconstruction fidelity relative to calculations over the full dataset.

Clustering High-Dimensional Data Stream: A Survey on Subspace Clustering, Projected Clustering on Bioinformatics Applications (Advanced Science, Engineering and Medicine, Vol. 8(9), pp. 749–757 (2016))

Clustering High-Dimensional Data Stream: A Survey on Subspace Clustering, Projected Clustering on Bioinformatics Applications (Advanced Science, Engineering and Medicine, Vol. 8(9), pp. 749–757 (2016))

CSTG: An Effective Framework for Cost-sensitive Sparse Online Learning.

Sparse online learning and cost-sensitive learning are two important areas of machine learning and data mining research. Each has been well studied with many interesting algorithms developed. However, very limited published work addresses the joint study of these two fields. In this paper, to tackle the high-dimensional data streams with skewed distributions, we introduce a framework of cost-sensitive sparse online learning. Our proposed framework is a substantial extension of the influential Truncated Gradient (TG) method by formulating a new convex optimization problem, where the two mutual restraint factors, misclassification cost and sparsity, can be simultaneously and favorably balanced. We theoretically analyze the regret and cost bounds of the proposed algorithm, and pinpoint its theoretical merit compared to the existing related approaches. Large-scale empirical comparisons to five baseline methods on eight real-world streaming datasets demonstrate the encouraging performance of the developed method. Algorithm implementation and datasets are available upon request.

A fast and effective principal singular subspace tracking algorithm

In this paper, we propose a fast and effective neural network algorithm to perform singular value decomposition (SVD) of a cross-covariance matrix between two high-dimensional data streams. Firstly, we derive a dynamical system from a newly proposed information criterion. This system exhibits a single stable stationary point if and only if the weight matrices of the left and right neural networks span the left and right principal singular subspace of a cross-covariance matrix, respectively, and the other stationary points are (unstable) saddle points. Then, a principal singular subspace (PSS) tracking algorithm is obtained from the dynamical system. Moreover, convergence analysis shows that the proposed algorithm converges to a stationary point that relates to the principal singular values. Thus, compared with traditional algorithms who can only track the PSS, the proposed algorithm can not only track the PSS but also estimate all of the corresponding principal singular values based on the extracted subspace. Finally, numerical simulations and practical application are carried to further demonstrate the efficiency of the proposed algorithm.

Actionable intelligence and online learning for semantic computing

As the world becomes more connected and instrumented, high dimensional, heterogeneous and time-varying data streams are collected and need to be analyzed on the fly to extract the actionable intelligence from the data streams and make timely decisions based on this knowledge. This requires that appropriate classifiers are invoked to process the incoming streams and find the relevant knowledge. Thus, a key challenge becomes choosing online, at run-time, which classifier should be deployed to make the best possible predictions on the incoming streams. In this paper, we survey a class of methods capable to perform online learning in stream-based semantic computing tasks: multi-armed bandits (MABs). Adopting MABs for stream mining poses, numerous new challenges requires many new innovations. Most importantly, the MABs will need to explicitly consider and track online the time-varying characteristics of the data streams and to learn fast what is the relevant information out of the vast, heterogeneous and possibly highly dimensional data streams. In this paper, we discuss contextual MAB methods, which use similarities in context (meta-data) information to make decisions, and discuss their advantages when applied to stream mining for semantic computing. These methods can be adapted to discover in real-time the relevant contexts guiding the stream mining decisions, and tract the best classifier in presence of concept drift. Moreover, we also discuss how stream mining of multiple data sources can be performed by deploying cooperative MAB solutions and ensemble learning. We conclude the paper by discussing the numerous other advantages of MABs that will benefit semantic computing applications.

On-The-Fly Processing of continuous high-dimensional data streams

A novel method and software system for rational handling of time series of multi-channel measurements is presented. This quantitative learning tool, the On-The-Fly Processing (OTFP), develops reduced-rank bilinear subspace models that summarise massive streams of multivariate responses, capturing the evolving covariation patterns among the many input variables over time and space. Thereby, a considerable data compression can be achieved without significant loss of useful systematic information.The underlying proprietary OTFP methodology is relatively fast and simple – it is linear/bilinear and does not require a lot of raw data or huge cross-correlation matrices to be kept in memory. Unlike conventional compression methods, the approach allows the high-dimensional data stream to be graphically interpreted and quantitatively utilised – in its compressed state. Unlike adaptive moving-window methods, it allows all past and recent time points to be reconstructed and displayed simultaneously.This new approach is applied to four different case-studies: (i) multi-channel Vis-NIR spectroscopy of the Belousov–Zhabotinsky reaction, a complex, ill understood chemical process; (ii) quality control of oranges by hyperspectral imaging; (iii) environmental monitoring by airborne hyperspectral imaging; (iv) multi-sensor process analysis in the petrochemical industry. These examples demonstrate that the OTFP can automatically develop high-fidelity subspace data models, which simplify the storage/transmission and the interpretation of more or less continuous time series of high-dimensional measurements – to the extent there are covariations among the measured variables.

A clustering algorithm for stream data with LDA-based unsupervised localized dimension reduction

We present an algorithm for clustering high dimensional streaming data. The algorithm incorporates dimension reduction into the stream clustering framework. When a new datum arrives, the algorithm performs dimension reduction to find a local projected subspace using unsupervised LDA (Linear Discriminant Analysis)-based method. The obtained local subspace would maximally separate the nearby micro-clusters with respect to the incoming point. Then, the incoming point is assigned to a micro-cluster in the projected space, rather than in the full dimensional space. The experimental results show that the proposed algorithm outperforms its counterpart streaming clustering algorithms. Moreover, when compared with traditional clustering algorithms which require the whole data set, the proposed algorithms shows comparable clustering performances with much less computation time for large data sets.

개념 변동 고차원 스트리밍 데이터에 대한 차원 감소 방법

고차원데이터에 대한 차원 감소 기법들은 많이 연구되어져 온 반면, 개념 변동을 가진 고차원 스트리밍 데이터에서 적용할 수 있는 차원 감소 기법에 대한 연구는 제한적이다. 이 논문에서는 스트리밍 데이터에서 적용할 수 있는 점층적 차원 감소 기법들을 살펴보고, 개념 변동 고차원 스트리밍 데이터에 대해 분류 성능을 향상시킬 수 있도록 차원 감소를 효과적으로 적용하는 방법을 제안한다. While dimension reduction methods on high dimensional data have been widely studied, research on dimension reduction methods for high dimensional streaming data with concept drift is limited. In this paper, we review incremental dimension reduction methods and propose a method to apply dimension reduction efficiently in order to improve classification performance on high dimensional streaming data with concept drift.

A streaming algorithm for 2-center with outliers in high dimensions

We study the 2-center problem with outliers in high-dimensional data streams. Given a stream of points in arbitrary d dimensions, the goal is to find two congruent balls of minimum radius covering all but at most z points. We present a (1.8+ε)-approximation streaming algorithm, improving over the previous (4+ε)-approximation algorithm available for the problem. The space complexity and update time of our algorithm are poly(d,z,1/ε), independent of the size of the stream.

Clustering Big Data streams: recent challenges and contributions

Abstract Traditional clustering algorithms merely considered static data. Today's various applications and research issues in big data mining have however to deal with continuous, possibly infinite streams of data, arriving at high velocity. Web traffic data, surveillance data, sensor measurements and stock trading are only some examples of these daily-increasing applications. Since the growth of data volumes is accompanied by a similar raise in their dimensionalities, clusters cannot be expected to completely appear when considering all attributes together. Subspace clustering is a general approach that solved that issue by automatically finding the hidden clusters within different subsets of the attributes rather than considering all attributes together. In this article, novel methods for an efficient subspace clustering of high-dimensional big data streams are presented. Approaches that efficiently combine the anytime clustering concept with the stream subspace clustering paradigm are discussed. Additionally, efficient and adaptive density-based clustering algorithms are presented for high-dimensional data streams. Novel open-source assessment framework and evaluation measures are additionally presented for subspace stream clustering.

The Local Convexity of Solving Systems of Quadratic Equations

This paper considers the recovery of a rank r positive semidefinite matrix \({X X^T \in \mathbb{R}^{n\times n}}\) from m scalar measurements of the form \({y_i := a_i^T X X^T a_i}\) (i.e., quadratic measurements of X). Such problems arise in a variety of applications, including covariance sketching of high-dimensional data streams, quadratic regression, quantum state tomography, among others. A natural approach to this problem is to minimize the loss function \({f(U) = \sum_i (y_i - a_{i}^{T}UU^{T}a_i)^2}\) which has an entire manifold of solutions given by \({\{XO\}_{O\in\mathcal{O}_r}}\) where \({\mathcal{O}_r}\) is the orthogonal group of \({r\times r}\) orthogonal matrices; this is non-convex in the \({n\times r}\) matrix U, but methods like gradient descent are simple and easy to implement (as compared to semidefinite relaxation approaches). In this paper we show that once we have \({m \geq Cnr\,{\rm log}^2(n)}\) samples from isotropic gaussian \({a_i}\), with high probability (a) this function admits a dimension-independent region of local strong convexity on lines perpendicular to the solution manifold, and (b) with an additional polynomial factor of r samples, a simple spectral initialization will land within the region of convexity with high probability. Together, this implies that gradient descent with initialization (but no re-sampling) will converge linearly to the correct X, up to an orthogonal transformation. We believe that this general technique (local convexity reachable by spectral initialization) should prove applicable to a broader class of nonconvex optimization problems.

Universal Nonlinear Regression on High Dimensional Data Using Adaptive Hierarchical Trees

We study online sequential regression with nonlinearity and time varying statistical distribution when the regressors lie in a high dimensional space. We escape the curse of dimensionality by tracking the subspace of the underlying manifold using a hierarchical tree structure. We use the projections of the original high dimensional regressor space onto the underlying manifold as the modified regressor vectors for modeling of the nonlinear system. By using the proposed algorithm, we reduce the computational complexity to the order of the depth of the tree and the memory requirement to only linear in the intrinsic dimension of the manifold. The proposed techniques are specifically applicable to high dimensional streaming data analysis in a time varying environment. We demonstrate the significant performance gains in terms of mean square error over the other state of the art techniques through simulated as well as real data.

Coupled cross-correlation neural network algorithm for principal singular triplet extraction of a cross-covariance matrix

This paper proposes a novel coupled neural network learning algorithm to extract the principal singular triplet (PST) of a cross-correlation matrix between two high-dimensional data streams. We firstly introduce a novel information criterion (NIC), in which the stationary points are singular triplet of the crosscorrelation matrix. Then, based on Newton's method, we obtain a coupled system of ordinary differential equations (ODEs) from the NIC. The ODEs have the same equilibria as the gradient of NIC, however, only the first PST of the system is stable (which is also the desired solution), and all others are (unstable) saddle points. Based on the system, we finally obtain a fast and stable algorithm for PST extraction. The proposed algorithm can solve the speed-stability problem that plagues most noncoupled learning rules. Moreover, the proposed algorithm can also be used to extract multiple PSTs effectively by using sequential method.

Dynamic Filtering of Time-Varying Sparse Signals via <inline-formula> <tex-math notation="LaTeX">$\ell _1$</tex-math> </inline-formula> Minimization

Despite the importance of sparsity signal models and the increasing prevalence of high-dimensional streaming data, there are relatively few algorithms for dynamic filtering of time-varying sparse signals. Of the existing algorithms, fewer still provide strong performance guarantees. This paper examines two algorithms for dynamic filtering of sparse signals that are based on efficient l1 optimization methods. We first present an analysis for one simple algorithm (BPDN-DF) that works well when the system dynamics are known exactly. We then introduce a novel second algorithm (RWL1-DF) that is more computationally complex than BPDN-DF but performs better in practice, especially in the case where the system dynamics model is inaccurate. Robustness to model inaccuracy is achieved by using a hierarchical probabilistic data model and propagating higher-order statistics from the previous estimate (akin to Kalman filtering) in the sparse inference process. We demonstrate the properties of these algorithms on both simulated data as well as natural video sequences. Taken together, the algorithms presented in this paper represent the first strong performance analysis of dynamic filtering algorithms for time-varying sparse signals as well as state-of-the-art performance in this emerging application.

Sliding Window-Based Fault Detection From High-Dimensional Data Streams

High-dimensional data streams are becoming increasingly ubiquitous in industrial systems. Efficient detection of system faults from these data can ensure the reliability and safety of the system. The difficulties brought about by high dimensionality and data streams are mainly the “curse of dimensionality” and concept drifting, and one current challenge is to simultaneously address them. To this purpose, this paper presents an approach to fault detection from nonstationary high-dimensional data streams. An angle-based subspace anomaly detection approach is proposed to detect low-dimensional subspace faults from high-dimensional datasets. Specifically, it selects fault-relevant subspaces by evaluating vectorial angles and computes the local outlier-ness of an object in its subspace projection. Based on the sliding window strategy, the approach is further extended to an online mode that can continuously monitor system states. To validate the proposed algorithm, we compared it with the local outlier factor-based approaches on artificial datasets and found the algorithm displayed superior accuracy. The results of the experiment demonstrated the efficacy of the proposed algorithm. They also indicated that the algorithm has the ability to discriminate low-dimensional subspace faults from normal samples in high-dimensional spaces and can be adaptive to the time-varying behavior of the monitored system. The online subspace learning algorithm for fault detection would be the main contribution of this paper.

A Novel High Dimensional and High Speed Data Streams Algorithm: HSDStream

This paper presents a novel high speed clustering scheme for high-dimensional data stream. Data stream clustering has gained importance in different applications, for example, network monitoring, intrusion detection, and real-time sensing. High dimensional stream data is inherently more complex when used for clustering because the evolving nature of the stream data and high dimensionality make it non-trivial. In order to tackle this problem, projected subspace within the high dimensions and limited window sized data per unit of time are used for clustering purpose. We propose a High Speed and Dimensions data stream clustering scheme (HSDStream) which employs exponential mov-ing averages to reduce the size of the memory and speed up the processing of projected subspace data stream. It works in three steps: i) initialization, ii) real-time maintenance of core and outlier micro-clusters, and iii) on-demand offline generation of the final clusters. The proposed algorithm is tested against high dimensional density-based projected clustering (HDDStream) for cluster purity, memory usage, and the cluster sensitivity. Experi-mental results are obtained for corrected KDD intrusion detection dataset. These results show that HSDStream outperforms the HDDStream in all performance metrics, especially, the memory usage and the processing speed.