Large Spatio-temporal Datasets Research Articles

To aggregate or not? Capturing the spatio-temporal complexity of the thermal regime

Freshwater stream systems are under immense pressure from various anthropogenic impacts, including climate change. Stream systems are increasingly being altered by changes to the magnitude, timing, frequency, and duration of their thermal regimes, which will have profound impacts on the life-history dynamics of resident biota within their home range. Although temperature regimes have a significant influence on the biology of instream fauna, large spatio-temporal temperature datasets are often reduced to a single metric at discrete locations and used to describe the thermal regime of a system; potentially leading to a significant loss of information crucial to stream management. Models are often used to extrapolate these metrics to unsampled locations, but it is unclear whether predicting actual daily temperatures or an aggregated metric of the temperature regime best describes the complexity of the thermal regime. We fit spatial statistical stream-network models (SSNMs), random forest and non-spatial linear models to stream temperature data from the Upper Condamine River in QLD, Australia and used them to semi-continuously predict metrics describing the magnitude, duration, and frequency of the thermal regime through space and time. We compared both daily and aggregated temperature metrics and found that SSNMs always had more predictive ability than the random forest models, but both models outperformed the non-spatial linear model. For metrics describing thermal magnitude and duration, aggregated predictions were most accurate, while metrics describing the frequency of heating events were better represented by metrics based on daily predictions generated using a SSNM. A more comprehensive representation of the spatio-temporal thermal regime allows researchers to explore new spatio-temporally explicit questions about the thermal regime. It also provides the information needed to generate a suite of ecologically meaningful metrics capturing multiple aspects of the thermal regime, which will increase our scientific understanding of how organisms respond to thermal cues and provide much-needed information for more effective management actions.

Fast multidimensional ensemble empirical mode decomposition for the analysis of big spatio-temporal datasets.

In this big data era, it is more urgent than ever to solve two major issues: (i) fast data transmission methods that can facilitate access to data from non-local sources and (ii) fast and efficient data analysis methods that can reveal the key information from the available data for particular purposes. Although approaches in different fields to address these two questions may differ significantly, the common part must involve data compression techniques and a fast algorithm. This paper introduces the recently developed adaptive and spatio-temporally local analysis method, namely the fast multidimensional ensemble empirical mode decomposition (MEEMD), for the analysis of a large spatio-temporal dataset. The original MEEMD uses ensemble empirical mode decomposition to decompose time series at each spatial grid and then pieces together the temporal–spatial evolution of climate variability and change on naturally separated timescales, which is computationally expensive. By taking advantage of the high efficiency of the expression using principal component analysis/empirical orthogonal function analysis for spatio-temporally coherent data, we design a lossy compression method for climate data to facilitate its non-local transmission. We also explain the basic principles behind the fast MEEMD through decomposing principal components instead of original grid-wise time series to speed up computation of MEEMD. Using a typical climate dataset as an example, we demonstrate that our newly designed methods can (i) compress data with a compression rate of one to two orders; and (ii) speed-up the MEEMD algorithm by one to two orders.

Spatial statistics and Gaussian processes: A beautiful marriage

Spatial analysis has grown at a remarkable rate over the past two decades. Fueled by sophisticated GIS software and inexpensive and fast computation, collection of data with spatially referenced information has increased. Recognizing that such information can improve data analysis has led to an explosion of modeling and model fitting.The contribution of this paper is to illustrate how Gaussian processes have emerged as, arguably, the most valuable tool in the toolkit for geostatistical modeling. Apart from the simplest versions, geostatistical modeling can be viewed as a hierarchical specification with Gaussian processes introduced appropriately at different levels of the specification. This naturally leads to adopting a Bayesian framework for inference and suitable Gibbs sampling/Markov chain Monte Carlo for model fitting.Here, we review twenty years of modeling work spanning multivariate spatial analysis, gradient analysis, Bayesian nonparametric spatial ideas, directional data, extremes, data fusion, and large spatial and spatio-temporal datasets. We demonstrate that Gaussian processes are the key ingredients in all of this work. Most of the content is focused on modeling with examples being limited due to length constraints for the article. Altogether, we are able to conclude that spatial statistics and Gaussian processes do, indeed, make a beautiful marriage.

Spatio-temporal analysis of factors controlling nitrate dynamics and potential denitrification hot spots and hot moments in groundwater of an alluvial floodplain

Nitrate (NO3−) contamination of freshwater systems is a global concern. In alluvial floodplains, riparian areas have been proven to be efficient in nitrate removal. In this study, a large spatio-temporal dataset collected during one year at monthly time steps within a meander area of the Garonne floodplain (France) was analysed in order to improve the understanding of nitrate dynamic and denitrification process in floodplain areas. The results showed that groundwater NO3− concentrations (mean 50mg NO3−L−1) were primarily controlled by groundwater dilution with river water (explaining 54% of NO3− variance), but also by nitrate removal process identified as denitrification (explaining 14% of NO3− variance). Dilution was controlled by hydrological flow paths and residence time linked to river-aquifer exchanges and flood occurrence, while potential denitrification (DEA) was controlled by oxygen, high dissolved organic carbon (DOC) and organic matter content in the sediment (31% of DEA variance). DOC can originate both from the river input and the degradation of organic matter (OM) located in topsoil and sediments of the alluvial plain. In addition, river bank geomorphology appeared to be a key element explaining potential denitrification hot spot locations. Low bankfull height (LBH) areas corresponding to wetlands exhibited higher denitrification rates than high bankfull height (HBH) areas less often flooded. Hydrology determined the timing of denitrification hot moments occurring after flood events. These findings underline the importance of integrating dynamic water interactions between river and aquifer, geomorphology, and dual carbon source (river and sediment) when assessing nitrate dynamics and denitrification patterns in floodplain environments.

Spatial online sampling and aggregation

The massive adoption of smart phones and other mobile devices has generated humongous amount of spatial and spatio-temporal data. The importance of spatial analytics and aggregation is ever-increasing. An important challenge is to support interactive exploration over such data. However, spatial analytics and aggregation using all data points that satisfy a query condition is expensive, especially over large data sets, and could not meet the needs of interactive exploration. To that end, we present novel indexing structures that support spatial online sampling and aggregation on large spatial and spatio-temporal data sets. In spatial online sampling, random samples from the set of spatial (or spatio-temporal) points that satisfy a query condition are generated incrementally in an online fashion. With more and more samples, various spatial analytics and aggregations can be performed in an online, interactive fashion, with estimators that have better accuracy over time. Our design works well for both memory-based and disk-resident data sets, and scales well towards different query and sample sizes. More importantly, our structures are dynamic, hence, they are able to deal with insertions and deletions efficiently. Extensive experiments on large real data sets demonstrate the improvements achieved by our indexing structures compared to other baseline methods.

The CASE histogram: privacy-aware processing of trajectory data using aggregates

Due to the high uptake of location-based services (LBSs), large spatio-temporal datasets of moving objects’ trajectories are being created every day. An important task in spatial data analytics is to service range queries by returning trajectory counts within a queried region. The question of how to keep an individual user’s data private whilst enabling spatial data analytics by third parties has become an urgent research direction. Indeed, it is increasingly becoming a concern for users. To preserve privacy we discard individual trajectories and aggregate counts over a spatial and temporal partition. However the privacy gained comes at a cost to utility: trajectories passing through multiple cells and re-entering a query region, lead to inaccurate query responses. This is known as the distinct counting problem. We propose the Connection Aware Spatial Euler (CASE) histogram to address this long-standing problem. The CASE histogram maintains the connectivity of a moving object path, but does not require the ID of an object to distinguish multiple entries into an arbitrary query region. Our approach is to process trajectories offline into aggregate counts which are sent to third parties, rather than the original trajectories. We also explore modifications of our aggregate counting approach that preserve differential privacy. Theoretically and experimentally we demonstrate that our method provides a high level of accuracy compared to the best known methods for the distinct counting problem, whilst preserving privacy. We conduct our experiments on both synthetic and real datasets over two competitive Euler histogram-based methods presented in the literature. Our methods enjoy improvements to accuracy from 10 % up to 70 % depending on trip data and query region size, with the greatest increase seen on the Microsoft T-Drive real dataset, representing a more than tripling of accuracy.

Visual mining of moving flock patterns in large spatio-temporal data sets using a frequent pattern approach

The popularity of tracking devices continues to contribute to increasing volumes of spatio-temporal data about moving objects. Current approaches in analysing these data are unable to capture collective behaviour and correlations among moving objects. An example of these types of patterns is moving flocks. This article develops an improved algorithm for mining such patterns following a frequent pattern discovery approach, a well-known task in traditional data mining. It uses transaction-based data representation of trajectories to generate a database that facilitates the application of scalable and efficient frequent pattern mining algorithms. Results were compared with an existing method (Basic Flock Evaluation or BFE) and are demonstrated for both synthetic and real data sets with a large number of trajectories. The results illustrate a significant performance increase. Furthermore, the improved algorithm has been embedded into a visual environment that allows manipulation of input parameters and interactive recomputation of the resulting flocks. To illustrate the visual environment a data set containing 30 years of tropical cyclone tracks with 6 hourly observations is used. The example illustrates how the visual environment facilitates exploration and verification of flocks by changing the input parameters and instantly showing the spatio-temporal distribution of the resulting flocks in the Space-Time Cube and interactively selecting, querying and saving the resulting flocks for further analysis and verification.

A Bayesian spatio-temporal geostatistical model with an auxiliary lattice for large datasets

When spatio-temporal datasets are large, the computational burden can lead to failures in the implementation of traditional geostatistical tools. In this pa- per, we propose a computationally efficient Bayesian hierarchical spatio-temporal model in which the spatial dependence is approximated by a Gaussian Markov random field (GMRF) while the temporal correlation is described using a vector autoregressive model. By introducing an auxiliary lattice on the spatial region of interest, the proposed method is not only able to handle irregularly spaced observa- tions in the spatial domain, but it is also able to bypass the missing data problem in a spatio-temporal process. Because the computational complexity of the proposed Markov chain Monte Carlo algorithm is of the order O(n) with n the total number of observations in space and time, our method can be used to handle very large spatio-temporal datasets with reasonable CPU times. The performance of the pro- posed model is illustrated using simulation studies and a dataset of precipitation data from the coterminous United States.

A temporal GIS for field based environmental modeling

Time in geographic information systems has been a research theme for more than two decades, resulting in comprehensive theoretical work, many research prototypes and several working solutions. However, none of the available solutions provides the ability to manage, analyze, process and visualize large environmental spatio-temporal datasets and the investigation and assessment of temporal relationships between them. We present in this paper a freely available field based temporal GIS (TGRASS) that fulfills these requirements. Our approach is based on the integration of time in the open source Geographic Resources Analysis Support System (GRASS). We introduce the concept of a space time dataset that is defined as a collection of time stamped raster, voxel or vector data. A dedicated set of spatio-temporal tools was implemented to manage, process and analyze space time datasets and their temporal and spatial relationships. We demonstrate the temporal GIS and environmental modeling capabilities of TGRASS by analyzing a multi-decadal European climate dataset.

Nanocubes for Real-Time Exploration of Spatiotemporal Datasets

Consider real-time exploration of large multidimensional spatiotemporal datasets with billions of entries, each defined by a location, a time, and other attributes. Are certain attributes correlated spatially or temporally? Are there trends or outliers in the data? Answering these questions requires aggregation over arbitrary regions of the domain and attributes of the data. Many relational databases implement the well-known data cube aggregation operation, which in a sense precomputes every possible aggregate query over the database. Data cubes are sometimes assumed to take a prohibitively large amount of space, and to consequently require disk storage. In contrast, we show how to construct a data cube that fits in a modern laptop's main memory, even for billions of entries; we call this data structure a nanocube. We present algorithms to compute and query a nanocube, and show how it can be used to generate well-known visual encodings such as heatmaps, histograms, and parallel coordinate plots. When compared to exact visualizations created by scanning an entire dataset, nanocube plots have bounded screen error across a variety of scales, thanks to a hierarchical structure in space and time. We demonstrate the effectiveness of our technique on a variety of real-world datasets, and present memory, timing, and network bandwidth measurements. We find that the timings for the queries in our examples are dominated by network and user-interaction latencies.

Hierarchical self-organizing maps for clustering spatiotemporal data

Spatial sciences are confronted with increasing amounts of high-dimensional data. These data commonly exhibit spatial and temporal dimensions. To explore, extract, and generalize inherent patterns in large spatiotemporal data sets, clustering algorithms are indispensable. These clustering algorithms must account for the distinct special properties of space and time to outline meaningful clusters in such data sets. Therefore, this research develops a hierarchical method based on self-organizing maps. The hierarchical architecture permits independent modeling of spatial and temporal dependence. To exemplify the utility of the method, this research uses an artificial data set and a socio-economic data set of the Ostregion, Austria, from the years 1961 to 2001. The results for the artificial data set demonstrate that the proposed method produces meaningful clusters that cannot be achieved when disregarding differences in spatial and temporal dependence. The results for the socio-economic data set show that the proposed method is an effective and powerful tool for analyzing spatiotemporal patterns in a regional context.

Parameter tuning for a multi-fidelity dynamical model of the magnetosphere

Geomagnetic storms play a critical role in space weather physics with the potential for far reaching economic impacts including power grid outages, air traffic rerouting, satellite damage and GPS disruption. The LFM–MIX is a state-of-the-art coupled magnetospheric–ionospheric model capable of simulating geomagnetic storms. Imbedded in this model are physical equations for turning the magnetohydrodynamic state parameters into energy and flux of electrons entering the ionosphere, involving a set of input parameters. The exact values of these input parameters in the model are unknown, and we seek to quantify the uncertainty about these parameters when model output is compared to observations. The model is available at different fidelities: a lower fidelity which is faster to run, and a higher fidelity but more computationally intense version. Model output and observational data are large spatiotemporal systems; the traditional design and analysis of computer experiments is unable to cope with such large data sets that involve multiple fidelities of model output. We develop an approach to this inverse problem for large spatiotemporal data sets that incorporates two different versions of the physical model. After an initial design, we propose a sequential design based on expected improvement. For the LFM–MIX, the additional run suggested by expected improvement diminishes posterior uncertainty by ruling out a posterior mode and shrinking the width of the posterior distribution. We also illustrate our approach using the Lorenz ‘96 system of equations for a simplified atmosphere, using known input parameters. For the Lorenz ‘96 system, after performing sequential runs based on expected improvement, the posterior mode converges to the true value and the posterior variability is reduced.

Mining trajectories of moving dynamic spatio-temporal regions in sensor datasets

When mining large spatio-temporal datasets, interesting patterns typically emerge where the dataset is most dynamic. These dynamic regions can be characterized by a location or set of locations that exhibit different behaviors from their neighbors and the time periods where these differences are most pronounced. Examples include locally intense areas of precipitation, anomalous sea surface temperature (SST) readings, and locally high levels of water pollution, to name a few. The focus of this paper is to find and analyze the pattern of moving dynamic spatio-temporal regions in large sensor datasets. The approach presented in this paper uses a measure of local spatial autocorrelation over time to determine how pronounced the difference in measurements taken at a spatial location is with those taken at neighboring locations. Dynamic regions are analyzed both globally, in the form of spatial locations and time periods that have the largest difference in local spatial autocorrelation, and locally, in the form of dynamic spatial locations for a particular time period or dynamic time periods for a particular spatial node. Then, moving dynamic regions are identified by determining the spatio-temporal connectivity, extent, and trajectory for groups of locally dynamic spatial locations whose position has shifted from one time period to the next. The efficacy of the approach is demonstrated on two real-world spatio-temporal datasets (a) NEXRAD precipitation and (b) SST. Promising results were found in discovering highly dynamic regions in these datasets depicting several real environmental phenomenon which are validated as actual events of interest.

Feature-based Analysis of Large-scale Spatio-Temporal Sensor Data on Hybrid Architectures.

Analysis of large sensor datasets for structural and functional features has applications in many domains, including weather and climate modeling, characterization of subsurface reservoirs, and biomedicine. The vast amount of data obtained from state-of-the-art sensors and the computational cost of analysis operations create a barrier to such analyses. In this paper, we describe middleware system support to take advantage of large clusters of hybrid CPU-GPU nodes to address the data and compute-intensive requirements of feature-based analyses in large spatio-temporal datasets.

An Algebra for Spatiotemporal Data: From Observations to Events

AbstractRecent technological advances in geospatial data gathering have created massive data sets with better spatial and temporal resolution than ever before. These large spatiotemporal data sets have motivated a challenge for Geoinformatics: how to model changes and design good quality software. Many existing spatiotemporal data models represent how objects and fields evolve over time. However, to properly capture changes, it is also necessary to describe events. As a contribution to this research, this article presents an algebra for spatiotemporal data. Algebras give formal specifications at a high‐level abstraction, independently of programming languages. This helps to develop reliable and expressive applications. Our algebra specifies three data types as generic abstractions built on real‐world observations: time series, trajectory and coverage. Based on these abstractions, it defines object and event types. The proposed data types and functions can model and capture changes in a large range of applications, including location‐based services, environmental monitoring, public health, and natural disasters.

Time-space Kriging to address the spatiotemporal misalignment in the large datasets

This paper presents a Bayesian hierarchical spatiotemporal method of interpolation, termed as Markov Cube Kriging (MCK). The classical Kriging methods become computationally prohibitive, especially for large datasets due to the O(n3) matrix decomposition. MCK offers novel and computationally efficient solutions to address spatiotemporal misalignment, mismatch in the spatiotemporal scales and missing values across space and time in large spatiotemporal datasets. MCK is flexible in that it allows for non-separable spatiotemporal structure and nonstationary covariance at the hierarchical spatiotemporal scales. Employing MCK we developed estimates of daily concentration of fine particulates matter ≤2.5 μm in aerodynamic diameter (PM2.5) at 2.5 km spatial grid for the Cleveland Metropolitan Statistical Area, 2000 to 2009. Our validation and cross-validation suggest that MCK achieved robust prediction of spatiotemporal random effects and underlying hierarchical and nonstationary spatiotemporal structure in air pollution data. MCK has important implications for environmental epidemiology and environmental sciences for exposure quantification and collocation of data from different sources, available at different spatiotemporal scales.

USER FRIENDLY OPEN GIS TOOL FOR LARGE SCALE DATA ASSIMILATION – A CASE STUDY OF HYDROLOGICAL MODELLING

Abstract. Open source software (OSS) coding has tremendous advantages over proprietary software. These are primarily fuelled by high level programming languages (JAVA, C++, Python etc...) and open source geospatial libraries (GDAL/OGR, GEOS, GeoTools etc.). Quantum GIS (QGIS) is a popular open source GIS package, which is licensed under GNU GPL and is written in C++. It allows users to perform specialised tasks by creating plugins in C++ and Python. This research article emphasises on exploiting this capability of QGIS to build and implement plugins across multiple platforms using the easy to learn – Python programming language. In the present study, a tool has been developed to assimilate large spatio-temporal datasets such as national level gridded rainfall, temperature, topographic (digital elevation model, slope, aspect), landuse/landcover and multi-layer soil data for input into hydrological models. At present this tool has been developed for Indian sub-continent. An attempt is also made to use popular scientific and numerical libraries to create custom applications for digital inclusion. In the hydrological modelling calibration and validation are important steps which are repetitively carried out for the same study region. As such the developed tool will be user friendly and used efficiently for these repetitive processes by reducing the time required for data management and handling. Moreover, it was found that the developed tool can easily assimilate large dataset in an organised manner.

Changes in intrapopulation resource use patterns of an endangered raptor in response to a disease‐mediated crash in prey abundance

1. A long-standing question in ecology is how natural populations respond to a changing environment. Emergent optimal foraging theory-based models for individual variation go beyond the population level and predict how its individuals would respond to disturbances that produce changes in resource availability. 2. Evaluating variations in resource use patterns at the intrapopulation level in wild populations under changing environmental conditions would allow to further advance in the research on foraging ecology and evolution by gaining a better idea of the underlying mechanisms explaining trophic diversity. 3. In this study, we use a large spatio-temporal scale data set (western continental Europe, 1968-2006) on the diet of Bonelli's Eagle Aquila fasciata breeding pairs to analyse the predator trophic responses at the intrapopulation level to a prey population crash. In particular, we borrow metrics from studies on network structure and intrapopulation variation to understand how an emerging infectious disease [the rabbit haemorrhagic disease (RHD)] that caused the density of the eagle's primary prey (rabbit Oryctolagus cuniculus) to dramatically drop across Europe impacted on resource use patterns of this endangered raptor. 4. Following the major RHD outbreak, substantial changes in Bonelli's Eagle's diet diversity and organisation patterns at the intrapopulation level took place. Dietary variation among breeding pairs was larger after than before the outbreak. Before RHD, there were no clusters of pairs with similar diets, but significant clustering emerged after RHD. Moreover, diets at the pair level presented a nested pattern before RHD, but not after. 5. Here, we reveal how intrapopulation patterns of resource use can quantitatively and qualitatively vary, given drastic changes in resource availability. 6. For the first time, we show that a pathogen of a prey species can indirectly impact the intrapopulation patterns of resource use of an endangered predator.

Spatio-temporal modeling of particulate matter concentration through the SPDE approach

In this work, we consider a hierarchical spatio-temporal model for particulate matter (PM) concentration in the North-Italian region Piemonte. The model involves a Gaussian Field (GF), affected by a measurement error, and a state process characterized by a first order autoregressive dynamic model and spatially correlated innovations. This kind of model is well discussed and widely used in the air quality literature thanks to its flexibility in modelling the effect of relevant covariates (i.e. meteorological and geographical variables) as well as time and space dependence. However, Bayesian inference—through Markov chain Monte Carlo (MCMC) techniques—can be a challenge due to convergence problems and heavy computational loads. In particular, the computational issue refers to the infeasibility of linear algebra operations involving the big dense covariance matrices which occur when large spatio-temporal datasets are present. The main goal of this work is to present an effective estimating and spatial prediction strategy for the considered spatio-temporal model. This proposal consists in representing a GF with Matern covariance function as a Gaussian Markov Random Field (GMRF) through the Stochastic Partial Differential Equations (SPDE) approach. The main advantage of moving from a GF to a GMRF stems from the good computational properties that the latter enjoys. In fact, GMRFs are defined by sparse matrices that allow for computationally effective numerical methods. Moreover, when dealing with Bayesian inference for GMRFs, it is possible to adopt the Integrated Nested Laplace Approximation (INLA) algorithm as an alternative to MCMC methods giving rise to additional computational advantages. The implementation of the SPDE approach through the R-library INLA ( www.r-inla.org ) is illustrated with reference to the Piemonte PM data. In particular, providing the step-by-step R-code, we show how it is easy to get prediction and probability of exceedance maps in a reasonable computing time.

A three-dimensional gap filling method for large geophysical datasets: Application to global satellite soil moisture observations

The presence of data gaps is always a concern in geophysical records, creating not only difficulty in interpretation but, more importantly, also a large source of uncertainty in data analysis. Filling the data gaps is a necessity for use in statistical modeling. There are numerous approaches for this purpose. However, particularly challenging are the increasing number of very large spatio-temporal datasets such as those from Earth observations satellites. Here we introduce an efficient three-dimensional method based on discrete cosine transforms, which explicitly utilizes information from both time and space to predict the missing values. To analyze its performance, the method was applied to a global soil moisture product derived from satellite images. We also executed a validation by introducing synthetic gaps. It is shown this method is capable of filling data gaps in the global soil moisture dataset with very high accuracy.