Large Spatio-temporal Datasets Research Articles

Unlocking massively parallel spectral proper orthogonal decompositions in the PySPOD package

We propose a parallel (distributed) version of the spectral proper orthogonal decomposition (SPOD) technique. The parallel SPOD algorithm distributes the spatial dimension of the dataset preserving time. This approach is adopted to preserve the non-distributed fast Fourier transform of the data in time, thereby avoiding the associated bottlenecks. The parallel SPOD algorithm is implemented in the PySPOD library and makes use of the standard message passing interface (MPI) library, implemented in Python via mpi4py. An extensive performance evaluation of the parallel package is provided, including strong and weak scalability analyses. The open-source library allows the analysis of large datasets of interest across the scientific community. Here, we present applications in fluid dynamics and geophysics, that are extremely difficult (if not impossible) to achieve without a parallel algorithm. This work opens the path toward modal analyses of big quasi-stationary data, helping to uncover new unexplored spatio-temporal patterns. Program summaryProgram Title: PySPODCPC Library link to program files:https://doi.org/10.17632/jf5bf26jcj.1Developer's repository link:https://github.com/MathEXLab/PySPODLicensing provisions: MIT LicenseProgramming language: PythonNature of problem: Large spatio-temporal datasets may contain coherent patterns that can be leveraged to better understand, model, and possibly predict the behaviour of complex dynamical systems. To this end, modal decomposition methods, such as the proper orthogonal decomposition (POD) and its spectral counterpart (SPOD), constitute powerful tools. The SPOD algorithm allows the systematic identification of space-time coherent patterns. This can be used to understand better the physics of the process of interest, and provide a path for mathematical modeling, including reduced order modeling. The SPOD algorithm has been successfully applied to fluid dynamics, geophysics and other domains. However, the existing open-source implementations are serial, and they prevent running on the increasingly large datasets that are becoming available, especially in computational physics. The inability to analyse via SPOD large dataset in turn prevents unlocking novel mechanisms and dynamical behaviours in complex systems.Solution method: We provide an open-source parallel (MPI distributed) code, namely PySPOD, that is able to run on large datasets (the ones considered in the present paper reach about 200 Terabytes). The code is built on the previous serial open-source code PySPOD that was published in https://joss.theoj.org/papers/10.21105/joss.02862.pdf. The new parallel implementation is able to scale on several nodes (we show both weak and strong scalability) and solve some of the bottlenecks that are commonly found at the I/O stage. The current parallel code allows running on datasets that was not easy or possible to analyse with serial SPOD algorithms, hence providing a path towards unlocking novel findings in computational physics.Additional comments including restrictions and unusual features: The code comes with a set of built-in postprocessing tools, for visualizing the results. It also comes with extensive continuous integration, documentation, and tutorials, as well as a dedicated website in addition to the associated GiHub repository. Within the package we also provide a parallel implementation of the proper orthogonal decomposition (POD), that leverages the I/O parallel capabilities of the SPOD algorithm.

A spatially adaptive multi-resolution generative algorithm: Application to simulating flood wave propagation

We propose a statistical model suitable for large spatio-temporal data sets exhibiting complex patterns such as simulated by physics-based hydraulic models over high resolution (HR) 2D meshes. Although necessary for impact studies such as urban flood hazard assessment, their long computation times limit their applicability leading to the development of statistical models that may emulate them quickly. Our model draws from the strengths of multi-resolution analysis and relies on an extension of the lifting scheme, a flexible implementation of discrete wavelet transforms, for spatio-temporal data. The extended lifting scheme exploits the idea that dominant spatial features, that may be identified with clustering, remain present through time. An easily interpretable non-parametric representation can be derived from the lifting scheme by combining a smoothed version of the data (obtained by simple averaging) with details (given by local regression residuals). A generative algorithm is built by introducing the information provided by a low resolution model, whose computation times are orders of magnitude smaller, yielding a downscaling model. This downscaling model assumes that sufficiently representative HR spatial patterns can be inferred from the training set. Our model is applied to a 2D dam break experiment using a synthetic urban configuration and to a field-scale test case simulating the propagation of a dike break flood wave into a Sacramento urban area. A comparison, carried out with spatial interpolation schemes and with a variant of our model based on principal component analysis, shows that the spatio-temporal lifting scheme based model is better at reproducing extreme events.

Integrating consumer risk perception and awareness with simulation-based quantitative microbial risk assessment using a coupled systems framework: A case study of private groundwater users in Ontario

Private well users in Ontario are responsible for ensuring the potability of their own private drinking water source through protective actions (i.e., water treatment, well maintenance, and regular water quality testing). In the absence of regulation and limited surveillance, quantitative microbial risk assessment (QMRA) represents the most practical and robust approach to estimating the human health burden attributable to private wells. For an increasingly accurate estimation, QMRA of private well water should be represented by a coupled model, which includes both the socio-cognitive and physical aspects of private well water contamination and microbial exposure. The objective of the current study was to determine levels of waterborne exposure via well water consumption among three sub-groups (i.e., clusters) of private well users in Ontario and quantify the risk of waterborne acute gastrointestinal illness (AGI) attributed to Giardia, shiga-toxin producing E. coli (STEC) and norovirus from private drinking water sources in Ontario. Baseline simulations were utilized to explore the effect of varying socio-cognitive scenarios on model inputs (i.e., increased awareness, protective actions, aging population). The current study uses a large spatio-temporal groundwater quality dataset and cross-sectional province-wide survey to create socio-cognitive-specific QMRA simulations to estimate the risk of waterborne AGI attributed to three enteric pathogens in private drinking waters source in Ontario. Findings suggest significant differences in the level of exposure among sub-groups of private well users. Private well users within Cluster 3 are characterised by higher levels of exposure and annual illness attributable to STEC, Giardia and norovirus than Clusters 1 and 2. Provincial incidence rates of 520.9 (1522 illness per year), 532.1 (2211 illness per year) and 605.5 (5345 illness per year) cases/100,000 private well users per year were predicted for private well users associated with Clusters 1 through 3. Established models will enable development of necessary tools tailored to specific groups of at-risk well users, allowing for preventative public health management of private groundwater sources.

Using unstable data from mobile phone applications to examine recent trajectories of retail centre recovery

The COVID-19 pandemic has changed the ways in which we shop, with significant impacts on retail and consumption spaces. Yet, empirical evidence of these impacts, specifically at the national level, or focusing on latter periods of the pandemic remain notably absent. Using a large spatio-temporal mobility dataset, which exhibits significant temporal instability, we explore the recovery of retail centres from summer 2021 to 2022, considering in particular how these responses are determined by the functional and structural characteristics of retail centres and their regional geography. Our findings provide important empirical evidence of the multidimensionality of retail centre recovery, highlighting in particular the importance of composition, e-resilience and catchment deprivation in determining such trajectories, and identifying key retail centre functions and regions that appear to be recovering faster than others. In addition, we present a use case for mobility data that exhibits temporal stability, highlighting the benefits of viewing mobility data as a series of snapshots rather than a complete time series. It is our view that such data, when controlling for temporal stability, can provide a useful way to monitor the economic performance of retail centres over time, providing evidence that can inform policy decisions, and support interventions to both acute and longer-term issues in the retail sector.

Surface time series models for large spatio-temporal datasets

The data observed in many phenomena have a spatial and a temporal component. Due to the rapid development of complex, performant technologies, spatio-temporal data can now be collected on a large scale. However, the statistical modeling of large sets of spatio-temporal data involves several challenging problems. For example, it is computationally challenging to deal with large datasets and spatio-temporal nonstationarity. Therefore, the development of novel statistical models is necessary. Here, we present a new methodology to model complex and large spatio-temporal datasets. In our approach, we estimate a continuous surface at each time point, and this captures the spatial dependence, possibly nonstationary. In this way, the spatio-temporal data result in a sequence of surfaces. Then, we model this sequence of surfaces using functional time series techniques. The functional time series approach allows us to obtain a computationally feasible methodology, and also provides extensive flexibility in terms of time-forecasting. We illustrate these advantages through a Monte Carlo simulation study. We also test the performance of our method using a high-resolution wind speed simulated dataset of over 4 million values. Overall, our method uses a new paradigm of data analysis in which the random fields are considered as a single entity, a very valuable approach in the context of big data.

SpatialEpi'2022 Workshop Report: The 3rd ACM SIGSPATIAL International Workshop on Spatial Computing for Epidemiology

Since the onset of the COVID-19 pandemic, researchers in the SIGSPATIAL community have utilized computational solutions to better explain, predict, and respond to infectious disease outbreaks. Using spatial computing for pandemic preparedness has also been highlighted as a major application of mobility data science [16]. At the beginning of the COVID-19 pandemic, the SIGSPATIAL community rapidly published ideas to improve our understanding of the spread of the virus in two SIGSPATIAL Special Newsletter Issues in March and July 2020 [28, 29]. These efforts led to the 1st and 2nd ACM SIGSPATIAL International Workshop on Spatial Computing for Epidemiology [5, 4] (formerly called Workshop on Modeling and Understanding the Spread of COVID-19 in 2020) which has provided authors of these newsletter articles a forum to present and discuss their solutions. Including both work published at the SIGSPATIAL Special Newsletter and regular peer-reviewed submissions, this workshop included topics such as the collection of large spatiotemporal datasets [20], leveraging data mining and spatial analysis techniques to analyze and visualize such data [2, 12, 25, 21, 11, 9, 8, 3], developing predictive spatial models and simulations [6, 1, 19, 13, 24, 10, 23, 14], and employing novel technologies towards contact tracing and surveillance [17, 26].

Deep learning based topic and sentiment analysis: COVID19 information seeking on social media

Social media platforms have become a common place for information exchange among their users. People leave traces of their emotions via text expressions. A systematic collection, analysis, and interpretation of social media data across time and space can give insights into local outbreaks, mental health, and social issues. Such timely insights can help in developing strategies and resources with an appropriate and efficient response. This study analysed a large Spatio-temporal tweet dataset of the Australian sphere related to COVID19. The methodology included a volume analysis, topic modelling, sentiment detection, and semantic brand score to obtain an insight into the COVID19 pandemic outbreak and public discussion in different states and cities of Australia over time. The obtained insights are compared with independently observed phenomena such as government-reported instances.

Scalable spatiotemporal regression model based on Moran’s eigenvectors

We propose a scalable regression model with spatially and temporally varying coefficients based on Moran’s eigenvectors and efficient computation algorithms. Regression models that consider spatiotemporal non-stationarity are important because many real-world datasets, such as housing prices, are tied to geographical and temporal locations. Although geographically weighted regression (GWR) and its variants are widely used to model spatially varying coefficients, they cannot handle large datasets. We employ an alternative modelling method of spatially varying coefficients based on Moran’s eigenvectors and extend it to handle large spatiotemporal datasets. Additionally, we introduce a scalable learning algorithm that exploits the model structures based on the Kalman filter and the expectation–maximisation algorithm. Our scalable algorithm is efficient even for large datasets that cannot be handled by GWR. To evaluate the performance of the proposed model, we applied it to a housing market dataset collected in Tokyo, Japan. The results show that the predictive performance of the proposed model is comparable to that of GWR while increasing the computational speed. Moreover, larger datasets can accelerate the algorithm convergence.

Prediction of Air Pollution in Smart Cities Using Machine Learning Techniques

Abstract: Air-pollution is one of the main threats for developed societies. According to the World Health Organization (WHO), pollution is the main cause of deaths among children aged under five years. Smart cities are called to play a decisive role to increase such pollution in real-time. The increase in air pollution due to fossil fuel consumption as well as its ill effects on the climate has made air pollution forecasting an important research area in today’s times. Deployment of the Internet of things (IoT) based sensors has considerably changed the dynamics of predicting air quality. prediction of spatio-temporal data has been one of the major challenges in creating a good predictive model. There are many different approaches which have been used to create an accurate predictive model. Primitive predictive machine learning algorithms like simple linear regression have failed to produce accurate results primarily due to lack of computing power but also due to lack of optimization techniques. A recent development in deep learning as well as improvements in computing resources has increased the accuracy of predicting time series data. However, with large spatio-temporal data sets spanning over years. Employing regression models on the entire data can cause per date predictions to be corrupted. In this work, we look at dealing with pre-processing the times series. However, pre-processing involves a similarity measure, we explore the use of Dynamic Time Warping (DTW). K-means is then used to classify the spatio-temporal pollution data over a period of 16 years from 2000 to 2016. Here Mean Absolute error (MAE) and Root Mean Square Error (RMSE) have been used as evaluation criteria for the comparison of regression models. Keywords: Spatio-temporal data, Primitive predictive machine learning algorithms, regression models

Coastal change patterns from time series clustering of permanent laser scan data

Abstract. Sandy coasts are constantly changing environments governed by complex, interacting processes. Permanent laser scanning is a promising technique to monitor such coastal areas and to support analysis of geomorphological deformation processes. This novel technique delivers 3-D representations of the coast at hourly temporal and centimetre spatial resolution and allows us to observe small-scale changes in elevation over extended periods of time. These observations have the potential to improve understanding and modelling of coastal deformation processes. However, to be of use to coastal researchers and coastal management, an efficient way to find and extract deformation processes from the large spatiotemporal data set is needed. To enable automated data mining, we extract time series of surface elevation and use unsupervised learning algorithms to derive a partitioning of the observed area according to change patterns. We compare three well-known clustering algorithms (k-means clustering, agglomerative clustering and density-based spatial clustering of applications with noise; DBSCAN), apply them on the set of time series and identify areas that undergo similar evolution during 1 month. We test if these algorithms fulfil our criteria for suitable clustering on our exemplary data set. The three clustering methods are applied to time series over 30 d extracted from a data set of daily scans covering about 2 km of coast in Kijkduin, the Netherlands. A small section of the beach, where a pile of sand was accumulated by a bulldozer, is used to evaluate the performance of the algorithms against a ground truth. The k-means algorithm and agglomerative clustering deliver similar clusters, and both allow us to identify a fixed number of dominant deformation processes in sandy coastal areas, such as sand accumulation by a bulldozer or erosion in the intertidal area. The level of detail found with these algorithms depends on the choice of the number of clusters k. The DBSCAN algorithm finds clusters for only about 44 % of the area and turns out to be more suitable for the detection of outliers, caused, for example, by temporary objects on the beach. Our study provides a methodology to efficiently mine a spatiotemporal data set for predominant deformation patterns with the associated regions where they occur.

Large-Scale Optical Reservoir Computing for Spatiotemporal Chaotic Systems Prediction

Reservoir computing is a relatively recent computational paradigm that originates from a recurrent neural network and is known for its wide range of implementations using different physical technologies. Large reservoirs are very hard to obtain in conventional computers, as both the computation complexity and memory usage grow quadratically. We propose an optical scheme performing reservoir computing over very large networks potentially being able to host several millions of fully connected photonic nodes thanks to its intrinsic properties of parallelism and scalability. Our experimental studies confirm that, in contrast to conventional computers, the computation time of our optical scheme is only linearly dependent on the number of photonic nodes of the network, which is due to electronic overheads, while the optical part of computation remains fully parallel and independent of the reservoir size. To demonstrate the scalability of our optical scheme, we perform for the first time predictions on large spatiotemporal chaotic datasets obtained from the Kuramoto-Sivashinsky equation using optical reservoirs with up to 50 000 optical nodes. Our results are extremely challenging for conventional von Neumann machines, and they significantly advance the state of the art of unconventional reservoir computing approaches, in general.

Analysis of a large spatiotemporal groundwater quality dataset, Ontario 2010–2017: Informing human health risk assessment and testing guidance for private drinking water wells

Approximately 1.5 million individuals in Ontario are supplied by private water wells (private groundwater supplies). Unlike municipal supplies, private well water quality remains unregulated, with owners responsible for testing, treating, and maintaining their own water supplies. The primary goal of this study was to assess the effect of repeat sampling of private well water in Ontario and investigate the efficacy of geographically- and/or temporally specific testing recommendations and health risk assessments. The current study combines the Well Water Information System Dataset and the Well Water Testing Dataset from 2010 to 2017, inclusive. These two large existing province-wide datasets collated over an eight-year period were merged using an integrated spatial fuzzy logic and (next)- nearest neighbour approach. Provincial sampling data from 239,244 wells (702,861 samples) were analyzed for Escherichia coli to study the relationship between sampling frequency and Escherichia coli detection. Dataset variables were delineated based on hydrogeological setting (e.g. aquifer type, overburden depth, well depth, bedrock type) and seasonality to provide an in-depth understanding of Escherichia coli detection in private well water. Findings reveal differences between detection rates in consolidated and unconsolidated aquifers (p = 0.0191), and across seasons (p < 0.0001). The variability associated with Escherichia coli detection rates was explored by estimating sentinel sampling rates for private wells sampled three times, twelve times and twenty-four times per year. As sample size increases on an annual basis, so too does detection rate, highlighting the need to address current testing frequency guidelines. Future health risk assessments for private well water should consider the impact of spatial and temporal factors on the susceptibility of this drinking water source, leading to an increasingly accurate depiction of private well water contamination and the estimated effects on human health.

Open‐source tools for the measurement of socio‐spatial segregation in activity spaces

AbstractOver the last decade, socio‐spatial segregation of different population groups in activity spaces has attracted considerable attention from geographers and sociologists. A careful examination of such activity‐space segregation can provide a more comprehensive description of dynamic intergroup relations in society than conventional residential segregation, as its measurement is not limited to a particular place. However, the evaluation of activity‐space segregation is a challenging task because it requires the manipulation of large spatiotemporal data sets. The lack of software tools that can assist researchers in exploring and analyzing individuals’ travel trajectories is one of the significant impediments to empirical studies. To address this practical limitation, this article proposes a combined use of two R packages, namely, slice and seg: the former provides tools for storing and manipulating activity‐space data, and the latter implements several existing measures of segregation that can be applied to various social places. The proposed approach is applied to the measurement of activity‐space segregation between high‐ and low‐income groups in Seoul to illustrate the key features of the packages. The results demonstrate that the use of these packages would make it more convenient to conduct activity‐space segregation research and enhance our understanding of the complex socio‐spatial phenomenon.

Data-Driven Approaches for Spatio-Temporal Analysis: A Survey of the State-of-the-Arts

With the advancement of telecommunications, sensor networks, crowd sourcing, and remote sensing technology in present days, there has been a tremendous growth in the volume of data having both spatial and temporal references. This huge volume of available spatio-temporal (ST) data along with the recent development of machine learning and computational intelligence techniques has incited the current research concerns in developing various data-driven models for extracting useful and interesting patterns, relationships, and knowledge embedded in such large ST datasets. In this survey, we provide a structured and systematic overview of the research on data-driven approaches for spatio-temporal data analysis. The focus is on outlining various state-of-the-art spatio-temporal data mining techniques, and their applications in various domains. We start with a brief overview of spatio-temporal data and various challenges in analyzing such data, and conclude by listing the current trends and future scopes of research in this multi-disciplinary area. Compared with other relevant surveys, this paper provides a comprehensive coverage of the techniques from both computational/methodological and application perspectives. We anticipate that the present survey will help in better understanding various directions in which research has been conducted to explore data-driven modeling for analyzing spatio-temporal data.

Scalable Algorithms for Maximizing Spatiotemporal Range Sum and Range Sum Change in Spatiotemporal Datasets

In this paper, we introduce the three-dimensional Maximum Range-Sum (3D MaxRS) problem and the Maximum Spatiotemporal Range-Sum Change (MaxStRSC) problem. The 3D MaxRS problem tries to find the 3D range where the sum of weights across all objects inside is maximized, and the MaxStRSC problem tries to find the spatiotemporal range where the sum of weights across all objects inside is maximally increased. The goal of this paper is to provide efficient methods for data analysts to find interesting spatiotemporal regions in a large historical spatiotemporal dataset by addressing two problems. We provide a mathematical explanation for each problem and propose several algorithms for them. Existing methods tried to find the optimal region over two-dimensional datasets or to monitor a burst region over two-dimensional data streams. The majority of them cannot directly solve our problems. Although some existing methods can be used or modified to solve the 3D MaxRS problems, they have limited scalability. In addition, none of them can be used to solve the MaxStRS-RC problem (a type of MaxStRSC problem). Finally, we study the performance of the proposed algorithms experimentally. The experimental results show that the proposed algorithms are scalable and much more efficient than existing methods.

From small sets of GPS trajectories to detailed movement profiles: quantifying personalized trip-dependent movement diversity

ABSTRACT The ubiquity of personal sensing devices has enabled the collection of large, diverse, and fine-grained spatio-temporal datasets. These datasets facilitate numerous applications from traffic monitoring and management to location-based services. Recently, there has been an increasing interest in profiling individuals' movements for personalized services based on fine-grained trajectory data. Most approaches identify the most representative paths of a user by analyzing coarse location information, e.g., frequently visited places. However, even for trips that share the same origin and destination, individuals exhibit a variety of behaviors (e.g., a school drop detour, a brief stop at a supermarket). The ability to characterize and compare the variability of individuals' fine-grained movement behavior can greatly support location-based services and smart spatial sampling strategies. We propose a TRip DIversity Measure --TRIM – that quantifies the regularity of users' path choice between an origin and destination. TRIM effectively captures the extent of the diversity of the paths that are taken between a given origin and destination pair, and identifies users with distinct movement patterns, while facilitating the comparison of the movement behavior variations between users. Our experiments using synthetic and real datasets and across geographies show the effectiveness of our method.

Investigating emergent nested geographic structure in consumer purchases: a Bayesian dynamic multi-scale spatiotemporal modeling approach

Spatial modeling of consumer response data has gained increased interest recently in the marketing literature. In this paper, we extend the (spatial) multi-scale model by incorporating both spatial and temporal dimensions in the dynamic multi-scale spatiotemporal modeling approach. Our empirical application with a US company’s catalog purchase data for the period 1997–2001 reveals a nested geographic market structure that spans geopolitical boundaries such as state borders. This structure identifies spatial clusters of consumers who exhibit similar spatiotemporal behavior, thus pointing to the importance of emergent geographic structure, emergent nested structure and dynamic patterns in multi-resolution methods. The multi-scale model also has better performance in estimation and prediction compared with several spatial and spatiotemporal models and uses a scalable and computationally efficient Markov chain Monte Carlo method that makes it suitable for analyzing large spatiotemporal consumer purchase datasets.

Technological advances in field studies of pollinator ecology and the future of e-ecology.

Our review looks at recent advances in technologies applied to studying pollinators in the field. These include RFID, radar and lidar for detecting and tracking pollinators; wireless sensor networks (e.g. 'smart' hives); automated visual and audio monitoring systems including vision motion software for monitoring fine-scale pollinator behaviours over extended periods; and automated species identification systems based on machine learning that can vastly reduce the bottleneck in (big) data analysis. An improved e-ecology platform that leverages these tools is needed for ecologists to acquire and understand large spatiotemporal datasets, and thus inform knowledge gaps in environmental policy-making. Developing the next generation of e-ecology tools will require synergistic partnerships between academia and industry and significant investment in a cross-disciplinary scientific consortia.

Extracting Patterns from Large Movement Datasets

Extracting useful information from large spatiotemporal datasets is a challenging task that requires suitable visual data representations. Big movement data are particularly hard to visualize since they are prone to visual clutter caused by overlapping and crisscrossing trajectories. Different data aggregation approaches have been developed to address this challenge and to provide analysts with better visualizations for data exploration and data-driven hypothesis generation. However, most approaches for extracting patterns, such as mobility graphs or generalized flow maps, cannot handle large input datasets. This paper presents a flow extraction algorithm that can be used in distributed computing environments and thus make it possible to explore movement patterns in large datasets. We demonstrate its usefulness in a use case exploring maritime vessel movements

Having your cake and eating it too: Flexibility and power with mass univariate statistics for ERP data.

ERP studies produce large spatiotemporal data sets. These rich data sets are key to enabling us to understand cognitive and neural processes. However, they also present a massive multiple comparisons problem, potentially leading to a large number of studies with false positive effects (a high Type I error rate). Standard approaches to ERP statistical analysis, which average over time windows and regions of interest, do not always control for Type I error, and their inflexibility can lead to low power to detect true effects. Mass univariate approaches offer an alternative analytic method. However, they have thus far been viewed as appropriate primarily for exploratory statistical analysis and only applicable to simple designs. Here, we present new simulation studies showing that permutation-based mass univariate tests can be employed with complex factorial designs. Most importantly, we show that mass univariate approaches provide slightly greater power than traditional spatiotemporal averaging approaches when strong a priori time windows and spatial regions are used. Moreover, their power decreases only modestly when more exploratory spatiotemporal parameters are used. We argue that mass univariate approaches are preferable to traditional spatiotemporal averaging analysis approaches for many ERP studies.