Spatio-temporal Operators Research Articles

EMO-MoviNet: Enhancing Action Recognition in Videos with EvoNorm, Mish Activation, and Optimal Frame Selection for Efficient Mobile Deployment.

The primary goal of this study is to develop a deep neural network for action recognition that enhances accuracy and minimizes computational costs. In this regard, we propose a modified EMO-MoviNet-A2* architecture that integrates Evolving Normalization (EvoNorm), Mish activation, and optimal frame selection to improve the accuracy and efficiency of action recognition tasks in videos. The asterisk notation indicates that this model also incorporates the stream buffer concept. The Mobile Video Network (MoviNet) is a member of the memory-efficient architectures discovered through Neural Architecture Search (NAS), which balances accuracy and efficiency by integrating spatial, temporal, and spatio-temporal operations. Our research implements the MoviNet model on the UCF101 and HMDB51 datasets, pre-trained on the kinetics dataset. Upon implementation on the UCF101 dataset, a generalization gap was observed, with the model performing better on the training set than on the testing set. To address this issue, we replaced batch normalization with EvoNorm, which unifies normalization and activation functions. Another area that required improvement was key-frame selection. We also developed a novel technique called Optimal Frame Selection (OFS) to identify key-frames within videos more effectively than random or densely frame selection methods. Combining OFS with Mish nonlinearity resulted in a 0.8-1% improvement in accuracy in our UCF101 20-classes experiment. The EMO-MoviNet-A2* model consumes 86% fewer FLOPs and approximately 90% fewer parameters on the UCF101 dataset, with a trade-off of 1-2% accuracy. Additionally, it achieves 5-7% higher accuracy on the HMDB51 dataset while requiring seven times fewer FLOPs and ten times fewer parameters compared to the reference model, Motion-Augmented RGB Stream (MARS).

AutoSTL: Automated Spatio-Temporal Multi-Task Learning

Spatio-temporal prediction plays a critical role in smart city construction. Jointly modeling multiple spatio-temporal tasks can further promote an intelligent city life by integrating their inseparable relationship. However, existing studies fail to address this joint learning problem well, which generally solve tasks individually or a fixed task combination. The challenges lie in the tangled relation between different properties, the demand for supporting flexible combinations of tasks and the complex spatio-temporal dependency. To cope with the problems above, we propose an Automated Spatio-Temporal multi-task Learning (AutoSTL) method to handle multiple spatio-temporal tasks jointly. Firstly, we propose a scalable architecture consisting of advanced spatio-temporal operations to exploit the complicated dependency. Shared modules and feature fusion mechanism are incorporated to further capture the intrinsic relationship between tasks. Furthermore, our model automatically allocates the operations and fusion weight. Extensive experiments on benchmark datasets verified that our model achieves state-of-the-art performance. As we can know, AutoSTL is the first automated spatio-temporal multi-task learning method.

Approximating the evolution of rotating moving regions using Bezier curves

The region interpolation methods proposed in the moving objects databases literature impose restrictions that can have a significant impact on the representation of the evolution of moving regions, in particular, when a rotation occurs between two observations. In this paper, we propose a data model for moving regions that allows moving segments to rotate and change their length during their evolution between two observations and uses quadratic Bezier curves to define the trajectories of their endpoints. This introduces a new class of moving regions called rotating moving regions (rmregions). We present algorithms for operations involving rmregions and we propose a strategy to allow different interpolation methods to be used in the context of moving objects databases by approximating the interpolations they create using rmregions. We demonstrate our strategy using a reference implementation and compare results obtained when using the strategy presented here and the region interpolation methods and the spatiotemporal operations proposed in the state-of-the-art. Experimental results show that our strategy can be used to complement the region interpolation methods proposed in the moving objects databases literature.

CRISPR/Cas9 delivery by NIR-responsive biomimetic nanoparticles for targeted HBV therapy

BackgroundCurrently, there are no curative drugs for hepatitis B virus (HBV). Complete elimination of HBV covalently closed circular DNA (cccDNA) is key to the complete cure of hepatitis B virus infection. The CRISPR/Cas9 system can directly destroy HBV cccDNA. However, a CRISPR/Cas9 delivery system with low immunogenicity and high efficiency has not yet been established. Moreover, effective implementation of precise remote spatiotemporal operations in CRISPR/Cas9 is a major limitation.ResultsIn this work, we designed NIR-responsive biomimetic nanoparticles (UCNPs-Cas9@CM), which could effectively deliver Cas9 RNP to achieve effective genome editing for HBV therapy. HBsAg, HBeAg, HBV pgRNA and HBV DNA along with cccDNA in HBV-infected cells were found to be inhibited. These findings were confirmed in HBV-Tg mice, which did not exhibit significant cytotoxicity and minimal off-target DNA damage.ConclusionsThe UCNPs-based biomimetic nanoplatforms achieved the inhibition of HBV replication via CRISPR therapy and it is a potential system for efficient treatment of human HBV diseases.Graphical

Evaluating preprocessing and interpolation strategies to create moving regions from real-world observations

Spatio-temporal data may be used to represent the evolution of real-world objects and phenomena. Such data can be represented in discrete time, which associates spatial information (like position and shape) to time instants, or in continuous time, in which the representation of the evolution of the phenomena is decomposed into slices and interpolation functions are used to estimate the intermediate position and shape at any time. The use of a discrete model may seem more straightforward, but a continuous representation provides potential gains in terms of data management, including in compression and spatio-temporal operations. In this work, we study the use of the continuous model to represent deformable moving regions captured at discrete snapshots. We use a dissimilarity distance-based strategy to select the observations that should be used to define the time slices of the continuous representation, thus transforming data acquired at discrete steps into a continuous model. We also study how the use of geometry simplification algorithms and simplification levels may impact on moving regions interpolation quality. We evaluate our proposals using a dataset composed by thousands of aerial bush-fires images. After applying object simplification and slice decomposition, we use two region interpolation algorithms to generate in-between observations and compare them with geometries representing real images. The results prove the effectiveness of our proposals and their importance in terms of interpolation accuracy.

Window Operators for Processing Spatio-Temporal Data Streams on Unmanned Vehicles

Abstract. Unmanned aerial and submersible vehicles are used in an increasing number of applications especially for data collection in misanthropic environments. During a mission, such vehicles generate multiple spatio-temporal data streams suitable to be processed by data stream management systems (DSMS). The main approach of a DSMS is limiting the elements of a stream by using sliding and tilting windows with time intervals as temporal condition. However, due to varying vehicle speed and limited on-board resources, such temporal windows do not provide adequate support for spatio-temporal problems. For solving this problem, we propose a set of six new spatio-temporal window operators in this paper. This set comprises of sliding distance, tilting distance, tilting waypoint, session distance, jumping distance and an area window to limit stream elements based on spatial conditions. Each of the listed operators provides an individual behaviour to support sophisticated applications like spatial interpolation and forecasting. An evaluation based on an example trajectory shows the benefit of the presented operators for spatio-temporal applications.

Convolutional LSTM based transportation mode learning from raw GPS trajectories

With the advancement of location acquisition technologies, a large amount of raw global positioning system (GPS) trajectory data is produced by many moving devices. Learning transportation modes from the GPS trajectory data is an important problem in the domain of trajectory data mining. Traditional supervised learning-based approaches rely heavily on data preprocessing and feature engineering, which require domain expertise and are time consuming. The authors propose a deep learning-based convolutional long short term memory (LSTM) model for transportation mode learning, in which the convolution neural network is first used to extract deep high-level features and then LSTM is used to learn the sequential patterns in the data that uses both GPS and weather features, thus making the full use of spatiotemporal operations. The authors have also analysed the impact of the geospatial region on human mobility. Experiments conducted on the Microsoft Geolife data set fused with the weather data set show that their model achieves the state-of-the-art results. The authors compare the performance of their model with the benchmark models, which shows the superiority of their model having 3% improvement in accuracy using only GPS features, and the accuracy is further improved by 4 and 7% on including the impact of geospatial region and weather attributes, respectively.

Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning

We propose a novel self-supervised method, referred to as Video Cloze Procedure (VCP), to learn rich spatial-temporal representations. VCP first generates “blanks” by withholding video clips and then creates “options” by applying spatio-temporal operations on the withheld clips. Finally, it fills the blanks with “options” and learns representations by predicting the categories of operations applied on the clips. VCP can act as either a proxy task or a target task in self-supervised learning. As a proxy task, it converts rich self-supervised representations into video clip operations (options), which enhances the flexibility and reduces the complexity of representation learning. As a target task, it can assess learned representation models in a uniform and interpretable manner. With VCP, we train spatial-temporal representation models (3D-CNNs) and apply such models on action recognition and video retrieval tasks. Experiments on commonly used benchmarks show that the trained models outperform the state-of-the-art self-supervised models with significant margins.

A Topology Based Spatio-Temporal Map Algebra for Big Data Analysis

Continental and global datasets based on earth observations or computational models challenge the existing map algebra approaches. The available datasets differ in their spatio-temporal extents and their spatio-temporal granularity, which makes it difficult to process them as time series data in map algebra expressions. To address this issue we introduce a new map algebra approach that is topology based. This topology based map algebra uses spatio-temporal topological operators (STTOP and STTCOP) to specify spatio-temporal operations between topological related map layers of different time-series data. We have implemented several topology based map algebra tools in the open source geoinformation system GRASS GIS and its open source cloud processing engine actinia. We demonstrate the application of our topology based map algebra by solving real world big data problems using a single algebraic expression. This included the massively parallel computation of the NDVI from a series of 100 Sentinel2A scenes organized as earth observation data cubes. The processing was performed and benchmarked on a many core computer setup and in a distributed container environment. The design of our topology based map algebra allows us to deploy it as a standardized service in the EU Horizon 2020 project openEO.

Accurate manipulation of optogenetic proteins with wavelength tunable femtosecond laser system

Photoactivated proteins controlled by optogenetic tools have broad application prospects in cell biology, neuroscience, and brain science. However, due to the narrow excitation wavelength width and the inflexibility of spatiotemporal operations, conventional sources such as visible light severely limit the further application of optogenetics. In this work, a femtosecond laser-operated system based on the optogenetic application was designed to address these limitations. The interaction between the photoreceptor and its partner protein can be triggered by a wavelength-tunable femtosecond laser. The results indicated that this process can be used to accurately manipulate optogenetic proteins in cells, which met spectral flexibility (700–1040 nm) and operational flexibility in time and space (a single cell to multiple cells). To demonstrate the practical applications of this process, the apoptotic signaling pathway of cancer cells was taken as an example. We believe that this wavelength-tunable femtosecond laser system will promote the development of optogenetics, making optics and even physics more powerful tools in biology.

EFFICIENT AND PRACTICAL HANDLING OF SPATIO-TEMPORAL DATA BASED ON TIME-DEPENDENT NET COMPONENTS

Abstract. The monitoring of spatio-temporal phenomena such as the movement of glaciers or volcanos produces huge amounts of data. Especially the handling of time as a fourth dimension usually requires terabytes of storage space. Therefore, an appropriate data management model for an efficient handling of such data is required and a key concept for any 3D/4D GIS. Within this paper, a comprehensive data management model is presented that optimizes the handling and storage of spatio-temporal data. The concept of time-dependent Net Components is introduced to handle regions that are monitored and modelled in individual time steps. Combined with the concept of Point Tubes the resulting data management model is used to manage spatio-temporal data based on d-simplicial complexes with dimensions d ∈ {0,1,2,3}. It is shown that the model is capable of handling various kinds of spatio-temporal applications. These include the proper handling of variable temporal discretizations of partial regions within a 4D model. The storage requirements are reduced and spatial as well as spatio-temporal operations are accelerated significantly. Critical issues such as the preservation of the net topology between time steps and the handling of boundary regions of adjacent net components can be solved.

Mining spatiotemporal co-occurrence patterns in non-relational databases

Spatiotemporal co-occurrence patterns (STCOPs) represent the subsets of feature types whose instances are frequently co-occurring both in space and time. Spatiotemporal co-occurrences reflect the spatiotemporal overlap relationships among two or more spatiotemporal instances both in spatial and temporal dimensions. STCOPs can be potentially used to predict and understand the generation and evolution of different types of interacting phenomena in various scientific fields such as astronomy, meteorology, biology, geosciences. Meaningful and statistically significant data analysis for these scientific fields requires processing sufficiently large datasets. Due to the computationally expensive nature of spatiotemporal operations required for mining spatiotemporal co-occurrences, it is increasingly difficult to identify spatiotemporal co-occurrences and discover STCOPs in centralized system settings. As a solution, we developed a cloud-based distributed mining system for discovering STCOPs. Our system uses Accumulo, a column-oriented non-relational database management system as its backbone. In order to efficiently mine the STCOPs, we propose three data models for managing trajectory-based spatiotemporal data in Accumulo. We introduce an in-memory join-index structure and a join algorithm for effectively performing spatiotemporal join operations on spatiotemporal trajectories in non-relational databases. Lastly, with the experiments with artificial and real life datasets, we evaluate the performance of the proposed models for STCOP mining.

Linking movement and environmental data: The need for representation

In several domains, there has been an increasing interest in analysing moving objects due to the recent ubiquity of location tracking devices. These locations are commonly abstracted and stored as trajectories: a finite set of ordered in time geometries. Tracked objects tend to move in a certain environment that influences their movement behaviour. Time dependent environmental data are commonly abstracted and stored in grid/array structures that have different granularities and characteristics compare to the trajectory datasets. Movement analysis requires linking these two data types. However, little consideration has been given to the issues of integration in the moving objects databases that is primarily dealing with trajectory storage and analysis as well as in array databases dealing mainly with the storage and retrieval of grid structures. In this paper, we propose a database model that utilizes abstract data types for combining trajectories and time dependent environmental data. We introduce a set of spatio-temporal operations for interacting with raster data and integrating them with the moving geometries. We demonstrate how the integration operations can be used for manipulation and analysis of moving objects, using trajectories of tropical cyclone and environmental data, using sea surface temperature (SST) for the period from 1980 to 2009 as a case study. Since tropical cyclones generally gain strength over the warmer seas, the proposed operations are used to answer questions about genesis and movement patterns of tropical cyclones, in relation to the changing patterns of the SST.

Spatiotemporal data model for network time geographic analysis in the era of big data

ABSTRACTThere has been a resurgence of interest in time geography studies due to emerging spatiotemporal big data in urban environments. However, the rapid increase in the volume, diversity, and intensity of spatiotemporal data poses a significant challenge with respect to the representation and computation of time geographic entities and relations in road networks. To address this challenge, a spatiotemporal data model is proposed in this article. The proposed spatiotemporal data model is based on a compressed linear reference (CLR) technique to transform network time geographic entities in three-dimensional (3D) (x, y, t) space to two-dimensional (2D) CLR space. Using the proposed spatiotemporal data model, network time geographic entities can be stored and managed in classical spatial databases. Efficient spatial operations and index structures can be directly utilized to implement spatiotemporal operations and queries for network time geographic entities in CLR space. To validate the proposed spatiotemporal data model, a prototype system is developed using existing 2D GIS techniques. A case study is performed using large-scale datasets of space-time paths and prisms. The case study indicates that the proposed spatiotemporal data model is effective and efficient for storing, managing, and querying large-scale datasets of network time geographic entities.

Spectrotemporal dynamics of the EEG during working memory encoding and maintenance predicts individual behavioral capacity.

We investigated the effect of memory load on encoding and maintenance of information in working memory. Electroencephalography (EEG) signals were recorded while participants performed a modified Sternberg visual memory task. Independent component analysis (ICA) was used to factorise the EEG signals into distinct temporal activations to perform spectrotemporal analysis and localisation of source activities. We found 'encoding' and 'maintenance' operations were correlated with negative and positive changes in α-band power, respectively. Transient activities were observed during encoding of information in the bilateral cuneus, precuneus, inferior parietal gyrus and fusiform gyrus, and a sustained activity in the inferior frontal gyrus. Strong correlations were also observed between changes in α-power and behavioral performance during both encoding and maintenance. Furthermore, it was also found that individuals with higher working memory capacity experienced stronger neural oscillatory responses during the encoding of visual objects into working memory. Our results suggest an interplay between two distinct neural pathways and different spatiotemporal operations during the encoding and maintenance of information which predict individual differences in working memory capacity observed at the behavioral level.

Querying a Trajectories Database About Sex Offenders

To combat against serial Sex Offenders (SOs), recent laws call for the use of the Global Positioning Satellite (GPS) technology to monitor their movements 24 hours a day. In this paper, we suggest the use of a spatio-temporal database suitable to store the trajectories of such a category of criminals, besides traditional data pertinent to the context (i.e., data about their home, their past history of offences, (pending) crimes, and sensible areas). To show the effectiveness of such a kind of database, then we focus on an investigative strategy that takes profit from the availability of the trips of the SOs in the database and implement it in terms of simple queries. As software platform, the SECONDO DataBase Management System (DBMS) is adopted, since it offers a data type about moving objects as well as a reach set of spatiotemporal operators that greatly simplify the management of trajectories. The solution recommended in our paper opens the frontier to a new generation of software applications, much more effective than those currently used by several criminal investigation departments all over the world, because it keeps together the two components that fully describe any criminal event, namely: space and time.

Investigative Strategies on Top of a Spatio-temporal Database about Sex Offenders

To combat against persistent sex offenders, recent laws call for the use of the GPS technology to monitor their movements. In [1], it has been proposed the adoption of a Spatio-Temporal DataBase (S-T DB) for archiving this type of complex data, besides traditional ones (i.e., data about their home, (pending) crimes, and sensible areas). In the present paper we discuss investigative strategies that can be easily implemented on top of such a kind of database by taking advantage of the history of the movements of sex offenders. As implementation platform, the SECONDO DB Management System (DBMS) is adopted since it supports a data type about moving objects as well as a reach set of spatio-temporal operators to query such complex data [2].

모바일 장치용 시공간 질의 처리 시스템의 개발

As the recent development of the ubiquitous computing environment, u-GIS is being highlighted as the core technology of the ubiquitous computing environment, and thereby, studies on spatio-temporal data are being actively conducted. In this u-GIS environment, it is still difficult for existing mobile devices to efficiently manage the massive spatio-temporal data of u-GIS that are increasing day by day. Therefore, this paper develops a spatio-temporal query processing system for mobile devices in order to solve the problem. The system provides various spatio-temporal operators to insert/delete/update/search spatio-temporal data and supports a query optimization function that uses a spatio-temporal index for the flash memory and a spatio-temporal histogram for guaranteeing query execution speed. Lastly, by applying the spatio-temporal query processing system developed in this paper to the virtual scenario, this paper has proved that the system can be utilized in various application fields necessary to process spatio-temporal data in the mobile environment.

모바일 환경을 위한 GML 기반 시공간 질의 처리 시스템

최근 무선 액세스 망의 범위가 증가하고 발전함에 따라 다양한 분야에서 u-GIS 서비스가 지원되고 있으며, 특히 모바일 환경에서의 u-GIS 서비스를 위해 시공간 데이터가 널리 활용되고 있다. 그러나 모바일 환경에서 활용되는 시공간 데이터에 대한 표준이 없으므로 서로 다른 시공간 데이터를 사용하는 모바일 u-GIS 서비스 간의 상호운용성을 위한 효율적인 시공간 데이터 처리 기술이 필요하다. 또한 모바일 장치의 저용량과 낮은 성능을 고려한 시공간 데이터의 수집, 저장, 관리 시스템이 필수적이다. 따라서 본 논문에서는 모바일 환경에서 시공간 데이터의 효율적인 관리를 위해 GML 기반의 질의 처리 시스템을 설계 및 구현하였다. GML 기반 시공간 질의 처리 시스템은 GML 문서의 특성인 상호운용성을 유지하고 저장 효율성을 높이기 위해 GML 스키마와 저장 테이블을 매핑하는 구조형 저장 방식과 Fast Infoset 기법을 이용한 바이너리 XML 저장 방식을 제공한다. 그리고 저장된 GML 문서의 시공간 데이터에 대한 신속한 질의 처리를 위하여 시공간 연산자를 제공한다. 마지막으로 본 논문에서 개발한 시스템을 가상 시나리오에 적용하여 본 시스템이 u-GIS 서비스를 위한 시스템으로 활용될 수 있음을 확인하였다. Recently, with increase and development of the wireless access network area, u-GIS Service is supported in various fields. Especially, spatio-temporal data is used in the mobile environment for the u-GIS service. However, there is no standard for the spatio-temporal data used in different spaces, spatio-temporal data processing technology is necessary to makes interoperability among mobile u-GIS services. Furthermore, it is also necessary to develop the system of gathering, storing, and managing the spatio-temporal data in consideration of small capacity and low performance of mobile devices. Therefore, in this paper, we designed and implemented a spatio-temporal query processing system based on GML to manage spatio-temporal data efficiently in the mobile environment. The spatio-temporal query processing system based on GML can offer a structured storage method which maps a GML schema to a storage table and a binary XML storage method which uses the Fast Infoset technique, so as to support interoperability that is an important feature of GML and increase storage efficiency. we can also provide spatio-temporal operators for rapid query processing of spatio-temporal data of GML documents. In addition, we proved that this system can be utilized for the u-GIS service to implement a virtual scenario.

A spatiotemporal extension for dealing with moving objects with extent in Oracle 11g

This paper deals with the design and implementation of a data model and operations for dealing with continuously changing spatial data in Oracle 11g object-relational DBMS. The data model relies on abstract data types but we introduce modifications to the internal structure of the spatiotemporal data representations proposed in the literature, to reduce storage requirements and to enable the reuse of data during the execution of queries. We show how to implement spatiotemporal operations relying on the spatial functions released by the underlying DBMS and how to use the alternative data representations to reduce the volume of temporary data created in the evaluation of spatiotemporal operations. We also demonstrate how to use the proposed data types and operations for storage and manipulation of moving objects data using SQL. Finally, we discuss on the advantages and disadvantages of the proposed solutions.