Multidimensional Data Structures Research Articles

MATAR: A performance portability and productivity implementation of data-oriented design with Kokkos

There is a need for simple, fast, and memory-efficient multidimensional data structures for dense and sparse storage that arise with numerical methods and in software applications. The data structures must perform equally well across multiple computer architectures, including CPUs and GPUs. For this purpose, we developed MATAR, a C++ software library that allows for simple creation and use of intricate data structures that is also portable across disparate architectures using Kokkos. The performance aspect is achieved by forcing contiguous memory layout (or as close to contiguous as possible) for multidimensional and multi-size dense or sparse MATrix and ARray (hence, MATAR) types. Our results show that MATAR has the capability to improve memory utilization, performance, and programmer productivity in scientific computing. This is achieved by fitting more work into the available memory, minimizing memory loads required, and by loading memory in the most efficient order. This document describes the purpose of the work, the implementation of each of the data types, and the resulting performance both in some simple baseline test cases and in an application code.

Assessing trends and predicting the cover management factor in a tropical island state using Enhanced Vegetation Index

Soil erosion is a complex process involving multiple factors that contribute to the amount of soil loss. The amount of vegetation cover is one of the main factors used to estimate soil loss and is an important risk factor in informing land use management and soil conservation policies. Cover/crop management (C-factor) is a dynamic soil loss factor and analysing C-factor trends in the context of both space and time, to build a multi-dimensional data structure, increased the value of the trend analysis. The objectives of the study were to (1) utilise EVI dependent static and dynamic predictive equations to compute the C-factor, and (2) to investigate the spatio-temporal changes of the C-factor and hotspots in a tropical small island developing state (SIDS) from 2010 to 2019. ArcGIS Model Builder was utilised to automate the computation of the C-factor using Moderate Resolution Imaging Spectroradiometer (MODIS) Enhanced Vegetation Index (EVI) data and compute the ordinary least square regression. Spatio-temporal analysis was performed using the novel emerging hot spot analysis in ArcGIS to identify statistically significant hot and cold spot trends of cover management to locate new, intensifying, persistent, or sporadic hot spot patterns at different time-step intervals. The regression output for the C-factor and EVI values indicated a strong r2, explaining on average 90% of the models. There was no statistically significant (P value > 0.05) increase in C-factor trend over a 10 year period (2010–2019, P-value = 0.92), a 5 year period (2015–2019; P-value = 0.59), or a 3 year period (2017–2019; P-Value = 0.31). Our results showed that intensifying hot spots (27%) were concentrated along the north–south corridor of the study area, highlighting areas with a statistically significant increase of the C-factor, thus a reduction in vegetation cover over the study period. Integrating spatio-temporal data and spatial technology provided vital cover management estimates for soil practitioners and farmers to guide conservation strategies.

Accurate and Fast Recovery of Network Monitoring Data With GPU-Accelerated Tensor Completion

Monitoring the performance of a large network would involve a high measurement cost. To reduce the overhead, sparse network monitoring techniques may be applied to select paths or time intervals to take the measurements, while the remaining monitoring data can be inferred leveraging the spatial-temporal correlations among data. The quality of missing data recovery, however, highly relies on the specific inference technique adopted. Tensor completion is a promising technique for more accurate missing data inference by exploiting the multi-dimensional data structure. However, data processing for higher dimensional tensors involves a large amount of computation, which prevents conventional tensor completion algorithms from practical application in the presence of large amount of data. This work takes the initiative to investigate the potential and methodologies of performing parallel processing for high-speed and high accuracy tensor completion over Graphics Processing Units (GPUs). We propose a GPU-accelerated parallel Tensor Completion scheme (GPU-TC) for accurate and fast recovery of missing data. To improve the data recovery accuracy and speed, we propose three novel techniques to well exploit the tensor factorization structure and the GPU features: grid-based tensor partition, independent task assignment based on Fisher-Yates shuffle, sphere facilitated and memory-correlated scheduling. We have conducted extensive experiments using network traffic trace data to compare the proposed GPU-TC with the state of art tensor completion algorithms and matrix-based algorithms. The experimental results demonstrate that GPU-TC can achieve significantly better performance in terms of two relative error ratio metrics and computation time.

Research on multi-dimensional analysis model of drilling operations safety evaluation system

Oil and gas drilling operation has the characteristics of large investment and high risk, so the effective identification and analysis of potential accidents is an important problem that needs to be solved urgently. In order to meet the requirements of complex analysis of drilling safety evaluation system, a multidimensional data analysis model based on data warehouse is constructed in view of the shortcomings of traditional database technology in information analysis ability, which transforms the application-oriented data structure into the analysis of multi-dimensional data structure, and provides a good environment and analysis method for drilling safety analysis and evaluation. According to the human, material, environmental and management factors of oil and gas drilling accidents, the relevant modelling topics were determined, and a multidimensional data model for analysing the potential accidents was constructed. This paper introduces the construction method of multidimensional data model with the theme of “analysis of potential accidents of drilling operation” as an example.

Structuring Multidimensional Data in the Study of the Development of Information Society in Russian Regions

In the article, the authors present the results of the analysis of the structure of multidimensional data on the factors of use of information and communication technologies (ICT) for the development of the information society in the Russian Federation. We chose the subjects of the Russian Federation as the objects of observations, and the average annual values of the corresponding factors of monitoring the development of the information society in the Russian Federation for the period from 2010 to 2017 as the factors of each object of observations. The study was carried out to reduce the set of factors of ICT use in the subjects of the Russian Federation and identify their structure. The analysis of the structure of factors is carried out by applying the Principal Component Analysis (PCA) method, implemented by the authors in the integrated development environment Rstudio. Through the use of the PCA method, we identified the structure of the initial factors of ICT use for the development of the information society in the Russian Federation and made their reduction. Then, we showed that the identified factor loads have specific semantic interpretations that consolidate the links of individual factors of ICT use for the development of the information society in the Russian Federation. The factor analysis carried out by us proves the effectiveness of the DCA method in research on the development of digitalisation of the regions of the Russian Federation for structuring the initial data and qualitative interpretation of the results. We have proved that factor loadings revealed employing the application of a method of RSA have the deterministic economic sense.

Accurate Recovery of Missing Network Measurement Data With Localized Tensor Completion

The inference of the network traffic data from partial measurements data becomes increasingly critical for various network engineering tasks. By exploiting the multi-dimensional data structure, tensor completion is a promising technique for more accurate missing data inference. However, existing tensor completion algorithms generally have the strong assumption that the tensor data have a global low-rank structure, and try to find a single and global model to fit the data of the whole tensor. In a practical network system, a subset of data may have stronger correlation. In this work, we propose a novel localized tensor completion model (LTC) to increase the data recovery accuracy by taking advantage of the stronger local correlation of data to form and recover sub-tensors each with a lower rank. Despite that it is promising to use local tensors, the finding of correlated entries faces two challenges, the data with adjacent indexes are not ones with higher correlation and it is difficult to find the similarity of data with missing tensor entries. To conquer the challenges, we propose several novel techniques: efficiently calculating the candidate anchor points based on locality-sensitive hash (LSH), building sub-tensors around properly selected anchor points, encoding factor matrices to facilitate the finding of similarity with missing entries, and similarity-aware local tensor completion and data fusion. We have done extensive experiments using real traffic traces. Our results demonstrate that LTC is very effective in increasing the tensor recovery accuracy without depending on specific tensor completion algorithms.

An efficient storage and service method for multi-source merging meteorological big data in cloud environment

With the development of the meteorological IoT (Internet of Things) and meteorological sensing network, the collected multi-source meteorological data have the characteristics of large amount of information, multidimensional and high accuracy. Cloud computing technology has been applied to the storage and service of meteorological big data. Although the constant evolution of big data storage technology is improving the storage and access of meteorological data, storage and service efficiency is still far from meeting multi-source big data requirements. Traditional methods have been used for the storage and service of meteorological data, and a number of problems still persist, such as a lack of unified storage structure, poor scalability, and poor service performance. In this study, an efficient storage and service method for multidimensional meteorological data is designed based on NoSQL big data storage technology and the multidimensional characteristics of meteorological data. In the process of data storage, multidimensional block compression technology and data structures are applied to store and transmit meteorological data. In service, heterogeneous NoSQL common components are designed to improve the heterogeneity of the NoSQL database. The results show that the proposed method has good storage transmission efficiency and versatility, and can effectively improve the efficiency of meteorological data storage and service in meteorological applications.

Perception of human skin conditions and imagestatistics.

The color and texture of human skin provide useful information in evaluating a person's health, emotion, attractiveness, and so on. Whereas the appearance of skin is a product of optical properties originating from its own biophysical substrates, recent psychophysical studies of surface material perception indicate a possibility that humans can perceive skin conditions from relatively simple image features such as texture statistics. The present study investigated the structure of multidimensional perceptual rating data for 289 female skin images and analyzed diagnostic low-level image statistics. We found that various skin appearances can be summarized into two dimensions, pleasantness and glossiness, and either dimension is well correlated with a small set of subband luminance and color statistics. We confirmed that manipulations of these critical image statistics significantly alter the appearance of skin in the expected direction. The results support a notion that humans perceive skin properties essentially based on two cardinal dimensions as predicted by simple image statistics.

CREATING NEURAL NETWORK FOR IDENTIFICATION OF POTENTIAL BANK CUSTOMERS

The article is devoted to the application of the Data Mining «Neural Network» algorithm for analyzing the available data of potential bank customers. The article discusses the possibilities and features of the creation and use of neural networks when conducting a comprehensive data analysis and building a system for further forecasting. The authors set the task of identifying the main characteristics of potential customers that affect a person’s desire to continue interacting with the bank after offering services to them. Based on the task, the neural network is created on the basis of the created multidimensional data structure to be able to predict the prospects of interaction with each client. The constructed neural network allows identifying potential customers based on new customer data to develop further marketing campaigns and optimize banking strategies aimed at improving long-term customer interaction.

The combined use of self-organizing map technique and fuzzy c-means clustering to evaluate urban groundwater quality in Seoul metropolitan city, South Korea

To make an overall assessment of the groundwater quality in Seoul city, we used the self-organizing map (SOM) technique in combination with fuzzy c-means (FCM) clustering. SOM visualizes complicate and multidimensional data structures on a 2D surface while the FCM algorithm creates overlapping cluster boundaries among samples that are continuously distributed over a data space. The combination of SOM and FCM clustering was expected to help characterize highly complicated urban groundwater quality. As a result, the SOM characterized 343 groundwater samples using 91 neurons, which were further classified by FCM clustering into three water groups. Group 1 addressed the least polluted groundwater (17% of the samples (n = 58), average TDS = 194.5 mg/L and NO3 = 6.9 mg/L) and occurred in the peripheral areas whose land cover is mainly occupied by forests. Increasing pH with increasing sodium and bicarbonate concentrations indicated that the hydrogeochemistry of Group 1 was largely controlled by water-rock interactions. Group 2 included the highly polluted groundwater (24% of the samples (n = 82), average TDS = 326.2 mg/L and NO3 = 42.6 mg/L), and sporadically occurred in Seoul, with no distinct spatial control. This group seemed to be affected by sewage from broken sewer pipes, which are a primary pollution source of Seoul groundwater and are ubiquitously distributed beneath the city. Group 3 water also represented the highly contaminated groundwater (30% of the samples (n = 103), average TDS = 527.1 mg/L), but contained low nitrate concentrations (average NO3 = 13.1 mg/L). Based on their spatial locations, intensive groundwater pumping from subway tunnels and other underground spaces at the city center seemed to drive the induced flow of organic contaminants, resulting in local reducing conditions sufficient for denitrification. The remaining 100 samples (29% of the samples) shared the hydrogeochemical properties of two or three groups. This study successfully characterized the spatial pattern of urban groundwater quality that is complicated by various contamination sources and hydrogeochemical processes. The combined use of SOM and FCM clustering was proven as a powerful tool to interpret nonlinear and highly heterogeneous environmental data for which it is difficult to define cluster boundaries. Taken together, our results contribute to a better management of urban groundwater in metropolitan cities under high risks of anthropogenic contamination.

A Speckle Filtering Method Based on Hypothesis Testing for Time-Series SAR Images

To improve the suppression effect for the speckle noise of synthetic aperture radar (SAR) images and the ability of spatiotemporal information preservation of the filtered image without losing the spatial resolution, a novel multitemporal filtering method based on hypothesis testing is proposed in this paper. A framework of a two-step similarity measure strategy is adopted to further enhance the filtering results. Firstly, bi-date analysis using a two-sample Kolmogorov-Smirnov (KS) test is conducted in step 1 to extract homogeneous patches for 3-D patch stacks generation. Subsequently, the similarity between patch stacks is compared by a sliding time-series likelihood ratio (STSLR) test algorithm in step 2, which utilizes the multi-dimensional data structure of the stacks to improve the accuracy of unchanged pixels detection. Finally, the filtered values are obtained by averaging the similar pixels in time-series. The experimental results and analysis of two multitemporal datasets acquired by TerraSAR-X show that the proposed method outperforms the other typical methods with regard to the overall filtering effect, especially in terms of the consistency between the filtered images and the original ones. Furthermore, the performance of the proposed method is also discussed by analyzing the results from step 1 and step 2.

Co-occurrence simplicial complexes in mathematics: identifying the holes of knowledge

In the last years complex networks tools contributed to provide insights on the structure of research, through the study of collaboration, citation and co-occurrence networks. The network approach focuses on pairwise relationships, often compressing multidimensional data structures and inevitably losing information. In this paper we propose for the first time a simplicial complex approach to word co-occurrences, providing a natural framework for the study of higher-order relations in the space of scientific knowledge. Using topological methods we explore the conceptual landscape of mathematical research, focusing on homological holes, regions with low connectivity in the simplicial structure. We find that homological holes are ubiquitous, which suggests that they capture some essential feature of research practice in mathematics. k-dimensional holes die when every concept in the hole appears in an article together with other k+1 concepts in the hole, hence their death may be a sign of the creation of new knowledge, as we show with some examples. We find a positive relation between the size of a hole and the time it takes to be closed: larger holes may represent potential for important advances in the field because they separate conceptually distant areas. We provide further description of the conceptual space by looking for the simplicial analogs of stars and explore the likelihood of edges in a star to be also part of a homological cycle. We also show that authors’ conceptual entropy is positively related with their contribution to homological holes, suggesting that polymaths tend to be on the frontier of research.

Sparse tensor neighbor embedding based pan-sharpening via N-way block pursuit

Most of the available pan-sharpening methods use vector or matrix based detail injection to enhance the resolution of MultiSpectral (MS) image, which may result in unavoidable spectral and spatial distortions. In this paper we explore the intrinsic tensor structure and local sparsity of MS images, to develop a novel Sparse Tensor Neighbor Embedding (STNE) based pan-sharpening method that reduces the distortions in the fused images. First, MS images are formulated as some spectral tensors, and each tensor and its nearest neighbor tensors are assumed to lie in a low-dimensional manifold. Then the tensor is sparsely coded under its neighbor tensors, and a joint sparse coding assumption is cast on bands to develop an N-way Block Pursuit algorithm for solving sparse tensor coefficients. Finally high resolution MS tensor can be obtained by weighting Panchromatic image with the sparse tensor coefficients. Tensors are higher order generalizations of vectors and matrices, and taking advantage of high-order structure of multi-dimensional data can help us understand them. The proposed method first combines a sparse tensor with neighbor embedding, to construct a new high-dimensional sparse tensor embedding for efficient pan-sharpening. Because tensor formulation can exploit the structural correlations in high-dimensional MS data, the proposed method can well preserve spectral correlation among different bands simultaneously and capture the underlying high-order statistical properties of MS image. Some experiments are performed on several real QuickBird and GeoEye datasets, and experimental results show that STNE is superior to its counterparts in reducing spectral and spatial distortions.

Efficient Privacy-Preserving Cube-Data Aggregation Scheme for Smart Grids

Efficient power management in smart grids requires obtaining power consumption data from each resident. However, data concerning user’s electricity consumption might reveal sensitive information, such as living habits and lifestyles. In order to solve this problem, this paper proposes a privacy-preserving cube-data aggregation scheme for electricity consumption. In our scheme, a data item is described as a multi-dimensional data structure ( $l$ -dimensional), and users form and live in multiple residential areas ( $m$ areas, and at most $n$ users in each area). Based on Horner’s Rule, for each user, we construct a user-level polynomial to store dimensional values in a single data space by using the first Horner parameter. After embedding the second Horner parameter into the polynomial, the polynomial is hidden by using Paillier cryptosystem. By aggregating data from $m$ areas, we hide the area-level polynomial into the final output. Moreover, we propose a batch verification scheme in multi-dimensional data to reduce authentication cost. Finally, our analysis shows that the proposed scheme is efficient in terms of computation and communication costs, suitable for massive user groups, and supports the flexible and rapid growth of residential scales in smart grids.

Fast Algorithms for Higher-Order Singular Value Decomposition from Incomplete Data

Higher-order singular value decomposition (HOSVD) is an efficient way for data reduction and also eliciting intrinsic structure of multi-dimensional array data. It has been used in many applications, and some of them involve incomplete data. To obtain HOSVD of the data with missing values, one can first impute the missing entries through a certain tensor completion method and then perform HOSVD to the reconstructed data. However, the two-step procedure can be inefficient and does not make reliable decomposition. In this paper, we formulate an incomplete HOSVD problem and combine the two steps into solving a single optimization problem, which simultaneously achieves imputation of missing values and also tensor decomposition. We also present two algorithms for solving the problem based on block coordinate update. Global convergence of both algorithms is shown under mild assumptions. The convergence of the second algorithm implies that of the popular higher-order orthogonality iteration (HOOI) method, and thus we, for the first time, give global convergence of HOOI. In addition, we compare the proposed methods to state-of-the-art ones for solving incomplete HOSVD and also low-rank tensor completion problems and demonstrate the superior performance of our methods over other compared ones. Furthermore, we apply them to face recognition and MRI image reconstruction to show their practical performance.

A Dynamic Data Structure to Efficiently Find the Points below a Line and Estimate Their Number

A basic question in computational geometry is how to find the relationship between a set of points and a line in a real plane. In this paper, we present multidimensional data structures for N points that allow answering the following queries for any given input line: (1) estimate in O ( log N ) time the number of points below the line; (2) return in O ( log N + k ) time the k ≤ N points that are below the line; and (3) return in O ( log N ) time the point that is closest to the line. We illustrate the utility of this computational question with GIS applications in air defense and traffic control.

Joins via Geometric Resolutions

We present a simple geometric framework for the relational join. Using this framework, we design an algorithm that achieves the fractional hypertree-width bound, which generalizes classical and recent worst-case algorithmic results on computing joins. In addition, we use our framework and the same algorithm to show a series of what are colloquially known as beyond worst-case results. The framework allows us to prove results for data stored in BTrees, multidimensional data structures, and even multiple indices per table. A key idea in our framework is formalizing the inference one does with an index as a type of geometric resolution, transforming the algorithmic problem of computing joins to a geometric problem. Our notion of geometric resolution can be viewed as a geometric analog of logical resolution. In addition to the geometry and logic connections, our algorithm can also be thought of as backtracking search with memoization.

On the Cost of Fixed Partial Match Queries in K-d Trees

Partial match queries constitute the most basic type of associative queries in multidimensional data structures such as $$K$$K-d trees or quadtrees. Given a query $$\mathbf {q}=(q_0,\ldots ,q_{K-1})$$q=(q0,?,qK-1) where s of the coordinates are specified and $$K-s$$K-s are left unspecified ($$q_i=*$$qi=?), a partial match search returns the subset of data points $$\mathbf {x}=(x_0,\ldots ,x_{K-1})$$x=(x0,?,xK-1) in the data structure that match the given query, that is, the data points such that $$x_i=q_i$$xi=qi whenever $$q_i\not =*$$qi??. There exists a wealth of results about the cost of partial match searches in many different multidimensional data structures, but most of these results deal with random queries. Only recently a few papers have begun to investigate the cost of partial match queries with a fixed query $$\mathbf {q}$$q. This paper represents a new contribution in this direction, giving a detailed asymptotic estimate of the expected cost $$P_{{n},\mathbf {q}}$$Pn,q for a given fixed query $$\mathbf {q}$$q. From previous results on the cost of partial matches with a fixed query and the ones presented here, a deeper understanding is emerging, uncovering the following functional shape for $$P_{{n},\mathbf {q}}$$Pn,q$$\begin{aligned} P_{{n},\mathbf {q}} = \nu \cdot \left( \prod _{i:q_i\text { is specified}}\, q_i(1-q_i)\right) ^{\alpha /2}\cdot n^\alpha + \text {l.o.t.} \end{aligned}$$Pn,q=?·?i:qiis specifiedqi(1-qi)?/2·n?+l.o.t.(l.o.t. lower order terms, throughout this work) in many multidimensional data structures, which differ only in the exponent $$\alpha $$? and the constant $$\nu $$?, both dependent on s and K, and, for some data structures, on the whole pattern of specified and unspecified coordinates in $$\mathbf {q}$$q as well. Although it is tempting to conjecture that this functional shape is universal, we have shown experimentally that it seems not to be true for a variant of $$K$$K-d trees called squarish $$K$$K-d trees.

Buildings and terrain unified – multidimensional dual data structure for GIS

3D city models are widely used in many disciplines and applications, such as urban planning, disaster management, and environmental simulation. Usually, the terrain and embedded objects like buildings are taken into consideration. A consistent model integrating these elements is vital for GIS analysis, especially if the geometry is accompanied by the topological relations between neighboring objects. Such a model allows for more efficient and errorless analysis. The memory consumption is another crucial aspect when the wide area of a city is considered – light models are highly desirable. Three methods of the terrain representation using the geometrical–topological data structure – the dual half-edge – are proposed in this article. The integration of buildings and other structures like bridges with the terrain is also presented.

Data Warehouse Software Used in the Decisional Process on the Capital Market

Capital market provides strong opportunities for investors who know how to adopt favorable decision in real time. The decision to invest on the capital market involves deep knowledge on the mechanisms and characteristics of this market, the factors that determine the consequences of investment decisions and the performance of allocated capital. As the data volume related to the capital market evolution reaches a sizable amount, we consider appropriate to propose a multidimensional data structure that would be able to assist the investor in analyzing the shares in which he would like to invest. This structure could be designed by using an accessible support system, as the dimensional model proposed is valid for any DSS based on OLAP.