Multidimensional Arrays Research Articles

Tensor Landmark Analysis With Application to ADNI data

ABSTRACTRecent advancements in data collection have facilitated the use of multidimensional arrays, also known as tensors, in prediction of health outcomes. In this article, we introduce a tensor landmark model for predicting survival outcomes using multiple longitudinal biomarkers as tensor covariates through CANDECOMP/PARAFAC (CP) decomposition. An iteratively reweighted least squares estimation is adopted for components of the tensor coefficients in the landmark model. We also present empirical results of the Akaike information criterion (AIC) and the Bayesian information criterion (BIC) for right‐censored data to select the CP rank. Simulations and Alzheimer's disease neuroimaging initiative (ADNI) data analysis demonstrate that our proposed model accurately estimates survival coefficients and predicts survival probabilities. The implementation code can be found online (https://github.com/sparkqkr/TensorCoxReg).

Partial Quantile Tensor Regression

Tensors, characterized as multidimensional arrays, are frequently encountered in modern scientific studies. Quantile regression has the unique capacity to explore how a tensor covariate influences different segments of the response distribution. In this work, we propose a partial quantile tensor regression (PQTR) framework, which novelly applies the core principle of the partial least squares technique to achieve effective dimension reduction for quantile regression with a tensor covariate. The proposed PQTR algorithm is computationally efficient and scalable to a large tensor covariate. Moreover, we uncover an appealing latent variable model representation for the PQTR algorithm, justifying a simple population interpretation of the resulting estimator. We further investigate the connection of the PQTR procedure with an envelope quantile tensor regression (EQTR) model, which defines a general set of sparsity conditions tailored to quantile tensor regression. We prove the root-n consistency of the PQTR estimator under the EQTR model, and demonstrate its superior finite-sample performance compared to benchmark methods through simulation studies. We demonstrate the practical utility of the proposed method via an application to a neuroimaging study of post traumatic stress disorder (PTSD). Results derived from the proposed method are more neurobiologically meaningful and interpretable as compared to those from existing methods.

Compact lossy compression of tensors via neural tensor-train decomposition

AbstractMany real-world datasets are represented as tensors, i.e., multi-dimensional arrays of numerical values. Storing them without compression often requires substantial space, which grows exponentially with the order. While many tensor compression algorithms are available, many of them rely on strong data assumptions regarding its order, sparsity, rank, and smoothness. In this work, we propose TensorCodec, a lossy compression algorithm for general tensors that do not necessarily adhere to strong input data assumptions.TensorCodec incorporates three key ideas. The first idea is neural tensor-train decomposition (NTTD) where we integrate a recurrent neural network into Tensor-Train Decomposition to enhance its expressive power and alleviate the limitations imposed by the low-rank assumption. Another idea is to fold the input tensor into a higher-order tensor to reduce the space required by NTTD. Finally, the mode indices of the input tensor are reordered to reveal patterns that can be exploited by NTTD for improved approximation. In addition, we extend TensorCodec to enable the lossy compression of tensors with missing entries, often found in real-world datasets. Our analysis and experiments on 8 real-world datasets demonstrate that TensorCodec is (a) Concise: it gives up to $$7.38 \times $$ 7.38 × more compact compression than the best competitor with similar reconstruction error, (b) Accurate: given the same budget for compressed size, it yields up to $$3.33\times $$ 3.33 × more accurate reconstruction than the best competitor, (c) Scalable: Its empirical compression time is linear in the number of tensor entries, and it reconstructs each entry in logarithmic time. Our code and datasets are available at https://github.com/kbrother/TensorCodec.

Interoperable and scalable echosounder data processing with Echopype

Abstract Echosounders are high-frequency sonar systems used to sense fish and zooplankton underwater. Their deployment on a variety of ocean observing platforms is generating vast amounts of data at an unprecedented speed from the oceans. Efficient and integrative analysis of these data, whether across different echosounder instruments or in combination with other oceanographic datasets, is crucial for understanding marine ecosystem response to the rapidly changing climate. Here we present Echopype, an open-source Python software library designed to address this need. By standardizing data as labeled, multi-dimensional arrays encoded in the widely embraced netCDF data model following a community convention, Echopype enhances the interoperability of echosounder data, making it easier to explore and use. By leveraging scientific Python libraries optimized for distributed computing, Echopype achieves computational scalability, enabling efficient processing in both local and cloud computing environments. Echopype’s modularized package structure further provides a unified framework for expanding support for additional instrument raw data formats and incorporating new analysis functionalities. We plan to continue developing Echopype by supporting and collaborating with the echosounder user community, and envision that the growth of this package will catalyze the integration of echosounder data into broader regional and global ocean observation strategies.

Rapid computation of survival signature for dynamic fault tree based on sequential binary decision diagram and multidimensional array

Many practical safety-critical systems typically exhibit sequence-dependent failure behaviors, limiting the efficiency of analyzing these systems. Although the survival signature-based method can address this problem to a certain extent, the dependence on Boolean states constrains its application to large systems. In this study, we present a novel method that leverages the sequential binary decision diagram (SBDD) and multidimensional array to rapidly compute survival signatures for dynamic fault trees (DFTs) of these systems. These dynamic nodes in the SBDD are represented through multidimensional arrays, which are then utilized as inputs for the subsequent computations. Ultimately, survival signatures are obtained by iteratively computing the multidimensional arrays. Additionally, two practical engineering cases are examined to highlight the superiority of the proposed methods over other methods. Compared with Boolean state vector-based methods, the proposed method achieves a 689-fold and 209-fold increase in efficiency for calculating survival signatures in their respective cases. Compared with the Monte Carlo (MC) simulation, the simulation efficiency for the reliability results improve by 60-fold and 201-fold in their respective cases.

Quantum Tensor DBMS and Quantum Gantt Charts: Towards Exponentially Faster Earth Data Engineering

Earth data is essential for global environmental studies. Many Earth data types are naturally modeled by multidimensional arrays (tensors). Array (Tensor) DBMSs strive to be the best systems for tensor-related workloads and can be especially helpful for Earth data engineering, which takes up to 80% of Earth data science. We present a new quantum Array (Tensor) DBMS data model and new quantum approaches that rely on the upcoming quantum memory and demonstrate exponential speedups when applied to many toughest Array (Tensor) DBMS challenges stipulated by classical computing and real-world Earth data use-cases. We also propose new types of charts: Quantum Gantt (QGantt) Charts and Quantum Network Diagrams (QND). QGantt charts clearly illustrate how multiple operations occur simultaneously across different data items and what are the input/output data dependencies between these operations. Unlike traditional Gantt charts, which typically track project timelines and resources, QGantt charts integrate specific data items and operations over time. A Quantum Network Diagram combines several QGantt charts to show dependencies between multistage operations, including their inputs/outputs. By using a static format, QGantt charts and Quantum Network Diagrams allow users to explore complex processes at their own pace, which can be beneficial for educational and R&D purposes.

Mitigating Disparate Elevation Differences between Adjacent Topobathymetric Data Models Using Binary Code

Integrating coastal topographic and bathymetric data for creating regional seamless topobathymetric digital elevation models of the land/water interface presents a complex challenge due to the spatial and temporal gaps in data acquisitions. The Coastal National Elevation Database (CoNED) Applications Project develops topographic (land elevation) and bathymetric (water depth) regional scale digital elevation models by integrating multiple sourced disparate topographic and bathymetric data models. These integrated regional models are broadly used in coastal and climate science applications, such as sediment transport, storm impact, and sea-level rise modeling. However, CoNED’s current integration method does not address the occurrence of measurable vertical discrepancies between adjacent near-shore topographic and bathymetric data sources, which often create artificial barriers and sinks along their intersections. To tackle this issue, the CoNED project has developed an additional step in its integration process that collectively assesses the input data to define how to transition between these disparate datasets. This new step defines two zones: a micro blending zone for near-shore transitions and a macro blending zone for the transition between high-resolution (3 m or less) to moderate-resolution (between 3 m and 10 m) bathymetric datasets. These zones and input data sources are reduced to a multidimensional array of zeros and ones. This array is compiled into a 16-bit integer representing a vertical assessment for each pixel. This assessed value provides the means for dynamic pixel-level blending between disparate datasets by leveraging the 16-bit binary notation. Sample site RMSE assessments demonstrate improved accuracy, with values decreasing from 0.203–0.241 using the previous method to 0.126–0.147 using the new method. This paper introduces CoNED’s unique approach of using binary code to improve the integration of coastal topobathymetric data.

Massively Parallel Computational Chemistry with the Super Instruction Architecture and ACES4.

The task of developing high-performing parallel software must be made easier and more cost-effective in order to fully exploit existing and emerging large-scale computer systems for the advancement of science. The Super Instruction Architecture (SIA) is a parallel programming platform geared toward applications that need to manage large amounts of data stored in potentially sparse multidimensional arrays during calculations. The SIA platform was originally designed for the quantum chemistry software package ACESIII. More recently, the SIA was reimplemented to overcome the limitations in the original ACESIII program. It has now been successfully employed in the new ACES4 quantum chemistry software package. This paper describes the SIA and ACES4 and illustrates their capabilities with some difficult quantum chemistry open-shell coupled-cluster benchmark calculations.

A Machine Learning Model for Training Your AI

Artificial Intelligence is playing an increasing role in solving some of the world’s biggest problems. Machine Learning Models, within the context of reinforcement learning, define and structure a problem in a format that can be used to learn about an environment in order to find an optimal solution. This includes the states, actions, rewards, and other elements in a learning environment. This also includes the logic and policies that guide learning agents to an optimal or nearly optimal solution to the problem. This paper outlines a process for developing machine learning models. The process is extensible and can be applied to solve various problems. This includes a process for implementing data models using multi-dimensional arrays for efficient data processing. We include an evaluation of learning policies, assessing their performance relative to manual and automated approaches.

Refactoring the elastic–viscous–plastic solver from the sea ice model CICE v6.5.1 for improved performance

Abstract. This study focuses on the performance of the elastic–viscous–plastic (EVP) dynamical solver within the sea ice model, CICE v6.5.1. The study has been conducted in two steps. First, the standard EVP solver was extracted from CICE for experiments with refactored versions, which are used for performance testing. Second, one refactored version was integrated and tested in the full CICE model to demonstrate that the new algorithms do not significantly impact the physical results. The study reveals two dominant bottlenecks, namely (1) the number of Message Parsing Interface (MPI) and Open Multi-Processing (OpenMP) synchronization points required for halo exchanges during each time step combined with the irregular domain of active sea ice points and (2) the lack of single-instruction, multiple-data (SIMD) code generation. The standard EVP solver has been refactored based on two generic patterns. The first pattern exposes how general finite differences on masked multi-dimensional arrays can be expressed in order to produce significantly better code generation by changing the memory access pattern from random access to direct access. The second pattern takes an alternative approach to handle static grid properties. The measured single-core performance improvement is more than a factor of 5 compared to the standard implementation. The refactored implementation of strong scales on the Intel® Xeon® Scalable Processors series node until the available bandwidth of the node is used. For the Intel® Xeon® CPU Max series, there is sufficient bandwidth to allow the strong scaling to continue for all the cores on the node, resulting in a single-node improvement factor of 35 over the standard implementation. This study also demonstrates improved performance on GPU processors.

Machine learning revealing overlooked conjunction of working volume and mixing intensity in anammox optimization

Extensive studies on improving anammox performance have taken place for decades with particular focuses on its operational and environmental factors, but such parameter-based optimization is difficult, because of the sheer number of possible combinations and multidimensional arrays of these factors. Utilizing machine-learning algorithm and published anammox data, Bayesian nonparametric general regression (BNGR) was applied to identify the possible governing variable(s) from among 11 operating and environmental parameters: reactor type, mixing type, working volume, hydraulic retention time, temperature, influent pH, nitrite, ammonium, nitrate concentration, nitrogen loading rate, and organic concentration. The results showed that working volume is a key but oft-overlooked governing parameter. By integrating the BNGR findings with computational fluid dynamics simulation, which assessed mixing properties, it became feasible to conclude that working volume and mixing intensity co-regulated flow fields in reactors and had a significant influence on anammox performance. Furthermore, this study experimentally validated how mixing intensity affected performance, and specific input power (x), a parameter that conjugates both working volume and mixing intensity, was used to establish the relationship with ammonium removal rate (NH4+-N RR, y) y = 49.90x+1.97 (R2 = 0.94). With specific input power increased from 3.4 × 10−4 to 2.6 × 10−2 kW m−3, the ammonium removal rate exhibited a rise from 1.8 to 3.2 mg L−1h−1. Following, a relationship among input power-working volume-nitrogen removal rate was also established with a view to determining the design variables for anammox reactor. Consequently, the study highlighted the necessity to consider the working volume-mixing intensity correlation when optimizing the anammox process.

Building reliable AI for quantifying uncertainty in particulate matter predictions with deep learning

Predicting the concentrations of particulate matter (PM) has recently become crucial because of its significant impact on air pollution. As the adoption of artificial intelligence (AI) in this domain increases, ensuring the reliability of AI-driven predictions has become paramount. Although many previous studies have harnessed the power of AI for PM prediction, the inherent uncertainty in the results, which is influenced by spatial data imbalances and varying meteorological factors, poses a challenge. Addressing this uncertainty is central to building reliable AI systems. Therefore, we employed the Monte Carlo dropout (MCDO) approach, which is a technique that integrates dropout layers in neural networks during both the training and inference phases, to estimate the prediction uncertainty. Our objective was to address the challenge of prediction uncertainty in PM concentrations, with the ultimate aim of enhancing the reliability and trustworthiness of AI-driven predictions in air quality forecasting. By applying the MCDO approach to grids formed from multidimensional arrays of air quality and weather data, we obtained a 95% confidence interval for the prediction results, thereby demonstrating the trustworthiness of our model. Our evaluation revealed an R-square greater than 0.97 for PM10 and PM2.5, showcasing the robust and reliable predictive performance of the model. This study highlights not only the accuracy of this approach but also the critical role of quantifying uncertainty in building reliable AI systems. Our findings mark a significant step towards a holistic approach to predictive modeling in which reliability and uncertainty quantification are at the forefront.

Abstractions for C++ code optimizations in parallel high-performance applications

Many computational problems consider memory throughput a performance bottleneck, especially in the domain of parallel computing. Software needs to be attuned to hardware features like cache architectures or concurrent memory banks to reach a decent level of performance efficiency. This can be achieved by selecting the right memory layouts for data structures or changing the order of data structure traversal. In this work, we present an abstraction for traversing a set of regular data structures (e.g., multidimensional arrays) that allows the design of traversal-agnostic algorithms. Such algorithms can easily optimize for memory performance and employ semi-automated parallelization or autotuning without altering their internal code. We also add an abstraction for autotuning that allows defining tuning parameters in one place and removes boilerplate code. The proposed solution was implemented as an extension of the Noarr library that simplifies a layout-agnostic design of regular data structures. It is implemented entirely using C++ template meta-programming without any nonstandard dependencies, so it is fully compatible with existing compilers, including CUDA NVCC or Intel DPC++. We evaluate the performance and expressiveness of our approach on the Polybench-C benchmarks.

Chemisorption of hydrogen on metal oxides with palladium (II) oxide additives

Chemisorption can lead to a change in electrical conductivity, which allows using semiconductor materials to create gas sensors. On the other hand, semiconductor sensors can be used as devices for studying the chemisorption of gases. In this work, we set two tasks. The first of them was to create a sensor for the selective determination of hydrogen in air, including in a mixture with other gases. The second task was to determine the mechanism of hydrogen chemisorption on the surface of metal oxides with palladium (II) oxide additives. We obtained and characterised nanodispersed materials based on SnO2 and WO3 with additions of 3% PdO by weight. We compared the electrical characteristics of these materials in stationary temperature conditions as well as with temperature modulation in the presence of hydrogen, methane, and their mixtures. The specific features of hydrogen chemisorption were determined on the surface of metal oxide semiconductors based on SnO2 and WO3 with PdO additives in the temperature modulation mode of the sensor. It was shown that the extrema in the dependence of the sensor’s electrical conductivity on temperature can be explained by the contribution of proton transfer to the total electrical conductivity. The presence of extrema in the electrical conductivity of sensors, which were observed in thermal modulation mode in the presence of hydrogen, allows conducting qualitative and quantitative analysis of not only single-component gas systems, but also of mixtures of hydrogen with other analyte gases. In particular, this work demonstrated that it was possible to determine the composition of a hydrogen-methane mixture in air using a single metal oxide sensor. The advantage of this approach is that multidimensional data arrays can be easily processed.

Pattern Recognition of Alkaloids by Inhibiting the Catalytic Activity of Dopzymes for Dopamine.

Exploiting the specific recognition probe for all of the biomolecules is difficult in "lock-and-key" biosensors. The cross-reaction or the semispecific probe in pattern recognition mode is an alternative strategy through extracting a multidimensional signal array from recognition elements. Here, we design a pattern recognition sensor array based on the alkaloid-inhibited catalytic activity of dopzymes for the discrimination and determination of six alkaloids. In this sensor array, three different G-rich sequences, i.e., G-triplex (G3), G-quadruplex (GQ1), and the G-quadruplex dimer (2GQ1) possessing various peroxidase activities, conjugated with a dopamine aptamer and the dopzymes (G3-d-apt, GQ1-d-apt, and 2GQ1-d-apt) were obtained with an enhanced catalytic performance for the substrate. Through the interactions between six target alkaloids and G3, GQ1, and 2GQ1 regions, the pattern signal (6 alkaloids × 3 dopzymes × 5 replicates) was obtained from the diverse inhibited effect for the dopzyme activity. In virtue of the statistical method principal component analysis (PCA), the data array was projected into a new dimensional space to acquire the three-dimensional (3D) canonical scores and grouped into their respective clusters. The sensor array exhibited an outstanding discrimination and classification capability for six alkaloids with different concentrations with 100% accuracy. In addition, the nonspecific recognition elements of the sensor array showed high selectivity even though other alkaloids with similar structures to targets existed in the samples. Importantly, the levels of the six targets can be analyzed by the most influential discrimination factor, which represented the vector with the highest variance, evidencing that the sensor array has potential in drug screening and clinical treatment.

Censored Least Squares for Imputing Missing Values in PARAFAC Tensor Factorization.

Tensor factorization is a dimensionality reduction method applied to multidimensional arrays. These methods are useful for identifying patterns within a variety of biomedical datasets due to their ability to preserve the organizational structure of experiments and therefore aid in generating meaningful insights. However, missing data in the datasets being analyzed can impose challenges. Tensor factorization can be performed with some level of missing data and reconstruct a complete tensor. However, while tensor methods may impute these missing values, the choice of fitting algorithm may influence the fidelity of these imputations. Previous approaches, based on alternating least squares with prefilled values or direct optimization, suffer from introduced bias or slow computational performance. In this study, we propose that censored least squares can better handle missing values with data structured in tensor form. We ran censored least squares on four different biological datasets and compared its performance against alternating least squares with prefilled values and direct optimization. We used the error of imputation and the ability to infer masked values to benchmark their missing data performance. Censored least squares appeared best suited for the analysis of high-dimensional biological data by accuracy and convergence metrics across several studies.

The tensor as an informational resource.

A tensor is a multidimensional array of numbers that can be used to store data, encode a computational relation, and represent quantum entanglement. In this sense, a tensor can be viewed as valuable resource whose transformation can lead to an understanding of structure in data, computational complexity, and quantum information. In order to facilitate the understanding of this resource, we propose a family of information-theoretically constructed preorders on tensors, which can be used to compare tensors with each other and to assess the existence of transformations between them. The construction places copies of a given tensor at the edges of a hypergraph and allows transformations at the vertices. A preorder is then induced by the transformations possible in a given growing sequence of hypergraphs. The new family of preorders generalizes the asymptotic restriction preorder which Strassen defined in order to study the computational complexity of matrix multiplication. We derive general properties of the preorders and their associated asymptotic notions of tensor rank and view recent results on tensor rank nonadditivity, tensor networks, and algebraic complexity in this unifying frame. We hope that this work will provide a useful vantage point for exploring tensors in applied mathematics, physics, and computer science but also from a purely mathematical point of view.

A Prediction‐Traversal Approach for Compressing Scientific Data on Unstructured Meshes with Bounded Error

AbstractWe explore an error‐bounded lossy compression approach for reducing scientific data associated with 2D/3D unstructured meshes. While existing lossy compressors offer a high compression ratio with bounded error for regular grid data, methodologies tailored for unstructured mesh data are lacking; for example, one can compress nodal data as 1D arrays, neglecting the spatial coherency of the mesh nodes. Inspired by the SZ compressor, which predicts and quantizes values in a multidimensional array, we dynamically reorganize nodal data into sequences. Each sequence starts with a seed cell; based on a predefined traversal order, the next cell is added to the sequence if the current cell can predict and quantize the nodal data in the next cell with the given error bound. As a result, one can efficiently compress the quantized nodal data in each sequence until all mesh nodes are traversed. This paper also introduces a suite of novel error metrics, namely continuous mean squared error (CMSE) and continuous peak signal‐to‐noise ratio (CPSNR), to assess compression results for unstructured mesh data. The continuous error metrics are defined by integrating the error function on all cells, providing objective statistics across nonuniformly distributed nodes/cells in the mesh. We evaluate our methods with several scientific simulations ranging from ocean‐climate models and computational fluid dynamics simulations with both traditional and continuous error metrics. We demonstrated superior compression ratios and quality than existing lossy compressors.

Advancing Artificial Intelligence (AI) and Machine Learning (ML) Based Soft Sensors for In-Cylinder Predictions with a Real-Time Simulator and a Crank Angle Resolved Engine Model

In a previous research effort by this group, pseudo engine dynamometer data in multi-dimensional arrays were combined with dynamic equations to form a crank angle resolved engine model compatible with a real-time simulator. The combination of the real-time simulator and external targets enabled the development of a software-in-the-loop (SIL) environment that enabled near-real-time development of AI/ML and real-time deployment of the resulting AI/ML. Military applications, in particular, are unlikely to possess large quantities of non-sparse operational data that span the full operational range of the system, stove-piping the ability to develop and deploy AI/ML which is sufficient for near-real-time or real-time control. AI/ML has been shown to be well suited for predicting highly non-linear mathematical phenomena and thus military systems could potentially benefit from the development and deployment of AI/ML acting as soft sensors. Given the non-sparse nature of the data, it becomes exceedingly important that AI/ML be developed and deployed in near-real-time or real-time in parallel to a real-time system to overcome the inadequacy of applicable data. This research effort used parallel processing to reduce the training duration of the shallow artificial neural networks (SANN) and forest algorithms forming ensemble models. This research is novel in that the SIL environment enables pre-developed AI/ML to be adapted in near-real-time or develop AI/ML in response to changes within the operation of the applied system, different load torques, engine speeds, and atmospheric conditions to name a few. Over time it is expected that the continued adaptation of the algorithms will lead to the development of AI/ML that is suitable for real-time control and energy management.

CP factor model for dynamic tensors

Abstract Observations in various applications are frequently represented as a time series of multidimensional arrays, called tensor time series, preserving the inherent multidimensional structure. In this paper, we present a factor model approach, in a form similar to tensor CANDECOMP/PARAFAC (CP) decomposition, to the analysis of high-dimensional dynamic tensor time series. As the loading vectors are uniquely defined but not necessarily orthogonal, it is significantly different from the existing tensor factor models based on Tucker-type tensor decomposition. The model structure allows for a set of uncorrelated one-dimensional latent dynamic factor processes, making it much more convenient to study the underlying dynamics of the time series. A new high-order projection estimator is proposed for such a factor model, utilizing the special structure and the idea of the higher order orthogonal iteration procedures commonly used in Tucker-type tensor factor model and general tensor CP decomposition procedures. Theoretical investigation provides statistical error bounds for the proposed methods, which shows the significant advantage of utilizing the special model structure. Simulation study is conducted to further demonstrate the finite sample properties of the estimators. Real data application is used to illustrate the model and its interpretations.