Neural Tensor Network Research Articles

Variational tensor neural networks for deep learning

Deep neural networks (NNs) encounter scalability limitations when confronted with a vast array of neurons, thereby constraining their achievable network depth. To address this challenge, we propose an integration of tensor networks (TN) into NN frameworks, combined with a variational DMRG-inspired training technique. This in turn, results in a scalable tensor neural network (TNN) architecture capable of efficient training over a large parameter space. Our variational algorithm utilizes a local gradient-descent technique, enabling manual or automatic computation of tensor gradients, facilitating design of hybrid TNN models with combined dense and tensor layers. Our training algorithm further provides insight on the entanglement structure of the tensorized trainable weights and correlation among the model parameters. We validate the accuracy and efficiency of our method by designing TNN models and providing benchmark results for linear and non-linear regressions, data classification and image recognition on MNIST handwritten digits.

Research on Grid Data Analysis and Intelligent Recommendation System by Introducing Neural Tensor Network Model

Research on Grid Data Analysis and Intelligent Recommendation System by Introducing Neural Tensor Network Model

3-D Quantum-Inspired Self-Supervised Tensor Network for Volumetric Segmentation of Medical Images.

This article introduces a novel shallow 3-D self-supervised tensor neural network in quantum formalism for volumetric segmentation of medical images with merits of obviating training and supervision. The proposed network is referred to as the 3-D quantum-inspired self-supervised tensor neural network (3-D-QNet). The underlying architecture of 3-D-QNet is composed of a trinity of volumetric layers, viz., input, intermediate, and output layers interconnected using an S -connected third-order neighborhood-based topology for voxelwise processing of 3-D medical image data, suitable for semantic segmentation. Each of the volumetric layers contains quantum neurons designated by qubits or quantum bits. The incorporation of tensor decomposition in quantum formalism leads to faster convergence of network operations to preclude the inherent slow convergence problems faced by the classical supervised and self-supervised networks. The segmented volumes are obtained once the network converges. The suggested 3-D-QNet is tailored and tested on the BRATS 2019 Brain MR image dataset and the Liver Tumor Segmentation Challenge (LiTS17) dataset extensively in our experiments. The 3-D-QNet has achieved promising dice similarity (DS) as compared with the time-intensive supervised convolutional neural network (CNN)-based models, such as 3-D-UNet, voxelwise residual network (VoxResNet), Dense-Res-Inception Net (DRINet), and 3-D-ESPNet, thereby showing a potential advantage of our self-supervised shallow network on facilitating semantic segmentation.

Optimal depth and a novel approach to variational unitary quantum process tomography

In this work, we present two new methods for variational quantum circuit (VQC) process tomography (PT) onto n qubits systems: unitary PT based on VQCs (PT_VQC) and unitary evolution-based variational quantum singular value decomposition (U-VQSVD). Compared to the state of the art, PT_VQC halves in each run the required amount of qubits for unitary PT and decreases the required state initializations from 4 n to just 2 n , all while ensuring high-fidelity reconstruction of the targeted unitary channel U. It is worth noting that, for a fixed reconstruction accuracy, PT_VQC achieves faster convergence per iteration step compared to quantum deep neural network and tensor network schemes. The novel U-VQSVD algorithm utilizes variational singular value decomposition to extract eigenvectors (up to a global phase) and their associated eigenvalues from an unknown unitary representing a universal channel. We assess the performance of U-VQSVD by executing an attack on a non-unitary channel quantum physical unclonable function. By using U-VQSVD we outperform an uninformed impersonation attack (using randomly generated input states) by a factor of 2 to 5, depending on the qubit dimension. For the two presented methods, we propose a new approach to calculate the complexity of the displayed VQC, based on what we denote as optimal depth.

Zero-shot reconstruction of ocean sound speed field tensors: A deep plug-and-play approach.

Reconstructing a three-dimensional ocean sound speed field (SSF) from limited and noisy measurements presents an ill-posed and challenging inverse problem. Existing methods used a number of pre-specified priors (e.g., low-rank tensor and tensor neural network structures) to address this issue. However, the SSFs are often too complex to be accurately described by these pre-defined priors. While utilizing neural network-based priors trained on historical SSF data may be a viable workaround, acquiring SSF data remains a nontrivial task. This work starts with a key observation: Although natural images and SSFs admit fairly different characteristics, their denoising processes appear to share similar traits-as both remove random components from more structured signals. This observation allows us to incorporate deep denoisers trained using extensive natural images to realize zero-shot SSF reconstruction, without any extra training or network modifications. To implement this idea, an alternating direction method of multipliers (ADMM) algorithm using such a deep denoiser is proposed, which is reminiscent of the plug-and-play scheme from medical imaging. Our plug-and-play framework is tailored for SSF recovery such that the learned denoiser can be simultaneously used with other handcrafted SSF priors. Extensive numerical studies show that the new framework largely outperforms state-of-the-art baselines, especially under widely recognized challenging scenarios, e.g., when the SSF samples are taken as tensor fibers. The code is available at https://github.com/OceanSTARLab/DeepPnP.

Sample complexity bounds for the local convergence of least squares approximation

We consider the problem of approximating a function in a general nonlinear subset of [Formula: see text], when only a weighted Monte Carlo estimate of the [Formula: see text]-norm is accessible. The concept of sample complexity, i.e. the number of sample points necessary to achieve a prescribed error with high probability, is of particular interest in this setting. Reasonable worst-case bounds for this quantity exist only for particular model classes, like linear spaces or sets of sparse vectors. However, the existing bounds are very pessimistic for more general sets, like tensor networks or neural networks. Restricting the model class to a neighborhood of the best approximation allows us to derive improved worst-case bounds for the sample complexity. When the considered neighborhood is a manifold with positive local reach, its sample complexity can be estimated through the sample complexities of the tangent and normal spaces and the manifold’s curvature.

Tensor decomposition method and neural network model for identifying side effects of polypragmasy

Tensor decomposition method and neural network model for identifying side effects of polypragmasy

Computing multi-eigenpairs of high-dimensional eigenvalue problems using tensor neural networks

In this paper, we propose a type of tensor-neural-network-based machine learning method to compute multi-eigenpairs of high dimensional eigenvalue problems without Monte-Carlo procedure. Solving multi-eigenvalues and their corresponding eigenfunctions is one of the basic tasks in mathematical and computational physics. With the help of tensor neural network, the high dimensional integrations included in the loss functions of the machine learning process can be computed with high accuracy. The high accuracy of high dimensional integrations can improve the accuracy of the machine learning method for computing multi-eigenpairs of high dimensional eigenvalue problems. Here, we introduce the tensor neural network and design the machine learning method for computing multi-eigenpairs of the high dimensional eigenvalue problems. The proposed numerical method is validated with plenty of numerical examples.

Ocean sound speed field reconstruction beyond tensor neural network

Obtaining accurate ocean sound speed fields (SSFs) across a three-dimensional (3D) geographic region is vital for various underwater acoustic tasks. However, the scarcity of measurements due to the high cost of underwater sensors, combined with the high dimensionality of complex 3D SSF, makes the reconstruction problem highly ill-conditioned, thus demanding advanced models and methods. Our recent work has analyzed the reconstruction error and identified one promising way: finding a representation model that is both concise and expressive. Following this path, we proposed a tensor neural network (TNN) model, which leverages the conciseness of tensor models and the expressive power of deep learning. However, existing TNN-based approaches have two limitations: (1) they are unable to capture long-range correlations within the SSF and (2) they can only handle discrete-indexed tensor data. To overcome these limitations and fully unleash the power of deep tensor learning, we seamlessly introduce attention schemes into the existing TNN framework without compromising its interpretability. Additionally, we employ Gaussian process models to evolve the original parameterized tensor model into a new functional tensor model, enabling reconstruction with a continuous grid. Numerical results obtained from real-life datasets demonstrate the superior performance of our approach compared to state-of-the-art methods.

Learned Tensor Neural Network Texture Prior for Photon-Counting CT Reconstruction.

Photon-counting computed tomography (PCCT) reconstructs multiple energy-channel images to describe the same object, where there exists a strong correlation among different channel images. In addition, reconstruction of each channel image suffers photon count starving problem. To make full use of the correlation among different channel images to suppress the data noise and enhance the texture details in reconstructing each channel image, this paper proposes a tensor neural network (TNN) architecture to learn a multi-channel texture prior for PCCT reconstruction. Specifically, we first learn a spatial texture prior in each individual channel image by modeling the relationship between the center pixels and its corresponding neighbor pixels using a neural network. Then, we merge the single channel spatial texture prior into multi-channel neural network to learn the spectral local correlation information among different channel images. Since our proposed TNN is trained on a series of unpaired small spatial-spectral cubes which are extracted from one single reference multi-channel image, the local correlation in the spatial-spectral cubes is considered by TNN. To boost the TNN performance, a low-rank representation is also employed to consider the global correlation among different channel images. Finally, we integrate the learned TNN and the low-rank representation as priors into Bayesian reconstruction framework. To evaluate the performance of the proposed method, four references are considered. One is simulated images from ultra-high-resolution CT. One is spectral images from dual-energy CT. The other two are animal tissue and preclinical mouse images from a custom-made PCCT systems. Our TNN prior Bayesian reconstruction demonstrated better performance than other state-of-the-art competing algorithms, in terms of not only preserving texture feature but also suppressing image noise in each channel image.

Theoretical Advancement of Xi Jinping’s Socialist Thought with Chinese Characteristics in the New Era under the Construction of Knowledge Map

Abstract This paper discusses the theory and practice of constructing knowledge graph under the guidance of Xi Jinping’s socialist thought with Chinese characteristics in the new era. The study first adopts the Transformer technique based on the BERT model for effectively extracting entity relationships. Then, the article proposes an entity alignment method based on Neural Tensor Network, which enhances the semantic and structural information utilization of entity embedding vectors through optimized NTN and joint knowledge representation learning. In addition, the study employs a knowledge graph link prediction method that combines semantics and paths to fill in missing relations, and applies probabilistic soft logic to solve the inference problem of non-deterministic knowledge. Ultimately, the effectiveness of the constructed knowledge graph is demonstrated through empirical analysis of Xi Jinping’s theoretical dataset of socialist thought with Chinese characteristics in the new era. The results show that the method is more effective in dealing with polysemous words and participle errors than traditional methods, improving the data structuring level. For example, the accuracy of entity alignment is significantly better than the baseline algorithm on different datasets, e.g., on the DBP-WE dataset, its Hits@1, Hits@10 and MRR values are higher than those of the baseline by 4.01-29.49, 1.99-31.05, and 0.036-0.289, respectively. The method of this paper is remarkably effective in improving the performance of the entity alignment task, which is helpful in understanding and applying the Xi Jinping’s thought of socialism with Chinese characteristics in the new era provides new theoretical perspectives and practical methods.

Stable tensor neural networks for efficient deep learning.

Learning from complex, multidimensional data has become central to computational mathematics, and among the most successful high-dimensional function approximators are deep neural networks (DNNs). Training DNNs is posed as an optimization problem to learn network weights or parameters that well-approximate a mapping from input to target data. Multiway data or tensors arise naturally in myriad ways in deep learning, in particular as input data and as high-dimensional weights and features extracted by the network, with the latter often being a bottleneck in terms of speed and memory. In this work, we leverage tensor representations and processing to efficiently parameterize DNNs when learning from high-dimensional data. We propose tensor neural networks (t-NNs), a natural extension of traditional fully-connected networks, that can be trained efficiently in a reduced, yet more powerful parameter space. Our t-NNs are built upon matrix-mimetic tensor-tensor products, which retain algebraic properties of matrix multiplication while capturing high-dimensional correlations. Mimeticity enables t-NNs to inherit desirable properties of modern DNN architectures. We exemplify this by extending recent work on stable neural networks, which interpret DNNs as discretizations of differential equations, to our multidimensional framework. We provide empirical evidence of the parametric advantages of t-NNs on dimensionality reduction using autoencoders and classification using fully-connected and stable variants on benchmark imaging datasets MNIST and CIFAR-10.

The cross-sectional stock return predictions via quantum neural network and tensor network

In this paper, we investigate the application of quantum and quantum-inspired machine learning algorithms to stock return predictions. Specifically, we evaluate the performance of quantum neural network, an algorithm suited for noisy intermediate-scale quantum computers, and tensor network, a quantum-inspired machine learning algorithm, against classical models such as linear regression and neural networks. To evaluate their abilities, we construct portfolios based on their predictions and measure investment performances. The empirical study on the Japanese stock market shows the tensor network model achieves superior performance compared to classical benchmark models, including linear and neural network models. Though the quantum neural network model attains the lowered risk-adjusted excess return than the classical neural network models over the whole period, both the quantum neural network and tensor network models have superior performances in the latest market environment, which suggests capability of model’s capturing non-linearity between input features.

Tensor Neural Network and Its Numerical Integration

Tensor Neural Network and Its Numerical Integration

Neural-based inexact graph de-anonymization

Graph de-anonymization is a technique used to reveal connections between entities in anonymized graphs, which is crucial in detecting malicious activities, network analysis, social network analysis, and more. Despite its paramount importance, conventional methods often grapple with inefficiencies and challenges tied to obtaining accurate query graph data. This paper introduces a neural-based inexact graph de-anonymization, which comprises an embedding phase, a comparison phase, and a matching procedure. The embedding phase uses a graph convolutional network to generate embedding vectors for both the query and anonymized graphs. The comparison phase uses a neural tensor network to ascertain node resemblances. The matching procedure employs a refined greedy algorithm to discern optimal node pairings. Additionally, we comprehensively evaluate its performance via well-conducted experiments on various real datasets. The results demonstrate the effectiveness of our proposed approach in enhancing the efficiency and performance of graph de-anonymization through the use of graph embedding vectors.

Striking the right balance: Three-dimensional ocean sound speed field reconstruction using tensor neural networks.

Accurately reconstructing a three-dimensional (3D) ocean sound speed field (SSF) is essential for various ocean acoustic applications, but the sparsity and uncertainty of sound speed samples across a vast ocean region make it a challenging task. To tackle this challenge, a large body of reconstruction methods has been developed, including spline interpolation, matrix/tensor-based completion, and deep neural networks (DNNs)-based reconstruction. However, a principled analysis of their effectiveness in 3D SSF reconstruction is still lacking. This paper performs a thorough analysis of the reconstruction error and highlights the need for a balanced representation model that integrates expressiveness and conciseness. To meet this requirement, a 3D SSF-tailored tensor DNN is proposed, which uses tensor computations and DNN architectures to achieve remarkable 3D SSF reconstruction. The proposed model not only includes the previous tensor-based SSF representation model as a special case but also has a natural ability to reject noise. The numerical results using the South China Sea 3D SSF data demonstrate that the proposed method outperforms state-of-the-art methods. The code is available at https://github.com/OceanSTARLab/Tensor-Neural-Network.

Analysis of the Recursive Locally-Adaptive Filtration of 3D Tensor Images

This work is focused on the computational complexity (CC) reduction of the locally-adaptive processing of 3D tensor images, based on recursive approaches. As a basis, a local averaging operation is used, implemented as a sliding mean 3D filter (SM3DF) with a central symmetric 3D kernel. Symmetry plays a very important role in constructing the working window. The presented theoretical approach could be adopted in various algorithms for locally-adaptive processing, such as additive noise reduction, sharpness enhancement, texture segmentation, etc. The basic characteristics of the recursive SM3DF are analyzed, together with the main features of the adaptive algorithms for filtration of Gaussian noises and unsharp masking where the filter is aimed at. In the paper, the ability of SM3DF implementation through recursive sliding mean 1D filters, sequentially bonded together, is also introduced. The computational complexity of the algorithms is evaluated for the recursive and non-recursive mode. The recursive SM3DF also suits the 3D convolutional neural networks which comprise sliding locally-adaptive 3D filtration in their layers. As a result of the lower CC, a promising opportunity is opened for higher efficiency of the 3D image processing through tensor neural networks.

Implicit sentiment analysis based on affective knowledge and event information

In recent years, implicit sentiment analysis, which aims to detect sentiment of a sentence that don’t contain obvious sentiment words, has become an attractive research topic. This paper focuses on event-centric implicit sentiment analysis, which utilizes the sentiment-aware event to infer the sentiment polarity of the sentence. Existing event-based implicit sentiment analysis methods typically treat entities or noun phrases in the text as events, or model contextual information to indirectly infer events using sophisticated models, but these methods fail to fully capture event information. To address these issues, this paper defines events as <subject, predicate, object>. Based on this event representation, neural tensor network was used to model the interaction between event elements and extract high-level semantic features of events. In addition, a novel affective enhanced graph model was proposed to capture sentiment-related dependencies between context words. Furthermore, this paper considers the case where a sentence contains multiple events, and constructs an event-centric implicit sentiment analysis dataset, where each sentence contains at least one event triplet. Experimental results on the constructed dataset demonstrate the effectiveness of our proposed approach.

Capsule neural tensor networks with multi-aspect information for Few-shot Knowledge Graph Completion

Few-shot Knowledge Graph Completion (FKGC) has recently attracted significant research interest due to its ability to expand few-shot relation coverage in Knowledge Graphs. Prevailing FKGC approaches focus on exploiting the one-hop neighbor information of entities to enhance few-shot relation embedding. However, these methods select one-hop neighbors randomly and neglect the rich multi-aspect information of entities. Although some methods have attempted to leverage Long Short-Term Memory (LSTM) to learn few-shot relation embedding, they are sensitive to the input order. To address these limitations, we propose the Capsule Neural Tensor Networks with Multi-Aspect Information approach (short for InforMix-FKGC). InforMix-FKGC employs a one-hop neighbor selection strategy based on how valuable they are and encodes multi-aspect information of entities, including one-hop neighbors, attributes and literal description. Then, a capsule network is responsible for integrating the support set and deriving few-shot relation embedding. Moreover, a neural tensor network is used to match the query set with the support set. In this way, InforMix-FKGC can learn few-shot relation embedding more precisely so as to enhance the accuracy of FKGC. Extensive experiments on the NELL-One and Wiki-One datasets demonstrate that InforMix-FKGC significantly outperforms ten state-of-the-art methods in terms of Mean Reciprocal Rank and Hits@K.

Pixel-wise mechanical damage detection of waxy maize using spectral–spatial feature extraction and hyperspectral image

Mechanical damage detection on waxy maize is critical to guarantee maize quality and meet the product requirements. A novel spectral–spatial feature extraction enhanced fully connected neural network (SSFE-FCNN) classification method is proposed in this paper to achieve pixel-wise mechanical damage detection of waxy maize using the hyperspectral image (HSI). The proposed SSFE-FCNN utilizes the online tensor dictionary learning algorithm to extract spectral–spatial tensor features from HSI cubes. Moreover, the designed fully connected neural network extracts high-level features from the tensor features and provides classification results. The SSFE-FCNN combines the advantages of sparse tensor representation and neural networks to exploit joint spectral–spatial information in HSI and improve detection performance. Experimental results on the HSI of waxy maize show that the proposed SSFE-FCNN is superior to the support vector machine and the currently popular neural network methods. The overall accuracy of SSFE-FCNN in the pixel-wise classification of waxy maize achieves as high as 98.09%. The false and missing detection rates are 2.08% and 1.96%, respectively. Furthermore, pixel-wise detection can detect slight damage and perform the maize grading operation according to the damage ratio.