Tensor Features Research Articles

Multi-exposure image fusion based on tensor decomposition

In this paper, a multi-exposure image fusion (MEF) method is proposed based on tensor decomposition and saliency model. The main innovation of the proposed method is to explore a tensor domain for MEF and define the fusion rules based on tensor feature of higher order singular value decomposition (HOSVD) and saliency. Specifically, RGB images are converted to YCbCr images to maintain the stability of color information. For luminance channels, luminance patches of luminance images are constructed 3-order sub-tensors, and HOSVD is used to extract features of sub-tensors. Then, the sum of absolute coefficients (SAC) of weight coefficients are defined. Meanwhile, considering the impact of saliency on visual perception, visual saliency maps (VSMs) is used to evaluate luminance patches quality and guide the fusion rules to define the rule of fusion. For chrominance channels, VSMs of the chrominance channels is used to define fused rule. The experimental results show that the fused image with more texture details and saturated color is successfully generated by proposed method.

Kernel flexible and displaceable convex hull based tensor machine for gearbox fault intelligent diagnosis with multi-source signals

Abstract The methods based on traditional pattern recognition and deep learning have been successfully applied in gearbox intelligent diagnosis. However, traditional pattern recognition methods cannot directly classify feature tensors of multi-source signals, and deep learning networks hardly handle the classification of small samples. Therefore, for the gearbox intelligent diagnosis with multi-source signals, a novel tensor classifier called kernel flexible and displaceable convex hull based tensor machine (KFDCH-TM) is proposed. In KFDCH-TM, the kernel flexible and displaceable convex hull of tensor samples in tensor feature space is defined firstly. Then, an optimal separating hyper-plane between two kernel flexible and displaceable convex hulls is constructed. Meanwhile, feature tensors extracted from multi-source signals through wavelet packet transform (WPT) are used to diagnose gearbox fault by KFDCH-TM. The results of two cases demonstrate that KFDCH-TM can effectively identify gearbox fault with multi-source signals and has better robustness.

Multi-Class ASD Classification Based on Functional Connectivity and Functional Correlation Tensor via Multi-Source Domain Adaptation and Multi-View Sparse Representation.

The resting-state functional magnetic resonance imaging (rs-fMRI) reflects functional activity of brain regions by blood-oxygen-level dependent (BOLD) signals. Up to now, many computer-aided diagnosis methods based on rs-fMRI have been developed for Autism Spectrum Disorder (ASD). These methods are mostly the binary classification approaches to determine whether a subject is an ASD patient or not. However, the disease often consists of several sub-categories, which are complex and thus still confusing to many automatic classification methods. Besides, existing methods usually focus on the functional connectivity (FC) features in grey matter regions, which only account for a small portion of the rs-fMRI data. Recently, the possibility to reveal the connectivity information in the white matter regions of rs-fMRI has drawn high attention. To this end, we propose to use the patch-based functional correlation tensor (PBFCT) features extracted from rs-fMRI in white matter, in addition to the traditional FC features from gray matter, to develop a novel multi-class ASD diagnosis method in this work. Our method has two stages. Specifically, in the first stage of multi-source domain adaptation (MSDA), the source subjects belonging to multiple clinical centers (thus called as source domains) are all transformed into the same target feature space. Thus each subject in the target domain can be linearly reconstructed by the transformed subjects. In the second stage of multi-view sparse representation (MVSR), a multi-view classifier for multi-class ASD diagnosis is developed by jointly using both views of the FC and PBFCT features. The experimental results using the ABIDE dataset verify the effectiveness of our method, which is capable of accurately classifying each subject into a respective ASD sub-category.

An Integration Framework on Cloud for Cyber-Physical-Social Systems Big Data

A tremendous challenge in the development of Cyber-Physical-Social Systems (CPSS) is to integrate the growing volume and wide variety of data generated from multiple sources. To address the challenge, this paper presents an integration framework on cloud consisting of five functionally complementary processes, namely data representation, dimensionality reduction, relation establishment, data rank and data retrieval. The unstructured, semi-structured and structured data in the cyber, physical and social space are first represented as low-order data tensors, and then transformed to a three-order feature tensor for relation establishment. A similarity-based multi-linear data rank approach is proposed to measure the importance of the CPSS big data, and an incremental method is explored to rapidly and accurately update the rank vector. This paper, through a smart home case study, illustrates a practical application of the proposed integration framework, and evaluates the performance of the multi-linear data rank approach as well as the incremental rank update method. The results reveal that the proposed framework is feasible and competitive to integrate the CPSS big data on cloud.

Fast depth map intra-mode selection for 3D-HEVC intra-coding

As the last effort of ISO/IEC MPEG and ITU-T Video Coding Experts Group for 3D video coding, high-efficiency video coding (HEVC)-based 3D video coding (3D-HEVC) is the new international emerging standard to compress multi-view texture videos plus depth maps data. In the test model of 3D-HEVC, the conventional HEVC intra-prediction modes and new additional intra-coding tools called depth modeling modes (DMMs) are exploited for depth map intra-coding. These intra-modes provide important coding improvement. However, the improvement of the coding efficiency is obtained at the expense of a computational complexity increase, which obstructs 3D-HEVC from practical real-world applications. Previously, we propose a depth map intra-model decision using tensor features as homogeneity determination. In this paper, we propose an amelioration of the previously proposed depth map intra-model decision using an efficient homogeneity determination. In most cases, the angular intra-coding and DMM could be skipped if some specific conditions are satisfied. The experimental results show that the proposed intra-model decision can reduce the encoding time significantly, with negligible loss of rate distortion performance.

Transferred CNN Based on Tensor for Hyperspectral Anomaly Detection

Nowadays, deep learning (DM) and tensor theory have become research hotspot in hyperspectral images (HSIs) processing. In this letter, transferred convolutional neural network (CNN) based on tensor (TCNNT) is proposed for hyperspectral anomaly detection (AD). TCNNT is an unsupervised DM framework and utilizes tensor structure to extract the spatial and spectral information of HSI effectively. First, the test tensor block centered at the test point is regarded as a tensor convolution kernel to convolve with the dictionary tensor blocks to extract deep feature, and the difference between the test tensor and the feature tensor is obtained. Then, the local neighboring tensor blocks are also regarded as tensor convolution kernels to convolve with the dictionary tensor blocks to extract deep feature, and the difference between the local neighboring tensor and the feature tensor is obtained. Finally, an adaptive model based on the above two differences is proposed for the detection output. Experiments conducted on one synthetic HSI and two real HSIs present superior performance of the proposed TCNNT.

Screen Content Video Quality Assessment: Subjective and Objective Study.

In this paper, we make the first attempt to study the subjective and objective quality assessment for the screen content videos (SCVs). For that, we construct the first large-scale video quality assessment (VQA) database specifically for the SCVs, called the screen content video database (SCVD). This SCVD provides 16 reference SCVs, 800 distorted SCVs, and their corresponding subjective scores, and it is made publicly available for research usage. The distorted SCVs are generated from each reference SCV with 10 distortion types and 5 degradation levels for each distortion type. Each distorted SCV is rated by at least 32 subjects in the subjective test. Furthermore, we propose the first full-reference VQA model for the SCVs, called the spatiotemporal Gabor feature tensor-based model (SGFTM), to objectively evaluate the perceptual quality of the distorted SCVs. This is motivated by the observation that 3D-Gabor filter can well stimulate the visual functions of the human visual system (HVS) on perceiving videos, being more sensitive to the edge and motion information that are often-encountered in the SCVs. Specifically, the proposed SGFTM exploits 3D-Gabor filter to individually extract the spatiotemporal Gabor feature tensors from the reference and distorted SCVs, followed by measuring their similarities and later combining them together through the developed spatiotemporal feature tensor pooling strategy to obtain the final SGFTM score. Experimental results on SCVD have shown that the proposed SGFTM yields a high consistency on the subjective perception of SCV quality and consistently outperforms multiple classical and state-of-the-art image/video quality assessment models.

Imbalanced Data Classification for Multi-Source Heterogenous Sensor Networks

Most of the traditional classification algorithms are based on the uniform distribution of samples, and the effect is not ideal when dealing with such data, which mainly shows that the classification results incline to the majority class. Therefore, we propose the imbalanced multi-source heterogeneous data classification algorithms in this paper, which are mainly based on the expansion and extension of Support Vector Machines. Considering that there are complex connections within multi-source data, express them as a unified, concise and efficient mathematical model can completely retain data information and improve data processing efficiency. We perform tensor representation and feature extraction on the heterogeneous data, and two different classification algorithms are proposed in this paper. In the first method, we represent multi-source heterogeneous data into a unified tensor form directly and obtain a high-quality core data through dimensionality reduction algorithm, then realize data classification by Support Tensor Machine. In the other method, we extract data from different data sources and classify them with Ensemble Deep Support Vector Machine (DSVM), which combined three DSVM with different kernel functions. The algorithms are compared on CUAVE data set, which contains two different modalities of sound and picture.

Tensor Completion Methods for Collaborative Intelligence

In the race to bring Artificial Intelligence (AI) to the edge, collaborative intelligence has emerged as a promising way to lighten the computation load on edge devices that run applications based on Deep Neural Networks (DNNs). Typically, a deep model is split at a given layer into edge and cloud sub-models. The deep feature tensor produced by the edge sub-model is transmitted to the cloud, where the remaining computationally intensive workload is performed by the cloud sub-model. The communication channel between the edge and cloud is imperfect, which will result in missing data in the deep feature tensor received at the cloud side, an issue that has mostly been ignored by existing literature on the topic. In this paper we study four methods for recovering missing data in the deep feature tensor. Three of the studied methods are existing, generic tensor completion methods, and are adapted here to recover deep feature tensor data, while the fourth method is newly developed specifically for deep feature tensor completion. Simulation studies show that the new method is 3-18 times faster than the other three methods, which is an important consideration in collaborative intelligence. For VGG16's sparse tensors, all methods produce statistically equivalent classification results across all loss levels tested. For ResNet34's non-sparse tensors, the new method offers statistically better classification accuracy (by 0.25%-6.30%) compared to other methods for matched execution speeds, and second-best accuracy among the four methods when they are allowed to run until convergence.

Fine-grained image classification with factorized deep user click feature

The advantages of user click data greatly inspire its wide application in fine-grained image classification tasks. In previous click data based image classification approaches, each image is represented as a click frequency vector on a pre-defined query/word dictionary. However, this approach not only introduces high-dimensional issues, but also ignores the part of speech (POS) of a specific word as well as the word correlations. To address these issues, we devise the factorized deep click features to represent images. We first represent images as the factorized TF-IDF click feature vectors to discover word correlation, wherein several word dictionaries of different POS are constructed. Afterwards, we learn an end-to-end deep neural network on click feature tensors built on these factorized TF-IDF vectors. We evaluate our approach on the public Clickture-Dog dataset. It shows that: 1) the deep click feature learned on click tensor performs much better than traditional click frequency vectors; and 2) compared with many state-of-the-art textual representations, the proposed deep click feature is more discriminative and with higher classification accuracies.

Multidimensional Sparse Representation for Multishot Person Reidentification

Person reidentification is known as recognizing a subject in diverse scenes obtained from nonoverlapping cameras. To the best of our knowledge, the existing sparse representation approaches for person reidentification need an additional stage for combining a different level of features. In this letter, we propose a new sparse representation approach based on tensor along with the dictionary learning that is able to tackle both the sparse representation of features and the combination of different level of features, simultaneously. First, we construct the feature tensors using images of people. Subsequently, we learn a single cross-view invariant dictionary for representing images from different viewpoints in each tensor mode. The tensor representations of images alleviate the computational complexity of the conventional feature combination approaches, and enhance the reidentification of high dimensional data. Experimental results on iLIDS-VID dataset show the superiority of our method compared to some recent methods.

Subspace Learning Network: An Efficient ConvNet for PolSAR Image Classification

Land cover classification is an important part of the polarimetric synthetic aperture radar (PolSAR) image interpretation. The convolutional neural network (CNN) has been utilized to improve the classification accuracy recently. However, how to efficiently train the classification model with limited training samples while keeping the generalization performance is still a challenge. In this letter, we devise a subspace learning network (SSLNet) for PolSAR image classification, which can be trained more efficiently. First, a third-order polarimetric feature tensor is constructed using five-target decompositions to make full use of the prior knowledge. The tensor is then fed into a two-layer CNN in which the principal component analysis (PCA) is employed to learn the convolutional filters. Finally, the output features of the network are obtained by binary hashing and block-wise histograms, followed by the nearest neighbor (NN) classifier to complete the classification. Due to the simple learning strategy, the proposed SSLNet can be easily designed and efficiently trained. Experimental results on benchmark PolSAR data reveal that the SSLNet can achieve higher classification accuracy with limited training samples than the conventional CNN method.

Enlarging smaller images before inputting into convolutional neural network: zero-padding vs. interpolation

The input to a machine learning model is a one-dimensional feature vector. However, in recent learning models, such as convolutional and recurrent neural networks, two- and three-dimensional feature tensors can also be inputted to the model. During training, the machine adjusts its internal parameters to project each feature tensor close to its target. After training, the machine can be used to predict the target for previously unseen feature tensors. What this study focuses on is the requirement that feature tensors must be of the same size. In other words, the same number of features must be present for each sample. This creates a barrier in processing images and texts, as they usually have different sizes, and thus different numbers of features. In classifying an image using a convolutional neural network (CNN), the input is a three-dimensional tensor, where the value of each pixel in each channel is one feature. The three-dimensional feature tensor must be the same size for all images. However, images are not usually of the same size and so are not their corresponding feature tensors. Resizing images to the same size without deforming patterns contained therein is a major challenge. This study proposes zero-padding for resizing images to the same size and compares it with the conventional approach of scaling images up (zooming in) using interpolation. Our study showed that zero-padding had no effect on the classification accuracy but considerably reduced the training time. The reason is that neighboring zero input units (pixels) will not activate their corresponding convolutional unit in the next layer. Therefore, the synaptic weights on outgoing links from input units do not need to be updated if they contain a zero value. Theoretical justification along with experimental endorsements are provided in this paper.

A sparse tensor-based classification method of hyperspectral image

Previous studies have demonstrated that spatial information can provide significant improvement for the accuracy hyperspectral image (HSI) classification. However, it remains a challenging task to extract three-dimensional (3-D) features from HSI directly. In this paper, a sparse tensor-based classification (STC) method for HSI is proposed. Different from the traditional vector-based or matrix-based methods, the STC utilizes tensor technique to extract the joint spatial-spectral tensor features. We exploit the principal component analysis (PCA) and the 3-D intrinsic spatial-spectral tensors of HSI to alleviate within-class spatial-spectral variation, and to improve the classification performance simultaneously. First, the HSI is segmented into a number of overlapping 3-D tensor patches, which are modelled as summation of intrinsic spatial-spectral tensors and corresponding variation tensors in the next step. Second, the intrinsic spatial-spectral tensor is decomposed into three matrices and a core tensor by the Tucker decomposition (TKD). Sparsity constraint is enforced on the core tensor to extract joint sparse spatial-spectral features. We utilize tensor-based dictionary learning algorithm to train three dictionaries, in order to extract more discriminative tensor features for classification. Finally, we use the support vector machine (SVM) to perform the pixel-wise classification. Experimental results on real HSI datasets demonstrate the proposed method can achieve accurate and robust classification results, and can provide competitive results to state-of-the-art methods.

FD-MIMO via Pilot-Data Superposition: Tensor-Based DOA Estimation and System Performance

Increased pilot overhead is one of the major issues for 5G full-dimension MIMO (FD-MIMO) systems. In this paper, we introduce the FD-MIMO system using pilot-data superposition to reduce the uplink pilot overhead and investigate the impact of superimposed pilots on the overall network performance. The pilot-data superposition can relatively increase the resources allocated to the uplink data transmission. However, in time division duplex (TDD) systems, it also negatively impacts the downlink throughput since the relatively reduced pilot power can affect the estimation of the channel state information (CSI) which is used for the downlink precoding. To improve the CSI estimation, the intrinsic tensor feature of the FD-MIMO channel is exploited in our method. The introduced CSI estimation algorithm is designed through the expectation-maximization (EM) framework via tensor as the processing data structure. Furthermore, Cramer-Rao lower bound (CRLB) is utilized as a metric in our evaluation. The overall system achievable rate which is a weighted sum of the uplink and downlink throughput is investigated to reveal the fundamental trade-off between the uplink and downlink transmission in the TDD-based FD-MIMO system. In our simulation, the results demonstrate the superior performance of the introduced strategy as opposed to the conventional orthogonal pilot-data approach. Meanwhile, the pilot power allocation, as well as different transmitting and receiving strategies are investigated to offer various trade-off points between the uplink and downlink transmission.

Spatial-spectral feature based approach towards convolutional sparse coding of hyperspectral images

The sparse coding approaches, which learn over-complete bases for efficiently representing a given set of data, are being extensively studied for various image analysis tasks. However, since sparse analysis of individual bands does not consider inter-spectral characteristics, adaptation of these approaches to hyperspectral domain has been quite limited. In this regard, we investigate convolutional frameworks for simultaneously learning sparse codes and dictionaries so as to efficiently represent the spectral–spatial characteristics of hypercubes. This study also reviews prominent convolutional sparse coding algorithms relevant to hyperspectral datasets. Unlike conventional approaches, in order to address the issues due to huge spectral dimension and high inter-band correlation, the proposed approaches implement sparse coding in a convolutional encoder–decoder framework. Two architectures, one based on shrinkage thresholding algorithm and other based on structured sparse regularization, are proposed. These architectures adopt 3D convolutions in such a way as to model both spectral and spatial features. Although Architecture-1 gives better results than other prominent approaches, its accuracy deteriorates with increase in spectral dimension of input. This can be attributed to the bottleneck due to shrinkage thresholding operation. To address the issue, Architecture-2 adopts structured sparse regularizations over filter weights and feature tensors. Results indicate that the proposed approaches achieve significant improvement in reconstruction accuracy when compared to the existing approaches. The frameworks are also evaluated with respect to super-resolution and dimensionality reduction of hyperspectral datasets. From different experiments, it is observed that the simultaneous usage of 3D and 2D filters along with sparse regularizations significantly improve learning capability of the networks.

Tensor Decomposition for Salient Object Detection in Images

The fundamental challenge of salient object detection is to find the decision boundary that separates the salient object from the background. Low-rank recovery models address this challenge by decomposing an image or image feature-based matrix into a low-rank matrix representing the image background and a sparse matrix representing salient objects. This method is simple and efficient in finding salient objects. However, it needs to convert high-dimensional feature space into a two-dimensional matrix. Therefore, it does not take full advantage of image features in discovering the salient object. In this article, we propose a tensor decomposition method which considers spatial consistency and tries to make full use of image feature information in detecting salient objects. First, we use high-dimensional image features in tensor to preserve spatial information about image features. Following this, we use a tensor low-rank and sparse model to decompose the image feature tensor into a low-rank tensor and a sparse tensor, where the low-rank tensor represents the background and the sparse tensor is used to identify the salient object. To solve the tensor low-rank and sparse model, we employed a heuristic strategy by relaxing the definition of tensor trace norm and tensor l1-norm. Experimental results on three saliency benchmarks demonstrate the effectiveness of the proposed tensor decomposition method.

Fast 3D-HEVC PU size decision algorithm for depth map intra-video coding

High-Efficiency Video Coding (HEVC)-based 3D video coding (3D-HEVC) is the most recent standard and last exertion of ISO/IEC MPEG and ITU-T Video Coding Experts Group (VCEG) for 3D video coding using a new data video format called Multi-View Video plus Depth map (MVD). This new standard achieves a high coding improvement. In any case, one of the most critical difficulties in 3D-HEVC is time computational complexity. The depth map intra-prediction is a critical factor in 3D-HEVC intra-coding, in which, the 3D-HEVC uses a highly adaptable Coding Unit (CU) structure with a specific end goal to expand the coding efficiency of all depth map characteristics. However, it results in an enormous Rate Distortion Optimization Cost (RDO-Cost) because of the broad recursive search for the best CU size from $$64\times 64$$ down to $$4\times 4$$ . This computational complexity excludes the 3D-HEVC from true and real-time application. Hence, it is imperative to build up an algorithm to diminish the complexity of the size decision in depth map intra-coding. To determine the previously mentioned issue, this paper proposes an effective 3D-HEVC PU size decision algorithm for depth map intra-video coding based on tensor features and statistical data analyses. The experimental results demonstrate that the proposed model diminishes the complexity of depth map size decision significantly with low rate distortion increase.

Layout Hotspot Detection With Feature Tensor Generation and Deep Biased Learning

Detecting layout hotspots is a key step in the physical verification flow. Although machine learning solutions show benefits over lithography simulation and pattern matching-based methods, it is still hard to select a proper model for large scale problems and inevitably, performance degradation occurs. To overcome these issues, in this paper, we develop a deep learning framework for high performance and large scale hotspot detection. First, we use feature tensor generation to extract representative layout features that fit well with convolutional neural networks while keeping the spatial relationship of the original layout pattern with minimal information loss. Second, we propose a biased learning (BL) algorithm to train the convolutional neural network to further improve detection accuracy with small false alarm penalties. In addition, to simplify the training procedure and seek a better tradeoff between accuracy and false alarms, we extend the original BL to a batch BL algorithm. Experimental results show that our framework outperforms previous machine learning-based hotspot detectors in both ICCAD 2012 Contest benchmarks and large scale industrial benchmarks. Source code and trained models are available at https://github.com/phdyang007/dlhsd .

Nanoparticle diffusion in sheared cellular blood flow

Using a multiscale blood flow solver, the complete diffusion tensor of nanoparticles (NPs) in sheared cellular blood flow is calculated over a wide range of shear rate and haematocrit. In the short-time regime, NPs exhibit anomalous dispersive behaviors under high shear and high haematocrit due to the transient elongation and alignment of the red blood cells (RBCs). In the long-time regime, the NP diffusion tensor features high anisotropy. Particularly, there exists a critical shear rate (${\sim}100~\text{s}^{-1}$) around which the shear-rate dependence of the diffusivity tensor changes from linear to nonlinear scale. Above the critical shear rate, the cross-stream diffusivity terms vary sublinearly with shear rate, while the longitudinal term varies superlinearly. The dependence on haematocrit is linear in general except at high shear rates, where a sublinear scale is found for the vorticity term and a quadratic scale for the longitudinal term. Through analysis of the suspension microstructure and numerical experiments, the nonlinear haemorheological dependence of the NP diffusion tensor is attributed to the streamwise elongation and cross-stream contraction of RBCs under high shear, quantified by a capillary number. The RBC size is shown to be the characteristic length scale affecting the RBC-enhanced shear-induced diffusion (RESID), while the NP submicrometre size exhibits negligible influence on the RESID. Based on the observed scaling behaviours, empirical correlations are proposed to bridge the NP diffusion tensor to specific shear rate and haematocrit. The characterized NP diffusion tensor provides a constitutive relation that can lead to more effective continuum models to tackle large-scale NP biotransport applications.