Deep Computation Model Research Articles

Tensor-Empowered LSTM for Communication-Efficient and Privacy-Enhanced Cognitive Federated Learning in Intelligent Transportation Systems

Multimedia cognitive computing as a revolutionary emerging concept of artificial intelligence emulating the reasoning process like human brains can facilitate the evolution of intelligent transportation systems (ITS) to be smarter, safer, and more efficient. Massive multimedia traffic big data is an important prerequisite for the success of cognitive computing in ITS. However, traditional data-centralized artificial intelligence approaches often face the problems of data islands and data famine due to concerns about data privacy and security. To this end, we propose the concept of cognitive federated learning leveraging federated learning as the learning paradigm for cognitive computing, which solves the preceding concerns by sharing updated models rather than raw data. Nevertheless, the exchange of numerous model parameters not only generates significant communication overhead but also suffers from the risk of privacy leakage due to inference attacks. This article aims to design a novel lightweight and privacy-enhanced cognitive federated learning architecture to facilitate the development of ITS. First, a privacy-enhanced model protection scheme with homomorphic encryption as the underlying technology is proposed to simultaneously defend against the inference attacks launched by external malicious attackers, honest-but-curious cognitive platforms, and internal participants. Furthermore, a novel tensor ring-block decomposition and its corresponding deep computation model converting the weight tensor into a set of matrices and third-order core tensors are proposed, which could reduce the communication overhead and storage requirements without compromising model performance. Experimental results on real-world datasets show that the proposed approach performs well.

Lightweight Tensor Deep Computation Model With Its Application in Intelligent Transportation Systems

Deep computation models (DCMs) are widely used in intelligent transportation systems (ITS), like driving behavior detection, intelligent parking navigation and real-time road condition detection. Due to the multi-source heterogeneous nature of big data of the ITS, it is difficult for traditional DCMs to learn effective multi-modal data features. Although, the DCMs in tensor space can efficiently represent multi-modal data, it further worsens the problem of model learning parameter explosion. In this paper, we propose a lightweight tensor DCM. The model compresses the redundant learning parameters of the model and reduces the consumption of computational resources while maintaining the learning characterization capability of the DCM in tensor space, thus making the network model more general and lightweight for deploying the DCM to smart cars and edge devices. The proposed lightweight tensor DCM is evaluated on several real datasets. The experimental results show that the number of learning parameters is massively compressed while keeping the performance of the network model almost constant, while also reducing the computational complexity and training time of the model.

Deep computation model to the estimation of sulphur dioxide for plant health monitoring in IoT

Deep computation model to the estimation of sulphur dioxide for plant health monitoring in IoT

A Distributed Hierarchical Deep Computation Model for Federated Learning in Edge Computing

Deep learning has recently garnered significant interest in many applications especially for big data analytics in the edge computing environment. Federated learning, as a novel machine learning technique, aims to build a shared learning model from training data on distributed edge nodes to protect data privacy. However, the model update in federated learning requires parameter exchanges among edge nodes, which is rather bandwidth-consuming. This article proposes a novel distributed hierarchical tensor deep computation model by condensing the model parameters from a high-dimensional tensor space into a set of low-dimensional subspaces to reduce the bandwidth consumption and storage requirement for federated learning. Moreover, an updating approach with a hierarchical tensor back-propagation algorithm is developed by directly computing the gradients of low-dimensional parameters to reduce the memory requirement of training for edge nodes and improve training efficiency. Finally, extensive simulations on classical datasets with different local data distributions are presented for the performance evaluation. The results demonstrate that the proposed model relieves the burden of communication bandwidth and reduces energy consumption at edge nodes for federated learning.

Hybrid Deep Model for Human Behavior Understanding on Industrial Internet of Video Things

Human behavior understanding is playing more and more important role in human-centered Industrial Internet of Video Things (IIoVT) system with the deep combination of artificial intelligence and video-based industrial Internet of Things. However, it requires expensively computational resources, including high-performance computing units and large memory, to train a deep computation model with a large number of parameters, which limits its effectiveness and efficiency for IIoVT applications. In this article, a tensor-train mechanism based deep model is presented for video human behavior understanding to meet the requirement of IIoVT applications. It can get competitive performance in accuracy and training efficiency with potentiality for combination of artificial intelligence and prefront IIoVT system. On the one hand, to achieve desirable accuracy, we improved the conventional CNN and adopted the recurrent neural network mechanism to enhance the video representation over time, which takes the correlation between consecutive deep feature into consideration. On the other hand, to enhance the inference capacity between the spatial and temporal features, we carry out the self-critical reinforcement learning mechanism in parameter learning stage. Meanwhile, to further reduce parameter storage size to meet requirement for the deployment of deep neural network and edge device, the tensor-train mechanism is used, which transforms the parameter matrix to a tensor space and carry out tensor decomposition mechanism to decrease the number of parameter generated in parameter training. Finally, we conduct extensive experiments to evaluate our scheme, and the results demonstrate that our method can improve the training efficiency and save the memory space for the deep computation model with better accuracy.

Differentially Private Tensor Train Deep Computation for Internet of Multimedia Things

The significant growth of the Internet of Things (IoT) takes a key and active role in healthcare, smart homes, smart manufacturing, and wearable gadgets. Due to complexness and difficulty in processing multimedia data, the IoT based scheme, namely Internet of Multimedia Things (IoMT) exists that is specialized for services and applications based on multimedia data. However, IoMT generated data are facing major processing and privacy issues. Therefore, tensor-based deep computation models proved a better platform to process IoMT generated data. A differentially private deep computation method working in the tensor space can attest to its efficacy for IoMT. Nevertheless, the deep computation model comprises a multitude of parameters; thus, it requires large units of memory and expensive computing units with higher performance levels, which hinders its performance for IoMT. Motivated by this, therefore, the paper proposes a deep private tensor train autoencoder (dPTTAE) technique to deal with IoMT generated data. Notably, the compression of weight tensors to manageable tensor train format is achieved through Tensor Train (TT) network. Moreover, TT format parameters are trained through higher-order back-propagation and gradient descent. We applied dPTTAE on three representative datasets. Comprehensive experimental evaluations and theoretical analysis show that dPTTAE enhances training time efficiency, and greatly improve memory utilization efficiency, attesting its potential for IoMT.

Video Scene Segmentation Using Tensor-Train Faster-RCNN for Multimedia IoT Systems

Video surveillance techniques like scene segmentation are playing an increasingly important role in multimedia Internet-of-Things (IoT) systems. However, existing deep learning-based methods face challenges in both accuracy and memory when deployed on edge computing devices with limited computing resources. To address these challenges, a tensor-train video scene segmentation scheme that compares the local background information in regional scene boundary boxes in adjacent frames is proposed. Compared to the existing methods, the proposed scheme can achieve competitive performance in both segmentation accuracy and parameter compression rate. In detail, first, an improved faster region convolutional neural network (faster-RCNN) model is proposed to recognize and generate a large number of region boxes with foreground and background to achieve boundary boxes. Then, the foreground boxes with sparse objects are removed and the rest are considered as optional background boxes used to measure the similarity between two adjacent frames. Second, to accelerate the training efficiency and reduce memory size, a general and efficient training way using tensor-train decomposition to factor the input-to-hidden weight matrix is proposed. Finally, experiments are conducted to evaluate the performance of the proposed scheme in terms of accuracy and model compression. Our results demonstrate that the proposed model can improve the training efficiency and save the memory space for the deep computation model with good accuracy. This work opens the potential for the use of artificial intelligence methods in edge computing devices for multimedia IoT systems.

A Cloud-Edge-Aided Incremental High-Order Possibilistic c-Means Algorithm for Medical Data Clustering

Medical Internet of Things are generating a big volume of data to enable smart medicine that tries to offer computer-aided medical and healthcare services with artificial intelligence techniques like deep learning and clustering. However, it is a challenging issue for deep learning, and clustering algorithms to analyze large medical data because of their high computational complexity, thus hindering the progress of smart medicine. In this article, we present an incremental high-order possibilistic c-means algorithm (IHoPCM) on a cloud-edge computing system to achieve medical data coclustering of multiple hospitals in different locations. Specifically, each hospital employs the deep computation model to learn a feature tensor of each medical data object on the local edge computing system, and then uploads the feature tensors to the cloud computing platform. The high-order possibilistic c-means algorithm is performed on the cloud system for medical data clustering on uploaded feature tensors. Once the new medical data feature tensors are arriving at the cloud computing platform, the incremental high-order possibilistic c-means algorithm (IHoPCM) is performed on the combination of the new feature tensors and the previous clustering centers to obtain clustering results for the feature tensors received to date. In this way, repeated clustering on the previous feature tensors is avoided to improve the clustering efficiency. In the experiments, we compare different algorithms on two medical datasets regarding clustering accuracy and clustering efficiency. Results show that the presented IHoPCM method achieves great improvements over the compared algorithms in clustering accuracy and efficiency.

Incremental Deep Computation Model for Wireless Big Data Feature Learning

Big data feature learning is a crucial issue for the service management for Internet of Things. However, big data collected from Internet of Things is of dynamic nature at a high speed, which poses an important challenge on wireless big data learning models, especially the deep computation model. In this paper, an incremental deep computation model is proposed for wireless big data feature learning in Internet of Things. First, two incremental tensor auto-encoders (ITAE) are developed by devising two incremental learning algorithms, namely parameter-based incremental learning algorithm (PI-TAE) and structure-based incremental learning algorithm (SI-TAE), when new wireless samples are available. PI-TAE only updates the network parameters while SI-TAE simultaneously adjusts the structure and updates the parameters to adapt to the new arriving wireless big data. Furthermore, an incremental deep computation model is constructed by stacking several ITAEs. Experiments are conducted to evaluate the performance of the proposed model by comparing with the conventional deep computation model and other two representative incremental learning algorithms, i.e., OANN and PIE. Results demonstrate that the presented model can modify the network in an incremental manner for new arriving data learning efficiently with preserving the prior knowledge for the previous data learning, proving its potential for dynamic wireless big data learning in Internet of Things.

An Edge-Cloud-Aided High-Order Possibilistic c-Means Algorithm for Big Data Clustering

In this article, a high-order possibilistic c-means algorithm (HOPCM) based on the double-layer deep computation model (DCM) is proposed for big data clustering. Specifically, an asymmetric tensor autoencoder is presented to efficiently train the double-layer DCM for big data feature learning. Furthermore, an edge-cloud computing system is developed to improve the clustering efficiency. In the edge-cloud system, the computation-intensive tasks including the parameters’ training and clustering are offloaded to the cloud while the task of feature learning is performed at the edge of network. Finally, we conduct extensive experiments to evaluate the performance of the presented algorithm by comparing it with other two representative big data clustering algorithms, i.e., the standard HOPCM and the HOPCM based on deep learning. Results demonstrate that the presented algorithm achieves higher accuracy than the two compared algorithms and furthermore the clustering efficiency are significantly improved by the developed edge-cloud computing system.

A hybrid deep computation model for feature learning on aero-engine data: applications to fault detection

Recently, the safety of aircraft has attracted much attention with some crashes occurring. Gas-path faults, as the most common faults of aircraft, pose a vast challenge for the safety of aircraft because of the complexity of the aero-engine structure. In this article, a hybrid deep computation model is proposed to effectively detect gas-path faults on the basis of the performance data. In detail, to capture the local spatial features of the gas-path performance data, an unfully connected convolutional neural network of one-dimensional kernels is used. Furthermore, to model the temporal patterns hidden in the gas-path faults, a recurrent computation architecture is introduced. Finally, extensive experiments are conducted on real aero-engine data. The results show that the proposed model can outperform the models with which it is compared.

A Unified Smart Chinese Medicine Framework for Healthcare and Medical Services.

Smart Chinese medicine has emerged to contribute to the evolution of healthcare and medical services by applying machine learning together with advanced computing techniques like cloud computing to computer-aided diagnosis and treatment in the health engineering and informatics. Specifically, smart Chinese medicine is considered to have the potential to treat difficult and complicated diseases such as diabetes and cancers. Unfortunately, smart Chinese medicine has made very limited progress in the past few years. In this paper, we present a unified smart Chinese medicine framework based on the edge-cloud computing system. The objective of the framework is to achieve computer-aided syndrome differentiation and prescription recommendation, and thus to provide pervasive, personalized, and patient-centralized services in healthcare and medicine. To accomplish this objective, we integrate deep learning and deep reinforcement learning into the traditional Chinese medicine. Furthermore, we propose a multi-modal deep computation model for syndrome recognition that is a crucial part of syndrome differentiation. Finally, we conduct experiments to validate the proposed model by comparing with the staked auto-encoder and multi-modal deep learning model for syndrome recognition of hypertension and cold.

An Adaptive Dropout Deep Computation Model for Industrial IoT Big Data Learning With Crowdsourcing to Cloud Computing

Deep computation, as an advanced machine learning model, has achieved the state-of-the-art performance for feature learning on big data in industrial Internet of Things (IoT). However, the current deep computation model usually suffers from overfitting due to the lack of public available labeled training samples, limiting its performance for big data feature learning. Motivated by the idea of active learning, an adaptive dropout deep computation model (ADDCM) with crowdsourcing to cloud is proposed for industrial IoT big data feature learning in this paper. First, a distribution function is designed to set the dropout rate for each hidden layer to prevent overfitting for the deep computation model. Furthermore, the outsourcing selection algorithm based on the maximum entropy is employed to choose appropriate samples from the training set to crowdsource on the cloud platform. Finally, an improved supervised learning from multiple experts scheme is presented to aggregate answers given by human workers and to update the parameters of the ADDCM simultaneously. Extensive experiments are conducted to evaluate the performance of the presented model by comparing with the dropout deep computation model and other state-of-the-art crowdsourcing algorithms. The results demonstrate that the proposed model can prevent overfitting effectively and aggregate the labeled samples to train the parameters of the deep computation model with crowdsouring for industrial IoT big data feature learning.

A multi-projection deep computation model for smart data in Internet of Things

The double-projection deep computation model (DPDCM) proved to be effective for big data feature learning. However, DPDCM cannot capture the underlying correlations over the different modalities enough since it projects the input data into only two subspaces. To tackle this problem, this paper presents a multi-projection deep computation model (MPDCM) to generalize DPDCM for smart data in Internet of Things. Specially, MPDCM maps the input data into multiple nonlinear subspaces to learn the interacted features of IoT big data by substituting each hidden layer with a multi-projection layer. Furthermore, a learning algorithm based on back-propagation and gradient descent is designed to train the parameters of the presented model. Finally, extensive experiments are conducted on two representative datasets, i.e, Animal-20 and NUS-WIDE-14, to verify the presented model by comparing with DPDCM. Results show that the presented model achieves higher classification accuracy than DPDCM, proving the potential of the presented model to drill smart data for Internet of Things.

An Improved Deep Computation Model Based on Canonical Polyadic Decomposition

Deep computation models achieve super performance for big data feature learning. However, training a deep computation model poses a significant challenge since a deep computation model typically involves a large number of parameters. Specially, it needs a high-performance computing server with a large-scale memory and a powerful computing unit to train a deep computation model, making it difficult to increase the size of a deep computation model further for big data feature learning on low-end devices such as conventional desktops and portable CPUs. In this paper, we propose an improved deep computation model based on the canonical polyadic decomposition scheme to compress the parameters and to improve the training efficiency. Furthermore, we devise a learning algorithm based on the back-propagation strategy to train the parameters of the proposed model. The learning algorithm can be directly performed on the compressed parameters to improve the training efficiency. Finally, we carry on the experiments on three representative datasets, i.e., CUAVE, SNAE2, and STL-10, to evaluate the performance of the proposed model by comparing with the conventional deep computation model and other two improved deep computation models based on the Tucker decomposition and the tensor-train network. Results demonstrate that the proposed model can compress parameters greatly and improve the training efficiency significantly with a low classification accuracy drop.

Dependable Deep Computation Model for Feature Learning on Big Data in Cyber-Physical Systems

With the ongoing development of sensor devices and network techniques, big data are being generated from the cyber-physical systems. Because of sensor equipment occasional failure and network transmission unreliability, a large number of low-quality data, such as noisy data and incomplete data, is collected from the cyber-physical systems. Low-quality data pose a remarkable challenge on deep learning models for big data feature learning. As a novel deep learning model, the deep computation model achieves superior performance for big data feature learning. However, it is difficult for the deep computation model to learn dependable features for low-quality data, since it uses the nonlinear function as the encoder. In this article, a dependable deep computation model is proposed for feature learning on low-quality big data in cyber-physical systems. Specially, a regularity is added into the objective function of the deep computation model to obtain reliable features in the intermediate-level representation space. Furthermore, a learning algorithm based on the back-propagation strategy is devised to train the parameters of the proposed model. Finally, experiments are conducted on three representative datasets and a real dataset to evaluate the effectiveness of the dependable deep computation model for low-quality big data feature learning. Results show that the proposed model achieves a remarkable result for the tasks of classification, restoration, and prediction, proving the potential of this work for practical applications in cyber-physical systems.

Privacy-Preserving Double-Projection Deep Computation Model With Crowdsourcing on Cloud for Big Data Feature Learning

Recent years have witness a considerable advance of Internet of Things with the tremendous progress of communication theories and sensing technologies. A large number of data, usually referring to big data, have been generated from Internet of Things. In this paper, we present a double-projection deep computation model (DPDCM) for big data feature learning, which projects the raw input into two separate subspaces in the hidden layers to learn interacted features of big data by replacing the hidden layers of the conventional deep computation model (DCM) with double-projection layers. Furthermore, we devise a learning algorithm to train the DPDCM. Cloud computing is used to improve the training efficiency of the learning algorithm by crowdsourcing the data on cloud. To protect the private data, a privacy-preserving DPDCM (PPDPDCM) is proposed based on the BGV encryption scheme. Finally, experiments are carried on Animal-20 and NUS-WIDE-14 to estimate the performance of DPDCM and PPDPDCM by comparing with DCM. Results demonstrate that DPDCM achieves a higher classification accuracy than DCM. More importantly, PPDPDCM can effectively improve the efficiency for training parameters, proving its potential for big data feature learning.

A Dropconnect Deep Computation Model for Highly Heterogeneous Data Feature Learning in Mobile Sensing Networks

Deep computation model, as a tensor deep learning model, outperforms multi-modal deep learning models for feature learning on heterogenous data. However, deep computation model is limited in generalization to small heterogeneous data sets since it typically requires many training objects to learn the parameters. In this article, we propose a dropconnect deep computation model (DDCM) for highly heterogeneous data feature learning in mobile sensing networks. Specifically, the dropconnect technique is used to generalize the large fully-connected layers in the deep computation model for small heterogeneous data sets. Furthermore, the rectified linear units (ReLU) are used as the activation function to reduce computation and prevent overfitting. Finally, we compare the classification accuracy and execution time for learning the parameters between our model and the traditional deep computation model on two highly heterogeneous data sets. Results illustrate that our model achieves 2 percent higher classification accuracy and performs more efficiently than the deep computation model, proving the potential of our proposed model for highly heterogeneous data learning in mobile sensing networks.

A Tensor-Train Deep Computation Model for Industry Informatics Big Data Feature Learning

The deep computation model has been proved to be effective for big data hierarchical feature and representation learning in the tensor space. However, it requires expensively computational resources including high-performance computing units and large memory to train a deep computation model with a large number of parameters, limiting its effectiveness and efficiency for industry informatics big data feature learning. In this paper, a tensor-train deep computation model is presented for industry informatics big data feature learning. Specially, the tensor-train network is used to compress the parameters significantly by converting the dense weight tensors into the tensor-train format. Furthermore, a learning algorithm is implemented based on gradient descent and back-propagation to train the parameters of the presented tensor-train deep computation model. Extensive experiments are carried on STL-10, CUAVE, and SNAE2 to evaluate the presented model in terms of the approximation error, classification accuracy drop, parameters reduction, and speedup. Results demonstrate that the presented model can improve the training efficiency and save the memory space greatly for the deep computation model with small accuracy drops, proving its potential for industry informatics big data feature learning.

Deep Convolutional Computation Model for Feature Learning on Big Data in Internet of Things

Currently, a large number of industrial data, usually referred to big data, are collected from Internet of Things (IoT). Big data are typically heterogeneous, i.e., each object in big datasets is multimodal, posing a challenging issue on the convolutional neural network (CNN) that is one of the most representative deep learning models. In this paper, a deep convolutional computation model (DCCM) is proposed to learn hierarchical features of big data by using the tensor representation model to extend the CNN from the vector space to the tensor space. To make full use of the local features and topologies contained in the big data, a tensor convolution operation is defined to prevent overfitting and improve the training efficiency. Furthermore, a high-order backpropagation algorithm is proposed to train the parameters of the deep convolutional computational model in the high-order space. Finally, experiments on three datasets, i.e., CUAVE, SNAE2, and STL-10 are carried out to verify the performance of the DCCM. Experimental results show that the deep convolutional computation model can give higher classification accuracy than the deep computation model or the multimodal model for big data in IoT.