Heterogeneous Convolutional Neural Network Research Articles

DraftFed: A Draft-Based Personalized Federated Learning Approach for Heterogeneous Convolutional Neural Networks

DraftFed: A Draft-Based Personalized Federated Learning Approach for Heterogeneous Convolutional Neural Networks

An Efficient Hybrid Deep Learning Accelerator for Compact and Heterogeneous CNNs

Resource-efficient Convolutional Neural Networks (CNNs) are gaining more attention. These CNNs have relatively low computational and memory requirements. A common denominator among such CNNs is having more heterogeneity than traditional CNNs. This heterogeneity is present at two levels: intra-layer type and inter-layer type. Generic accelerators do not capture these levels of heterogeneity, which harms their efficiency. Consequently, researchers have proposed model-specific accelerators with dedicated engines. When designing an accelerator with dedicated engines, one option is to dedicate one engine per CNN layer. We refer to accelerators designed with this approach as single-engine single-layer (SESL). This approach enables optimizing each engine for its specific layer. However, such accelerators are resource-demanding and unscalable. Another option is to design a minimal number of dedicated engines such that each engine handles all layers of one type. We refer to these accelerators as single-engine multiple-layer (SEML). SEML accelerators capture the inter-layer-type but not the intra-layer-type heterogeneity. We propose the Fixed Budget Hybrid CNN Accelerator (FiBHA), a hybrid accelerator composed of an SESL part and an SEML part, each processing a subset of CNN layers. FiBHA captures more heterogeneity than SEML while being more resource-aware and scalable than SESL. Moreover, we propose a novel module, Fused Inverted Residual Bottleneck (FIRB), a fine-grained and memory-light SESL architecture building block. The proposed architecture is implemented and evaluated using high-level synthesis (HLS) on different Field Programmable Gate Arrays representing various resource budgets. Our evaluation shows that FiBHA improves the throughput by up to 4 x and 2.5 x compared to state-of-the-art SESL and SEML accelerators, respectively. Moreover, FiBHA reduces memory and energy consumption compared to an SEML accelerator. The evaluation also shows that FIRB reduces the required memory by up to 54%, and energy requirements by up to 35% compared to traditional pipelining.

A New Siamese Heterogeneous Convolutional Neural Networks Based on Attention Mechanism and Feature Pyramid.

Accuracy and speed are the most important indexes for evaluating many object tracking algorithms. However, when constructing a deep fully convolutional neural network (CNN), the use of deep network feature tracking will cause tracking drift due to the effects of convolution padding, receptive field (RF), and overall network step size. The speed of the tracker will also decrease. This article proposes a fully convolutional siamese network object tracking algorithm that combines the attention mechanism with the feature pyramid network (FPN), and uses heterogeneous convolution kernels to reduce the amount of calculations (FLOPs) and parameters. The tracker first uses a new fully CNN to extract image features, and introduces a channel attention mechanism in the feature extraction process to improve the representation ability of convolutional features. Then use the FPN to fuse the convolutional features of high and low layers, learn the similarity of the fused features, and train the fully CNNs. Finally, the heterogeneous convolutional kernel is used to replace the standard convolution kernel to improve the speed of the algorithm, thereby making up for the efficiency loss caused by the feature pyramid model. In this article, the tracker is experimentally verified and analyzed on the VOT-2017, VOT-2018, OTB-2013, and OTB-2015 datasets. The results show that our tracker has achieved better results than the state-of-the-art trackers.

Heterogeneous Convolutional Neural Networks for Emotion Recognition Combined with Multimodal Factorised Bilinear Pooling and Mobile Application Recommendation

The field of emotion recognition has garnered considerable interest due to its diverse applications in mental health, personalised advertising and enhancing user experiences. This research paper introduces a unique and innovative method for emotion recognition by integrating heterogeneous convolutional neural networks (CNNs) with multimodal factorised bilinear pooling. Furthermore, the paper also incorporates the integration of mobile application recommendations as part of the overall approach. The proposed method leverages the power of CNNs to extract high-level features from different modalities, including facial expressions, speech signals and physiological signals. By using heterogeneous CNNs, each modality is processed independently to capture modality-specific emotional cues effectively. To fuse the extracted features, multimodal factorised bilinear pooling is employed, which captures the complex interactions between different modalities while reducing the computational complexity. This pooling technique efficiently combines the modality-specific features, resulting in a compact and discriminative representation of the emotional state. In addition to emotion recognition, this paper also introduces the integration of mobile app recommendations. By leveraging the recognised emotion, the system recommends relevant mobile applications that are tailored to the user’s emotional state. This integration enhances user experience and facilitates emotion regulation through the utilisation of appropriate mobile apps. Experimental evaluations are conducted on benchmark emotion recognition datasets, including the DEAP and MAHNOB_HCI datasets. The findings of the study highlight the effectiveness of the proposed methodology in terms of accuracy and robustness, surpassing existing approaches in the field. Additionally, the integration of the mobile app recommendation system showcases encouraging outcomes by offering personalised recommendations tailored to the user’s emotional state.

Hashing for secure optical information compression in a heterogeneous convolutional neural network

In recent years, heterogeneous machine learning accelerators have become of significant interest to science, engineering, and industry. At the same time, the looming post-quantum encryption era instigates the demand for increased data security. From a hardware processing point of view, electronic computing hardware is challenged by electronic capacitive interconnect delay and associated energy consumption. In heterogeneous systems, such as electronic–photonic accelerators, parasitic domain crossings limit throughput and speed. With analog optical accelerators exhibiting a strong potential for high throughput (up to petaoperations per second) and operation efficiency, their ability to perform machine learning classification tasks on encrypted data has not been broadly recognized. This work is a significant step in that direction. Here, we present an optical hashing and compression scheme that is inspired by SWIFFT, a post-quantum hashing family of algorithms. High degree optical hardware-to-algorithm homomorphism allows one to optimally harvest the potential of free-space data processing: innate parallelism, low latency tensor by-element multiplication, and zero-energy Fourier transformation operations. The algorithm can provide several orders of magnitude increase in processing speed as compared to optical machine learning accelerators with non-compressed input. This is achieved by replacing slow, high-resolution CMOS cameras with ultra-fast and signal-triggered CMOS detector arrays. Additionally, information acquired in this way will require much lower transmission throughput, less in silico processing power, storage, and will be pre-hashed, facilitating optical information security. This concept has the potential to allow heterogeneous convolutional Fourier classifiers to approach the performance of their fully electronic counterparts and enables data classification on hashed data.

Deep learning for skillful long-lead ENSO forecasts

El Niño-Southern Oscillation (ENSO) is one of the fundamental drivers of the Earth's climate variability. Thus, its skillful prediction at least a few months to years ahead is of utmost importance to society. Using both dynamical and statistical methods, several studies reported skillful ENSO predictions at various lead times. Predictions with long lead times, on the other hand, remain difficult. In this study, we propose a convolutional neural network (CNN)-based statistical ENSO prediction system with heterogeneous CNN parameters for each season with a modified loss function to predict ENSO at least 18–24 months ahead. The developed prediction system indicates that the CNN model is highly skillful in predicting ENSO at long lead times of 18–24 months with high skills in predicting extreme ENSO events compared with the Scale Interaction Experiment-Frontier ver. 2 (SINTEX-F2) dynamical system and several other statistical prediction systems. The analysis indicates that the CNN model can overcome the spring barrier, a major hindrance to dynamical prediction systems, in predicting ENSO at long lead times. The improvement in the prediction skill can partly be attributed to the heterogeneous parameters of seasonal CNN models used in this study and also to the use of a modified loss function in the CNN model. In this study, we also attempted to identify various precursors to ENSO events using CNN heatmap analysis.

Human behaviour analysis based on spatio-temporal dual-stream heterogeneous convolutional neural network

Human behaviour analysis based on spatio-temporal dual-stream heterogeneous convolutional neural network

PerHeFed: A general framework of personalized federated learning for heterogeneous convolutional neural networks

In conventional federated learning, each device is restricted to train a network model of the same structure. This greatly hinders the application of federated learning where the data and devices are quite heterogeneous because of their different hardware equipment and communication networks. At the same time, existing studies have shown that transmitting all of the model parameters not only has heavy communication costs, but also increases risk of privacy leakage. We propose a general framework for personalized federated learning (PerHeFed), which enables the devices to design their local model structures autonomously and share sub-models without structural restrictions. In PerHeFed, a simple-but-effective mapping relation and a novel personalized sub-model aggregation method are proposed for heterogeneous sub-models to be aggregated. By dividing the aggregations into two primitive types (i.e., inter-layer and intra-layer), PerHeFed is applicable to any combination of heterogeneous convolutional neural networks, and we believe that this can satisfy the personalized requirements of heterogeneous models. Experiments show that, compared to the state-of-the-art method (e.g., FLOP), in non-IID data sets our method compress ≈ 50% of the shared sub-model parameters with only a 4.38% drop in accuracy on SVHN dataset and on CIFAR-10, PerHeFed even achieves a 0.3% improvement in accuracy. To the best of our knowledge, our work is the first general personalized federated learning framework for heterogeneous convolutional networks, even cross different networks, addressing model structure unity in conventional federated learning.

FedStack: Personalized activity monitoring using stacked federated learning

Recent advances in remote patient monitoring (RPM) systems can recognize various human activities to measure vital signs, including subtle motions from superficial vessels. There is a growing interest in applying artificial intelligence (AI) to this area of healthcare by addressing known limitations and challenges such as predicting and classifying vital signs and physical movements, which are considered crucial tasks. Federated learning is a relatively new AI technique designed to enhance data privacy by decentralizing traditional machine learning modeling. However, traditional federated learning requires identical architectural models to be trained across the local clients and global servers. This limits global model architecture due to the lack of local models’ heterogeneity. To overcome this, a novel federated learning architecture, FedStack, which supports ensembling heterogeneous architectural client models was proposed in this study. This work offers a protected privacy system for hospitalized in-patients in a decentralized approach and identifies optimum sensor placement. The proposed architecture was applied to a mobile health sensor benchmark dataset from 10 different subjects to classify 12 routine activities. Three AI models, artificial neural network (ANN), convolutional neural network (CNN), and bidirectional long short-term memory (Bi-LSTM) were trained on individual subject data. The federated learning architecture was applied to these models to build local and global models capable of state-of-the-art performances. The local CNN model outperformed ANN and Bi-LSTM models on each subject data. Our proposed work has demonstrated better performance for heterogeneous stacking of the local models compared to homogeneous stacking. Further analysis of the global heterogeneous CNN model determined that the optimum placement of the sensors on human limbs resulted in better activity recognition. This work sets the stage to build an enhanced RPM system that incorporates client privacy to assist with clinical observations for patients in an acute mental health facility and ultimately help to prevent unexpected death.

Emotion recognition using heterogeneous convolutional neural networks combined with multimodal factorized bilinear pooling

Multimodal emotion recognition is one of the challenging topics in the field of knowledge-based systems and many methods have been studied successfully. Nevertheless, multimodal emotion recognition needs effective fusion representations of multimodal domains, and available methods still have problems on this challenging task. In view of this, this paper proposes a new deep learning model for emotion recognition based on heterogeneous convolutional neural networks (HCNNs) and multimodal factorized bilinear pooling (MFB). In the proposed model, firstly, we select the channels of electroencephalogram (EEG) signals to reduce the interference caused by the redundant channels. Secondly, the HCNNs extract the convolutional features of each modality, and then the MFB method fuses the deep convolution features of the different modalities. Finally, the ensembled strategy is used to verify the model proposed in this paper and explore the influence of various bands on the experiment. The proposed method allows all elements of each component to effectively contact with each other to express the complex internal relationship of each component modality. The experimental results show that the best average result of our proposed method achieves the accuracy of 91.84% on DEAP dataset and 90.17% on MAHNOB-HCI dataset, which proves that the proposed method can improve the performance of multimodal emotion recognition and significantly outperform the state-of-the-art.

Video behavior recognition algorithm based on two-stream heterogeneous convolutional neural network

Abstract In video behavior recognition, making full use of the spatio-temporal information contained in video frame is the critical point to further improve the recognition accuracy. In our study, we propose a video behavior recognition algorithm based on two-stream heterogeneous network, so that we can further extract the spatio-temporal feature information in the video frame, and finally take full advantage of the spatio-temporal features in the video sequence. In view of the characteristics of RGB and optical flow images, this paper uses DenseNet121 and Inception-V4 to structure a two-stream network, to fully extract the spatio-temporal feature information in the video. In this paper, UCF101 dataset is used to evaluate the algorithm, and the recognition accuracy is 91.7%, which verified the reliability of the proposed algorithm.

Human activity recognition using wearable sensors by heterogeneous convolutional neural networks

Recent researches on sensor based human activity recognition (HAR) are mostly devoted to designing various network architectures to enhance their feature representation capacity for raw sensor data. In this paper, we focus on strengthening the vanilla convolution without adjusting the model architectures in HAR scenario. Inspired by the idea of grouped convolution, we propose a novel heterogeneous convolution for activity recognition task, where all filters within a specific convolutional layer are separated into two uneven groups. Specifically, the sensor input is down-sampled into a low-dimensional embedding, which is then convolved by one filter group to recalibrate normal filters within the other group. The two filter groups can complement each other, which is very beneficial for augmenting the receptive field of sensor signals for HAR task. Extensive experiments are conducted on several benchmark HAR datasets, which consists of OPPORTUNITY, PAMAP2, UCI-HAR, USC-HAD as well as the Weakly Labeled HAR dataset. The results show that the baseline models can be significantly improved. Our heterogeneous convolution is simple and can easily be integrated into standard convolutional layers without increasing extra parameters and computational overhead. Finally, the actual operation of heterogeneous convolution is evaluated on an embedded Raspberry Pi platform.

Grafting Heterogeneous Neural Networks for a Hierarchical Object Classification

Convolutional neural networks (CNNs) are deep learning architectures used for image classification that have been improved in recent years to increase their accuracies and reduce their computation times. Hierarchical approaches are based on a step-by-step strategy and aim to optimize performance on difficult tasks by solving successive subtasks. The gain provided by these solutions must be relativized with the explosion in the number of parameters they imply, which makes their implementation on embedded systems difficult. New constraints also appear in the choice of the architectures of the branches when one seeks to have a global network providing predictions at different levels. We propose a strategy that allows the merging of heterogeneous CNNs by following a hierarchical approach in which the information extracted by first-level networks can be fed back at any location into second-level networks. Despite the differences in the number and size of the feature maps, such grafting can be done by using clustering, dimension reduction, and interpolation techniques. This strategy eliminates the computational redundancy induced by the recalculation of low-level features. The proposed grafting approach significantly reduces the inference time of the second network without impacting the accuracy. Tests performed on MNIST, CIFAR-10, and PlantVillage datasets with several CNNs illustrate the possibility of implementation in various situations. Our solution allows us to consider in an innovative way the implementation of hierarchical solutions on devices with limited capacities.

Human behaviour analysis based on spatio-temporal dual-stream heterogeneous convolutional neural network

Human behaviour analysis based on spatio-temporal dual-stream heterogeneous convolutional neural network

Cross-Model Deep Feature Fusion for Face Detection

Face detectors using the single shot multibox detector have a small number of discontinuous fixed anchor scales that cannot fully match continuous faces, and these may degrade the accuracy when detecting tiny faces that are not fully matched. To address this issue, we utilize the capabilities and complementary features of the heterogeneous submodel to help the main model extract more abundant features and scale-invariant features from the heterogeneous submodel. First, we propose a deep feature fusion model of the heterogeneous convolutional neural networks to refine and integrate different levels of semantics and obtain a semantically enhanced feature pyramid. We then present a composite anchor loss function that trains the generated network end-to-end, which solves the accuracy-diversity tradeoff of these hybrid submodels. Finally, we apply the proposed approach to the a benchmark for face detection in unconstrained settings (FDDB) and WIDER FACE benchmarks and present the results using their respective standard evaluation protocols. Experiments result in our method surpassing the existing single-shot face detector and further improves the average accuracy of the original model.

Energy-Guided Temporal Segmentation Network for Multimodal Human Action Recognition.

To achieve the satisfactory performance of human action recognition, a central task is to address the sub-action sharing problem, especially in similar action classes. Nevertheless, most existing convolutional neural network (CNN)-based action recognition algorithms uniformly divide video into frames and then randomly select the frames as inputs, ignoring the distinct characteristics among different frames. In recent years, depth videos have been increasingly used for action recognition, but most methods merely focus on the spatial information of the different actions without utilizing temporal information. In order to address these issues, a novel energy-guided temporal segmentation method is proposed here, and a multimodal fusion strategy is employed with the proposed segmentation method to construct an energy-guided temporal segmentation network (EGTSN). Specifically, the EGTSN had two parts: energy-guided video segmentation and a multimodal fusion heterogeneous CNN. The proposed solution was evaluated on a public large-scale NTU RGB+D dataset. Comparisons with state-of-the-art methods demonstrate the effectiveness of the proposed network.

Tensor-Based Emotional Category Classification via Visual Attention-Based Heterogeneous CNN Feature Fusion.

The paper proposes a method of visual attention-based emotion classification through eye gaze analysis. Concretely, tensor-based emotional category classification via visual attention-based heterogeneous convolutional neural network (CNN) feature fusion is proposed. Based on the relationship between human emotions and changes in visual attention with time, the proposed method performs new gaze-based image representation that is suitable for reflecting the characteristics of the changes in visual attention with time. Furthermore, since emotions evoked in humans are closely related to objects in images, our method uses a CNN model to obtain CNN features that can represent their characteristics. For improving the representation ability to the emotional categories, we extract multiple CNN features from our novel gaze-based image representation and enable their fusion by constructing a novel tensor consisting of these CNN features. Thus, this tensor construction realizes the visual attention-based heterogeneous CNN feature fusion. This is the main contribution of this paper. Finally, by applying logistic tensor regression with general tensor discriminant analysis to the newly constructed tensor, the emotional category classification becomes feasible. Since experimental results show that the proposed method enables the emotional category classification with the F1-measure of approximately 0.6, and about 10% improvement can be realized compared to comparative methods including state-of-the-art methods, the effectiveness of the proposed method is verified.

Application of Multi-Column Heterogeneous Convolutional Neural Networks in image classification

Image classification is an important research direction of computer vision. Convolutional neural network is a deep feedforward neural network model. It uses the deep learning idea and shows good performance in multiple image classification fields such as speech recognition, face recognition, motion analysis, and medical diagnosis. However, a single-structure convolutional neural network is prone to overfitting problems. The main reason for the overfitting problem is that the learning model overfits the training set and results in the lack of generalization performance, which affects the feature extraction and judgment of the test set. This paper presents a structure model for Multi-Column Heterogeneous Convolutional Neural Networks. Multi-Column Heterogeneous Convolutional Neural Networks are used in image classification. We construct several convolutional neural networks with different structures by setting different size of convolution kernels and different number of feature maps. Image features are learned from multiple perspectives. Each convolutional neural network model is trained on the training set, and the different network models are fitted to the training set. Finally, through the sliding window, the output of each network is fused to obtain a relatively better prediction result. Experiments show that Multi-Column Heterogeneous Convolutional Neural Networks reduce the overfitting problem to a certain extent, and the accuracy of object recognition is improved compared to the single structure convolutional neural network.

Semi-supervised person re-identification using multi-view clustering

Person Re-Identification (Re-Id) is a challenging task focusing on identifying the same person among disjoint camera views. A number of deep learning algorithms have been reported for this task in fully-supervised fashion which requires a large amount of labeled training data, while obtaining high quality labels for Re-Id is extremely time consuming. To address this problem, we propose a semi-supervised Re-Id framework by using only a small portion of labeled data and some additional unlabeled samples. This paper approaches the problem by constructing a set of heterogeneous Convolutional Neural Networks (CNNs) fine-tuned using the labeled portion, and then propagating the labels to the unlabeled portion for further fine-tuning the overall system. In this work, label estimation is a key component during the propagation process. We propose a novel multi-view clustering method, which integrates features of multiple heterogeneous CNNs to cluster and generate pseudo labels for unlabeled samples. Then we fine-tune each of the multiple heterogeneous CNNs by minimizing an identification loss and a verification loss simultaneously, using training data with both true labels and pseudo labels. The procedure is iterated until the estimation of pseudo labels no longer changes. Extensive experiments on three large-scale person Re-Id datasets demonstrate the effectiveness of the proposed method.