Multi-stream Deep Neural Networks Research Articles

A Multi-Stream Deep Neural Network to Predict the Energy Consumption of Smart Home Appliances

A novel ensemble machine learning approach for predicting energy consumption in smart appliances is presented in this paper. The main objective is minimizing the number of necessary sensors and improving the prediction accuracy at the same time. The proposed method combines the Fuzzy C-Means method with a multi-stream deep neural network to achieve this goal. The method focuses on prediction accuracy and aims to extract the minimum number of essential features from the dataset corresponding to the required sensors. These selected features are then scatter-reduced and homogenized into subsets. Each subset is used to train a cluster-specific deep neural network designed exclusively for that subset. The final prediction is obtained by computing the fuzzy-weighted sum of these cluster-specific network outputs. Numerical results show that the proposed prediction method outperforms conventional methods in terms of root mean square error and mean absolute percentage error criteria, despite using fewer sensors. This improvement can be attributed to the reduced dataset scatter, which improves the learning speed and model performance. Furthermore, the fuzzy combination of the outputs improves the final prediction accuracy. Overall, the proposed approach provides a more cost-effective and accurate solution for predicting the energy consumed by smart appliances.

Soybean seed composition prediction from standing crops using PlanetScope satellite imagery and machine learning

Soybean is a pivotal agricultural commodity around the world, primarily because of its high seed protein and oil concentration. Therefore, farmers, breeders and end-users are highly interested in understanding and predicting the soybean seed composition traits from the individual field level or agroecosystem. Seed composition traits are the proportions of different chemical and physical makeup of soybean seeds. Frequent daily coverage of PlanetScope (PS) satellite provides a unique opportunity of estimating seed composition due to its ability to track crop growth and development with its unique combination of high spatial and temporal resolution. We aim to predict six different soybean seed composition traits (i.e., protein, oil, sucrose, fiber, ash, starch) using PS imagery of standing soybean crops and machine learning algorithms. We developed multi-stream deep neural network which is based on two types of recurrent neural networks, i.e., long short-term memory (LSTM) and gated recurrent unit (GRU) that utilize temporal phenology observed from PS. Four statistical machine learning algorithms, i.e., partial least squares (PLSR), random forest (RFR), gradient boosting machine (GBM), support vector machine (SVR) were used for comparison. Our results show that GRU worked well for protein (R2 0.36, NRMSE 3.62%) and oil (R2 0.53, NRMSE 4.78%), SVR showed the best results for sucrose (R2 0.74, NRMSE 8.34%), fiber (R2 0.21, NRMSE 4.20%), and starch (R2 0.15, NRMSE 16.84%), and PLSR provided the best result for ash (R2 0.60, NRMSE 1.70%). Among the features, vegetation indices at later reproductive stages were found as the most important variables compared to texture features. Overall, the study reveals the feasibility and efficiency of PS images and machine learning for plot-level seed composition estimation.

Multistream-Based Marked Point Process With Decomposed Cumulative Hazard Functions.

When applying a point process to a real-world problem, an appropriate intensity function model should be designed based on physical and mathematical prior knowledge. Recently, a fully trainable deep learning-based approach has been developed for temporal point processes. In this approach, a cumulative hazard function (CHF) capable of systematic computation of adaptive intensity function is modeled in a data-driven manner. However, in this approach, although many applications of point processes generate various kinds of information such as location, magnitude, and depth, the mark information of events is not considered. To overcome this limitation, we propose a fully trainable marked point process method for modeling decomposed CHFs for time and mark prediction using multistream deep neural networks. We demonstrate the effectiveness of the proposed method through experiments with synthetic and real-world event data.

A multi-stream deep neural network with late fuzzy fusion for real-world anomaly detection

Abnormal event detection in video is alternatively known as outlier detection, where machine learning can be highly effective. While testing an unknown video, the objective of such methods is to verify the video’s category, e.g. normal or abnormal. This paper exploits visual information from normal as well as abnormal videos to train a deep multiple instance learning classifier for classification of videos. Existing multiple instance learning classifiers presume that the training videos contain only short-duration anomalous events. This assumption may not be valid for all real-world anomalies. Also, multiple occurrences of anomalies in training videos cannot be ruled out. This paper shows that by injecting temporal information in feature extraction, anomaly detection performance can be improved. To accomplish this, two spatio-temporal deep feature extractors have been applied in parallel on the training videos. These streams are then used to train a modified multiple instance learning-based classifier. Finally, a fuzzy aggregation is employed to fuse the anomaly scores. Additionally, two lightweight deep-learning classifiers have been used to substantiate the model’s efficacy for classifying fire and accident events. To understand the reliability and performance of the proposed method, extensive experiments have been carried out using UCF-Crime video dataset containing 13 anomaly categories. The dataset has been restructured into five broad categories based on the severity of actions to study the robustness of the proposed method. The paper provides sufficient empirical evidence which proves that by incorporating temporal features in the pipeline, anomaly detection accuracy can be significantly improved. Moreover, the model helps to detect long-duration anomalies in videos, which was not possible using existing methods. The proposed end-to-end multi-stream architecture performs abnormal event detection with accuracy as high as 84.48%, which is better than the performance of existing video anomaly detection methods. Moreover, the class-wise detection accuracy has improved by 6%–14% across various broad categories.

Multi-Stream Deep Neural Network for Diabetic Retinopathy Severity Classification Under a Boosting Framework

Diabetic Retinopathy (DR) is an eye disorder in patients with diabetes. Detection of DR presence and its complications using fundus images at an early stage helps prevent its progression to the advanced levels. In the recent years, several well-designed Convolutional Neural Networks (CNN) have been proposed to detect the presence of DR with the help of publicly available datasets. However, these existing CNN-based classifiers focus on utilizing different architectural settings to improve the performance of detection task only i.e. presence or absence of DR. The further classification of the severity and type of the disease, however, remains a non-trivial task. To this end, we propose a multi-stream ensemble deep network to classify diabetic retinopathy severity. The proposed approach takes advantages of the deep networks and principal component analysis (PCA) to learn inter-class and intra-class variations from the raw image features. Ensemble machine learning classifiers are then applied to achieve high classification accuracy and robust performance on the obtained deep features. Specifically, a multi-stream network is made using pre-trained deep learning architectures i.e. ResNet-50 and DenseNet-121 to serve as the main feature extractors. Further application of PCA reduces the dimensionality of features and effectively separates the variation space of inter-class and intra-class images. Finally, an ensemble machine learning classifier using AdaBoost and random forest algorithms is built to further improve classification accuracy. The proposed approach has been compared with multiple conventional CNN-based approaches on Messidor-2 (two categories) and EyePACS (two, five categories) datasets. The experiment results show that our proposed approach achieves superior performance (upto 95.58% accuracy) and can be considered a promising method for automatic diabetic retinopathy detection.

Real-time space occupancy sensing and human motion analysis using deep learning for indoor air quality control

This study proposed a novel indoor air quality control methodology that uses occupancy sensing and motion recognition techniques in combination with human motion analysis. The automated occupancy sensing systems that are most prevalent today analyze either environmental conditions (i.e., room temperature or data from entities in the room, such as states of electronic devices) to make human occupancy predictions. Since little emphasis is placed on observing humans directly, the estimations of these sensing systems are often inaccurate. Erroneous occupancy estimation leads to poor control of building resources such as HVAC (Heating, Ventilation, and Air-conditioning) and lighting systems. To address this issue, the paper puts forth a lean, vision-based system that estimates the number of occupants and recognizes their activities using the stacked history of unconstrained non-deterministic human movements over transient intervals. The system implements a multi-stream deep neural network to identify human activities and uses the YOLO V3 deep neural network for object detection to estimate occupancy count in a room. The study uses a publicly available action recognition dataset – NADA – to train the neural networks and experiment with a variety of video classification techniques to achieve higher accuracies.

Deep Learning-Based Computational Color Constancy With Convoluted Mixture of Deep Experts (CMoDE) Fusion Technique

In the human and computer vision, color constancy is the ability to perceive the true color of objects in spite of changing illumination conditions. Color constancy is remarkably benefitting human and computer vision issues such as human tracking, object and human detection and scene understanding. Traditional color constancy approaches based on the gray world assumption fall short of performing a universal predictor, but recent color constancy methods have greatly progressed with the introduction of convolutional neural networks (CNNs). Yet, shallow CNN-based methods face learning capability limitations. Accordingly, this article proposes a novel color constancy method that uses a multi-stream deep neural network (MSDNN)-based convoluted mixture of deep experts (CMoDE) fusion technique in performing deep learning and estimating local illumination. In the proposed method, the CMoDE fusion technique is used to extract and learn spatial and spectral features in an image space. The proposed method distinctively piles up layers both in series and in parallel, selects and concatenates effective paths in the CMoDE-based DCNN, as opposed to previous works where residual networks stack multiple layers linearly and concatenate multiple paths. As a result, the proposed CMoDE-based DCNN brings significant progress towards efficiency of using computing resources, as well as accuracy of estimating illuminants. In the experiments, Shi's Reprocessed, gray-ball and NUS-8 Camera datasets are used to prove illumination and camera invariants. The experimental results establish that this new method surpasses its conventional counterparts.

Multi-Stream Deep Neural Networks for RGB-D Egocentric Action Recognition

In this paper, we investigate the problem of RGB-D egocentric action recognition. Unlike conventional human action videos that are passively recorded by static cameras, egocentric videos are self-generated from wearable sensors that are more flexible and provide the close-ups with the visual attention of the wearers when they act. Moreover, RGB-D videos contain the spatial appearance and temporal information in the RGB modality and reflect the 3D structure of the scenes in the depth modality. To adequately learn the nonlinear structure of heterogeneous representations from different modalities and exploit their complementary characteristics, we develop a multi-stream deep neural networks (MDNN) method, which aims to preserve the distinctive property for each modality and simultaneously explore their sharable information in a unified deep architecture. Specifically, we deploy a Cauchy estimator to maximize the correlations of the sharable components and enforce the orthogonality constraints on the distinctive components to guarantee their high independencies. Since the egocentric action recognition is usually sensitive to hand poses, we extend our MDNN by integrating with the hand cues to enhance the recognition accuracy. Extensive experimental results on a newly collected data set and two additional benchmarks are presented to demonstrate the effectiveness of our proposed method for RGB-D egocentric action recognition.

Joint Dynamic Pose Image and Space Time Reversal for Human Action Recognition from Videos

Human action recognition aims to classify a given video according to which type of action it contains. Disturbance brought by clutter background and unrelated motions makes the task challenging for video frame-based methods. To solve this problem, this paper takes advantage of pose estimation to enhance the performances of video frame features. First, we present a pose feature called dynamic pose image (DPI), which describes human action as the aggregation of a sequence of joint estimation maps. Different from traditional pose features using sole joints, DPI suffers less from disturbance and provides richer information about human body shape and movements. Second, we present attention-based dynamic texture images (att-DTIs) as pose-guided video frame feature. Specifically, a video is treated as a space-time volume, and DTIs are obtained by observing the volume from different views. To alleviate the effect of disturbance on DTIs, we accumulate joint estimation maps as attention map, and extend DTIs to attention-based DTIs (att-DTIs). Finally, we fuse DPI and att-DTIs with multi-stream deep neural networks and late fusion scheme for action recognition. Experiments on NTU RGB+D, UTD-MHAD, and Penn-Action datasets show the effectiveness of DPI and att-DTIs, as well as the complementary property between them.

Fusing multi-stream deep neural networks for facial expression recognition

Among the factors contributing to conveying emotional state of an individual is facial expression. It represents the most important nonverbal communication and a challenging task in the field of computer vision. In this work, we propose a combined deep architecture model for facial expression recognition that uses appearance and geometric features extracted separately using convolution layers and supervised decent method, respectively. The proposed model is trained on three public databases [the Extended Cohn Kanade (CK+), the OULU-CASIA VIS, and the JAFFE]. The three databases contain a limited amount of data that we enlarge by adding a step of data augmentation to original images. For further comparison, two additional models that use appearance features only and geometric features only are trained on the same subset of data, to show how the combination of the two deep architectures influences results. On the other hand, in order to investigate the generalization of the combined model, a cross-database evaluation is performed. The obtained results achieve the state-of-the-art and improve recent work, especially in case of cross-database evaluation.

DAAL: Deep activation-based attribute learning for action recognition in depth videos

In this paper, we propose a joint semantic preserving action attribute learning framework for action recognition from depth videos, which is built on multi-stream deep neural networks. More specifically, this paper describes the idea to explore action attributes learned from deep activations. Multiple stream deep neural networks rather than conventional hand-crafted low-level features are employed to learn the deep activations. An undirected graph is utilized to model the complex semantics among action attributes and is integrated into our proposed joint action attribute learning algorithm. Experiments on several public datasets for action recognition demonstrate that 1) the deep activations achieve the state-of-the-art discriminative performance as feature vectors and 2) the attribute learner can produce generic attributes, and thus obtains decent performance on zero-shot action recognition.