Simple Deep Neural Network Research Articles

Automated detection of contractual risk clauses from construction specifications using bidirectional encoder representations from transformers (BERT)

Detecting contractual risk information from construction specifications is crucial to succeeding in construction projects. This paper describes clause classification using the Bidirectional Encoder Representations from Transformers (BERT) method in natural language processing. Seven risk categories are determined from a literature review, including payment, temporal, procedure, safety, role and responsibility, definition, and reference. Using 2807 clauses from 56 construction specifications, the BERT-based clause classification model returns noticeable performances with 0.889 accuracy for validation and a 0.934 F1 score on testing. The model is evaluated by comparing the clause classification performance with other machine learning methods, including the support vector machine and a simple deep neural network, and shows dominant performance on every risk category. Practitioners in the construction industry are the primary beneficiaries of the research as the model will contribute to improving the construction specification review process and risk management during construction projects.

Gammatone Filter Bank-Deep Neural Network-based Monaural speech enhancement for unseen conditions

Speech signal enhancement achieves high-level performance in recent years using deep learning techniques. However, the deep learning technique in the speech enhancement algorithm degrades the performance of speech, particularly for unseen noises, unseen speakers and moreover, deep learning models are limited to the small number of speakers. Hence, we propose a Gammatone filterbank (GTFB) – simple deep neural network (SDNN) based speech enhancement algorithm to improve the quality of speech for three different unseen conditions. The use of GTFB gives a finer resolution in low-frequency regions of speech, and the SDNN model extracts a noisy GTFB frame as input and maps it to a clean speech GTFB frame. The experimental results are measured objectively using signal-noise-ratio, perceptual evaluation of speech quality, short time objective intelligibility, and subjectively using mean opinion score. The experimental results are carried out using a variety of training and testing models. The performance results show that the proposed GTFB-SDNN are robust to a variety of test situations and outperform existing methods.

Machine Learning for Touch Localization on an Ultrasonic Lamb Wave Touchscreen.

Classification and regression employing a simple Deep Neural Network (DNN) are investigated to perform touch localization on a tactile surface using ultrasonic guided waves. A robotic finger first simulates the touch action and captures the data to train a model. The model is then validated with data from experiments conducted with human fingers. The localization root mean square errors (RMSE) in time and frequency domains are presented. The proposed method provides satisfactory localization results for most human–machine interactions, with a mean error of 0.47 cm and standard deviation of 0.18 cm and a computing time of 0.44 ms. The classification approach is also adapted to identify touches on an access control keypad layout, which leads to an accuracy of 97% with a computing time of 0.28 ms. These results demonstrate that DNN-based methods are a viable alternative to signal processing-based approaches for accurate and robust touch localization using ultrasonic guided waves.

Beyond Supervised Power Control in Massive MIMO Network: Simple Deep Neural Network Solutions

Power control in massive multiple input multiple output (MIMO) systems is an appealing technique to improve network performance and reliability. The traditional methods to solve such problems are based on the convex optimization theory, which incurs high computational complexity. In contrast, this work leverages deep neural networks to maximize the minimum data rate of the downlink users in the massive MIMO network. Based on the different setups in the practical systems, namely <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in-distribution case</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">weakly out-of-distribution case</i> , and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">out-of-distribution case</i> , we propose three approaches to solve them. Specifically, first, we establish a densely connected neural network for solving the power control problem. Then, for the case where the test data and training data come from the same distributions (the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in-distribution case</i> ), we use the established neural network to obtain an approximately optimal power control strategy, and propose a self-training-based algorithm to use the unlabeled samples collected in the actual system to further improve the system performance. For the case that the test data and training data come from different distributions and a small amount of labeled target domain data can be obtained (the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">weakly out-of-distribution case</i> ), we propose a pre-training and fine-tuning algorithm to solve the problem. For the issue that we can not obtain the labels of the target domain data (the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">out-of-distribution case</i> ), we transform the problem into a covariate shift problem and propose an algorithm by weighting the loss function. Numerical results demonstrate that the proposed schemes not only match the optimal solution well but also have low online inference complexity.

Detection of feature-poor, small sized objects on noisy images

Abstract Visual identification of objects is an important challenge today. Main target of frequently applied methods is to identify or classify complex objects. These methods are far less effective when objects are small and less complex, and thus less descriptor features are on hand. The main reason for this is that these features can significantly change on object occlusion or appearance of noise. The presented solution performs identification of simple, small (size is 17 × 13 pixels) objects with elliptical shape. High pass filtered normalized cross correlation is used for region of interest detection and a simple deep neural network is used for classification of selected regions. The proposed method detected objects on a noisy image with accuracy of 96.2%.

Calib-Net: Calibrating the Low-Cost IMU via Deep Convolutional Neural Network.

The low-cost Inertial Measurement Unit (IMU) can provide orientation information and is widely used in our daily life. However, IMUs with bad calibration will provide inaccurate angular velocity and lead to rapid drift of integral orientation in a short time. In this paper, we present the Calib-Net which can achieve the accurate calibration of low-cost IMU via a simple deep convolutional neural network. Following a carefully designed mathematical calibration model, Calib-Net can output compensation components for gyroscope measurements dynamically. Dilation convolution is adopted in Calib-Net for spatio-temporal feature extraction of IMU measurements. We evaluate our proposed system on public datasets quantitively and qualitatively. The experimental results demonstrate that our Calib-Net achieves better calibration performance than other methods, what is more, and the estimated orientation with our Calib-Net is even comparable with the results from visual inertial odometry (VIO) systems.

Analysis of Soft Decision Trees for Passive-Expert Reinforcement Learning

This paper explores the use of soft decision trees [1] in basic reinforcement applications to examine the efficacy of using passive-expert like networks for optimal Q-Value learning on Artificial Neural Networks (ANN). The soft decision tree networks were built using the PyTorch machine learning and the OpenAi’s Gym environment frameworks. The conducted research study aimed at assessing the performance of soft decision tree networks on Cartpole as provided in the OpenAi Gym software package. The baseline performance metric that the soft decision tree networks were compared against was a simple Deep Neural Network using several linear layers with ReLU and Softmax activation functions for the input and output layers, respectively. All networks were trained using the Backpropagation algorithm provided generically by PyTorch’sAutograd module.

A Multifeatured Data-Driven Homogenization for Heterogeneous Elastic Solids

A computational homogenization of heterogeneous solids is presented based on the data-driven approach for both linear and nonlinear elastic responses. Within the Double-Scale Finite Element Method (FE2) framework, a data-driven model is proposed to substitute the micro-level Finite Element (FE) simulations to reduce computational costs in multiscale simulations. The heterogeneity of porous solids at the micro-level is considered in various material properties and geometrical attributes. For material properties, elastic constants, which are Lame’s coefficients, are subjected to be heterogeneous in the linear elastic responses. For geometrical features, different numbers, sizes, and locations of voids are considered to reflect the heterogeneity of porous solids. A database for homogenized microstructural responses is constructed from a series of micro-level FE simulations, and machine learning is used to train and test our proposed model. In particular, four geometrical descriptors are designed, based on N-probability and lineal-path functions, to clearly reflect the geometrical heterogeneity of various microstructures. This study indicates that a simple deep neural networks model can capture diverse microstructural heterogeneous responses well when given proper input sources, including the geometrical descriptors, are considered to establish a computational data-driven homogenization scheme.

Visual Smoke Detection Based on Ensemble Deep CNNs

Smoke usually refers to a visible aerosol produced by fuel combustion, sometimes commingled with carbon and sulfur particles because of incomplete burning. The long-term accumulation of smoke aerosols is an important factor in the formation of haze, which is an environmental concern in many countries. Developing methods of intelligent smoke detection in the air would be greatly beneficial for such applications as industrial safety monitoring and prevention of air pollution for the purpose of wholly replacing manual processes. In this paper, we concentrate on resolving this difficulty. Unlike existing deep learning frameworks which lack generalization ability because they were developed for specific data instances, our proposed model aggregates simple deep convolutional neural networks but attains excellent performance. In order to capture the different aspects of smoke, we define a set of feature maps that are fed to a set of subnetworks. Each subnetwork is independently trained to deliver good detection performance. A final output is obtained by selectively aggregating the subnetwork responses via majority voting. In the experiment we conducted on two newly established noisy smoke image datasets corrupted by compression, our proposed model achieves a very high consistency beyond 97% on average between detection results and human judgements, outperforming other state-of-the-art smoke detection algorithms based on deep learning.

Cloud Computing Virtual Machine Workload Prediction Method Based on Variational Autoencoder

The paper proposes a method for predicting the workload of virtual machines in the cloud infrastructure. Reconstruction probabilities of variational autoencoders were used to provide the prediction. Reconstruction probability is a probability criterion that considers the variability in the distribution of variables. In the proposed approach, the values of the reconstruction probabilities of the variational autoencoder show the workload level of the virtual machines. The results of the experiments showed that variational autoencoders gave better results in predicting the workload of virtual machines compared to simple deep neural networks. The generative characteristics of the variational autoencoders determine the workload level by the data reconstruction.

Towards Extracting Graph Neural Network Models via Prediction Queries (Student Abstract)

Graph data has been widely used to represent data from various domain, e.g., social networks, recommendation system. With great power, the GNN models, usually as valuable properties of their owners, also become attractive targets of the adversary who covets to steal them. While existing works show that simple deep neural networks can be reproduced by so-called Model Extraction Attacks, how to extract a GNN model has not been explored. In this paper, we exploit the threat of model extraction attacks against GNN models. Unlike ordinary attacks which obtain model information via only the input-output query pairs, we utilize both the node queries and the graph structure to extract the GNNs. Furthermore, we consider the stealthiness of the attack and propose to generate legitimate queries so the extraction can be applied discreetly. We implement our attack by leveraging the responses of these queries, as well as other accessible knowledge, e.g., neighbor connectives of the queried nodes. By evaluating over three real-world datasets, our attack is shown to effectively produce a surrogate model with more than 80% equivalent predictions as the target model.

Automatic Extraction of Tree Crown for the Estimation of Biomass from UAV Imagery Using Neural Networks

The biomass content of vegetation has great significance for efficient balancing of biomass cycle. Estimation of biomass in vegetation requires quantification of tree canopy. Automatic extraction of tree canopy helps in faster and efficient estimation of the biomass content of a vegetative area in comparison to manual mapping methods. An agricultural farm at Papparapati Village in Dharmapuri, District, Tamil Nadu, with an Area of 1.2 sq.km was chosen as study area and multispectral UAV imagery at 4-cm spatial resolution with 4 Bands viz., NIR, Green, Red Edge, RED was acquired. In this study, an attempt has been made to automatically extract tree canopy using simple deep learning convolution neural network (CNN) using training sites as input from the band spectral information. The extracted tree canopy was refined using linear cluster algorithm to delineate the exact tree crown. The biomass of the tree is estimated using the tree properties (crown diameter, height) derived from UAV data and using the allometric equations. A field survey was conducted to validate the classification results achieved from UAV data of individual sample trees. It is observed that the accuracy of extracted tree properties was in agreement with the field measurements of sample trees.

Deep Neural Learning Adaptive Sequential Monte Carlo for Automatic Image and Speech Recognition

To enhance the performance of image classification and speech recognition, the optimizer is considered an important factor for achieving high accuracy. The state-of-the-art optimizer can perform to serve in applications that may not require very high accuracy, yet the demand for high-precision image classification and speech recognition is increasing. This study implements an adaptive method for applying the particle filter technique with a gradient descent optimizer to improve model learning performance. Using a pretrained model helps reduce the computational time to deploy an image classification model and uses a simple deep convolutional neural network for speech recognition. The applied method results in a higher speech recognition accuracy score—89.693% for the test dataset—than the conventional method, which reaches 89.325%. The applied method also performs well on the image classification task, reaching an accuracy of 89.860% on the test dataset, better than the conventional method, which has an accuracy of 89.644%. Despite a slight difference in accuracy, the applied optimizer performs well in this dataset overall.

Assessment of COVID-19 in lung ultrasound by combining anatomy and sonographic artifacts using deep learning

When assessing severity of COVID19 from lung ultrasound (LUS) frames, both anatomical phenomena (e.g., the pleural line, presence of consolidations), as well as sonographic artifacts, such as A-lines and B-lines are of importance. While ultrasound devices aim to provide an accurate visualization of the anatomy, the orientation of the sonographic artifacts differ between probe types. This difference poses a challenge in designing a unified deep artificial neural network capable of handling all probe types. In this work we improve upon Roy et al. (2020): We train a simple deep neural network to assess the severity of COVID-19 from LUS data. To address the challenge of handling both linear and convex probes in a unified manner we employed two strategies: First, we augment the input frames of convex probes with a “rectified” version in which A-lines and B-lines assume a horizontal/vertical aspect close to that achieved with linear probes. Second, we explicitly inform the network on the presence of important anatomical features and artifacts. We use a known Radon-based method for detecting the pleural line and B-lines and feed the detected lines as inputs to the network. Preliminary experiments yielded f1 = 68.7% compared to f1 = 65.1% reported by Roy et al.

Deep convolutional neural network for hand sign language recognition using model E

An image processing system that based computer vision has received many attentions from science and technology expert. Research on image processing is needed in the development of human-computer interactions such as hand recognition or gesture recognition for people with hearing impairments and deaf people. In this research we try to collect the hand gesture data and used a simple deep neural network architecture that we called model E to recognize the actual hand gestured. The dataset that we used is collected from kaggle.com and in the form of ASL (American Sign Language) datasets. We doing accuracy comparison with another existing model such as AlexNet to see how robust our model. We find that by adjusting kernel size and number of epoch for each model also give a different result. After comparing with AlexNet model we find that our model E is perform better with 96.82% accuracy.

Short-Term Electricity Demand Forecasting: Impact Analysis of Temperature for Thailand

Accurate electricity demand forecasting for a short horizon is very important for day-to-day control, scheduling, operation, planning, and stability of the power system. The main factors that affect the forecasting accuracy are deterministic variables and weather variables such as types of days and temperature. Due to the tropical climate of Thailand, the marginal impact of weather variables on electricity demand is worth analyzing. Therefore, this paper primarily focuses on the impact of temperature and other deterministic variables on Thai electricity demand. Accuracy improvement is also considered during model design. Based on the characteristics of demand, the overall dataset is divided into four different subgroups and models are developed for each subgroup. The regression models are estimated using Ordinary Least Square (OLS) methods for uncorrelated errors, and General Least Square (GLS) methods for correlated errors, respectively. While Feed Forward Artificial Neural Network (FF-ANN) as a simple Deep Neural Network (DNN) is estimated to compare the accuracy with regression methods, several experiments conducted for determination of training length, selection of variables, and the number of neurons show some major findings. The first finding is that regression methods can have better forecasting accuracy than FF-ANN for Thailand’s dataset. Unlike much existing literature, the temperature effect on Thai electricity demand is very interesting because of their linear relationship. The marginal impacts of temperature on electricity demand are also maximal at night hours. The maximum impact of temperature during night hours happens at 11 p.m., is 300 MW/ ° C, about 4 % rise in demand while during day hours, the temperature impact is only 10 MW/ ° C to 200 MW/ ° C about 1.4 % to 2.6 % rise.

Decision making methods for formulating the reserve scheme of spare parts based on deep neural network

Determination of the variety and quantity of Spares is the primary step to guarantee the spares supply. Firstly, this paper analyzes the factors affecting the reserve scheme of Spares. Then by analyzing the inherent five attributes of Spares, two methods are proposed to determine the variety and quantity of Spares based on deep neural network. The first method ranks Spares according to their importance. A relatively simple deep neural network is used to analyze every attribute of the Spares in turn. The second method inputs all the attributes of Spares into a relatively complex deep neural network to make the decision. The experimental results show the advantages of the two methods in terms of efficiency and accuracy for formulating the reserve scheme of Spares.

Deep Learning With EEG Spectrograms in Rapid Eye Movement Behavior Disorder.

REM Behavior Disorder (RBD) is now recognized as the prodromal stage of α-synucleinopathies such as Parkinson's disease (PD). In this paper, we describe deep learning models for diagnosis/prognosis derived from a few minutes of eyes-closed resting electroencephalography data (EEG) collected at baseline from idiopathic RBD patients (n = 121) and healthy controls (HC, n = 91). A few years after the EEG acquisition (4±2 years), a subset of the RBD patients were eventually diagnosed with either PD (n = 14) or Dementia with Lewy bodies (DLB, n = 13), while the rest remained idiopathic RBD. We describe first a simple deep convolutional neural network (DCNN) with a five-layer architecture combining filtering and pooling, which we train using stacked multi-channel EEG spectrograms from idiopathic patients and healthy controls. We treat the data as in audio or image classification problems where deep networks have proven successful by exploiting invariances and compositional features in the data. For comparison, we study a simple deep recurrent neural network (RNN) model using three stacked Long Short Term Memory network (LSTM) cells or gated-recurrent unit (GRU) cells—with very similar results. The performance of these networks typically reaches 80% (±1%) classification accuracy in the balanced HC vs. PD-conversion classification problem. In particular, using data from the best single EEG channel, we obtain an area under the curve (AUC) of 87% (±1%)—while avoiding spectral feature selection. The trained classifier can also be used to generate synthetic spectrograms using the DeepDream algorithm to study what time-frequency features are relevant for classification. We find these to be bursts in the theta band together with a decrease of bursting in the alpha band in future RBD converters (i.e., converting to PD or DLB in the follow up) relative to HCs. From this first study, we conclude that deep networks may provide a useful tool for the analysis of EEG dynamics even from relatively small datasets, offering physiological insights and enabling the identification of clinically relevant biomarkers.

Using Deep Neural Networks to Reconstruct Non-uniformly Sampled NMR Spectra

Non-uniform and sparse sampling of multi-dimensional NMR spectra has over the last decade become an important tool to allow for fast acquisition of multi-dimensional NMR spectra with high resolution. The success of non-uniform sampling NMR hinge on both the development of algorithms to accurately reconstruct the sparsely sampled spectra and the design of sampling schedules that maximise the information contained in the sampled data. Traditionally, the reconstruction tools and algorithms have aimed at reconstructing the full spectrum and thus ‘fill out the missing points’ in the time-domain spectrum, although other techniques are based on multi-dimensional decomposition and extraction of multi-dimensional shapes. Also over the last decade, machine learning, deep neural networks, and artificial intelligence have seen new applications in an enormous range of sciences, including analysis of MRI spectra. As a proof-of-principle, it is shown here that simple deep neural networks can be trained to reconstruct sparsely sampled NMR spectra. For the reconstruction of two-dimensional NMR spectra, reconstruction using a deep neural network performs as well, if not better than, the currently and widely used techniques. It is therefore anticipated that deep neural networks provide a very valuable tool for the reconstruction of sparsely sampled NMR spectra in the future to come.

Feasibility Study of Deep Neural Network for Heart Rate Estimation from Wearable Photoplethysmography and Acceleration Signals.

Heart rate (HR) estimation using wearable reflectance-type photoplethysmographic (PPG) signals is challenging due to low signal-to-noise ratio (SNR). Especially during intensive exercise, motion artifacts (MAs) overwhelm PPG signals in an unpredictable way. To overcome the issue, an acceleration signal as a reference signal has been adopted by simultaneously measuring with PPG signal. However, MAs are frequently uncorrelated with accelerometer signals under various activities. In this paper, we present a learning-based framework for HR estimation. The proposed framework is based on the deep neural network (DNN). For the feasibility study, we presented a simple network with two fully connected layers. We first extracted power spectra from the measured PPG signal and the acceleration signal. The two power spectra were then used for the input layer in the network. In addition, to inform the PPG signal quality, we added the acceleration signal intensity for the input layer. The proposed simple DNN network was trained for 10 epochs in IEEE Signal Processing Cup 2015 (ISPC) dataset (n=23). Subsequently, the trained network provided low mean absolute error (MAE) of 2.31 bpm in the ISPC dataset. We further tested the network on the new BAMI dataset (n=5), and found that it provided 4.72 bpm of MAE. On the other hand, the MAE without the learning frame was 15.73 bpm. In this study, we found that the simple DNN technique is effective. More training issues were also discussed.