Number Of Output Neurons Research Articles

Scheduling for Minimizing the Age of Information in Multisensor Multiserver Industrial Internet of Things Systems

Real-time data delivery is significant for the Industrial Internet of Things (IIoT). Age of information (AoI), a popular real-time metric, is usually used to measure the data freshness of the IIoT systems. If the data most recently received by the destination at time <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$t$</tex-math></inline-formula> was generated at time <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$t_{1}$</tex-math></inline-formula> , then the AoI is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$t-t_{1}$</tex-math></inline-formula> . In this paper, we consider a multi-sensor multi-server IIoT system and develop scheduling algorithms to minimize the average AoI. The challenge lies in the strong coupling between link scheduling, server selection, and service preemption. To address this issue, we propose a guided exploration-based deep Q-Network (GE-DQN) algorithm utilizing a fixed advantage policy, which has a faster learning speed compared to classical deep Q-Network. Moreover, we use a shared decision module followed by several network branches to transform the structure of GE-DQN and propose a guided exploration-based Branching Dueling Q-Network (GE-BDQN) algorithm. Since the branch structure of GE-BDQN can decompose the high-dimensional action, GE-BDQN can reduce the approximate exponential growth of the number of output neurons with the increase of the number of sensors to linear growth compared to GE-DQN, ensuring the applicability of the algorithm under large-scale systems. From the simulation results, it can be found that the proposed two algorithms can achieve better average AoI compared to the advanced algorithms, and the GE-BDQN algorithm can achieve up to 36% performance gain.

Comparative Analysis of the Clustering Quality in Self-Organizing Maps for Human Posture Classification.

The objective of this article is to develop a methodology for selecting the appropriate number of clusters to group and identify human postures using neural networks with unsupervised self-organizing maps. Although unsupervised clustering algorithms have proven effective in recognizing human postures, many works are limited to testing which data are correctly or incorrectly recognized. They often neglect the task of selecting the appropriate number of groups (where the number of clusters corresponds to the number of output neurons, i.e., the number of postures) using clustering quality assessments. The use of quality scores to determine the number of clusters frees the expert to make subjective decisions about the number of postures, enabling the use of unsupervised learning. Due to high dimensionality and data variability, expert decisions (referred to as data labeling) can be difficult and time-consuming. In our case, there is no manual labeling step. We introduce a new clustering quality score: the discriminant score (DS). We describe the process of selecting the most suitable number of postures using human activity records captured by RGB-D cameras. Comparative studies on the usefulness of popular clustering quality scores-such as the silhouette coefficient, Dunn index, Calinski-Harabasz index, Davies-Bouldin index, and DS-for posture classification tasks are presented, along with graphical illustrations of the results produced by DS. The findings show that DS offers good quality in posture recognition, effectively following postural transitions and similarities.

A novel multistage damage detection method for trusses using time-history data based on model order reduction and deep neural network

This article first proposes a multistage damage identification approach for trusses using time-series data relied upon model order reduction (MOR) and deep neural network (DNN). In the first step, an acceleration–displacement-based strain energy indicator (ADSEI), which is computed from the acceleration and displacement data incompletely measured at limited sensors and unmeasured ones inferred from a second-order Neumann series expansion (SNSE)-relied MOR technique, is utilized to eliminate low-risk damage candidates, aiming to only keep high-potential ones. This can dramatically reduce the number of output neurons of the DNN model in the second step. The input data employed to build such a DNN are the finite element method (FEM)-simulated acceleration and displacement signals corresponding to measured degrees of freedom (DOFs). Low-risk flawed candidates predicted by this DNN are then excluded via a suggested damage threshold. By repeating such a manner, the accuracy of the DNN models constructed in the subsequent stages is therefore enhanced continuously, although these upgraded DNNs only require a moderate dataset, simple architectures, and much less computational cost for their training and testing processes. Accordingly, both the location and severity of damaged members can be reliably and precisely diagnosed by time-series data measured in a fairly short interval at a few sensors, even with high noises. Several numerical examples of 3D trusses are tested to affirm the efficiency and feasibility of the current approach.

Stable recovery of entangled weights: Towards robust identification of deep neural networks from minimal samples

In this paper we approach the problem of unique and stable identifiability from a finite number of input-output samples of generic feedforward deep artificial neural networks of prescribed architecture with pyramidal shape up to the penultimate layer and smooth activation functions. More specifically we introduce the so-called entangled weights, which compose weights of successive layers intertwined with suitable diagonal and invertible matrices depending on the activation functions and their shifts. We prove that instances of entangled weights are completely and stably approximated by an efficient and robust algorithm as soon as O(D2×m) nonadaptive input-output samples of the network are collected, where D is the input dimension and m is the number of neurons of the network. Moreover, we empirically observe that the approach applies to networks with up to O(D×mL) neurons, where mL is the number of output neurons at layer L. Provided knowledge of layer assignments of entangled weights and of remaining scaling and shift parameters, which may be further heuristically obtained by least squares fitting, the entangled weights identify the network completely and uniquely. To highlight the relevance of the theoretical result of stable recovery of entangled weights, we present numerical experiments, which demonstrate that multilayered networks with generic weights can be robustly identified and therefore uniformly approximated by the presented algorithmic pipeline.

Fuzzy adaptive learning control network (FALCN) for image clustering and content-based image retrieval on noisy dataset

&lt;abstract&gt; &lt;p&gt;It has been demonstrated that fuzzy systems are beneficial for classification and regression. However, they have been mainly utilized in controlled settings. An image clustering technique essential for content-based picture retrieval in big image datasets is developed using the contents of color, texture and shape. Currently, it is challenging to label a huge number of photos. The issue of unlabeled data has been addressed. Unsupervised learning is used. K-means is the most often used unsupervised learning algorithm. In comparison to fuzzy c-means clustering, K-means clustering has lower-dimensional space resilience and initialization resistance. The dominating triple HSV space was shown to be a perceptual color space made of three modules, S (saturation), H (hue) and V (value), referring to color qualities that are significantly connected to how human eyes perceive colors. A deep learning technique for segmentation (RBNN) is built on the Gaussian function, fuzzy adaptive learning control network (FALCN), clustering and the radial basis neural network. The segmented image and critical information are fed into a radial basis neural network classifier. The suggested fuzzy adaptive learning control network (FALCN) fuzzy system, also known as the unsupervised fuzzy neural network, is very good at clustering images and can extract image properties. When a conventional fuzzy network system receives a noisy input, the number of output neurons grows needlessly. Finally, random convolutional weights extract features from data without labels. Furthermore, the state-of-the-art uniting the proposed FALCN with the RBNN classifier, the proposed descriptor also achieves comparable performance, such as improved accuracy is 96.547 and reduced mean squared error of 36.028 values for the JAFE, ORL, and UMIT datasets.&lt;/p&gt; &lt;/abstract&gt;

A novel integrated BPNN/SNN artificial neural network for predicting the mechanical performance of green fibers for better composite manufacturing

Since the mechanical properties of green cellulosic fibers are only determined experimentally with high diversity, introducing prediction methods for such intrinsic properties of fibers based upon chemical composition is of paramount importance. In this work, double integrated artificial neural networks are utilized to perform novel predictions and classifications of the intrinsic mechanical properties of natural fibers to enhance their implementations in green bio-composites. The predictions and classifications were based upon the green fibers’ chemical composition. The developed model was built up utilizing experimental data for mechanical properties of various cellulosic fiber types commonly used in natural fiber reinforced composites. The first stage back propagation neural network (BPNN) was developed to predict the intrinsic mechanical properties, while the second stage was used to classify the natural fibers based upon shallow neural networks (SNN). The second layer of the model was built based upon shallow neural networks that was capable of classifying the natural fibers. The SNN model was built with 20 hidden layer size and the number of output neurons was set to 11, which equals to the number of fibers classes specified by the dataset. The developed model was capable of predicting all of Young’s modulus, ultimate tensile strength, and elongation at break properties from only two intrinsic properties of fibers; cellulose and moisture content. The overall percentage for the correct classifications was 95.6%, which had demonstrated that machine learning offers a robust approach to classify the fibers based on their intrinsic mechanical properties. The presented model would dramatically help designers and engineers to enhance their selections of various cellulosic fibers for better reliable green composite material performance.

Machine Learning Method for Functional Assessment of Retinal Models.

Challenges in the field of retinal prostheses motivate the development of retinal models to accurately simulate Retinal Ganglion Cells (RGCs) responses. The goal of retinal prostheses is to enable blind individuals to solve complex, reallife visual tasks. In this paper, we introduce the functional assessment (FA) of retinal models, which describes the concept of evaluating the performance of retinal models on visual understanding tasks. We present a machine learning method for FA: we feed traditional machine learning classifiers with RGC responses generated by retinal models, to solve object and digit recognition tasks (CIFAR-10, MNIST, Fashion MNIST, Imagenette). We examined critical FA aspects, including how the performance of FA depends on the task, how to optimally feed RGC responses to the classifiers and how the number of output neurons correlates with the model's accuracy. To increase the number of output neurons, we manipulated input images - by splitting and then feeding them to the retinal model and we found that image splitting does not significantly improve the model's accuracy. We also show that differences in the structure of datasets result in largely divergent performance of the retinal model (MNIST and Fashion MNIST exceeded 80% accuracy, while CIFAR-10 and Imagenette achieved ∼40%). Furthermore, retinal models which perform better in standard evaluation, i.e. more accurately predict RGC response, perform better in FA as well. However, unlike standard evaluation, FA results can be straightforwardly interpreted in the context of comparing the quality of visual perception.

Managing Distributed Flexibility Under Uncertainty by Combining Deep Learning With Duality

In modern power systems, small distributed energy resources (DERs) are considered a valuable source of flexibility towards accommodating high penetration of Renewable Energy Sources (RES). In this paper we consider an economic dispatch problem for a community of DERs, where energy management decisions are made online and under uncertainty. We model multiple sources of uncertainty such as RES, wholesale electricity prices as well as the arrival times and energy needs of a set of Electric Vehicles. The economic dispatch problem is formulated as a multi-agent Markov Decision Process. The difficulties lie in the curse of dimensionality and in guaranteeing the satisfaction of constraints under uncertainty. A novel method, that combines duality theory and deep learning, is proposed to tackle these challenges. In particular, a Neural Network (NN) is trained to return the optimal dual variables of the economic dispatch problem. By training the NN on the dual problem instead of the primal, the number of output neurons is dramatically reduced, which enhances the performance and reliability of the NN. Finally, by treating the resulting dual variables as prices, each distributed agent can self-schedule, which guarantees the satisfaction of its constraints. As a result, our simulations show that the proposed scheme performs reliably and efficiently.

Photonic Reservoir Computer with Output Expansion for Unsupervized Parameter Drift Compensation.

We present a method to improve the performance of a reservoir computer by keeping the reservoir fixed and increasing the number of output neurons. The additional neurons are nonlinear functions, typically chosen randomly, of the reservoir neurons. We demonstrate the interest of this expanded output layer on an experimental opto-electronic system subject to slow parameter drift which results in loss of performance. We can partially recover the lost performance by using the output layer expansion. The proposed scheme allows for a trade-off between performance gains and system complexity.

Modeling of Correlated Complex Sea Clutter Using Unsupervised Phase Retrieval

The spatially and temporally correlated sea clutter with phase information is valuable for marine radar applications. The major difficulty of coherent sea clutter modeling is the generation of the continuous phases. This article presents a new phase retrieval approach for modeling the correlated complex sea clutter based on unsupervised neural networks. The unsupervised short-term and long-term neural networks have been developed for the phase retrieval on different term scales. Both these networks have the same input layer and feature extraction module, and however, the number of output neurons is different. The amplitude sea clutter series and the desired Doppler spectrum are fed into the network in parallel, and their features are extracted by two parallel bidirectional long short-term memory (Bi-LSTM) networks which sufficiently utilize the correlations of sea clutter data. These features are concatenated and fused by a residual network (ResNet). The phases can be successfully obtained by constraining to the desired Doppler spectrum and the given amplitudes of sea clutter series. This proposed approach has been verified by the measured Ice Multiparameter Imaging X-Band (IPIX) radar data, and it can precisely model the complex sea clutter with specified statistic characteristics and Doppler properties. The amplitude root mean square error (RMSE) between the obtained and measured Doppler spectra is only 1.5065 with the interval between adjacent frames equals to 32. The RMSE of Doppler central frequency and spectrum width is 6.9306 and 1.2293 Hz, respectively. It shows robustness with the change of range resolution and interval.

Analysis of backpropagation neural neural network algorithm on student ability based cognitive aspects

Back propagation is one of the supervised learning and multi-layered training program and uses errors during the process of changing the weight value in the backward process as well as the forward propagation. In the method for predicting cognitive abilities backpropagation the first step is to set the input neuron number, the number of neurons that are hidden, and the number of output neurons. The number of neurons used in the program is 6 neurons consisting of cognitive criteria, 6 hidden neuron layers, and 2 neuron outputs. The highest level of accuracy is in the binary sigmoid and bipolar sigmoid activation functions at the 64th epoch with the accuracy of each function of 82.93% +/- 37.63% and 85.37% +/- 35.34%. The smallest root mean squared error value was found in binary sigmoid of 0.266 with a tolerance of +/- 0.258 on the 100th epoch with an accuracy of 80.49% while for the sigmoid bipolar activation function the smallest root mean squared error value was obtained at the epoch 500 of 0.282 with tolerance +/- 0.353.

Initial synaptic weight distribution for fast learning speed and high recognition rate in STDP-based spiking neural network

We analyze that the initial synaptic weight distribution affects the performance, such as the learning speed, recognition rate and the power consumption in the spiking neural networks (SNNs) based on spike-timing-dependent plasticity (STDP) learning rule. A thin-film transistor (TFT)-type NOR flash memory is used as a synaptic device. In this fully connected two-layer neuromorphic system using the proposed pulse scheme, the results with and without the homeostasis functionality were analyzed separately. In addition, power consumption of the network in various initial synaptic weight distributions, and recognition rate that varies with the number of output neurons are also investigated. In pattern recognition for 28 × 28 MNIST handwritten patterns, higher performance is achieved in various aspects when the initial synaptic weights are distributed near the maximum value.

Multi-Task Learning for Multi-Dimensional Regression: Application to Luminescence Sensing

The classical approach to non-linear regression in physics is to take a mathematical model describing the functional dependence of the dependent variable from a set of independent variables, and then using non-linear fitting algorithms, extract the parameters used in the modeling. Particularly challenging are real systems, characterized by several additional influencing factors related to specific components, like electronics or optical parts. In such cases, to make the model reproduce the data, empirically determined terms are built in the models to compensate for the difficulty of modeling things that are, by construction, difficult to model. A new approach to solve this issue is to use neural networks, particularly feed-forward architectures with a sufficient number of hidden layers and an appropriate number of output neurons, each responsible for predicting the desired variables. Unfortunately, feed-forward neural networks (FFNNs) usually perform less efficiently when applied to multi-dimensional regression problems, that is when they are required to predict simultaneously multiple variables that depend from the input dataset in fundamentally different ways. To address this problem, we propose multi-task learning (MTL) architectures. These are characterized by multiple branches of task-specific layers, which have as input the output of a common set of layers. To demonstrate the power of this approach for multi-dimensional regression, the method is applied to luminescence sensing. Here, the MTL architecture allows predicting multiple parameters, the oxygen concentration and temperature, from a single set of measurements.

Investigation of Thermal Performance of Cylindrical Heat Pipe Using Silver Nanofluid: Part-II: Prediction Using Artificial Neural Network

Heat pipes are one of the heat transfer devices adopted in various applications due to its high heat transfer capability. Recently, heat pipes are charged with different nanofluids to increases its performance. Thermal performance of heat pipe charged with nanofluids depends of various factors related to operating conditions, working fluids, heat pipe, etc. Thus, extensive experimentation work is required to recommend appropriate parameters for different industrial applications. In such cases, in order to reduce experiments and to propose the appropriate combination of parameters, experimentation results need to be modeled using fundamentals of simulation. In the Part I of the paper, extensive experiments have been carried out on cylindrical heat pipe with silver nanofluid as working medium. In the present paper, total of 96 artificial neural network models have been created and trained on the basis of the total 700 data sets presented in Part I of the paper. Out of 700 data sets, eight data sets have been randomly separated for post-training simulation of the model, while 692 data sets have been employed for formulating the ANN model using the ANN tool of MATLAB student version available at the college. The models have been prepared to predict the thermal performance of cylindrical heat pipe under study when one, two and three parameters have been considered as output neurons one by one and six parameters as input neurons. Out of six input parameters, heat load, average evaporator and average condenser temperature are related to operating condition, size and concentration of silver nanoparticles are related to working fluid, and inclination angle is related to position of heat pipe. Trained 96 ANN models have been simulated using eight data sets, and the best ANN model has been proposed on the basis of minimum value of error in prediction in terms of MSE, NMSE, MAE, MARD and MRE. The best ANN model identified for one, two and three output neurons is single-layer feed-forward backpropagation network with six neurons of hidden layer, cascade feed-forward backpropagation network having three neurons at hidden layer and single-layer feed-forward backpropagation network with 12 neurons of hidden layer, respectively. It is also concluded that for single output neuron, any one type of error in prediction can be considered to select the best ANN model. But for more number of output neurons, comparison of various types of error in prediction is essential to judge the best ANN model.

Investigating Raw Wave Deep Neural Networks for End-to-End Speaker Spoofing Detection

Recent advances in automatic speaker verification (ASV) lead to an increased interest in securing these systems for real-world applications. Malicious spoofing attempts against ASV systems can lead to serious security breaches. A spoofing attack within the context of ASV is a condition in which a (potentially harmful) person successfully masks as another, to the ASV system already known person by falsifying or manipulating data. While most previous work focuses on enhanced, spoof-aware features, end-to-end models can be a potential alternative. In this paper, we investigate the training of a raw wave front-ends for deep convolutional, long short-term memory (LSTM) and vanilla neural networks, which are analyzed for their suitability toward spoofing detection, regarding the influence of frame size, number of output neurons, and sequence length. A joint convolutional LSTM neural network (CLDNN) is proposed, which outperforms previous attempts on the BTAS2016 dataset (0.82% $\rightarrow$ 0.19% HTER), placing itself as the current state-of-the-art model for the dataset. We show that end-to-end approaches are appropriate for the important replay detection task and show that the proposed model is capable of distinguishing device-invariant spoofing attempts. Regarding the ASVspoof2015 dataset, the end-to-end solution achieves an equal error rate (EER) of 0.00% for the S1-S9 conditions. We show that the end-to-end approach based on a raw waveform input can outperform common cepstral features, without the use of context-dependent frame extensions. In addition, a cross-database (domain mismatch) scenario is also evaluated, which shows that the proposed CLDNN model trained on the BTAS2016 dataset achieves an EER of 25.7% on the ASVspoof2015 dataset.

From Winner-Takes-All to Winners-Share-All: Exploiting the Information Capacity in Temporal Codes.

In this letter, we have implemented and compared two neural coding algorithms in the networks of spiking neurons: Winner-takes-all (WTA) and winners-share-all (WSA). Winners-Share-All exploits the code space provided by the temporal code by training a different combination of [Formula: see text] out of [Formula: see text] neurons to fire together in response to different patterns, while WTA uses a one-hot-coding to respond to distinguished patterns. Using WSA, the maximum value of [Formula: see text] in order to maximize information capacity using [Formula: see text] output neurons was theoretically determined and utilized. A small proof-of-concept classification problem was applied to a spiking neural network using both algorithms to classify 14 letters of English alphabet with an image size of 15 [Formula: see text] 15 pixels. For both schemes, a modified spike-timing-dependent-plasticity (STDP) learning rule has been used to train the spiking neurons in an unsupervised fashion. The performance and the number of neurons required to perform this computation are compared between the two algorithms. We show that by tolerating a small drop in performance accuracy (84% in WSA versus 91% in WTA), we are able to reduce the number of output neurons by more than a factor of two. We show how the reduction in the number of neurons will increase as the number of patterns increases. The reduction in the number of output neurons would then proportionally reduce the number of training parameters, which requires less memory and hence speeds up the computation, and in the case of neuromorphic implementation on silicon, would take up much less area.

SpikeTemp: An Enhanced Rank-Order-Based Learning Approach for Spiking Neural Networks With Adaptive Structure.

This paper presents an enhanced rank-order-based learning algorithm, called SpikeTemp, for spiking neural networks (SNNs) with a dynamically adaptive structure. The trained feed-forward SNN consists of two layers of spiking neurons: 1) an encoding layer which temporally encodes real-valued features into spatio-temporal spike patterns and 2) an output layer of dynamically grown neurons which perform spatio-temporal classification. Both Gaussian receptive fields and square cosine population encoding schemes are employed to encode real-valued features into spatio-temporal spike patterns. Unlike the rank-order-based learning approach, SpikeTemp uses the precise times of the incoming spikes for adjusting the synaptic weights such that early spikes result in a large weight change and late spikes lead to a smaller weight change. This removes the need to rank all the incoming spikes and, thus, reduces the computational cost of SpikeTemp. The proposed SpikeTemp algorithm is demonstrated on several benchmark data sets and on an image recognition task. The results show that SpikeTemp can achieve better classification performance and is much faster than the existing rank-order-based learning approach. In addition, the number of output neurons is much smaller when the square cosine encoding scheme is employed. Furthermore, SpikeTemp is benchmarked against a selection of existing machine learning algorithms, and the results demonstrate the ability of SpikeTemp to classify different data sets after just one presentation of the training samples with comparable classification performance.

Skimming Digits: Neuromorphic Classification of Spike-Encoded Images.

The growing demands placed upon the field of computer vision have renewed the focus on alternative visual scene representations and processing paradigms. Silicon retinea provide an alternative means of imaging the visual environment, and produce frame-free spatio-temporal data. This paper presents an investigation into event-based digit classification using N-MNIST, a neuromorphic dataset created with a silicon retina, and the Synaptic Kernel Inverse Method (SKIM), a learning method based on principles of dendritic computation. As this work represents the first large-scale and multi-class classification task performed using the SKIM network, it explores different training patterns and output determination methods necessary to extend the original SKIM method to support multi-class problems. Making use of SKIM networks applied to real-world datasets, implementing the largest hidden layer sizes and simultaneously training the largest number of output neurons, the classification system achieved a best-case accuracy of 92.87% for a network containing 10,000 hidden layer neurons. These results represent the highest accuracies achieved against the dataset to date and serve to validate the application of the SKIM method to event-based visual classification tasks. Additionally, the study found that using a square pulse as the supervisory training signal produced the highest accuracy for most output determination methods, but the results also demonstrate that an exponential pattern is better suited to hardware implementations as it makes use of the simplest output determination method based on the maximum value.

OCR Recognition System Using Feed Forward and Back Propagation Neural Network

In this paper, a for Tamil, English characters, and digits recognition is proposed by applying algorithms to the network. The can effectively recognize various English, Tamil characters and digits with good recognition rates. A feed forward, Back Propagation, and classifying algorithm is capable of reducing the number of neurons and increasing recognition rates for the fixed number of output neurons. An Artificial Neural Network (ANN), often just called a neural network (NN), is a mathematical model or computational model based on biological networks. It consists of an interconnected group of artificial neurons and processes information using a connectionist approach to computation. In most cases an ANN is an adaptive system that changes its structure based on external or internal information that flows through the during the learning phase. In more practical terms networks are non-linear statistical data modeling tools. They can be used to model complex relationships between inputs and outputs or to find patterns in data. The simulations have indicated that the can effectively recognize various English, Tamil characters and digits with good recognition rates.

Data adjustment for the purposes of self-teaching of the neural network, and its application for the model-reduction of classification of patients suffering from the ischemic heart disease

Neural networks present a modern, very effective and practical instrument designated for decision-making support. To make use of them, we not only need to select the neural network type and structure, but also a corresponding data adjustment. One consequence of unsuitable data use can be an inexact or absolutely mistaken function of the model. The need for a certain adjustment of input data comes from the features of the chosen neural network type, from the use of various metrics systems of object attributes, but also from the weight, i.e., the importance of individual attributes, but also from establishing representatives of classifying sets and learning about their characteristics. For the purposes of the classification itself, we can suffice with a model in which the number of output neurons equals the number of classifying sets. Nonetheless, the model with a greater number of neurons assembled into a matrix can testify more about the problem, and provides clearer visual information.