Real-valued Input Research Articles

Brain-Inspired Architecture for Spiking Neural Networks.

Spiking neural networks (SNNs), using action potentials (spikes) to represent and transmit information, are more biologically plausible than traditional artificial neural networks. However, most of the existing SNNs require a separate preprocessing step to convert the real-valued input into spikes that are then input to the network for processing. The dissected spike-coding process may result in information loss, leading to degenerated performance. However, the biological neuron system does not perform a separate preprocessing step. Moreover, the nervous system may not have a single pathway with which to respond and process external stimuli but allows multiple circuits to perceive the same stimulus. Inspired by these advantageous aspects of the biological neural system, we propose a self-adaptive encoding spike neural network with parallel architecture. The proposed network integrates the input-encoding process into the spiking neural network architecture via convolutional operations such that the network can accept the real-valued input and automatically transform it into spikes for further processing. Meanwhile, the proposed network contains two identical parallel branches, inspired by the biological nervous system that processes information in both serial and parallel. The experimental results on multiple image classification tasks reveal that the proposed network can obtain competitive performance, suggesting the effectiveness of the proposed architecture.

Frequency and phase correction of GABA-edited magnetic resonance spectroscopy using complex-valued convolutional neural networks

PurposeTo determine the significance of complex-valued inputs and complex-valued convolutions compared to real-valued inputs and real-valued convolutions in convolutional neural networks (CNNs) for frequency and phase correction (FPC) of GABA-edited magnetic resonance spectroscopy (MRS) data. MethodsAn ablation study using simulated data was performed to determine the most effective input (real or complex) and convolution type (real or complex) to predict frequency and phase shifts in GABA-edited MEGA-PRESS data using CNNs. The best CNN model was subsequently compared using both simulated and in vivo data to two recently proposed deep learning (DL) methods for FPC of GABA-edited MRS. All methods were trained using the same experimental setup and evaluated using the signal-to-noise ratio (SNR) and linewidth of the GABA peak, choline artifact, and by visually assessing the reconstructed final difference spectrum. Statistical significance was assessed using the Wilcoxon signed rank test. ResultsThe ablation study showed that using complex values for the input represented by real and imaginary channels in our model input tensor, with complex convolutions was most effective for FPC. Overall, in the comparative study using simulated data, our CC-CNN model (that received complex-valued inputs with complex convolutions) outperformed the other models as evaluated by the mean absolute error. ConclusionOur results indicate that the optimal CNN configuration for GABA-edited MRS FPC uses a complex-valued input and complex convolutions. Overall, this model outperformed existing DL models.

Expressive power of ReLU and step networks under floating-point operations

The study of the expressive power of neural networks has investigated the fundamental limits of neural networks. Most existing results assume real-valued inputs and parameters as well as exact operations during the evaluation of neural networks. However, neural networks are typically executed on computers that can only represent a tiny subset of the reals and apply inexact operations, i.e., most existing results do not apply to neural networks used in practice. In this work, we analyze the expressive power of neural networks under a more realistic setup: when we use floating-point numbers and operations as in practice. Our first set of results assumes floating-point operations where the significand of a float is represented by finite bits but its exponent can take any integer value. Under this setup, we show that neural networks using a binary threshold unit or ReLU can memorize any finite input/output pairs and can approximate any continuous function within an arbitrary error. In particular, the number of parameters in our constructions for universal approximation and memorization coincides with that in classical results assuming exact mathematical operations. We also show similar results on memorization and universal approximation when floating-point operations use finite bits for both significand and exponent; these results are applicable to many popular floating-point formats such as those defined in the IEEE 754 standard (e.g., 32-bit single-precision format) and bfloat16.

Fault detection and identification in induction motor using weightless neural network.

This paper presents a novel multiclass, multiparameter real-valued weightless neural network-based classifier for fault detection and identification. In contrast to primitive variants of weightless neural nets, the proposed method is capable of multiclass identification with real-valued input features and has improved recognition and generalization capabilities. The major accomplishments of the proposed method include an improved input-to-address mapping strategy that is suitable for address assignment within discriminator units and an effective memory expansion scheme based on similarity metric-based membership value. The developed classifier is utilized for fault detection and identification in single-phase and three-phase induction motors. The sensitivity analysis of the method is investigated for design parameter variation and the fault diagnosis is performed for multiple faults of the motor. The proposed method achieves the highest accuracy of 99.6% and 89.25% for single-phase and three-phase induction motor fault identification, respectively.

Maximal chain-based Choquet-like integrals

The original idea of the discrete Choquet integral on nonnegative reals is based on a distinguished permutation making the inputs reordering nonincreasingly or nondecreasingly. This permutation generates the weights from the given capacity, and then a weighted arithmetic mean yields the value of the corresponding Choquet integral. Since the permutation relates to a maximal chain, we introduce a concept of MCC-integrals using the setting of maximal chains on the power set of a finite set. This approach provides a unified framework for several integrals of nonnegative vectors with respect to games. We study basic properties and various representations of these MCC-integrals subsuming known results for the Choquet integral as a special case. For supermodular games it is shown that the Choquet integral is the minimum of all product-based MCC-integrals. Further, we discuss symmetric and asymmetric extensions for real-valued inputs providing new insight into the construction of fuzzy integral quadruplets. During the way, we point out certain inaccuracies in the literature.

Results on finite time stability of various fractional order systems

This article deals with the existence of the solution of Hilfer–Katugampola fractional derivatives, and we prove the stability results of the equilibrium point of the presented problem. Numerous authors have made substantial use of the concepts of fractional derivatives and fixed-point theory in order to produce stability results in neural networks containing complex-valued or real-valued inputs. In this connection, we discuss the finite stability of Caputo fractional derivatives as well as Caputo fractional-order complex-valued neural networks. We provide a numerical result that supports the theoretical discussion.

A Deep Learning Approach for Reconstruction in Millimeter-Wave Imaging Systems

In millimeter-wave (MMW) imaging, the objects of interest are oftentimes modeled as 2-D binary (black and white) shapes with white pixels representing the reflecting interior of the object. However, due to the propagation of the scattered waves, the continuous-domain binary images are convolved with a so-called point-spread function (PSF) before being digitized by means of sampling. As the 2-D PSF is both nonseparable and nonvanishing in the case of MMW imaging, exact recovery is quite complicated. In this communication, we propose a deep-learning approach for image reconstruction. We should highlight that the wave scatterings are suitably represented with complex-valued quantities, while standard deep neural networks (DNNs) accept real-valued inputs. To overcome this challenge, we separate the real and imaginary parts as if we had two imaging modalities and concatenate them to form a real-valued input with a larger size. Fortunately, the network automatically learns how to combine the mutual information between these modalities to reconstruct the final image. Among the advantages of the proposed method are improved robustness against additive noise and mismatch errors of imaging frequency and object to antenna distance; indeed, the method works well in wideband imaging scenarios over a wide range of objects to antenna distances even in the presence of high noise levels without requiring a separate calibration stage. We test the method with synthetic data simulated with software as well as real recordings in the laboratory.

Collective computational intelligence in biology – Emergence of memory in somatic tissues

Role of memory in the function of biological tissues, organs and organisms remains unexplored with many unanswered questions. In this study, the emergence of associative memory in somatic (non-neural) tissues and its potential relation to tissue function was explored using a number of biologically plausible network topologies in in silico tissues with computing cells. These topologies were local cooperation; complete system-wide cooperation or inhibition; and local cooperation and short- or long-range inhibition. These were tested with and without self-feedback on two-dimensional (2D) three-dimensional (3D) cell networks, resulting in various forms of fully and partially connected networks. Further, both binary inputs with threshold processing and real-valued inputs with nonlinear processing were considered. Results revealed the emergence of diverse forms of tissue memory. In full cooperation, networks produced one fixed attractor indicating the propensity towards a stable memory pattern which in a real tissue could correspond to an invariable physiological state, such as bioelectric homeostasis. The local neighbourhood cooperation produced both a fixed and a limit cycle attractor that could be beneficial for a tissue to hold few associative memories including circadian rhythms. Most interesting results were found for the local cooperation with short- or long-range inhibition topologies that produced a cluster of fixed and limit cycle attractors offering diverse memories. Fixed attractors could correspond to inactive tissue states and active nonrhythmic functional states and limit cycles could correspond to circadian rhythms such as pumping in heart, kidney or liver in various oscillatory regimes. In all topologies, self-feedback abolished or drastically reduced the limit cycles in favour of fixed stable state. These attractor patterns were found to be largely invariant to scale (2D or 3D) and type of inputs and processing. We also explored the self-optimising ability of the ‘local cooperation with global (short- or long-range) inhibition’ 2D topologies with Hebbian learning with fixed and flexible topologies. The fixed topology learned to self-model to consolidate memory towards fewer more stable attractors. The flexible topology even formed new connections to bring the system to a single fixed state. Thus local cooperation with global inhibition topology can offer greater freedom to create diverse memory pattens that can be tempered by learning, self-feedback, and to some extent continuous processing to simplify and consolidate memory towards manageable forms.

Elastic-Link for Binarized Neural Networks

Recent work has shown that Binarized Neural Networks (BNNs) are able to greatly reduce computational costs and memory footprints, facilitating model deployment on resource-constrained devices. However, in comparison to their full-precision counterparts, BNNs suffer from severe accuracy degradation. Research aiming to reduce this accuracy gap has thus far largely focused on specific network architectures with few or no 1 × 1 convolutional layers, for which standard binarization methods do not work well. Because 1 × 1 convolutions are common in the design of modern architectures (e.g. GoogleNet, ResNet, DenseNet), it is crucial to develop a method to binarize them effectively for BNNs to be more widely adopted. In this work, we propose an “Elastic-Link” (EL) module to enrich information flow within a BNN by adaptively adding real-valued input features to the subsequent convolutional output features. The proposed EL module is easily implemented and can be used in conjunction with other methods for BNNs. We demonstrate that adding EL to BNNs produces a significant improvement on the challenging large-scale ImageNet dataset. For example, we raise the top-1 accuracy of binarized ResNet26 from 57.9% to 64.0%. EL also aids con-vergence in the training of binarized MobileNet, for which a top-1 accuracy of 56.4% is achieved. Finally, with the integration of ReActNet, it yields a new state-of-the-art result of 71.9% top-1 accuracy.

Multicore Photonic Complex-Valued Neural Network with Transformation Layer

Photonic neural network chips have been widely studied because of their low power consumption, high speed and large bandwidth. Using amplitude and phase to encode, photonic chips can accelerate complex-valued neural network computations. In this article, a photonic complex-valued neural network (PCNN) chip is designed. The scale of the single-core PCNN chip is limited because of optical losses, and the multicore architecture of the chip is used to improve computing capability. Further, for improving the performance of the PCNN, we propose the transformation layer, which can be implemented by the designed photonic chip to transform real-valued encoding to complex-valued encoding, which has richer information. Compared with real-valued input, the transformation layer can effectively improve the classification accuracy from 93.14% to 97.51% of a 64-dimensional input on the MNIST test set. Finally, we analyze the multicore computation of the PCNN. Compared with the single-core architecture, the multicore architecture can improve the classification accuracy by implementing larger neural networks and has better phase noise robustness. The proposed architecture and algorithms are beneficial to promote the accelerated computing of photonic chips for complex-valued neural networks and are promising for use in many applications, such as image recognition and signal processing.

Interval deep learning architecture with rough pattern recognition and fuzzy inference for short-term wind speed forecasting

In recent decades, wind power is rapidly becoming a significant energy resource due to environmental considerations. The accuracy of wind energy forecasts is closely dependent on the prediction of wind speed time series. In this paper, a novel solution for ultra-short-term and short-term wind speed forecasting is introduced. The proposed method consists of a novel real-valued Deep Belief Network (DBN) with a new Rough feature extraction layer (RFEL) and a Fuzzy Type II Inference System (FT2IS) for robust supervised regression. To learn meaningful unsupervised features from the underlying wind speed data, real-valued input units are computed to better approximate the wind data distribution compared to the existing deep learning models. The proposed differentiable RFEL can be applied to any neural network to efficiently extract noise invariant features. A Takagi-Sugeno-Kang (TSK) system with interval Gaussian membership functions is employed for the supervised forecasting task. The high generalization capacity of the proposed unsupervised feature learning model incorporated into the robust RFEL and FT2IS leads to accurate predictions for highly varying wind speed time series. Numerical results on the Western Wind Dataset reveal significant performance improvements compared to recently proposed Deep Learning Architectures (DLAs), including the DBN, Stacked Autoencoder (SAE), and hybrid methodologies that leverage backtracking and metaheuristic optimization.

Mechanisms to Alleviate Over-Generalization in XCS for Continuous-Valued Input Spaces

In the field of rule-based approaches to Machine Learning, the XCS classifier system (XCS) is a well-known representative of the learning classifier systems family. By using a genetic algorithm (GA), the XCS aims at forming rules or so-called classifiers which are as general as possible to achieve an optimal performance level. A too high generalization pressure may lead to over-general classifiers degrading the performance of XCS. To date, no method exists for XCS for real-valued input spaces (XCSR) and XCS for function approximation (XCSF) to handle over-general classifiers ensuring an accurate population. The Absumption mechanism and the Specify operator, both developed for XCS with binary inputs, provide a promising basis for over-generality handling in XCSR and XCSF. This paper introduces adapted versions of Absumption and Specify by proposing different identification and specialization strategies for the application in XCSR and XCSF. To determine their potential, the adapted techniques are evaluated in different classification problems, i.e., common benchmarks and real-world data from the agricultural domain, in a multi-step problem as well as different regression tasks. Our experimental results show that the application of these techniques leads to significant improvements of the accuracy of the generated classifier population in the applied benchmarks, data sets, multi-step problems and regression tasks, especially when they tend to form over-general classifiers. Furthermore, considering the working principle of the proposed techniques, the intended decrease in overall classifier generality can be confirmed.

Internalizing Knowledge for Anticipatory Classifier Systems in Discretized Real-Valued Environments

Real-valued environments are challenging for learning systems because of a significant increase in the input space size of the problem. This work demonstrates that Anticipatory Learning Classifier Systems (ALCS) can successfully build sets of conditional rules foreseeing the consequences of executed actions. Three major classes of Learning Classifier Systems - Anticipatory Classifier System (ACS), ACS2, Yet Another Classifier System (YACS), alongside the traditional Dyna-Q algorithm implementations were adapted to handle real-valued input signal discretization. Aspects like the ability to capture all possible interactions, model generalization capabilities, size of the solution of relative execution times were compared in four different problems using probabilistic modelling, providing unbiased judgments. Results proved that the examined ALCS are capable of solving selected problems. Despite increased input size, all possible environmental transitions were learned latently, without obtaining any explicit incentives. Such an internal representation provides a more compact solution representation and can optimize learning speed further by executing imaginary environmental interactions or performing action planning for a new set of potential problems.

Medium-term Load Forecasting Method with Improved Deep Belief Network for Renewable Energy

With the continuous transition of the traditional power system to the newpower system, the composition of the power generation side in the powersystem has gradually begun to be dominated by renewable energy (at leastmore than 50%). Among the renewable energy sources, wind power is themost susceptible to weather and environmental influences. These factorsincrease the complexity of the power generation mode, and put forwardhigher requirements for the accuracy and stability of load forecasting. Thispaper proposes a medium-term renewable energy load forecasting methodbased on an improved deep belief network (IDBN-NN). The method includesthe construction of a deep belief network, the layer-by-layer pre-trainingand fine-tuning of model parameters, and the application of the model.In the process of model parameter pre-training, Gauss-Bernoulli RestrictedBoltzmann Machine (GB-RBM) is used as the first part of the stacked deepbelief network, so that it can process multiple types of real-valued input data more effectively. In addition, IDBN-NN uses a combination of unsupervisedtraining and supervised training for pre-training. Finally, the actual load datais used to analyze the calculation example. When the number of RBM layersis 3, the number of fully connected layers is 1, and Dropout is equal to0.2, the MSE and loss values are optimal, which are 0.0037 and 0.0104,respectively. The experimental results show that the proposed method hashigher prediction accuracy when the training sample is large and the loadinfluencing factors are complex.

Classifying flexible and integer data in two-stage network data envelopment analysis

In conventional DEA models, it is supposed real-valued inputs and outputs while every measure must be determined as an input or output. However, in some cases, there are flexible measures which can only take integer values in two-stage network DEA. In all previous researches has not been mentioned the classification flexible measures while some inputs, outputs, and flexible measures can only take integer values in two-stage network DEA. So in this paper, we propose integer-valued FNDEA approach is based on envelopment form of CCR that evaluates the relative efficiency of DMUs and determines the status of flexible measures in the presence of integer data in basic and general two-stage network structures. Our model can determine projection points of inputs, outputs, and flexible measures in the presence of integer data in two-stage network DEA. Numerical examples are used to illustrate the procedure.

Computing the partition function of the Sherrington–Kirkpatrick model is hard on average

We establish the average-case hardness of the algorithmic problem of exact computation of the partition function associated with the Sherrington–Kirkpatrick model of spin glasses with Gaussian couplings and random external field. In particular, we establish that unless P=#P, there does not exist a polynomial-time algorithm to exactly compute the partition function on average. This is done by showing that if there exists a polynomial time algorithm, which exactly computes the partition function for inverse polynomial fraction (1/nO(1)) of all inputs, then there is a polynomial time algorithm, which exactly computes the partition function for all inputs, with high probability, yielding P=#P. The computational model that we adopt is finite-precision arithmetic, where the algorithmic inputs are truncated first to a certain level N of digital precision. The ingredients of our proof include the random and downward self-reducibility of the partition function with random external field; an argument of Cai et al. (In STACS 99 (Trier) (1999) 90–99 Springer) for establishing the average-case hardness of computing the permanent of a matrix; a list-decoding algorithm of Sudan (In 37th Annual Symposium on Foundations of Computer Science (Burlington, VT, 1996) (1996) 164–172 IEEE Comput. Soc. Press), for reconstructing polynomials intersecting a given list of numbers at sufficiently many points; and near-uniformity of the log-normal distribution, modulo a large prime p. To the best of our knowledge, our result is the first one establishing a provable hardness of a model arising in the field of spin glasses. Furthermore, we extend our result to the same problem under a different real-valued computational model, for example, using a Blum–Shub–Smale machine (In [Proceedings 1988] 29th Annual Symposium on Foundations of Computer Science (1988) 387–397 IEEE) operating over real-valued inputs. We establish that, if there exists a polynomial time algorithm which exactly computes the partition function for 34+1nO(1) fraction of all inputs, then there exists a polynomial time algorithm, which exactly computes the partition function for all inputs, with high probability, yielding P=#P. Our proof uses the random self-reducibility of the partition function, together with a control over the total variation distance for log-normal random variables in presence of a convex perturbation, and the Berlekamp–Welch algorithm.

Optimal dynamic pricing for an electricity retailer in the price-responsive environment of smart grid

The main purpose of this study is to support a retail electric provider (REP) to make the best day-ahead dynamic pricing decisions in a realistic scenario. These decisions are made with the aim of maximizing the profit achieved by the REP under the assumption that mixed types of customers with different behaviors in the electricity market are considered. While some of the customers have installed smart meters with an embedded home energy management system (HEMS) in their home, others do not participate in the demand response (DR) programs. For this purpose, a bi-level hybrid demand modeling framework is proposed. It firstly uses an optimal energy management algorithm with bill minimization in order to model the behavior of customers with smart meters. Then, using a customers' behavior learning machine (CBLM), the behavior of other groups without smart meters is modeled. Therefore, the proposed hybrid model cannot only schedule usage of home appliances to the interests of customers with smart meters but can also be used to understand electricity usage behavior of customers without smart meters. The proposed model includes a stacked auto-encoder (SAE), one of the deep learning (DL) methods suitable for real-valued inputs, and adaptive neuro-fuzzy inference system (ANFIS). Based on the established hybrid demand model for all customers, a profit maximization algorithm is developed in order to achieve optimal prices for the REP under relevant market constraints. The results of the case studies confirm the applicability and effectiveness of the proposed model.

Circular Complex-Valued GMDH-Type Neural Network for Real-Valued Classification Problems.

Recently, applications of complex-valued neural networks (CVNNs) to real-valued classification problems have attracted significant attention. However, most existing CVNNs are black-box models with poor explanation performance. This study extends the real-valued group method of data handling (RGMDH)-type neural network to the complex field and constructs a circular complex-valued group method of data handling (C-CGMDH)-type neural network, which is a white-box model. First, a complex least squares method is proposed for parameter estimation. Second, a new complex-valued symmetric regularity criterion is constructed with a logarithmic function to represent explicitly the magnitude and phase of the actual and predicted complex output to evaluate and select the middle candidate models. Furthermore, the property of this new complex-valued external criterion is proven to be similar to that of the real external criterion. Before training this model, a circular transformation is used to transform the real-valued input features to the complex field. Twenty-five real-valued classification data sets from the UCI Machine Learning Repository are used to conduct the experiments. The results show that both RGMDH and C-CGMDH models can select the most important features from the complete feature space through a self-organizing modeling process. Compared with RGMDH, the C-CGMDH model converges faster and selects fewer features. Furthermore, its classification performance is statistically significantly better than the benchmark complex-valued and real-valued models. Regarding time complexity, the C-CGMDH model is comparable with other models in dealing with the data sets that have few features. Finally, we demonstrate that the GMDH-type neural network can be interpretable.

Real-Valued Neural Networks for Complex-Valued Impairment Compensation Using Digital Up-Conversion

Analog distortion compensation based on digital signal processing methods are widely applied for transmitter or receiver impairments in various digital communication systems. Recently, several neural network methods were developed for distortion compensation. However, while digital communication systems are typically complex-valued, neural networks are mostly designed to work with real-valued inputs. Thus, adaptations of the network architecture or input data should be applied. In this article a method for using a single-input real-valued neural network for digital communication-based complex-valued signals without any modifications to the neural network is proposed. The method transforms the complex-valued signal to a real-valued one by taking the real component of a complex frequency offset applied through digital up-conversion, without affecting the distortion, therefore allowing standard neural network-based functionality with significant reduction in size. The method is tested with a multi-layer perceptron and gated recurrent unit architectures applied over a generic Wiener-Hammerstein model for the case of equalization of a coherent optical system. A reduction of about 7% and 26% in network size is shown for the gated recurrent unit-based and multi-layer perceptron-based architectures respectively, without any significant change in performance. This size reduction capability shows the high potential in applying the proposed method in neural network-based equalization and pre-distortion operations.

A Robust and Scalable Harmonic Cancellation Digital Predistortion Technique for HF Transmitters

High frequency (HF) transmitter implemented with the multioctave broadband power amplifier (PA) enables long-distance communication and has strong survivability. However, it is faced with the problem of harmonics because of the nonlinearity of the PA. We first consider canceling the harmonics using a harmonic cancellation digital predistortion (HC-DPD) technique, instead of lossy and bulky analog filter banks. A uniform complex-valued Hilbert transform-based polynomial (HTBP) function is proposed for the modeling of the harmonics, as well as the fundamental component in the real-valued input and output signals. In order to validate this idea, detailed experiments were performed for carrier frequencies located at 5 and 10 MHz, respectively. After the proposed HC-DPD, the harmonics were all canceled to satisfy the linearization requirements. In particular, the improvement of the second harmonic is more than 40 dB, which is nearly 20 dB better than that of the memoryless polynomial (MLP) model. Furthermore, the experimental results demonstrate that the proposed technique is scalable, in which the model coefficients extracted at a specific power level, signal bandwidth, and carrier frequency can be extended to other scenarios. Both the second and third harmonics were canceled for being below −45 dBc after the HC-DPD.