Neural Network Operators Research Articles

Q-Deformed and L-parametrized hyperbolic tangent function relied complex valued multivariate trigonometric and hyperbolic neural network approximations

Here we study the multivariate quantitative approximation of complex valued continuous functions on a box of RN , N ∈ N, by the multivariate normalized type neural network operators. We investigate also the case of approximation by iterated multilayer neural network operators. These approximations are achieved by establishing multidimen-sional Jackson type inequalities involving the multivariate moduli of continuity of the en- gaged function and its partial derivatives. Our multivariate operators are defined by using a multidimensional density function induced by a q-deformed and λ-parametrized hyper-bolic tangent function, which is a sigmoid function. The approximations are pointwise and uniform. The related feed-forward neural network are with one or multi hidden layers. The basis of our theory are the introduced multivariate Taylor formulae of trigonometric and hyperbolic type.

Multiple general sigmoids based Banach space valued neural network multivariate approximation

Here we present multivariate quantitative approximations of Banach space valued continuous multivariate functions on a box or \(\mathbb{R}^{N},\) \(N\in \mathbb{N}\), by the multivariate normalized, quasi-interpolation, Kantorovich type and quadrature type neural network operators. We treat also the case of approximation by iterated operators of the last four types. These approximations are derived by establishing multidimensional Jackson type inequalities involving the multivariate modulus of continuity of the engaged function or its high order Fréchet derivatives. Our multivariate operators are defined by using a multidimensional density function induced by several different among themselves general sigmoid functions. This is done on the purpose to activate as many as possible neurons. The approximations are pointwise and uniform. The related feed-forward neural network is with one hidden layer. We finish with related \(L_{p}\) approximations.

Q-Deformed and λ-parametrized A-generalized logistic function based Banach space valued ordinary and fractional neural network approximation

Here we research the univariate quantitative approximation, ordinary and fractional, of Banach space valued continuous functions on a compact interval or all the real line by quasi-interpolation Banach space valued neural network operators. These approximations are derived by establishing Jackson type inequalities involving the modulus of continuity of the engaged function or its Banach space valued high order derivative of fractional derivatives. Our operators are defined by using a density function generated by a q-deformed and λ-parametrized A-generalized logistic function, which is a sigmoid function. The approximations are pointwise and of the uniform norm. The related Banach space valued feed-forward neural networks are with one hidden layer.

Modeling of the Hodgkin–Huxley neural oscillators dynamics using an artificial neural network

The purpose of this study — to represent a detailed description of the procedure for creating and training a neural network mapping on the example of the dynamics modeling of a neural oscillator of the Hodgkin–Huxley type; to show that the neural network mappings trained for single oscillators can be used as elements of a coupled system that simulate the behavior of coupled oscillators. Methods. Numerical method is used for solving stiff systems of ordinary differential equations. Also a procedure for training neural networks based on the method of back propagation of error is employed together with the Adam optimization algorithm, that is a modified version of the gradient descent supplied with an automatic step adjustment. Results. It is shown that the neural network mappings built according to the described procedure are able to reproduce the dynamics of single neural oscillators. Moreover, without additional training, these mappings can be used as elements of a coupled system for the dynamics modeling of coupled neural oscillator systems. Conclusion. The described neural network mapping can be considered as a new universal framework for complex dynamics modeling. In contrast to models based on series expansion (power, trigonometric), neural network mapping does not require truncating of the series. Consequently, it allows modeling processes with arbitrary order of nonlinearity, hence there are reasons to believe that in some aspects it will be more effective. The approach developed in this paper based on the neural network mapping can be considered as a sort of an alternative to the traditional numerical methods of modeling of dynamics. What makes this approach topical is the current rapid development of technologies for creating fast computing equipment that supports neural network training and operation.

Some density results by deep Kantorovich type neural network operators

In this paper, we prove density results by using deep Kantorovich type neural network operators. Firstly, we define a two layer neural network operator and prove the density results in the spaces C(I) and Lp(I) for p≥1, where I:=[−1,1]. Then we extend it to a multi-layer neural network operator and prove the corresponding density results. Our study provides a generalizations of the well known single layer Kantorovich type neural network operator in terms of its deeper version.

Optimized injection of noise in activation functions to improve generalization of neural networks

This paper proposes a flexible probabilistic activation function that enhances the training and operation of artificial neural networks by intentionally injecting noise to gain additional control over the response of each neuron. During the learning phase, the level of injected noise is iteratively optimized by gradient-descent, realizing a form of adaptive stochastic resonance. From simple hard-threshold non-differentiable neuronal responses, controlled injection of noise gives access to a wide range of useful activation functions, with sufficient differentiability to enable gradient-descent learning for both the neuron and the injected-noise levels. Experimental results on function approximation demonstrate injected noise generally converging to non-vanishing optimal levels associated with improved generalization abilities in the neural networks. A theoretical explanation of the generalization improvement based on the path norm bound is presented. With injected noise in the deep neural network, experimental results on classifying images also obtain non-vanishing optimal noise levels to achieve better testing accuracies. The proposed probabilistic activation functions show the potential of adaptive stochastic resonance for useful applications in machine learning.

HSH-UNet: Hybrid selective high order interactive U-shaped model for automated skin lesion segmentation

The use of computer-assisted clinical dermatologists to diagnose skin diseases is an important aid. And computer-assisted techniques mainly use deep neural networks. Recently, the proposal of higher-order spatial interaction operations in deep neural networks has attracted a lot of attention. It has the advantages of both convolution and transformers, and additionally has the advantages of efficient, extensible and translation-equivariant. However, the selection of the interaction order in higher-order interaction operations requires tedious manual selection of a suitable interaction order. In this paper, a hybrid selective higher-order interaction U-shaped model HSH-UNet is proposed to solve the problem that requires manual selection of the order. Specifically, we design a hybrid selective high-order interaction module HSHB embedded in the U-shaped model. The HSHB adaptively selects the appropriate order for the interaction operation channel-by-channel under the computationally obtained guiding features. The hybrid order interaction also solves the problem of fixed order of interaction at each level. We performed extensive experiments on three public skin lesion datasets and our own dataset to validate the effectiveness of our proposed method. The ablation experiments demonstrate the effectiveness of our hybrid selective higher order interaction module. The comparison with state-of-the-art methods also demonstrates the superiority of our proposed HSH-UNet performance. The code is available at https://github.com/wurenkai/HSH-UNet.

Neural network operators of generalized fractional integrals equipped with a vector-valued function

In this paper, motivated by the recent development of the approximation capability of neural network operators, we aim to construct neural network (NN) operators for multivariate fractional integrals of order α, which are equipped with a vector-valued function ψ and its Jacobian matrix. In this system, the connection strengths between output neurons are represented by fractional mean values, which depend on the vector-valued function ψ, of the approximated multivariate function. To estimate the rate of convergence (learning rate), the connection weights of NN are equipped with a decreasing sequence (λn). The activation function of the system is constructed using a linear collection ΩΨ, which consists of density functions generated by multivariate sigmoidal functions. Our goal is to construct a flexible and productive hybrid system by leveraging a diverse selection of the function ψ, the parameter α, the sequence (λn) and the activation function ΩΨ. The quantitative estimates of the operators are examined by means of the multivariate modulus of continuity. Moreover, we provide some illustrative examples with graphs to demonstrate the approximation performance of the operators through the selected activation functions. Finally, we present numerical results consisting of the maximum absolute errors of approximation for the proposed operators based on various selections of the vector-valued function ψ.

Binary‐Stochasticity‐Enabled Highly Efficient Neuromorphic Deep Learning Achieves Better‐than‐Software Accuracy

In this work, the requirement of using high‐precision (HP) signals is lifted and the circuits for implementing deep learning algorithms in memristor‐based hardware are simplified. The use of HP signals is required by the backpropagation learning algorithm since the gradient descent learning rule relies on the chain product of partial derivatives. However, it is both challenging and biologically implausible to implement such an HP algorithm in noisy and analog memristor‐based hardware systems. Herein, it is demonstrated that the requirement for HP signals handling is not necessary and more efficient deep learning can be achieved when using a binary stochastic learning algorithm. The new algorithm proposed in this work modifies elementary neural network operations, which improves energy efficiency by two orders of magnitude compared to traditional memristor‐based hardware and three orders of magnitude compared to complementary metal–oxide–semiconductor‐based hardware. It also provides better accuracy in pattern recognition tasks than the HP learning algorithm benchmarks.

Dual‐logic‐in‐memory implementation with orthogonal polarization of van der Waals ferroelectric heterostructure

AbstractThe rapid advancement of AI‐enabled applications has resulted in an increasing need for energy‐efficient computing hardware. Logic‐in‐memory is a promising approach for processing the data stored in memory, wherein fast and efficient computations are possible owing to the parallel execution of reconfigurable logic operations. In this study, a dual‐logic‐in‐memory device, which can simultaneously perform two logic operations in four states, is demonstrated using van der Waals ferroelectric field‐effect transistors (vdW FeFETs). The proposed dual‐logic‐in‐memory device, which also acts as a two‐bit storage device, is a single bidirectional polarization‐integrated ferroelectric field‐effect transistor (BPI‐FeFET). It is fabricated by integrating an in‐plane vdW ferroelectric semiconductor SnS and an out‐of‐plane vdW ferroelectric gate dielectric material—CuInP2S6. Four reliable resistance states with excellent endurance and retention characteristics were achieved. The two‐bit storage mechanism in a BPI‐FeFET was analyzed from two perspectives: carrier density and carrier injection controls, which originated from the out‐of‐plane polarization of the gate dielectric and in‐plane polarization of the semiconductor, respectively. Unlike conventional multilevel FeFETs, the proposed BPI‐FeFET does not require additional pre‐examination or erasing steps to switch from/to an intermediate polarization, enabling direct switching between the four memory states. To utilize the fabricated BPI‐FeFET as a dual‐logic‐in‐memory device, two logical operations were selected (XOR and AND), and their parallel execution was demonstrated. Different types of logic operations could be implemented by selecting different initial states, demonstrating various types of functions required for numerous neural network operations. The flexibility and efficiency of the proposed dual‐logic‐in‐memory device appear promising in the realization of next‐generation low‐power computing systems.image

SmartLite: A DBMS-Based Serving System for DNN Inference in Resource-Constrained Environments

Many IoT applications require the use of multiple deep neural networks (DNNs) to perform various tasks on low-cost edge devices with limited computation resources. However, existing DNN model serving platforms, such as TensorFlow Serving and TorchServe, are resource-intensive and require high-performance GPUs that are often not available on low-cost edge devices. In this paper, we propose SmartLite, a lightweight DBMS that addresses these challenges by storing the parameters and structural information of neural networks as database tables and implementing neural network operators inside the DBMS engine. SmartLite quantizes model parameters as binarized values, applies neural pruning techniques to compress the models, and transforms tensor manipulations into value lookup operations of the DBMS to reduce computation overhead. Experimental results show that SmartLite requires 98% less memory while achieving about a 134% performance speedup compared to Torch-Serve. Our proposed solution addresses the challenges of running multiple DNN models on low-cost edge devices and provides a significant contribution to the field of IoT applications.

Accelerating AI performance with the incorporation of TVM and MediaTek NeuroPilot

The continuing prominence of machine learning has led to an increased focus on enhancing the inference performance of edge devices to reduce latency and improve efficiency. Two widely adopted strategies for accelerating computational performance are quantisation and the utilisation of AI hardware accelerators. Each type of accelerator or inference engine offers distinct advantages, with accelerators primarily designed to optimise neural network operations. In this paper, we present an innovative method for integrating TVM's quantisation flow with the MediaTek Neuropilot AI accelerator. We outline the process of converting the TVM relay intermediate-representation quantised neural network dialect model to a tensor-oriented quantisation format, with the aim of harnessing the full potential of both TVM and MediaTek NeuroPilot. This integration enables more efficient neural network inference while preserving the accuracy of the results. We assessed the effectiveness of our proposed integration by conducting a series of experiments and comparing the performance of our approach with that of TVM equipped with an autotuning mechanism. The findings indicate that our approach substantially outperforms TVM in both floating-point model inference and quantised model inference, with inference speedups of up to 11× and up to 70×, respectively. These results underscore the potential of our approach in accelerating AI performance across a diverse range of applications and edge devices. Moreover, a key contribution of our work is providing a valuable practical method for other hardware companies interested in integrating TVM with their own accelerators to achieve performance gains.

SliceSamp: A Promising Downsampling Alternative for Retaining Information in a Neural Network

Downsampling, which aims to improve computational efficiency by reducing the spatial resolution of feature maps, is a critical operation in neural networks. Many downsampling methods have been proposed to address the challenge of retaining feature map information. However, some detailed information is still lost, even though these methods can extract features with stronger semantics. In this paper, we propose a novel downsampling method which combines feature slicing and depthwise separable convolution for information-retaining downsampling. It slices the input feature map into multiple non-overlapping sub-feature maps by using indexes with a stride of two in the spatial dimension and applies depthwise separable convolution on each slice to extract feature information. To demonstrate the effectiveness of SliceSamp, we compare it with classical downsampling methods on image classification, object detection, and semantic segmentation tasks using several benchmark datasets, including ImageNet-1K, COCO, VOC, and ADE20K. Extensive experiments demonstrate that SliceSamp outperforms classical downsampling methods with consistent improvements in various computer vision tasks. The proposed SliceSamp shows advanced model performance with lower computational costs and memory requirements. By replacing the downsampling layers in different network architectures (including ResNet (Residual Network), YOLOv5, and Swin Transformer), SliceSamp brings different degrees of performance gains (+0.54~3.64%) compared to these baseline models. Additionally, SliceUpsamp enables high-resolution feature reconstruction and alignment during upsampling. SliceSamp and SliceUpsamp can be plug-and-play-integrated into existing neural network architectures. As a promising downsampling alternative to traditional methods, SliceSamp can also provide a reference for designing lightweight and high-performance model architectures in the future.

The study and application of brain tumors classification based on artificial intelligence

The application of AI in the medical field has been increasingly popular due to its ability to enhance the accuracy and efficiency of diagnoses. A plurality of individuals employs diverse techniques and refine existing methods in order to achieve greater precision. This paper provides an overview of the methodology utilized in early research papers for brain tumors classification. This includes the input of datasets, preprocessing, model building, training and testing, evaluation, and application. In addition, this paper presents four models used in early research, including the Artificial neural network (ANN) which mimics the organization and operation of neural networks in living organisms by processing information through interconnected nodes or neurons. Another model is k-Nearest Neighbor (KNN), an instance-based learning algorithm that labels new data points by comparing them to the K closest labeled data points in the training dataset. The third model is Visual Geometry Group-16 (VGG-16), a 16-layer Convolutional Neural Network (CNN) that is highly regarded for its simplicity and effectiveness and is one of the top-performing CNN models used by VGG. Lastly, GN-AlexNet is a hybrid learning mode that combines the GoogleNet structure with the AlexNet model. The remainder of the article explains the utilization of this AI in three application scenarios, namely as an assistant for doctors, patients, and insurance companies. Moreover, the paper highlights potential challenges that may arise in both the present and future, such as patient mistrust and the emergence of new models like Vision Transformer (ViT) that could potentially outperform CNNs.

Approximation of Brownian Motion on Simple Graphs

This article is based on chapters 9 and 19 of the new neural network approximation monograph written by the first author. We use the approximation properties coming from the parametrized and deformed neural networks based on the parametrized error and q-deformed and β-parametrized half-hyperbolic tangent activation functions. So, we implement a univariate theory on a compact interval that is ordinary and fractional. The result is the quantitative approximation of Brownian motion on simple graphs: in particular over a system S of semiaxes emanating from a common origin radially arranged and a particle moving randomly on S. We produce a large variety of Jackson-type inequalities, calculating the degree of approximation of the engaged neural network operators to a general expectation function of this kind of Brownian motion. We finish with a detailed list of approximation applications related to the expectation of important functions of this Brownian motion. The differentiability of our functions is taken into account, producing higher speeds of approximation.

DeepFire2: A Convolutional Spiking Neural Network Accelerator on FPGAs

Brain-inspired spiking neural networks (SNNs) replace the multiply-accumulate operations of traditional neural networks by integrate-and-fire neurons, with the goal of achieving greater energy efficiency. Specialized hardware implementations of those neurons clearly have advantages over general-purpose devices in terms of power and performance, but exhibit poor scalability when it comes to accelerating large neural networks. DeepFire2 introduces a hardware architecture which can map large network layers efficiently across multiple super logic regions in a multi-die FPGA. That gives more control over resource allocation and parallelism, benefiting both throughput and energy consumption. Avoiding the use of lookup tables to implement the <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AND</monospace> operations of an SNN, prevents the layer size to be limited by logic resources. A deep pipeline does not only lead to an increased clock speed of up to 600 MHz. We double the throughput and power efficiency compared to our previous version of DeepFire, which equates to an almost 10-fold improvement over other previous implementations. Importantly, we are able to deploy a large ImageNet model, while maintaining a throughput of over 1500 frames per second.

Theoretical Enhancement of Energy Production Performance in PV Arrays through Effective Shadow Detection Using Hybrid Technique

This manuscript proposes an effective hybrid approach for shadow detection to increase the energy production of solar photovoltaic (PV) arrays. The proposed method is the joint operation of a deep neural network (DNN), and a gradient boosting decision tree (GBDT), together called as DNN-GBDT. The key aim is to improve the energy production performance of photovoltaic arrays. The proposed method dynamically reconfigures the PV system by tracking the shadow. The proposed method contains the following: (1) a camera system constantly monitors the photovoltaic array for moving the shadow track, (2) the image processing unit identifies the shading parts as full or partial, (3) the control decision unit finds the optimum panel layout. Then, by merging the original image and the shadow prior map, the deep neural network (DNN) efficiently trains the shadow samples. The output received is given to the GDBT classification and is processed by the identified and the background image to detect the foreground region. By this process, the normal region and the shadow region are detected. The performance of the proposed method is implemented in MATLAB platform and is compared with various existing methods. The maximum power of PV is 490 W, which is higher than the existing methods.

Neural network operators with hyperbolic tangent functions

Neural network operators with hyperbolic tangent functions

An energy-efficient 10T SRAM in-memory computing macro for artificial intelligence edge processor

In-Memory Computing (IMC) is emerging as a new paradigm to address the von-Neumann bottleneck (VNB) in data-intensive applications. In this paper, an energy-efficient 10T SRAM-based IMC macro architecture is proposed to perform logic, arithmetic, and In-memory Dot Product (IMDP) operations. The average write margin and read margins of the proposed 10T SRAM are improved by 40% and 2.5%, respectively, compared to the 9T SRAM. The write energy and leakage power of the proposed 10T SRAM are reduced by 89% and 83.8%, respectively, with aproximatelly similar read energy compared to 9T SRAM. Additionally, a 4 Kb SRAM array based on 10T SRAM is implemented in 180-nm SCL technology to analyze the operation and performance of the proposed IMC macro architecture. The proposed IMC architecture achieves an energy efficiency of 5.3 TOPS/W for 1-bit logic, 4.1 TOPS/W for 1-bit addition, and 3.1 TOPS/W for IMDP operations at 1.8 V and 60 MHz. The area efficiency of 65.2% is achieved for a 136 × 32 array of proposed IMC macro architecture. Further, the proposed IMC macro is also tested for accelerating the IMDP operation of neural networks by importing linearity variation analysis in Tensorflow for image classification on MNIST and CIFAR datasets. According to Monte-Carlo simulations, the IMDP operation has a standard deviation of 0.07 percent in accumulation, equating to a classification accuracy of 97.02% on the MNIST dataset and 88.39% on the CIFAR dataset.

Graph Neural Network Operators: a Review

Graph Neural Network Operators: a Review