ReLU Activation Function Research Articles

Efficient quantum circuits for machine learning activation functions including constant T-depth ReLU

In recent years, Quantum Machine Learning (QML) has increasingly captured the interest of researchers. Among the components in this domain, activation functions hold a fundamental and indispensable role. Our research focuses on the development of activation functions quantum circuits for integration into fault-tolerant quantum computing architectures, with an emphasis on minimizing T-depth. Specifically, we present novel implementations of ReLU and leaky ReLU activation functions, achieving constant T-depths of 4 and 8, respectively. Leveraging quantum lookup tables, we extend our exploration to other activation functions such as the sigmoid. This approach enables us to customize precision and T-depth by adjusting the number of qubits, making our results more adaptable to various application scenarios. This study represents a significant advancement towards enhancing the practicality and application of quantum machine learning. Published by the American Physical Society 2024

Developing a machine learning-based methodology for optimal hyperparameter determination—A mathematical modeling of high-pressure and high-temperature drilling fluid behavior

Drilling fluids exhibit complex rheological behavior due to a non-linear response to shear rate variations and high sensitivity to changes in temperature, time, and pressure conditions. The prediction of drilling fluid rheological behavior is crucial for the success of oil well drilling, and it directly impacts the fluid's performance. The dataset used in this study was obtained from extensive rheometric tests of water-based and olefin-based drilling fluids in steady-state flow curves. The optimal hyperparameters were guided by performance metrics and compared with alternative models such as Power-law and Herschel-Bulkley rheological models. Different configurations with different hidden layers, using neuron sequences of 16, 32, and 64, learning rates of 0.001 and 0.01, and the ReLU activation function were used to improve the model's performance. Additionally, the paper delved into the impact of the number of training epochs on the accuracy of shear stress predictions. Finding this equilibrium was identified as a crucial factor in achieving precise results. The neural network model demonstrated remarkable accuracy when using the ML-C3 configuration, with MAE values of 0.535 and R2 of 0.987 in predicting the steady-state flow curves of drilling fluids, establishing itself as a powerful tool for forecasting the rheological behavior of these fluids under diverse operational conditions. The present research significantly contributes to the field of drilling fluid rheology and provides valuable insights for optimizing drilling operations in HPHT environments.

Analytical Neural Network System for the Helicopter Turboshaft Engines Operating Modes Classification

A novel neural network-based analytical system has been developed for classifying helicopter turboshaft engine operating modes during flight operations. This system utilizes a neural network architecture comprising an input layer with three neurons, two fully connected hidden layers, and an output layer with two neurons. The proposed approach demonstrates an exceptional recognition accuracy of 0.997 (99.7%) across steady-state, unsteady, and transient operating modes of helicopter turboshaft engines. A new method for training this neural network has been introduced, employing forward propagation, loss calculation, backpropagation, and weight updates, enhanced by an adaptive learning rate and the cross-entropy function as the loss criterion. The method also incorporates a novel modified Smooth ReLU activation function for hidden layer neurons. This innovation led to a near-perfect accuracy in network training and reduced the loss to 0.025 (2.5%), highlighting the high quality and reliability of the neural network in classifying engine operating modes during flight. Furthermore, it has been empirically shown that the application of this neural network significantly reduces type I errors by a factor of 2.09 to 2.14 and type II errors by 2.05 to 2.21 times compared to traditional classifiers based on ART-1 and BAM networks. This advancement marks a substantial improvement in classification accuracy and error minimization for helicopter turboshaft engine operating modes.

A Convolutional Neural Network Approach for Precision Fish Disease Detection

Background: Detecting and classifying fish diseases is crucial for maintaining the health and sustainability of aquaculture systems. This study employs deep learning techniques, particularly Convolutional Neural Networks (CNNs), to automate the detection of various fish diseases using image data. Methods: The study utilizes a carefully curated dataset sourced from the Kaggle database, comprising images representing seven distinct types of fish diseases, along with images of healthy fish. Data preprocessing techniques, including resizing, rescaling, denoising, sharpening, and smoothing, are applied to enhance image quality and facilitate accurate disease detection. Data augmentation is employed to increase the model's ability to generalize to unseen data. The CNN architecture is designed with cascading convolutional layers, ReLU activation functions, and pooling operations to extract high-level features associated with fish infections. The model architecture, implemented using the Keras Sequential API, includes convolutional layers, max pooling layers, and densely connected layers for classification. Results: Experimental results demonstrated promising performance across various disease categories, with high accuracy and balanced precision and recall values for most classes. The study also discussed the impact of climate change on fish disease incidence and underscores the importance of effective monitoring and management practices facilitated by technological innovations such as Big Data, IoT, sensors, and robotics in ensuring sustainable fisheries management.

Deep learning Methods for ECG-Based Heart Disease Detection

Cardiovascular disease (CVD) continues to be a primary cause of death globally, and early detection plays a critical role in improving patient outcomes. This research introduces the development of a deep learning model designed to automatically categorize heart diseases using Electrocardiogram (ECG) data. The model utilizes a 1D Convolutional Neural Network (CNN) structure and makes use of the MIT-BIH Arrhythmia dataset from Physionet. The dataset was split into training, validation, and testing subsets. Our proposed design incorporates convolutional layers, max-pooling, ReLU activation functions, and dropout layers to prevent overfitting. Comparative assessment against conventional methods such as logistic regression and Support Vector Machines (SVM) shows superior performance, achieving an accuracy of 98.29%, recall of 87.60%, precision of 93.75%, and F1 score of 90.37%. The potential of deep learning to enhance the accuracy and efficiency of diagnosing CVD from ECG data is highlighted in this study, introducing a reliable tool for clinical application.

Phase unwrapping in digital holography based on SRDU-net

In digital holographic measurement, the hologram phase is extracted using an inverse tangent function, resulting in a wrapped phase that is constrained to be (-π,π]. However, the phase contains the height information of the object, which is crucial for accurate measurement of the object contour. Therefore, a digital holographic phase unwrapping method based on SRDU-Net (Separable-Residual-Dense-Inverted U-Net) is proposed in this paper. The SRDU-Net network is constructed by introducing depth-separable convolution, inverted residuals and dense blocks with U-Net as the network framework to achieve high-precision hologram phase recovery. This network adopts depth-separable convolution instead of traditional convolution, combines the inverse residual connection to construct a lightweight convolution structure, and defines dense blocks with this structure to form a lightweight grouped deep convolution network. Meanwhile, the Leaky ReLU activation function is used to introduce the learning rate mechanism to optimize the network parameters with Huber and MSE (mean square error) as the combined loss function. The simulated phase dataset is used to train the network, the trained network model is tested for speckle noise immunity, and phase unwrapping experiments are performed on the collected holograms of the test samples. The results show that SRDU-Net improves the SSIM by 0.1% and reduces the RMSE by 86% over Res-UNet (Residual-UNet). Therefore, the proposed method can realize high-precision recovery of digital holographic wrapped phases, and has a good robustness to phase unwrapping of holograms containing a high degree of speckle noise.

Machine Learning-Based Remote Sensing Inversion of Non-Photosynthetic/Photosynthetic Vegetation Coverage in Desertified Areas and Its Response to Drought Analysis

Determining the responses of non-photosynthetic vegetation (NPV) and photosynthetic vegetation (PV) communities to climate change is crucial in illustrating the sensitivity and sustainability of these ecosystems. In this study, we evaluated the accuracy of inverting NPV and PV using Landsat imagery with random forest (RF), backpropagation neural network (BPNN), and fully connected neural network (FCNN) models. Additionally, we inverted MODIS NPV and PV time-series data using spectral unmixing. Based on this, we analyzed the responses of NPV and PV to precipitation and drought across different ecological regions. The main conclusions are as follows: (1) In NPV remote sensing inversion, the softmax activation function demonstrates greater advantages over the ReLU activation function. Specifically, the use of the softmax function results in an approximate increase of 0.35 in the R2 value. (2) Compared with a five-layer FCNN with 128 neurons and a three-layer BPNN with 12 neurons, a random forest model with over 50 trees and 5 leaf nodes provides better inversion results for NPV and PV (R2_RF-NPV = 0.843, R2_RF-PV = 0.861). (3) Long-term drought or heavy rainfall events can affect the utilization of precipitation by NPV and PV. There is a high correlation between extreme precipitation events following prolonged drought and an increase in PV coverage. (4) Under long-term drought conditions, the vegetation in the study area responded to precipitation during the last winter and growing season. This study provides an illustration of the response of semi-arid ecosystems to drought and wetting events, thereby offering a data basis for the effect evaluation of afforestation projects.

Development and performance evaluation of fully automated deep learning-based models for myocardial segmentation on T1 mapping MRI data

To develop a deep learning-based model capable of segmenting the left ventricular (LV) myocardium on native T1 maps from cardiac MRI in both long-axis and short-axis orientations. Models were trained on native myocardial T1 maps from 50 healthy volunteers and 75 patients using manual segmentation as the reference standard. Based on a U-Net architecture, we systematically optimized the model design using two different training metrics (Sørensen-Dice coefficient = DSC and Intersection-over-Union = IOU), two different activation functions (ReLU and LeakyReLU) and various numbers of training epochs. Training with DSC metric and a ReLU activation function over 35 epochs achieved the highest overall performance (mean error in T1 10.6 ± 17.9 ms, mean DSC 0.88 ± 0.07). Limits of agreement between model results and ground truth were from -35.5 to + 36.1 ms. This was superior to the agreement between two human raters (-34.7 to + 59.1 ms). Segmentation was as accurate for long-axis views (mean error T1: 6.77 ± 8.3 ms, mean DSC: 0.89 ± 0.03) as for short-axis images (mean error ΔT1: 11.6 ± 19.7 ms, mean DSC: 0.88 ± 0.08). Fully automated segmentation and quantitative analysis of native myocardial T1 maps is possible in both long-axis and short-axis orientations with very high accuracy.

A performance neural network model for conventional solar stills via transfer learning

Predictive solar-desalination models are becoming more widespread using ML and AI. However, forecasting solar still water productivity based on numerous designs still needs to be improved. Herein, we used transfer learning to create precise supervised predictive ANN regression models for water productivity (L/m2.day) predictions based on literature findings. Such observation datasets from single-basin solar stills were utilized to build the random initialization ANN model. The transfer learning method was applied to the latter model by taking the learned network (weights) for fine-tuning the hyperparameters from the earlier developed novel hybrid solar still known as the source (pre-trained) ANN model, to predict the target ANN model. Based on most minor statistical errors, the pre-trained model with 5–64–64-1 architecture and ReLU activation function was the most appropriate for water productivity prediction. All created ANN models were compared to the MLR model. The results revealed that the generated target ANN model outperformed the ANN RI and MLR with OI values of 0.872, 0.834, and 0.803, respectively, in all modeling stages. The target ANN model's accuracy and generalization were sufficient. The target ANN model had residuals of forecasted distillate values of around 1%. This work discusses the significance of transfer learning to generate accurate target ANN models for predicting freshwater outputs in single-slope solar stills, which can be integrated with established theoretically tuned parameters to enhance performance and maximize distillate water yields.

A Robust Deep Learning Model for Brain Tumor Detection and Classification Using Efficient Net: A Brief Meta-Analysis

Accurately detecting and classifying brain tumors’, is critical for timely diagnosis and effective treatment planning. The purpose of this paper is to provide a comprehensive examination utilizing the EfficientNet family of deep learning architectures to automatically identify and categorize three forms of brain tumors from magnetic resonance imaging (MRI) scans. The primary aim of the study is to assess the performance of different EfficientNet models (ranging from EfficientNet-B0 to EfficientNet-B7) and determine their capability to achieve high accuracy in brain tumor classification. In the implementation, a diverse dataset is compiled comprising approximately 18,500 MRI images, representing various types of brain tumors. EfficientNet models are trained, validated, and tested using a combination of Softmax and ReLU activation functions, along with the Adam optimizer by employing learning rates of 0.0005 and 0.00001 for model training and optimization. The experimental results underscore the significant potential of the EfficientNet family in brain tumor classification. The findings reveal a consistent improvement in tumor detection accuracy as model complexity increases. The attained accuracies for different EfficientNet models are; 96.07% (EfficientNet-B0), 97.86% (EfficientNet-B1), 98.21% (EfficientNet-B2), 97.86% (EfficientNet-B3), 98.93% (EfficientNet-B4), 99.64% (EfficientNet-B5), 98.57% (EfficientNet-B6), and 99.64% (EfficientNet-B7), respectively. The proposed research not only validates the efficiency of the EfficientNet architecture in classifying brain tumors but also offers valuable insights into how model complexity influences classification performance. The notably high accuracy rates emphasize the clinical promise of employing deep learning methods to aid radiologists and medical experts in precise and efficient brain tumor diagnosis. Additionally, the paper's scope adds to the growing body of knowledge regarding the application of deep learning techniques to enhance medical image analysis and diagnostic capabilities.

Improved research on coral bleaching detection model based on FCOS model

Coral bleaching detection is vital for assessing coral reef health. This paper introduces FCOS_EfficientNET, an improved model that enhances accuracy, recall, and real-time performance in coral bleaching detection. Utilizing EfficientNet as the backbone, the model optimizes parameter throughput. We adopt the ReLU activation function and utilize cosine similarity and softmax to assign weights to datasets, modifying the attention structure to reduce memory consumption. The model also integrates BiFPN for better feature extraction and employs an improved training method to enhance detection accuracy. To cater to different scenarios, we have developed four variants: FCOS_EfficientNETb0, FCOS_EfficientNETb1, FCOS_EfficientNETb2, and FCOS_EfficientNETb3. Experimental results on the MS COCO dataset show that FCOS_EfficientNETb3 achieves a mean average precision (mAP) of 48.5%, while FCOS_EfficientNETb0 reaches a frame rate of 167.17 fps, highlighting the superior performance of the series. On a custom coral bleaching detection dataset, FCOS_EfficientNETb3 achieves 81.5% accuracy and a 59.3% recall rate, demonstrating the effectiveness of the model. FCOS_EfficientNETb1 and FCOS_EfficientNETb2 offer a balance between operations per second, frame rate, and mAP, making them suitable for mobile and edge computing. These models effectively track movement or changes in marine traffic around coral reefs with moderate fps and recall rates.

An improved long short-term memory with denoising autoencoder for solving text classification problems

In the past decade, text classification has been a popular research field for automatically categorizing documents into relevant categories. Predicting classes for input samples based on their features is the goal of classification. To limit the dimension of the feature space for precise text categorization, feature selection approaches are widely used for recognizing important and affective features while ignoring unimportant, weak, and chaotic components. This paper proposes a new model referred to as DAE-LSTM for reducing data dimension through the use of a denoising autoencoder. (Long Short-Term Memory (LSTM) is one of the popular text classification approaches, particularly when dealing with sequential datasets. Additionally, the Rectified Linear Unit (Relu) activation function was used instead of the hyperbolic tangent activation function (tanh) to improve the accuracy of the model. Four benchmark datasets were used to evaluate the proposed model against 4 cutting-edge text classification methods, including conventional Bidirectional LSTM, Bidirectional GRU, CNN-LSTM, and CNN-GRU. The suggested model has been found to perform much better than current methods using several kinds of performance assessment measures.

Deep Learning Application as a Method for Classification Equipment Failure on Natural Gas Pipeline

Predicting maintenance and preventing failures in natural gas pipelines is critical for safety and efficiency in the oil and gas industry. This research utilizes machine learning (ML) and deep learning (DL) methods to predict failures in natural gas transmission pipelines. Using a dataset from the U.S. Department of Transportation’s Pipeline and Hazardous Materials Safety Administration (PHMSA), the study examines failure incidents from 2010-2019 and non-failure data from 2020-2023. The research involves comprehensive data preprocessing, addressing missing values and dataset imbalances. Key pipeline attributes such as diameter, wall thickness, age, and operating pressure are used for failure prediction. The ML approach employs logistic regression, optimized for binary classification tasks, to establish the probability of pipeline failures. For the DL approach, an Artificial Neural Network (ANN) with Feed Forward and Back Propagation algorithms is developed. Key parameters, including activation functions and hyperparameters, are finely tuned to enhance predictive performance. ReLU activation functions are used in input and hidden layers, while the sigmoid function is applied to the output layer. Performance evaluation of both ML and DL models is based on accuracy, F1 Score, and Area Under Curve (AUC). In summary, ML showed better performance than DL, with value of accuracy is 0.74, F1 Score is 0.85, and AUC is 0.55 while the value of accuracy, F1 score, AUC from DL is 0.61, 0.003, and 0.58 respectively. DL offers superior performance in handling complex and huge amounts of data. In the other hand, ML offers more robust models and simpler adjustable models than DL. Because of the amount of computed data in this research, it is better to have ML as a predictive algorithm.

Plant leaf disease recognition based on improved SinGAN and improved ResNet34.

The identification of plant leaf diseases is crucial in precision agriculture, playing a pivotal role in advancing the modernization of agriculture. Timely detection and diagnosis of leaf diseases for preventive measures significantly contribute to enhancing both the quantity and quality of agricultural products, thereby fostering the in-depth development of precision agriculture. However, despite the rapid development of research on plant leaf disease identification, it still faces challenges such as insufficient agricultural datasets and the problem of deep learning-based disease identification models having numerous training parameters and insufficient accuracy. This paper proposes a plant leaf disease identification method based on improved SinGAN and improved ResNet34 to address the aforementioned issues. Firstly, an improved SinGAN called Reconstruction-Based Single Image Generation Network (ReSinGN) is proposed for image enhancement. This network accelerates model training speed by using an autoencoder to replace the GAN in the SinGAN and incorporates a Convolutional Block Attention Module (CBAM) into the autoencoder to more accurately capture important features and structural information in the images. Random pixel Shuffling are introduced in ReSinGN to enable the model to learn richer data representations, further enhancing the quality of generated images. Secondly, an improved ResNet34 is proposed for plant leaf disease identification. This involves adding CBAM modules to the ResNet34 to alleviate the limitations of parameter sharing, replacing the ReLU activation function with LeakyReLU activation function to address the problem of neuron death, and utilizing transfer learning-based training methods to accelerate network training speed. This paper takes tomato leaf diseases as the experimental subject, and the experimental results demonstrate that: (1) ReSinGN generates high-quality images at least 44.6 times faster in training speed compared to SinGAN. (2) The Tenengrad score of images generated by the ReSinGN model is 67.3, which is improved by 30.2 compared to the SinGAN, resulting in clearer images. (3) ReSinGN model with random pixel Shuffling outperforms SinGAN in both image clarity and distortion, achieving the optimal balance between image clarity and distortion. (4) The improved ResNet34 achieved an average recognition accuracy, recognition precision, recognition accuracy (redundant as it's similar to precision), recall, and F1 score of 98.57, 96.57, 98.68, 97.7, and 98.17%, respectively, for tomato leaf disease identification. Compared to the original ResNet34, this represents enhancements of 3.65, 4.66, 0.88, 4.1, and 2.47%, respectively.

Infinite-dimensional reservoir computing

Reservoir computing approximation and generalization bounds are proved for a new concept class of input/output systems that extends the so-called generalized Barron functionals to a dynamic context. This new class is characterized by the readouts with a certain integral representation built on infinite-dimensional state–space systems. It is shown that this class is very rich and possesses useful features and universal approximation properties. The reservoir architectures used for the approximation and estimation of elements in the new class are randomly generated echo state networks with either linear or ReLU activation functions. Their readouts are built using randomly generated neural networks in which only the output layer is trained (extreme learning machines or random feature neural networks). The results in the paper yield a recurrent neural network-based learning algorithm with provable convergence guarantees that do not suffer from the curse of dimensionality when learning input/output systems in the class of generalized Barron functionals and measuring the error in a mean-squared sense.

MultiTumor Analyzer (MTA-20–55): A network for efficient classification of detected brain tumors from MRI images

Brain cancer, one of the leading causes of mortality worldwide, is caused by brain tumors. Early diagnosis of tumors and predicting their progression can help doctors to save lives. In this article, we have designed an automated approach for locating and classifying tumors from MRI images. The novelties of the research work include the following two stages: Developing an encoder-decoder type 20-Layered deep neural network (DNN) named MultiTumor Analyzer (MTA-20) with 15 down-sampling layers and 4 up-sampling layers, the segmentation is performed in the initial stage. Here, we have adhered a Leaky ReLU activation function instead of ReLU which learn a parameter with negative values that may have valuable information which is essential specifically for image segmentation. Further, a 55-layered DNN using multistage feature fusion is developed in the second stage of the work for the classification of localized tumors. The classification is performed using developed MultiTumor Analyzer (MTA-55) DNN with Softmax classifier. The efficacy of the designed network is validated using highly cited quantitative measures such as accuracy, sensitivity, specificity, dice similarity coefficient (DSC), precision, and F1-measure. It is observed that the proposed MTA-20 DNN attains the average accuracy, sensitivity, specificity, DSC, and precision of 99.2 %, 94.6 %, 99.3 %, 88 %, and 82.5 % respectively against seven state-of-the-art techniques. Also, it is found that, the proposed MTA-55 DNN provides the overall accuracy, recall, specificity, F1-measure, precision, and DSC of 99.8 %, 99.633 %, 99.844 %, 99.659 %, 99.689 %, and 99.656 % respectively as compared to thirteen state-of-the-art techniques. These results corroborate the superiority of the proposed technique.

Comparative Analysis of Recurrent Neural Network Models Performance in Predicting Bitcoin Prices

The recurring neural network is a deep learning algorithm that is commonly used to develop prediction systems. There are many variants of RNN such as RNN itself, long-short-term memory (LSTM), and gated recurring unit, so it is frequently debatable which algorithm from the RNN family has the most optimal efficiency and computation time. When developing a prediction system, sequential or time series data is required so that an accurate prediction can be made. Sequential or time series data involve data arranged in a time sequence, such as weather data, financial data, carbon emission data, and traffic data recorded over time. This research will be carried out by predicting the three RNN models against historical Bitcoin value data. The research method used is Experimental Design by comparing the performance between the three models on bitcoin value time series data, testing is done by involving hyperparameters such as Tanh, Sigmoid, and ReLU activation functions, batch size, and epochs. The aim of this research is to find out which RNN model can produce the most optimal performance and find out what performance measures can be used to evaluate and compare the performance between the three models. The results of the study show that LSTM is the most effective model with RMSE 0.012441 and MSE 0.000155 but inefficient because it takes 3 minutes 24 seconds to run the computation; in the meantime, the Tanh activation function gives the most optimal prediction than Sigmoid and RelU and therefore should be the main candidate to be used with RNN models when predicting Bitcoin prices.

Detection of fake face images using lightweight convolutional neural networks with stacking ensemble learning method.

Images and videos containing fake faces are the most common type of digital manipulation. Such content can lead to negative consequences by spreading false information. The use of machine learning algorithms to produce fake face images has made it challenging to distinguish between genuine and fake content. Face manipulations are categorized into four basic groups: entire face synthesis, face identity manipulation (deepfake), facial attribute manipulation and facial expression manipulation. The study utilized lightweight convolutional neural networks to detect fake face images generated by using entire face synthesis and generative adversarial networks. The dataset used in the training process includes 70,000 real images in the FFHQ dataset and 70,000 fake images produced with StyleGAN2 using the FFHQ dataset. 80% of the dataset was used for training and 20% for testing. Initially, the MobileNet, MobileNetV2, EfficientNetB0, and NASNetMobile convolutional neural networks were trained separately for the training process. In the training, the models were pre-trained on ImageNet and reused with transfer learning. As a result of the first trainings EfficientNetB0 algorithm reached the highest accuracy of 93.64%. The EfficientNetB0 algorithm was revised to increase its accuracy rate by adding two dense layers (256 neurons) with ReLU activation, two dropout layers, one flattening layer, one dense layer (128 neurons) with ReLU activation function, and a softmax activation function used for the classification dense layer with two nodes. As a result of this process accuracy rate of 95.48% was achieved with EfficientNetB0 algorithm. Finally, the model that achieved 95.48% accuracy was used to train MobileNet and MobileNetV2 models together using the stacking ensemble learning method, resulting in the highest accuracy rate of 96.44%.

A comparative study on ReLU Implementation using TMDFETs

In this study, we compare the implementation of the rectified linear (ReLU) activation function using transition metal dichalcogenide field-effect transistors (TMDFETs) and metal-oxide-semiconductor FETs (MOSFETs). Five TMDs - MoS 2, MoSe 2, MoTe 2, WS 2, WSe 2 along with three variants (low-power, high-performance, and multi-gate) of the MOSFETs are simulated. Three ReLU circuits utilizing these FETs are employed for the comparison. The power consumption, speed, and accuracy of the ReLU implementation are measured and compared for each circuit and each FET. Our simulation results show that the MOSFETs consume much less power than the TMDFETs and deliver more accurate ReLU functionality. However, the TMDFETs are much faster than the MOSFETs. Among the TMDFETs, the WS 2 FET stands out, as it has higher accuracy, consumes the least power and its power consumption is comparable to the MOSFETs. Additionally, WS 2 is faster compared to MOSFETs, resulting in a trade-off between power efficiency and speed. This makes WS 2 an attractive option for implementing the ReLU activation function.

Capacity bounds for hyperbolic neural network representations of latent tree structures

We study the representation capacity of deep hyperbolic neural networks (HNNs) with a ReLU activation function. We establish the first proof that HNNs can ɛ-isometrically embed any finite weighted tree into a hyperbolic space of dimension d at least equal to 2 with prescribed sectional curvature κ<0, for any ɛ>1 (where ɛ=1 being optimal). We establish rigorous upper bounds for the network complexity on an HNN implementing the embedding. We find that the network complexity of HNN implementing the graph representation is independent of the representation fidelity/distortion. We contrast this result against our lower bounds on distortion which any ReLU multi-layer perceptron (MLP) must exert when embedding a tree with L>2d leaves into a d-dimensional Euclidean space, which we show at least Ω(L1/d); independently of the depth, width, and (possibly discontinuous) activation function defining the MLP.