Composition of Activation Functions and the Reduction to Finite Domain at Fractional and Fuzzy Framework

Deep Neural Network (DNN) stands for multilayered Neural Network (NN) that is capable of progressively learn the more abstract and composite representations of the raw features of the input data received, with no need for any feature engineering. They are advanced NNs having repetitious hidden layers between the initial input and the final layer. The working principle of such a standard deep classifier is based on a hierarchy formed by the composition of linear functions and a defined nonlinear Activation Function (AF). It remains uncertain (not clear) how the DNN classifier can function so well. But it is clear from many studies that within DNN, the AF choice has a notable impact on the kinetics of training and the success of tasks. In the past few years, different AFs have been formulated. The choice of AF is still an area of active study. Hence, in this study, a novel deep Feed forward NN model with four AFs has been proposed for breast cancer classification: hidden layer 1: Swish, hidden layer, 2:-LeakyReLU, hidden layer 3: ReLU, and final output layer: naturally Sigmoidal. The purpose of the study is twofold. Firstly, this study is a step toward a more profound understanding of DNN with layer-wise different AFs. Secondly, research is also aimed to explore better DNN-based systems to build predictive models for breast cancer data with improved accuracy. Therefore, the benchmark UCI dataset WDBC was used for the validation of the framework and evaluated using a ten-fold CV method and various performance indicators. Multiple simulations and outcomes of the experimentations have shown that the proposed solution performs in a better way than the Sigmoid, ReLU, and LeakyReLU and Swish activation DNN in terms of different parameters. This analysis contributes to producing an expert and precise clinical dataset classification method for breast cancer. Furthermore, the model also achieved improved performance compared to many established state-of-the-art algorithms/models.

This work takes up the task of the determination of the rate of pointwise and uniform convergences to the unit operator of the “normalized cusp neural network operators”. The cusp is a compact support activation function, which is the composition of two general activation functions having as domain the whole real line. These convergences are given via the modulus of continuity of the engaged function or its derivative in the form of Jackson type inequalities. The composition of activation functions aims to more flexible and powerful neural networks, introducing for the first time the reduction in infinite domains to the one domain of compact support.

Deep learning has achieved unprecedented success in recent years. This approach essentially uses the composition of nonlinear functions to model the complex relationship between input features and output labels. However, a comprehensive theoretical understanding of why the hierarchical layered structure can exhibit superior expressive power is still lacking. In this paper, we provide an explanation for this phenomenon by measuring the approximation efficiency of neural networks with respect to discontinuous target functions. We focus on deep neural networks with rectified linear unit (ReLU) activation functions. We find that to achieve the same degree of approximation accuracy, the number of neurons required by a single‐hidden‐layer (SHL) network is exponentially greater than that required by a multi‐hidden‐layer (MHL) network. In practice, discontinuous points tend to contain highly valuable information (i.e., edges in image classification). We argue that this may be a very important reason accounting for the impressive performance of deep neural networks. We validate our theory in extensive experiments.

In this article, we determine analytical upper bounds on the local Lipschitz constants of feedforward neural networks with rectified linear unit (ReLU) activation functions. We do so by deriving Lipschitz constants and bounds for ReLU, affine-ReLU, and max pooling functions, and combining the results to determine a network-wide bound. Our method uses several insights to obtain tight bounds, such as keeping track of the zero elements of each layer, and analyzing the composition of affine and ReLU functions. Furthermore, we employ a careful computational approach which allows us to apply our method to large networks such as AlexNet and VGG-16. We present several examples using different networks, which show how our local Lipschitz bounds are tighter than the global Lipschitz bounds. We also show how our method can be applied to provide adversarial bounds for classification networks. These results show that our method produces the largest known bounds on minimum adversarial perturbations for large networks such as AlexNet and VGG-16.

IntroductionAnterior cruciate ligament injury and reconstruction (ACLR) is often associated with pain, functional loss, poor quality of life and accelerated knee osteoarthritis development. The effectiveness of interventions to enhance outcomes...

Individuals undergoing chronic hemodialysis represent a population with high morbidity and mortality, primarily due to poor nutritional status, chronic inflammation, and cardiovascular disease. However, additional factors, such as low physical activity and impaired functionality, have also been identified as directly associated with increased mortality. This study was conceived as a pilot study to investigate whether creatine supplementation (5g/day) for eight weeks could provide benefits in terms of physical functionality, handgrip and body composition in a group of adult patients on chronic hemodialysis. On dialysis days, creatine was administered immediately post-dialysis, while on non-dialysis days, patients took the supplement at home. Measurements were taken using bioimpedance analysis, handgrip strength (via dynamometry), and the Short Physical Performance Battery (SPPB), both before starting creatine supplementation and at week 8 of treatment. After performing robust statistical analysis, following creatine supplementation, an increase in SPPB scores was observed, with a mean improvement of 0.78 points [95% CI: 0.17-1.44] and an effect size of 0.53. Skeletal muscle mass increased by an average of 1.31 kg [95% CI: 0.55 to 2.23], with an effect size of 0.66. Fat-free mass showed a mean increase of 2.11 kg [95% CI: 0.75 to 3.58] with an effect size of 0.64, while phase angle rose by 0.52 degrees [95% CI: 0.27 to 0.76], corresponding to an effect size of 0.90. Regarding volumetric estimates, total body water increased by 1.17 L [95% CI: 0.26 to 2.13] with an effect size of 0.54, and intracellular water increased by 0.97 L [95% CI: 0.48 to 1.51] with an effect size of 0.81. No significant differences were observed in extracellular water with change of 0.20 L [95% CI: -0.30 to 0.70] or handgrip strength with an increment of 0.67 kgF, [95% CI: -0.67 to 2.11]. Oral creatine supplementation in HD patients for eight weeks improved muscular and functional outcomes and may be proposed as a strategy to mitigate the elevated morbidity observed in this group of patients.

Changes in glycerophospholipid metabolism with age and disease can have a profound effect on immune cell activation and effector function. We previously demonstrated that glycerol-3-phosphate acyltransferase-1, the first and rate limiting step in de novo glycerophospholipid synthesis, plays a role in modulating murine T cell function. The resultant phenotype is characterized by decreased IL-2 production, increased propensity toward apoptosis, and altered membrane glycerophospholipid mass similar to that of an aged T cell. Since T cells in previous experiments were harvested from GPAT-1(-/-) mice, questions remained as to what extent the macro environment of the model influenced the observed cellular phenotype. Therefore, we generated and phenotypically characterized a mitochondrial glycerol-3-phosphate acyltransferase (GPAM) deficient Jurkat T cell. Furthermore, this line was used to probe possible mechanisms by which GPAT-1/GPAM regulates T cell function. We report here that many of the key dysfunctional characteristics of murine GPAT-1(-/-) T cells are recapitulated in the GPAMKD Jurkat T cell. We found striking decreased IL-2 production along with altered phospholipid mass and increased incidence of apoptosis. Since PtdOH is an indirect downstream product of GPAM, we attempted to rescue IL-2 production with PtdOH supplementation; however, this addition did not return IL-2 production to normal levels. Interestingly, we did find significantly decreased Zap-70 phosphorylation following stimulation, suggesting that GPAM deficiency may alter membrane based stimulatory signaling. These data show for the first time that GPAM deficiency results in an inherent defect in Jurkat T cell function and glycerophospholipid composition and that this defect cannot be rescued by addition of exogenous PtdOH.

Background & Aims: Bacterial translocation and intestinal dysbiosis due to gut barrier dysfunction are widely recognized as major causes of the initiation and development of intra-abdominal sepsis. Systemic bacterial translocation and hepatic activation of the myeloid differentiation primary response gene 88 (MyD88) can disturb bile acid (BAs) metabolism, further exacerbating intestinal dysbiosis. The Farnesoid X receptor (FXR) and fibroblast growth factor (FGF) 15/19 are well known to be involved in the control of BAs synthesis and enterohepatic circulation. However, the influence of intestinal microbiota on intestinal MyD88 signaling, the FXR/FGF15 axis, as well as gut-liver crosstalk during sepsis remains unclear. The present study aims to decipher the role of intestinal MyD88 in abdominal sepsis, its impact on intestinal FXR signaling and FGF15-mediated gut-liver crosstalk. Methods: Expression levels of FXR and FGF15 in the liver and intestines, alongside assessments of gut barrier function, were evaluated in septic wild-type (WT) mice 24 hours post-caecal ligation and puncture (CLP) surgery. Subsequently, the FXR agonist INT-747 was administered to explore the relationship between FXR activation and gut barrier function. Further investigations involved MyD88-deficient mice with specific deletion of MyD88 in intestinal epithelial cells (MyD88△IEC), subjected to CLP to examine the interplay among intestinal MyD88, FXR, gut barrier function, microbiota, and BA composition. Additionally, fecal microbiota transplantation (FMT) from septic mice to MyD88△IEC mice was conducted to study the impact of dysbiosis on intestinal MyD88 expression during sepsis, using floxed (MyD88fl/fl) mice as controls. Finally, the effects of the probiotic intervention on gut barrier function and sepsis outcomes in CLP mice were investigated. Results: Induction of sepsis via CLP led to hepatic cholestasis, suppressed FXR-FGF15 signaling, altered gut microbiota composition, and compromised gut barrier function. Administration of INT-747 increased intestinal FXR and FGF15 expression, strengthened gut barrier function, and enhanced barrier integrity. Interestingly, MyD88△IEC mice exhibited partial reversal of sepsis-induced changes in FXR signaling, BA metabolism, and intestinal function, suggesting enhanced FXR expression upon MyD88 knockdown. Moreover, FMT from septic mice activated intestinal MyD88, subsequently suppressing FXR-FGF15 signaling, exacerbating cholestasis, and ultimately compromising gut barrier function. Probiotic treatment during abdominal sepsis mitigated flora disturbances, reduced MyD88 activation in the intestinal epithelium, increased FXR expression, alleviated cholestasis, and consequently reduced barrier damage. Conclusions: This study highlights the critical role of MyD88/FXR signaling in intestinal epithelial cells as a pivotal mediator of the detrimental effects induced by sepsis-related intestinal dysbiosis on barrier function and bile acid metabolism. Probiotics show promise in restoring intestinal homeostasis by leveraging intestinal MyD88 and FXR signaling to preserve barrier function and improve survival.

Background and aimsBacterial translocation and intestinal dysbiosis due to gut barrier dysfunction are widely recognized as major causes of the initiation and development of intra-abdominal sepsis. Systemic bacterial translocation and hepatic activation of the myeloid differentiation primary response gene 88 (Myd88) can disturb bile acids (BAs) metabolism, further exacerbating intestinal dysbiosis. The farnesoid X receptor (FXR) and fibroblast growth factor (FGF) 15/19 are well known to be involved in the control of BAs synthesis and enterohepatic circulation. However, the influence of intestinal microbiota on intestinal Myd88 signaling, the FXR/FGF15 axis, as well as gut-liver crosstalk during sepsis remains unclear. The present study aims to decipher the role of intestinal Myd88 in abdominal sepsis, its impact on intestinal FXR signaling and FGF15-mediated gut-liver crosstalk.MethodsExpression levels of FXR and FGF15 in the liver and intestines, alongside assessments of gut barrier function, were evaluated in septic wild-type (WT) mice 24 h post-cecal ligation and puncture (CLP) surgery. Subsequently, the FXR agonist INT-747 was administered to explore the relationship between FXR activation and gut barrier function. Further investigations involved Myd88-deficient mice with specific deletion of Myd88 in intestinal epithelial cells (Myd88△IEC), subjected to CLP to examine the interplay among intestinal Myd88, FXR, gut barrier function, microbiota, and BA composition. Additionally, fecal microbiota transplantation (FMT) from septic mice to Myd88△IEC mice was conducted to study the impact of dysbiosis on intestinal Myd88 expression during sepsis, using floxed (Myd88fl/fl) mice as controls. Finally, the effects of the probiotic intervention on gut barrier function and sepsis outcomes in CLP mice were investigated.ResultsInduction of sepsis via CLP led to hepatic cholestasis, suppressed FXR-FGF15 signaling, altered gut microbiota composition, and compromised gut barrier function. Administration of INT-747 increased intestinal FXR and FGF15 expression, strengthened gut barrier function, and enhanced barrier integrity. Interestingly, Myd88△IEC mice exhibited partial reversal of sepsis-induced changes in FXR signaling, BA metabolism, and intestinal function, suggesting enhanced FXR expression upon Myd88 knockdown. Moreover, FMT from septic mice activated intestinal Myd88, subsequently suppressing FXR-FGF15 signaling, exacerbating cholestasis, and ultimately compromising gut barrier function. Probiotic treatment during abdominal sepsis mitigated flora disturbances, reduced Myd88 activation in the intestinal epithelium, increased FXR expression, alleviated cholestasis, and consequently reduced barrier damage.ConclusionsThis study highlights the critical role of Myd88/FXR signaling in intestinal epithelial cells as a pivotal mediator of the detrimental effects induced by sepsis-related intestinal dysbiosis on barrier function and bile acid metabolism. In summary, disordered intestinal flora in septic mice specifically triggers intestinal epithelial Myd88 activation, inhibit the FXR-FGF15 axis, and then worsen intestinal barrier function impairment.Graphical

A traditional radial basis function (RBF) network takes Gaussian functions as its basis functions and adopts the least squares (LS) criterion as the objective function. However, it is difficult to use Gaussian functions to approximate constant values. If a function has nearly constant values in some intervals, the RBF network will be found inefficient in approximating these values. In this paper an RBF network which uses composite of sigmoidal functions to replace the Gaussian functions as the basis function of the network is proposed. It is also illustrated that the shape of the activation function can be constructed to be a similar rectangular or Gaussian function. Thus, the constant-valued functions can be approximated accurately by an RBF network. A robust objective function is also adopted in the network to replace the LS objective function. Experimental results demonstrated that the proposed network has better capability of approximation to underlying functions with a fast learning speed and high robustness to outliers.

A general lack of understanding pertaining to deep feedforward neural networks (DNNs) can be attributed partly to a lack of tools with which to analyze the composition of non-linear functions, and partly to a lack of mathematical models applicable to the diversity of DNN architectures. In this study, we analyze DNNs using directed acyclic graphs (DAGs) under a number of basic assumptions pertaining to activation functions, non-linear transformations, and DNN architectures. DNNs that satisfy these assumptions are referred to as general DNNs. Our construction of an analytic graph was based on an axiomatic method in which DAGs are built from the bottom–up through the application of atomic operations to basic elements in accordance with regulatory rules. This approach allowed us to derive the properties of general DNNs via mathematical induction. We demonstrate that the proposed analysis method enables the derivation of some properties that hold true for all general DNNs, namely that DNNs “divide up” the input space, “conquer” each partition using a simple approximating function, and “sparsify” the weight coefficients to enhance robustness against input perturbations. This analysis provides a systematic approach with which to gain theoretical insights into a wide range of complex DNN architectures.

This paper presents NeurEPDiff, a novel network to fast predict the geodesics in deformation spaces generated by a well known Euler-Poincaré differential equation (EPDiff). To achieve this, we develop a neural operator that for the first time learns the evolving trajectory of geodesic deformations parameterized in the tangent space of diffeomorphisms (a.k.a velocity fields). In contrast to previous methods that purely fit the training images, our proposed NeurEPDiff learns a nonlinear mapping function between the time-dependent velocity fields. A composition of integral operators and smooth activation functions is formulated in each layer of NeurEPDiff to effectively approximate such mappings. The fact that NeurEPDiff is able to rapidly provide the numerical solution of EPDiff (given any initial condition) results in a significantly reduced computational cost of geodesic shooting of diffeomorphisms in a high-dimensional image space. Additionally, the properties of discretiztion/resolution-invariant of NeurEPDiff make its performance generalizable to multiple image resolutions after being trained offline. We demonstrate the effectiveness of NeurEPDiff in registering two image datasets: 2D synthetic data and 3D brain resonance imaging (MRI). The registration accuracy and computational efficiency are compared with the state-of-the-art diffeomophic registration algorithms with geodesic shooting.

With the increasing popularity of nonconvex deep models, developing a unifying theory for studying the optimization problems that arise from training these models becomes very significant. Toward this end, we present in this paper a unifying landscape analysis framework that can be used when the training objective function is the composite of simple functions. Using the local openness property of the underlying training models, we provide simple sufficient conditions under which any local optimum of the resulting optimization problem is globally optimal. We first completely characterize the local openness of the symmetric and nonsymmetric matrix multiplication mapping. Then we use our characterization to (1) provide a simple proof for the classical result of Burer-Monteiro and extend it to noncontinuous loss functions; (2) show that every local optimum of two-layer linear networks is globally optimal. Unlike many existing results in the literature, our result requires no assumption on the target data matrix [Formula: see text], and input data matrix [Formula: see text]; (3) develop a complete characterization of the local/global optima equivalence of multilayer linear neural networks (we provide various counterexamples to show the necessity of each of our assumptions); and (4) show global/local optima equivalence of overparameterized nonlinear deep models having a certain pyramidal structure. In contrast to existing works, our result requires no assumption on the differentiability of the activation functions and can go beyond “full-rank” cases.

There was an error in the original publication [...]

Artificial Neural Network (ANN) is a parallel connection of a set of nodes called neurons which mimic biological neural system. Statistically, ANN represents a class of non-parametric models which is capable of approximating a non-linear function by a composition of low dimensional ridge functions. This study aimed at modeling diabetes mellitus among adult Kenyan population using 2015 stepwise survey data from Kenya National Bureau of Statistics. Data analysis was carried out using R statistical software version 3.5.0. Among the input variables Age, Sex, Alcoholic status, Sugar consumption, Physical Inactivity, Obesity status, Systolic and Diastolic blood pressure had a significant relationship with diabetic status at 5% level of significance. A multi layered feed-forward neural network with a back propagation algorithm and a logistic activation function was used. Considering a parsimonious model, the model selected had the eight input variables with two neurons in the hidden layer since it gave a minimum MSE of 0.0580 reported. 75% of data was used for training while 25% was used for testing. The sensitivity of the trained network was reported as 75% while specificity was 94.29%. The overall accuracy of the model was 84.64% . This implied that the model could correctly classify an individual as either diabetic or not with an accuracy rate of 84.64%. A 10-fold cross validation was carried out and an average MSE of 0.0686 reported. Kolmogorov-Smirnov test of normality was carried out and at 5% level of significance, for most parameter estimates, we failed to reject the null hypothesis and concluded that the network parameter estimates were asymptotically normal and consistent. With a good choice of risk factors for diabetes, neural network structures could be successfully used to accurately model diabetes melitus among Kenyan adult population.