Function Gradient Descent Research Articles

Comparative study of integer-order and fractional-order artificial neural networks: Application for mathematical function generation

This paper investigates the impact of fractional derivatives on the activation functions of an artificial neural network (ANN). Based on the results and analysis, a three-layer backpropagation neural network model with fractional and integer derivatives employed in the activation function and fractional gradient descent method backpropagation learning algorithm has been proposed. Specifically, three perceptrons have been proposed based on fractional and integer derivatives applied to activation functions and learning algorithms. They are fractional derivative activation function (FDAF) perceptron, integer derivative activation function (IDAF)-fractional derivative learning algorithm (FDLA) perceptron, and fractional derivative learning algorithm (FDLA) perceptron. The Riemann-Liouville (RL) fractional derivative, Grunwald-Letnikov (GL) derivative, Caputo-Fabrizio (CF) fractional derivative, Caputo (C) fractional derivative, and Atangana-Baleanu (AB) fractional derivative have been employed. The impact of these derivatives’ fractional-order (FO) is investigated in a wide range from 0.1−0.9 on testing mean square error (MSE) and time required to train the perceptron. FO-based and IO-based perceptrons are compared with the help of performance metrics such as testing MSE and the time required to train the models. The training and testing simulation results illustrate that Caputo-Fabrizio derivative-based perceptrons outperform other fractional derivatives. Also, the performance of the FO-based perceptron is better in terms of the least MSE.

Rand-Index Target Projective Gradient Deep Belief Network for Software Fault Prediction

Predicting defective software modules before testing is a valuable operation that reduces time and cost of software testing. Source code fault prediction plays a vital role in improving software quality that effectively assists in optimization testing resource allocation. Several machine learning and ensemble learning techniques has been extensively evolved over the recent few years to predict defect at an early stage. These techniques made predictions based on historical defect data, the software metrics. Nevertheless, the time efficient and accurate fault predictions are the major challenging tasks that yet have to be addressed. In order to ensure accurate software fault prediction, a method called Rand-Index Target Projective Gradient Deep Belief Network (RTPGDBN) is designed. The proposed RTPGDBN method comprises of three processes namely acquiring JAVA packages, software metric selection and classification. First, the number of JAVA packages is used as input from the dataset. Second with the JAVA packages obtained as input, Rand similarity indexive target projection function is applied for selecting the most significant software metrics in order to minimize time complexity of fault prediction. Third, with the selected metrics, classification is performed using Tversky Gradient Deep Belief neural network. Also, gradient descent function is applied to minimize classification error, therefore ensuring accurate software fault classification results obtained at the output layer. Experimental setup of proposed RPGDDBN and existing methods are implemented in Java language and the dataset collected from smell prediction replication package. Performance analysis is carried out with different quantitative metrics such as accuracy, precision, recall, F-measure, and time complexity and space complexity. Through extensive experiments on repository data, experimental results indicate that our RPGDDBN method outperforms two state-of-the-art defect detection methods in terms of different performance metrics.

Application of Neural Network in Seismic Risk Survey

The accuracy of inversion data is determined by the reliability of linear inversion method. Therefore, the study of nonlinear seismic inversion method is a meaningful work. This paper mainly aims at the application of nonlinear neural network inversion method in Fujian Risk Survey, the following work has been accomplished: (1) The theory and algorithm of the conventional multilayer feedforward neural network and some problems in the design of the model are analyzed and summarized. Aiming at the gradient descent function search algorithm in the BP algorithm, a method for global optimization is proposed by combining simulated annealing (global search algorithm) with Conjugate gradient method (local search algorithm) , the principle and implementation of simulated annealing and conjugate gradient algorithm are analyzed in detail. (2) Based on the analysis of conventional multilayer feedforward neural networks, the model structure and basic principle of probabilistic neural networks are studied. The object function search algorithm and its realization are analyzed. The method is applied to the risk elimination prediction for the first time, and the results are satisfactory. (3) The method and principle of inversion of reservoir parameters by using various seismic attribute information and neural network technology are discussed. In order to make full use of the effective information contained in seismic signal, the method of seismic attribute extraction and optimization is studied, the principle and implementation method of attribute optimal ranking by stepwise recursive method and selecting the most important attribute combination by interactive verification method are presented.

A Generalized Shuffle Framework for Privacy Amplification: Strengthening Privacy Guarantees and Enhancing Utility

The shuffle model of local differential privacy is an advanced method of privacy amplification designed to enhance privacy protection with high utility. It achieves this by randomly shuffling sensitive data, making linking individual data points to specific individuals more challenging. However, most existing studies have focused on the shuffle model based on (ε0,0)-Locally Differentially Private (LDP) randomizers, with limited consideration for complex scenarios such as (ε0,δ0)-LDP or personalized LDP (PLDP). This hinders a comprehensive understanding of the shuffle model's potential and limits its application in various settings. To bridge this research gap, we propose a generalized shuffle framework that can be applied to PLDP setting. This generalization allows for a broader exploration of the privacy-utility trade-off and facilitates the design of privacy-preserving analyses in diverse contexts. We prove that the shuffled PLDP process approximately preserves μ-Gaussian Differential Privacy with μ = O(1/√n). This approach allows us to avoid the limitations and potential inaccuracies associated with inequality estimations. To strengthen the privacy guarantee, we improve the lower bound by utilizing hypothesis testing instead of relying on rough estimations like the Chernoff bound or Hoeffding's inequality. Furthermore, extensive comparative evaluations clearly show that our approach outperforms existing methods in achieving strong central privacy guarantees while preserving the utility of the global model. We have also carefully designed corresponding algorithms for average function, frequency estimation, and stochastic gradient descent.

Temporal convex combination‐based secure distributed estimation against cyberattacks and noisy input

AbstractThis paper proposes a new secure distributed estimation for cyber‐physical systems against adversarial attacks with noisy input. To mitigate the effect of attacks, a novel distributed attack detection based on a reliable reference estimation obtained by temporal convex combination is proposed. Furthermore, for more effective and robust performance, an adaptation rule to adjust convex combination weights is presented, in which the generalized correntropy method with nonlinear loss function and stochastic gradient descent are utilized. Besides, to eliminate input noise, a bias‐compensation method in local adaptation of the secure distributed estimation is proposed. Simulations show superior dynamic and real‐time adaptability of the proposed algorithm under complex attacking scenarios.

Differential privacy model for blockchain based smart home architecture

Secure and private communications using the Internet of Things (IoT) pose several challenges for smart home systems. In particular, data collected from IoT devices comprise sensitive personal information such as biomedical data, financial data, and location and activity data. Recent research looks into the use of blockchain in smart home systems, protecting the privacy of the data in use. Such solutions need to address the issue of privacy using a formal and mathematical model for data privacy due to the vulnerability associated with privacy-preserving blockchain networks. In the present paper, our approach aims to provide a privacy-preserving data aggregation mechanism in the context of Smart Homes that agree to contribute their data to a cloud server using machine learning to improve services for home users. We propose the use of differential privacy, a powerful concept in privacy preserving schemes to provide formal assurances about how much information is leaked using a privacy budget. The main purpose of using such a privacy-preserving scheme is to limit what can be inferred about individual training data from the model. Our techniques use a R′enyi differential privacy (RDP) machine learning scheme and are based on a variant of the stochastic gradient descent function. The performance of our proposed framework is evaluated using three public datasets: UNSW-NB15, NSL-KDD, and ToN-IoT datasets. Our findings show that differential private models can provide privacy protection against attackers by sacrificing a substantial amount of model utility. Therefore, we propose an empirical value of ϵ, that can optimally balances utility and privacy for the current smart home scenario datasets.

Artificial Neural Network Models for Image Recognition

Image recognition is an important application of artificial intelligence. Due to the continuous development of chips and algorithms, this field has made great progress in the past decade. At the same time, AI based image recognition has been widely used in the emerging fields like robotics, autonomous vehicles, and surveillance cameras, and their demand for AI has promoted the development of image recognition technology. Resent research has found that the convolutional neural network model is particularly effective for image classification and detection and smaller convolution kernels with deeper network structures are conducive to improving the accuracy. However, problems such as overfitting and activation function gradient descent need to be solved during the operation process. The latest convolutional neural network model ResNet applies residual units to reduce the redundant calculations and improve the efficiency of the model. In general, different variants of convolutional neural network structures have different effects on image recognition, but regional convolutional neural network structures are preferred in engineering applications for its balance between processing speed and recognition accuracy.

Logistic Regression for Stroke Prediction: An Evaluation of its Accuracy and Validity

A stroke, also known as a brain attack, is a serious medical condition that occurs when the blood supply to the brain is disrupted. It is a leading cause of death globally, accounting for about 11% of all deaths. In this paper, the author opts to use logistic regression for predicting the stroke. The paper starts with introducing the methods used to preprocess the raw dataset, including data cleaning, label encoding, oversampling, splitting the dataset and finally feature scaling. Then it goes to the modeling section. In this section, the author introduces the process of constructing a logistic regression model in detail. The first step is building the sigmoid function, a foundation of the model. Then the cost function is built to measure the difference between the model's predicted values and the true values. After that, the author constructs the gradient-computing function to determine the rate and direction at each iteration when implementing gradient descent. Based on this function, the gradient descent function is finally built, and predictions are made according to the outcome after putting testing variables into the model. By analyzing the results, the author also compares the different performances of the model between the one without regularization and the one equipped with regularization and then draw a conclusion that using regularization can help improve the performance of the model. At the end of the study, the author gets a rather satisfying result with a prediction accuracy of over 95% in the collected dataset.

Differentially Private Neural Networks with Bounded Activation Function

Deep learning has shown outstanding performance in various fields, and it is increasingly deployed in privacy-critical domains. If sensitive data in the deep learning model are exposed, it can cause serious privacy threats. To protect individual privacy, we propose a novel activation function and stochastic gradient descent for applying differential privacy to deep learning. Through experiments, we show that the proposed method can effectively protect the privacy and the performance of proposed method is better than the previous approaches.

RETRACTED ARTICLE: Prewitt Logistic Deep Recurrent Neural Learning for Face Log Detection by Extracting Features from Images

Face log detection (FLD) in the surveillance video extracts a new face image from the video sequences (VS). FLD utilizes biometric techniques for humans’ recognition. To improve the precise FLD with less complexity of our proposed method is Prewitt Logistic Deep Recurrent Neural Learning (PLDRNL) used. The input VS was received from the video database. Next, the keyframes are extracted from the VS. This proposed deep recurrent neural learning method uses four hidden layers to remove the facial features such as the face, eyes, nose, and mouth in the form of an edge. The edges of each element are derived using the Prewitt edge detector through the horizontal and vertical mask. Finally, the relevant features are fed into the output layer. The PLDRNL uses a logistic activation function at the output layer for matching the extracted related elements with the pre-stored testing feature vector. If two features are matched, then the face in the given VF is detected. The error in the FD is minimized using gradient descent function at the output layer. Based on the results, the human face effectively identified with the minimum false-positive rate (FPR). Experimental evaluation is carried out using different factors such as FLD, FPR, and time complexity.

Optimasi Waktu Akuisisi Data Satelit Noaa18 Menggunakan Jaringan Syaraf Tiruan Backpropagation

Earth stations are built to monitor the presence of satellites starting from satellite data, monitoring satellites, and carry out orders and corrections if needed. On the earth station there is a satellite data receiving antenna, the more elevation angle of the current satellite data receiver antenna can affect the time duration of the satellite data. The purpose of this research is to apply the Artificial Neural Network (ANN) method to design a time optimization system for satellite data at the LAPAN Pekayon earth station, East Jakarta. The data used as input is the elevation angle. The benefit of this research is expected to make it easier for operators and technicians to measure the time optimization of satellite data at earth stations. The best training results with learning rate = 0.2, error = 0.0001, max. epoch = 100000, neuron hidden layer = 15. The MSE value obtained is 0.0001 reaching the goal at epoch 68810. Regret the training / training reverse sequence reaches 0.99878. The best test result is to use learning speed 0.2 hidden layer neurons 15 comparison of training data = 54 and test data = 18. The accurate result is exactly the same as the specified error, namely 0.0001. The difference in the average target duration is 3 seconds compared to the ANN target. Artificial Neural Network (ANN) with the back propagation method of training function gradient descent (traingd), was successfully used to an optimization system for satellite data acquisition time at earth stations.

Predictive modelling of fracture behaviour in silica-filled polymer composite subjected to impact with varying loading rates using artificial neural network

In the present work, the dynamic fracture toughness of silica filled polymer composites subjected to impact loading was studied using three different loading rates corresponding to different pulse shaper conditions. These loading rates were ~107 times higher as compared to the rates usually attained in quasi-static condition for the same material. The further analysis was done using the framework of artificial neural network for neat epoxy and 10% silica filled polymer composites. Multi-layer perceptron was used to predict the crack initiation toughness of resulting composites using feed forward network. Loading rate, shear wave speed, longitudinal wave speed, volume fraction of the silica fillers and time were used as the input parameters and gradient descent function was used to estimate the optimized synaptic weights. Predicted values were compared with the experimental ones and a good agreement was found between the two. After time, loading rate was found to be the most important factor in the prediction of stress intensity factor followed by shear wave speed, longitudinal wave speed and volume fraction of the fillers used.

Deep Fuzzy Cognitive Maps for Interpretable Multivariate Time Series Prediction

The fuzzy cognitive map (FCM) is a powerful model for system state prediction and interpretable knowledge representation. Recent years have witnessed the tremendous efforts devoted to enhancing the basic FCM, such as introducing temporal factors, uncertainty or fuzzy rules to improve interpretation, and introducing fuzzy neural networks or wavelets to improve time series prediction. But how to achieve high-precision yet interpretable prediction in cross-domain real-life applications remains a great challenge. In this article, we propose a novel FCM extension called deep FCM (DFCM) for multivariate time series forecasting, in order to take both the advantage of FCM in interpretation and the advantage of deep neural networks in prediction. Specifically, to improve the predictive power, DFCM leverages a fully connected neural network to model connections (relationships) among concepts in a system, and a recurrent neural network to model unknown exogenous factors that have influences on system dynamics. Moreover, to foster model interpretability encumbered by the embedded deep structures, a partial derivative-based approach is proposed to measure the connection strengths between concepts in DFCM. An alternate function gradient descent algorithm is then proposed for parameter inference. The effectiveness of DFCM is validated over four publicly available datasets with the presence of seven baselines. DFCM indeed provides an important clue to building interpretable predictors for real-life applications.

Deep neural learning techniques with long short-term memory for gesture recognition

Gesture recognition is a kind of biometric which has assumed great significance in the field of computer vision for communicating information through human activities. To recognize the various gestures and achieve efficient classification, an efficient computational machine learning technique is required. The Shift Invariant Convolutional Deep Structured Neural Learning with Long Short-Term Memory (SICDSNL–LSTM) and Bivariate Fully Recurrent Deep Neural Network with Long Short-Term Memory (BFRDNN–LSTM) have been introduced for classifying human activities with efficient accuracy and minimal time complexity. The SICDSNL–LSTM technique collects gesture data (a kind of biometric) from the dataset and gives it to the input layers of Shift Invariant Convolutional Deep Structured Neural Learning. The SICDSNL–LSTM technique uses two hidden layers for performing regression and classification. In the regression process, dice similarity is used for measuring the relationship between data and output classes. In the second process, the input data are classified into dissimilar classes for each time step using LSTM unit with soft-step activation function. The soft-step activation function uses ‘forget gate’ for removing the less significant data from the cell state. This is also used to make a decision to display the output at a particular time step and to remove other class results. Then, LSTM output is given to the output layers, and convolutional deep neural learning is performed to classify the gesture. Based on the classification results, human activity and gesture are recognized with high accuracy. The BFRDNN–LSTM technique also performs regression in the first hidden layers using bivariate correlation to find relationship between data and classes. The LSTM unit in BFRDNN–LSTM technique uses Gaussian activation function in the second hidden layers for categorizing incoming data into various classes at each time step. In this proposed BFRDNN–LSTM method, fully recurrent deep neural network utilizes gradient descent function to minimize the error rate at the output layers and to increase the accuracy of the gesture recognition. Both SICDSNL–LSTM and BFRDNN–LSTM techniques automatically learn the features and the data to minimize time complexity in gesture recognition. Experimental evaluation is carried out using factors such as gesture recognition accuracy, false-positive rate and time complexity with a number of data.

Hybridized Gradient Descent Spectral Graph and Local global Louvain Based Clustering of Temporal Relational Data

Temporal data clustering examines the time series data to determine the basic structure and other characteristics of the data. Many methodologies simply process the temporal dimension of data but it still faces the many challenges for extracting useful patterns due to complex data types. In order to analyze the complex temporal data, Hybridized Gradient Descent Spectral Graph and Local-Global Louvain Clustering (HGDSG-LGLC) technique are designed. The number of temporal data is gathered from input dataset. Then the HGDSG-LGLC technique performs graph-based clustering to partitions the vertices i.e. data into different clusters depending on similarity matrix spectrum. The distance similarity is measured between the data and cluster mean. The Gradient Descent function find minimum distance between data and cluster mean. Followed by, the Local-Global Louvain method performs the merging and filtering of temporal data to connect the local and global edges of the graph with similar data. Then for each data, the change in modularity is calculated for filtering the unwanted data from its own cluster and merging it into the neighboring cluster. As a result, optimal ‘k’ numbers of clusters are obtained with higher accuracy with minimum error rate. Experimental analysis is performed with various parameters like clustering accuracy ( ), error rate ( ), computation time ( ) and space complexity ( ) with respect to number of temporal data. The proposed HGDSG-LGLC technique achieves higher and lesser , minimum as well as than conventional methods.

On the Banach Spaces Associated with Multi-Layer ReLU Networks: Function Representation, Approximation Theory and Gradient Descent Dynamics

We develop Banach spaces for ReLU neural networks of finite depth $L$ and infinite width. The spaces contain all finite fully connected $L$-layer networks and their $L^2$-limiting objects under bounds on the natural path-norm. Under this norm, the unit ball in the space for $L$-layer networks has low Rademacher complexity and thus favorable generalization properties. Functions in these spaces can be approximated by multi-layer neural networks with dimension-independent convergence rates. The key to this work is a new way of representing functions in some form of expectations, motivated by multi-layer neural networks. This representation allows us to define a new class of continuous models for machine learning. We show that the gradient flow defined this way is the natural continuous analog of the gradient descent dynamics for the associated multi-layer neural networks. We show that the path-norm increases at most polynomially under this continuous gradient flow dynamics.

Stochastic Embedded Probit Regressive Reweight Boost Classifier for Software Quality Examination

In software development, Software quality analysis plays a considerable process. Through the software testing, the quality analysis is performed for efficient prediction of defects in the code. Due to the complicated structure of software projects, code examination has become a demanding issue that has to be addressed at the initial stage of testing for achieving the quality improved results. In order to resolve these issues, the Stochastic Gaussian Neighbor Embedding based Probit Regressive Reweight Boost Classification (SGNE-PRRBC) is introduced for accurate quality prediction system through code examination proficient system. The SGNE-PRRBC technique considers the number of program files as input for software quality analysis through feature selection and classification. Initially, the number of program files is taken from the dataset (DS). After collecting the files, the Gaussian distributive stochastic neighbor embedding technique choose the features (i.e. code metrics) based on the distance similarity. With the assist of Pearson correlative probit regressed reweight boost technique, the classification of program files is performed. The boosting algorithm creates ‘m’ number of weak classifiers i.e. Pearson correlative probit regression to categorize the input program files as normal or defected through analyze the source codes and chosen metrics. After that, the weak learners results are combined into strong through minimizing the out of sample error with gradient descent function. This enhances the accuracy of quality prediction and lessens the false positive rate (FPR). Experimental analysis is performed with various metrics namely accuracy, FPR and computation time (CT) with number of program files. Experimental results evident that the SGNE-PRRBC technique achieves better performance in terms of accuracy, CT and FPR as compared to the conventional methods.

Large Margin Dimensionality Reduction for Action Similarity Labeling

Action recognition in videos is receiving extensive research interest due to its wide applications. This task needs to assign a specific action class for each video. In this paper, we study the problem of action similarity labeling (ASLAN) that is to verify whether two action videos present the same type of action or not. We show that both Fisher vector (FV) and vector of locally aggregated descriptors (VLAD) with dense trajectory features can achieve state-of-the-art performance on the ASLAN benchmark. Our main contribution is to develop a large margin dimensionality reduction (LMDR) method to compress high-dimensional FV and VLAD. Specially, we leverage the hinge loss objective function and stochastic gradient descent to optimize the discriminative projection matrix of these vectors. Extensive experiments on the ASLAN dataset indicate that our LMDR method not only reduces the dimension significantly but also improves the verification performance.

A Multiple Cause Mixture Model for Unsupervised Learning

This paper presents a formulation for unsupervised learning of clusters reflecting multiple causal structure in binary data. Unlike the “hard” k-means clustering algorithm and the “soft” mixture model, each of which assumes that a single hidden event generates each data point, a multiple cause model accounts for observed data by combining assertions from many hidden causes, each of which can pertain to varying degree to any subset of the observable dimensions. We employ an objective function and iterative gradient descent learning algorithm resembling the conventional mixture model. A crucial issue is the mixing function for combining beliefs from different cluster centers in order to generate data predictions whose errors are minimized both during recognition and learning. The mixing function constitutes a prior assumption about underlying structural regularities of the data domain; we demonstrate a weakness inherent to the popular weighted sum followed by sigmoid squashing, and offer alternative forms of the nonlinearity for two types of data domain. Results are presented demonstrating the algorithm's ability successfully to discover coherent multiple causal representations in several experimental data sets.