Activation Function Research Articles

Flame Combustion State Detection Method of Cement Rotary Furnace Based on Improved RE-DDPM and DAF-FasterNet

It is of great significance to effectively identify the flame-burning state of cement rotary kilns to optimize the calcination process and ensure the quality of cement. However, high-temperature and smoke-filled environments bring about difficulties with respect to accurate feature extraction and data acquisition. To address these challenges, this paper proposes a novel approach. First, an improved denoising diffusion probability model (RE-DDPM) is proposed. By applying a mask to the burning area and mixing it with the actual image in the denoising process, local diversity generation in the image was realized, and the problem of limited and uneven data was solved. Secondly, this article proposes the DAF-FasterNet model, which incorporates a deformable attention mechanism (DAS) and replaces the ReLU activation function with FReLU so that it can better focus on key flame features and extract finer spatial details. The RE-DDPM method exhibits faster convergence and lower FID scores, indicating that the generated images are more realistic. DAF-FasterNet achieves 98.9% training accuracy, 98.1% test accuracy, and a 22.3 ms delay, making it superior to existing methods in flame state recognition.

Dynamics of convolutional recurrent neural networks near their critical point

We examine the dynamical properties of a single-layer convolutional recurrent network with a smooth sigmoidal activation function, for small values of the inputs and when the convolution kernel is unitary, so all eigenvalues lie exactly at the unit circle. Such networks have a variety of hallmark properties: the outputs depend on the inputs via compressive nonlinearities such as cubic roots and both the and the depend sensitively on the inputs as power laws, both diverging as the input →0. The basic dynamical mechanism is that inputs to the network generate ongoing activity, which in turn controls how additional inputs or signals propagate spatially or attenuate in time. We present analytical solutions for the steady states when the network is forced with a single oscillation and when a background value creates a steady state of “ongoing activity” and derive the relationships shaping the value of the temporal decay and spatial propagation length as a function of this background value. Published by the American Physical Society 2024

Mobile Phone Intelligent Recognition of Human Activity Based on Deep Neural Network

Abstract. Because of the rapid development of smartphone sensor technology and the continuous progress of machine learning algorithms, it has become possible to use smartphones for human activity recognition. Sensors such as accelerometers, gyroscopes, and magnetometers built into smartphones are able to collect human motion data in different activities, which provides a rich data source for activity recognition through mobile phones. In this paper, we change the activation function of the hidden layer of the neural network to observe the effect of this variable on the mobile phone to recognize human activities. Taking Rectified Linear Unit (ReLU)function and Sigmoid function as examples, it is found that the neural network using the ReLU function shows higher accuracy (0. 9518 and 0. 9332) and lower loss value in the early stage and middle to late stage of training. In addition, This paper finds that training for 100 epochs takes significantly less time than the model with the Sigmoid activation function (47. 041 seconds vs 179. 022 seconds). The neural network system built by the relu function has faster operation speed and better performance. This study shows that different hidden layer functions have great influence on the quality of the same neural network, and selecting the best hidden layer function will be an important point for the success of the study.

Functional Outcome after Clinical Recovery from Moderate to Critical COVID-19 among Patients Admitted to the Philippine General Hospital: A Prospective Cohort Study

Background and Objective. COVID-19 is a novel disease primarily affecting the respiratory system. Of those infected, approximately 20% require management in a hospital-setting which may lead to deconditioning. Measures implemented to control spread of the virus also restricted mobility both in the hospital and community setting. The goal of this study was to describe the patient characteristics (age, sex, comorbidities), hospitalization (length of hospital stay, ICU stay, referral to Rehabilitation Medicine), and long-term functional outcome of patients who have clinically recovered from moderate to critical COVID-19 in terms of participation in activities of daily living. Methods. This was a descriptive prospective cohort study conducted at a tertiary government hospital with participant recruitment from September 2020 to February 2021 consisting of clinically recovered adult patients managed as COVID-19 Confirmed via rRT-PCR with moderate, severe, or critical risk status. Descriptive statistics were obtained and multiple regression analysis was done to determine associations between patient demographics and their Barthel Index Scores on follow-up at discharge, one month post-discharge, and six months post-discharge. Results. A total of 63 patients were recruited to our study with an average age of 52.4 years. More recovered patients had fulfilled the criteria of moderate illness (46%), with the most common comorbidity being chronic lung disease (42.1%) and diabetes (42.1%). Almost all had total independence pre-morbidly with better baseline functional scores for the COVID-19 severe population. Majority of the patients (63.5%) were not referred for Rehabilitation services. Across all patients, Barthel Index Score at discharge indicated a significant decline from slight dependence to moderate dependence in performing activities of daily living with the pre-morbid status significantly predicting scores at discharge (β = 0.621, p = 0.001) on multiple regression analysis. Patient demographics, hospitalization and ICU stay and outcome, and referral to Rehabilitation Medicine were not found to be significant factors. In the course of follow up, a high dropout rate was observed across the population and by the end of the study, 57.1% of the participants were alive while among those lost to follow up, 20.6% had expired and the remaining 22.2% had an unknown status. Conclusion. COVID-19 significantly affects the functional outcome of patients in terms of activities of daily living as measured by the Barthel Index. Preliminary data gathered from our study and the high dropout rate supports the need for better follow-up and selecting a tool that is better able to describe the non-demographic factors affecting functionality and participation in activities of daily living.

Applications of Catalan Neural Network (CaNN) model

In this paper, we construct Catalan neural network (CaNN) model and examine the nonlinear solvers of the ordinary nonlinear differential equation using the CaNN model with linear and nonlinear activation functions. Using a single-layer functional linking neural network (FLANN) architecture, we extend input models using the first five Catalan polynomials, update network parameters that we initially randomly select, and use error backpropagation algorithms. The CaNN trial solution for solving nonlinear differential equations is in excellent agreement with the analytical solution. Also, the validity of the CaNN method is investigated by treating second-order differential equations of Lane–Emden type as boundary and initial value problems.

An intelligent magnetic resonance imagining-based multistage Alzheimer's disease classification using swish-convolutional neural networks.

Alzheimer's disease (AD) refers to a neurological disorder that causes damage to brain cells and results in decreasing cognitive abilities and memory. In brain scans, this degeneration can be seen in different ways. The disease can be classified into four stages: Non-demented (ND), moderate demented (MoD), mild demented (MiD), and very mild demented (VMD). To prepare the raw dataset for analysis, the collected magnetic resonance imaging (MRI) images are subjected to several pre-processing techniques in order to improve the performance accuracy of the proposed model. Medical images generally have poor contrast and get affected by noise, which ends up with inaccurate diagnosis. For the different phases of AD to be detected, a clear image is necessary. To address this issue, the influence of the artefacts must be reduced, enhance the contrast, and reduce the loss of information. A novel framework for image enhancement is suggested to increase the accuracy in the detection and identification of AD. In this study, the raw MRI dataset from the Alzheimer's disease neuroimaging initiative (ADNI) database is subjected to skull stripping, contrast enhancement, and image filtering followed by data augmentation to balance the dataset with four types of Alzheimer's classes. The pre-processed data are subjected to five different pre-trained models such as AlexNet, ResNet, VGG 16, EfficientNet, and Inceptionv3 achieving a testing accuracy rate of 91.2%, 88.21%, 92.34%, 93.45%, and 85.12%, respectively. These pre-trained models are compared with the proposed CNN (convolutional neural network) model designed with Adam optimizer and Flatten Swish activation function which reaches the highest accuracy of 96.5% with a learning rate of 0.000001. The five pre-trained CNNmodels along with the proposed swish-based AD-CNN were tested using various performance metrics to evaluate the model efficiency in classifying and identifying the AD classes. From the result analysis, it is evident that the proposed AD-CNN model outperforms all the other models.

Photonic Neural Activation Function Based on High-Speed Electro-Absorption Modulated Laser

Photonic Neural Activation Function Based on High-Speed Electro-Absorption Modulated Laser

Photonic Neuron With on Frequency-Domain ReLU Activation Function

Photonic Neuron With on Frequency-Domain ReLU Activation Function

10 GHz Optoelectronic Circuit With Variable Sigmoid Shaped Activation Function for Photonic Neural Networks

10 GHz Optoelectronic Circuit With Variable Sigmoid Shaped Activation Function for Photonic Neural Networks

Examining heterogeneity in dementia using data-driven unsupervised clustering of cognitive profiles.

Dementia is characterized by a decline in memory and thinking that is significant enough to impair function in activities of daily living. Patients seen in dementia specialty clinics are highly heterogenous with a variety of different symptoms that progress at different rates. Recent research has focused on finding data-driven subtypes for revealing new insights into dementia's underlying heterogeneity, rather than assuming that the cohort is homogenous. However, current studies on dementia subtyping have the following limitations: (i) focusing on AD-related dementia only and not examining heterogeneity within dementia as a whole, (ii) using only cross-sectional baseline visit information for clustering and (iii) predominantly relying on expensive imaging biomarkers as features for clustering. In this study, we seek to overcome such limitations, using a data-driven unsupervised clustering algorithm named SillyPutty, in combination with hierarchical clustering on cognitive assessment scores to estimate subtypes within a real-world clinical dementia cohort. We use a longitudinal patient data set for our clustering analysis, instead of relying only on baseline visits, allowing us to explore the ongoing temporal relationship between subtypes and disease progression over time. Results showed that subtypes with very mild or mild dementia were more heterogenous in their cognitive profiles and risk of disease progression.

Myc 9aaTAD activation domain binds to mediator of transcription with superior high affinity.

The overexpression of MYC genes is frequently found in many human cancers, including adult and pediatric malignant brain tumors. Targeting MYC genes continues to be challenging due to their undruggable nature. Using our prediction algorithm, the nine-amino-acid activation domain (9aaTAD) has been identified in all four Yamanaka factors, including c-Myc. The predicted activation function was experimentally demonstrated for all these short peptides in transactivation assay. We generated a set of c-Myc constructs (1-108, 69-108 and 98-108) in the N-terminal regions and tested their ability to initiate transcription in one hybrid assay. The presence and absence of 9aaTAD (region 100-108) in the constructs strongly correlated with their activation functions (5-, 3- and 67-times respectively). Surprisingly, we observed co-activation function of the myc region 69-103, called here acetyl-TAD, previously described by Faiola et al. (Mol Cell Biol 25:10220-10234, 2005) and characterized in this study as a new domain collaborating with the 9aaTAD. We discovered strong interactions on a nanomolar scale between the Myc-9aaTAD activation domains and the KIX domain of CBP coactivator. We showed conservation of the 9aaTADs in the MYC family. In summary for the c-Myc oncogene, the acetyl-TAD and the 9aaTAD domains jointly mediated activation function. The c-Myc protein is largely intrinsically disordered and therefore difficult to target with small-molecule inhibitors. For the c-Myc driven tumors, the strong c-Myc interaction with the KIX domain represents a promising druggable target.

Simple Recurrent Networks are Interactive.

There is disagreement among cognitive scientists as to whether a key computational framework - the Simple Recurrent Network (SRN; Elman, Machine Learning, 7(2), 195-225, 1991; Elman, Cognitive Science, 14(2), 179-211, 1990) - is a feedforward system. SRNs have been essential tools in advancing theories of learning, development, and processing in cognitive science for more than three decades. If SRNs were feedforward systems, there would be pervasive theoretical implications: Anything an SRN can do would therefore be explainable without interaction (feedback). However, despite claims that SRNs (and by extension recurrent neural networks more generally) are feedforward (Norris, 1993), this is not the case. Feedforward networks by definition are acyclic graphs - they contain no loops. SRNs contain loops - from hidden units back to hidden units with a time delay - and are therefore cyclic graphs. As we demonstrate, they are interactive in the sense normally implied for networks with feedback connections between layers: In an SRN, bottom-up inputs are inextricably mixed with previous model-internal computations. Inputs are transmitted to hidden units by multiplying them by input-to-hidden weights. However, hidden units simultaneously receive their own previous activations as input via hidden-to-hidden connections with a one-step time delay (typically via context units). These are added to the input-to-hidden values, and the sums are transformed by an activation function. Thus, bottom-up inputs are mixed with the products of potentially many preceding transformations of inputs and model-internal states. We discuss theoretical implications through a key example from psycholinguistics where the status of SRNs as feedforward or interactive has crucial ramifications.

Enhanced T-cell activation and chemokine-associated function in CD14-positive cells from venous sinus blood in sub-acute cerebral venous sinus thrombosis

BackgroundPatients with sub-acute cerebral venous sinus thrombosis experience (SA.CVST) severe symptoms compared to two other venous sinus-related diseases, including chronic cerebral venous sinus thrombosis (C.CVST) and idiopathic intracranial hypertension (IIH).ObjectiveThis study aimed to determine whether the different immune reactions in different venous sinuses are related.MethodsStagnant blood in the cerebral venous sinuses was extracted by passing a microcatheter and CD14-positive cells were sorted by magnetic beads and subjected to RNA-seq sequencing.ResultsCompared to patients with IIH, 128 genes were significantly down-regulated and 373 genes were significantly up-regulated in the sub-acute CVST samples. The functions of these genes were mainly focused on “immune response”, “T cell activation” and “plasma membrane”. Gene Set Enrichment Analysis (GSEA) showed T cell survival and activation-related function significantly unregulated in sub-acute CVST. On the other hand, there were 366 genes down-regulated in chronic CVST and 75 genes up-regulated in chronic CVST. In functional annotation, these differently expressed genes were enriched in the “extracellular region”, “chemokine-mediated signaling pathway” and “immune response”. GSEA analysis confirmed that chemokine-related functions were all up-regulated in sub-acute CVST and monocyte-macrophage adhesion functions were also significantly up-regulated.ConclusionThis study suggested the CD14-positive created an activated immune response in sub-acute CVST.

Physics-aware cross-domain fusion aids learning-driven computer-generated holography

The rapid advancement of computer-generated holography has bridged deep learning with traditional optical principles in recent years. However, a critical challenge in this evolution is the efficient and accurate conversion from the amplitude to phase domain for high-quality phase-only hologram (POH) generation. Existing computational models often struggle to address the inherent complexities of optical phenomena, compromising the conversion process. In this study, we present the cross-domain fusion network (CDFN), an architecture designed to tackle the complexities involved in POH generation. The CDFN employs a multi-stage (MS) mechanism to progressively learn the translation from amplitude to phase domain, complemented by the deep supervision (DS) strategy of middle features to enhance task-relevant feature learning from the initial stages. Additionally, we propose an infinite phase mapper (IPM), a phase-mapping function that circumvents the limitations of conventional activation functions and encapsulates the physical essence of holography. Through simulations, our proposed method successfully reconstructs high-quality 2K color images from the DIV2K dataset, achieving an average PSNR of 31.68 dB and SSIM of 0.944. Furthermore, we realize high-quality color image reconstruction in optical experiments. The experimental results highlight the computational intelligence and optical fidelity achieved by our proposed physics-aware cross-domain fusion.

KANQAS: Kolmogorov-Arnold Network for Quantum Architecture Search

AbstractQuantum architecture Search (QAS) is a promising direction for optimization and automated design of quantum circuits towards quantum advantage. Recent techniques in QAS emphasize Multi-Layer Perceptron (MLP)-based deep Q-networks. However, their interpretability remains challenging due to the large number of learnable parameters and the complexities involved in selecting appropriate activation functions. In this work, to overcome these challenges, we utilize the Kolmogorov-Arnold Network (KAN) in the QAS algorithm, analyzing their efficiency in the task of quantum state preparation and quantum chemistry. In quantum state preparation, our results show that in a noiseless scenario, the probability of success is 2× to 5× higher than MLPs. In noisy environments, KAN outperforms MLPs in fidelity when approximating these states, showcasing its robustness against noise. In tackling quantum chemistry problems, we enhance the recently proposed QAS algorithm by integrating curriculum reinforcement learning with a KAN structure. This facilitates a more efficient design of parameterized quantum circuits by reducing the number of required 2-qubit gates and circuit depth. Further investigation reveals that KAN requires a significantly smaller number of learnable parameters compared to MLPs; however, the average time of executing each episode for KAN is higher.

Three-Dimensional Instance Segmentation Using the Generalized Hough Transform and the Adaptive n-Shifted Shuffle Attention

The progress of 3D instance segmentation techniques has made it essential for several applications, such as augmented reality, autonomous driving, and robotics. Traditional methods usually have challenges with complex indoor scenes made of multiple objects with different occlusions and orientations. In this work, the authors present an innovative model that integrates a new adaptive n-shifted shuffle (ANSS) attention mechanism with the Generalized Hough Transform (GHT) for robust 3D instance segmentation of indoor scenes. The proposed technique leverages the n-shifted sigmoid activation function, which improves the adaptive shuffle attention mechanism, permitting the network to dynamically focus on relevant features across various regions. A learnable shuffling pattern is produced through the proposed ANSS attention mechanism to spatially rearrange the relevant features, thus augmenting the model’s ability to capture the object boundaries and their fine-grained details. The integration of GHT furnishes a vigorous framework to localize and detect objects in the 3D space, even when heavy noise and partial occlusions are present. The authors evaluate the proposed method on the challenging Stanford 3D Indoor Spaces Dataset (S3DIS), where it establishes its superiority over existing methods. The proposed approach achieves state-of-the-art performance in both mean Intersection over Union (IoU) and overall accuracy, showcasing its potential for practical deployment in real-world scenarios. These results illustrate that the integration of the ANSS and the GHT yields a robust solution for 3D instance segmentation tasks.

Bayesian-Optimized Deep Learning for Predicting Compressive Strength of Sustainable Eggshell Foamed Concrete

<p><span lang="EN-US">The use of eggshell-based foamed concrete represents a sustainable approach to enhancing environmental friendliness in construction materials. This study investigates the predictive modelling of the compressive strength of eggshell-based foamed concrete through a deep learning model, fine-tuned via Bayesian optimization. Utilizing a dataset of 360 samples with diverse input parameters, the model was optimized with four hidden layers (28, 21, 28, and 21 neurons) and the Rectified Linear Unit (ReLU) activation function. The model demonstrated excellent predictive accuracy, achieving a mean squared error of 0.0522, a mean absolute error of 0.0382, and an R² value of 0.9548 over 200 epochs. Notably, the water/cement ratio emerged as the most influential factor in prediction accuracy. This research provides a robust, AI-driven method for predicting the compressive strength of sustainable construction materials, contributing to advancements in environmental technology and the optimization of eco-friendly construction practices.</span></p>

Power Quality Improvement for Hybrid Microgrid Integrated with Renewable Energy Sources

Given the present energy crisis and depletion of fossil fuel reserves, there is a pressing need to enhance the integration of Renewable Energy Sources (RES) into the power system. This research focuses on designing and managing a Hybrid Microgrid (HMG) in fluctuating RES. The Direct Current (DC) sub-grid comprises a Wind Turbine (WT), a solar Photovoltaic (PV) array equipped with a Perturb-and-Observation (P&O) Maximum Power Point Tracking (MPPT) method, a boost conversion, and a Battery Energy Storage System (B-ESS) connected to DC demands. The Alternative Current (AC) sub-grid comprises a Permanent Magnet Synchronous Generator (PMSG) WT and a Fuel Cell (FC) integrated with an inverter circuit synced with the grid to fulfill its load requirements. A reversible Interlinking Conversion (IC) connects the AC and DC sub-grid, enabling efficient power interchange across the two grids. The IC's control technique allows it to function as an active energy filter and exchange operator. The IC's active energy filtering function ensures that the power supply of the microgrid complies with the criteria set by IEEE 519. It does this by offering reactive power assistance and decreasing the harmonic levels to below 5%. The suggested method enables the HMG to function in grid-connected and islanded states. During grid-connected operation, power is transferred across the DC and AC sub-grids, ensuring that all load requirements are fulfilled. In the islanded method, a diesel generator provides power to the AC sub-grid to fulfill the needs of essential loads, while the B-ESS sustains the DC microgrid. The suggested model is constructed and simulated with MATLAB, and its outcomes are examined.

Synergistic involvement of the NZF domains of the LUBAC accessory subunits HOIL-1L and SHARPIN in the regulation of LUBAC function.

The linear ubiquitin chain assembly complex (LUBAC) plays crucial roles in NF-κB signaling and protection against cell death by generating linear ubiquitin chains. Its accessory subunits, HOIL-1L and SHARPIN, regulate LUBAC function by binding to ubiquitin chains via their Npl4 zinc finger (NZF) domains. However, the synergistic effects of the two NZF domains on LUBAC function remain unclear. Here, we demonstrate that the ubiquitin-binding activity of the two NZF domains cooperatively regulates LUBAC functions. Simultaneous loss of the ubiquitin-binding activity of the NZF domains profoundly impaired both NF-κB activation and cell death protection functions. HOIL-1L NZF robustly binds to linear ubiquitin chains, whereas SHARPIN NZF binds to Lys(K)63-linked ubiquitin chains in addition to linear chains. Binding of both NZF domains to linear ubiquitin chains regulated NF-κB signaling, whereas SHARPIN NZF predominantly regulated the cell death protection function independently of the ubiquitin chain type, K63-linked or linear ubiquitin. However, concomitant loss of linear ubiquitin binding by HOIL-1L NZF drastically impaired cell death protection. A screen of compounds capable of inhibiting binding between HOIL-1L NZF and linear ubiquitin chains identified a small compound that inhibited SHARPIN NZF as well as HOIL-1L NZF binding to linear ubiquitin chains, supporting the synergistic effect of the two NZF domains on cell death protection and suggesting a potential therapeutic strategy for targeting increased LUBAC activity in diseases such as cancer.

SSFAN: A Compact and Efficient Spectral-Spatial Feature Extraction and Attention-Based Neural Network for Hyperspectral Image Classification

Hyperspectral image (HSI) classification is a crucial technique that assigns each pixel in an image to a specific land cover category by leveraging both spectral and spatial information. In recent years, HSI classification methods based on convolutional neural networks (CNNs) and Transformers have significantly improved performance due to their strong feature extraction capabilities. However, these improvements often come with increased model complexity, leading to higher computational costs. To address this, we propose a compact and efficient spectral-spatial feature extraction and attention-based neural network (SSFAN) for HSI classification. The SSFAN model consists of three core modules: the Parallel Spectral-Spatial Feature Extraction Block (PSSB), the Scan Block, and the Squeeze-and-Excitation MLP Block (SEMB). After preprocessing the HSI data, it is fed into the PSSB module, which contains two parallel streams, each comprising a 3D convolutional layer and a 2D convolutional layer. The 3D convolutional layer extracts spectral and spatial features from the input hyperspectral data, while the 2D convolutional layer further enhances the spatial feature representation. Next, the Scan Block module employs a layered scanning strategy to extract spatial information at different scales from the central pixel outward, enabling the model to capture both local and global spatial relationships. The SEMB module combines the Spectral-Spatial Recurrent Block (SSRB) and the MLP Block. The SSRB, with its adaptive weight assignment mechanism in the SToken Module, flexibly handles time steps and feature dimensions, performing deep spectral and spatial feature extraction through multiple state updates. Finally, the MLP Block processes the input features through a series of linear transformations, GELU activation functions, and Dropout layers, capturing complex patterns and relationships within the data, and concludes with an argmax layer for classification. Experimental results show that the proposed SSFAN model delivers superior classification performance, outperforming the second-best method by 1.72%, 5.19%, and 1.94% in OA, AA, and Kappa coefficient, respectively, on the Indian Pines dataset. Additionally, it requires less training and testing time compared to other state-of-the-art deep learning methods.