Conventional Neural Network Architectures Research Articles

An interpretable deep learning model predicts RNA-small molecule binding sites

The intricate interplay between RNA molecules and small ligands plays a pivotal role in regulating biological processes, underscoring the necessity for accurate prediction models of RNA- small molecule binding sites. In this study, we develop CapBind, a novel interpretive deep learning framework utilizing a Capsule Network architecture, to precisely identify RNA- small molecule binding sites. This framework stands out for its multifaceted approach, which leverages both the sequence and structural information of RNA as input features, allowing for a thorough understanding of the complex interactions between RNA and small molecules. Leveraging multi-source bioinformatics data, including the distribution of RNA secondary structures, our model characterizes sequence and structural features with unparalleled depth. Through a unified alignment of these multi-source feature representations, we achieve dimensional homogeneity in the potential feature maps for both types of biological data. The crux of our methodology lies in the deployment of a multi-level Capsule Network, engineered to identify the nuanced interactions of RNA sequences with small molecules. The essence of our approach resides in the implementation of a multi-layered Capsule Network. Utilizing a sophisticated routing mechanism, this network demonstrates unparalleled efficacy in learning and identifying the intricate interactions between RNA sequences and small molecules. This mechanism enables dynamic adjustment of information flow across multiple levels, thereby capturing subtle features and patterns that might elude conventional neural network architectures. The experimental results demonstrate the superior performance of our model, particularly highlighting its progress in evaluation metrics compared to current state-of-the-art models. Case studies further validate the model’s precision in identifying binding sites, offering potential insights into gene expression pathways and novel therapeutic avenues for neurodegenerative diseases. Moreover, an extensive interpretability analysis of our silicon model reveals the internal interaction mechanisms between specific RNA regions and small molecules, contributing valuable perspectives for drug design and molecular biology research. These findings underscore the model’s predictive accuracy and biological applicability, establishing our approach as a significant leap forward in the computational prediction of RNA- small molecule binding sites.

Operator-Learning-Inspired Modeling of Neural Ordinary Differential Equations

Neural ordinary differential equations (NODEs), one of the most influential works of the differential equation-based deep learning, are to continuously generalize residual networks and opened a new field. They are currently utilized for various downstream tasks, e.g., image classification, time series classification, image generation, etc. Its key part is how to model the time-derivative of the hidden state, denoted dh(t)/dt. People have habitually used conventional neural network architectures, e.g., fully-connected layers followed by non-linear activations. In this paper, however, we present a neural operator-based method to define the time-derivative term. Neural operators were initially proposed to model the differential operator of partial differential equations (PDEs). Since the time-derivative of NODEs can be understood as a special type of the differential operator, our proposed method, called branched Fourier neural operator (BFNO), makes sense. In our experiments with general downstream tasks, our method significantly outperforms existing methods.

Exploring elliptic geometry in neural networks : The concept of elliptic neural networks (ENN)

The exploration of non-Euclidean geometries in the context of neural network architectures presents a novel avenue for enhancing the processing of complex data structures. This paper introduces the concept of Elliptic Neural Networks (ENN), a framework that integrates the intrinsic properties of elliptic geometry into the fabric of neural computation. Characterized by constant positive curvature and closed geodesic paths, elliptic geometry provides a unique perspective for representing data, especially advantageous for configurations exhibiting cyclic and dense hierarchical interconnections. We begin by delineating the fundamental aspects of elliptic geometry, focusing on its impact on conventional notions of distance and global structural interpretation within data sets. This foundational understanding paves the way for our proposed mathematical schema for ENNs. Within this framework, we articulate the methodologies for embedding data in elliptic spaces and adapt neural network operations by encompassing neuron activation, signal propagation, and learning paradigms to conform to the topological idiosyncrasies of elliptic geometry. In this paper addressing the implementation of ENNs, we discuss the computational challenges and the development of novel algorithms requisite for navigating the elliptic geometric landscape. The prospective applications of ENNs are extensive and diverse, ranging from the enhancement of cyclic pattern recognition in time-series analysis to more sophisticated representations in graph-based data and natural language processing tasks. This theoretical exposition aims to set the groundwork for subsequent empirical research to validate the proposed model and assess its practical performance relative to conventional neural network architectures. By harnessing the distinctive characteristics of elliptic geometry, ENNs mark a significant stride towards enriching the arsenal of machine learning methodologies, potentially leading to more versatile and efficacious computational tools in data analysis and artificial intelligence.

Improving snore detection under limited dataset through harmonic/percussive source separation and convolutional neural networks

Snoring, an acoustic biomarker commonly observed in individuals with Obstructive Sleep Apnoea Syndrome (OSAS), holds significant potential for diagnosing and monitoring this recognized clinical disorder. Irrespective of snoring types, most snoring instances exhibit identifiable harmonic patterns manifested through distinctive energy distributions over time. In this work, we propose a novel method to differentiate monaural snoring from non-snoring sounds by analyzing the harmonic content of the input sound using harmonic/percussive sound source separation (HPSS). The resulting feature, based on the harmonic spectrogram from HPSS, is employed as input data for conventional neural network architectures, aiming to enhance snoring detection performance even under a limited data learning framework. To evaluate the performance of our proposal, we studied two different scenarios: 1) using a large dataset of snoring and interfering sounds, and 2) using a reduced training set composed of around 1% of the data material. In the former scenario, the proposed HPSS-based feature provides competitive results compared to other input features from the literature. However, the key advantage of the proposed method lies in the superior performance of the harmonic spectrogram derived from HPSS in a limited data learning context. In this particular scenario, using the proposed harmonic feature significantly enhances the performance of all the studied architectures in comparison to the classical input features documented in the existing literature. This finding clearly demonstrates that incorporating harmonic content enables more reliable learning of the essential time-frequency characteristics that are prevalent in most snoring sounds, even in scenarios where the amount of training data is limited.

Automated Sleep Stage Classification in Home Environments: An Evaluation of Seven Deep Neural Network Architectures.

Sleep is an essential human physiological need that has garnered increasing scientific attention due to the burgeoning prevalence of sleep-related disorders and their impact on public health. Among contemporary challenges, the demand for authentic sleep monitoring outside the confines of specialized laboratories, ideally within the home environment, has arisen. Addressing this, we explore the development of pragmatic approaches that facilitate implementation within domestic settings. Such approaches necessitate the deployment of streamlined, computationally efficient automated classifiers. In pursuit of a sleep stage classifier tailored for home use, this study rigorously assessed seven conventional neural network architectures prominent in deep learning (LeNet, ResNet, VGG, MLP, LSTM-CNN, LSTM, BLSTM). Leveraging sleep recordings from a cohort of 20 subjects, we elucidate that LeNet, VGG, and ResNet exhibit superior performance compared to recent advancements reported in the literature. Furthermore, a comprehensive architectural analysis was conducted, illuminating the strengths and limitations of each in the context of home-based sleep monitoring. Our findings distinctly identify LeNet as the most-amenable architecture for this purpose, with LSTM and BLSTM demonstrating relatively lesser compatibility. Ultimately, this research substantiates the feasibility of automating sleep stage classification employing lightweight neural networks, thereby accommodating scenarios with constrained computational resources. This advancement aims at revolutionizing the field of sleep monitoring, making it more accessible and reliable for individuals in their homes.

Exploiting graph neural networks to perform finite-difference time-domain based optical simulations

Having an artificial neural network that solves Maxwell’s equations in a general setting is an intellectual challenge and a great utility. Recently, there have been multiple successful attempts to use artificial neural networks to predict electromagnetic fields, given a specific source and interacting material distribution. However, many of these attempts are limited in domain size and restricted to object shapes similar to the learned ones. Here, we overcome these restrictions by using graph neural networks (GNNs) that adapt the propagation scheme of the finite-difference time-domain (FDTD) method to solve Maxwell’s equations for a distinct time step. GNNs yield a significant advantage, i.e., size invariance, over conventional neural network architectures, such as convolutional or linear neural networks. Once trained, a GNN can work on graphs of arbitrary size and connectivity. This allows us to train them on the propagation procedure of electromagnetic fields on small domain sizes and, finally, expand the domain to an arbitrary scale. Moreover, GNNs can adapt to any material shape and work not only on structured grids, such as FDTD, but also on arbitrary meshes. This work may be seen as the first benchmark for field predictions with graph networks and could be expanded to more complex mesh-based optical simulations, e.g., those based on finite elements.

Optical performance monitoring using lifelong learning with confrontational knowledge distillation in 7-core fiber for elastic optical networks

We propose a novel optical performance monitoring (OPM) scheme, including modulation format recognition (MFR) and optical signal-to-noise ratio (OSNR) estimation, for 7-core fiber in elastic optical networks (EONs) by using the specific Stokes sectional images of the received signals. Meanwhile, MFR and OSNR estimation in all channels can be utilized by using a lightweight neural network via lifelong learning. In addition, the proposed scheme saves the computational resources for real implementation through confrontational knowledge distillation, making it easy to deploy the proposed neural network in the receiving end and intermediate node. Five modulation formats, including BPSK, QPSK, 8PSK, 8QAM, and 16QAM, were recognized by the proposed scheme within the OSNR of 10-30 dB over 2 km weakly coupled 7-core fiber. Experimental results show that 100% recognition accuracy of all these five modulation formats can be achieved while the RMSE of the estimation is below 0.1 dB. Compared with conventional neural network architectures, the proposed neural network achieves better performance, whose runtime is merely 20.2 ms, saving the computational resource of the optical network.

Quantum Enhancements for Deep Reinforcement Learning in Large Spaces

In the past decade, the field of quantum machine learning has drawn significant attention due to the prospect of bringing genuine computational advantages to now widespread algorithmic methods. However, not all domains of machine learning have benefited equally from quantum enhancements. Notably, deep learning and reinforcement learning, despite their tremendous success in the classical domain, both individually and combined, remain relatively unaddressed by the quantum community. Arguably, one reason behind this is the systematic use in these domains of models and methods without prominent computational bottlenecks, leaving little room for quantum improvements. In this work, we study the state-of-the-art neural-network approaches for reinforcement learning with quantum enhancements in mind. We demonstrate the substantial learning advantage that models with a sampling bottleneck can provide over conventional neural network architectures in complex learning environments. These so-called energy-based models, like deep energy-based reinforcement learning, and deep projective simulation that we also introduce in this work, effectively allow to trade off learning performance for efficiency of computation. To alleviate the additional computational costs, we propose to leverage future and near-term quantum algorithms, resulting in overall more advantageous learning algorithms. This is achieved using cutting-edge and new quantum computing machinery to speed-up classical sampling methods and by employing generalized models to gain an additional quantum advantage.

Improved SpikeProp for Using Particle Swarm Optimization

A spiking neurons network encodes information in the timing of individual spike times. A novel supervised learning rule for SpikeProp is derived to overcome the discontinuities introduced by the spiking thresholding. This algorithm is based on an error-backpropagation learning rule suited for supervised learning of spiking neurons that use exact spike time coding. The SpikeProp is able to demonstrate the spiking neurons that can perform complex nonlinear classification in fast temporal coding. This study proposes enhancements of SpikeProp learning algorithm for supervised training of spiking networks which can deal with complex patterns. The proposed methods include the SpikeProp particle swarm optimization (PSO) and angle driven dependency learning rate. These methods are presented to SpikeProp network for multilayer learning enhancement and weights optimization. Input and output patterns are encoded as spike trains of precisely timed spikes, and the network learns to transform the input trains into target output trains. With these enhancements, our proposed methods outperformed other conventional neural network architectures.

Preintegration lateral inhibition enhances unsupervised learning.

A large and influential class of neural network architectures uses postintegration lateral inhibition as a mechanism for competition. We argue that these algorithms are computationally deficient in that they fail to generate, or learn, appropriate perceptual representations under certain circumstances. An alternative neural network architecture is presented here in which nodes compete for the right to receive inputs rather than for the right to generate outputs. This form of competition, implemented through preintegration lateral inhibition, does provide appropriate coding properties and can be used to learn such representations efficiently. Furthermore, this architecture is consistent with both neuroanatomical and neurophysiological data. We thus argue that preintegration lateral inhibition has computational advantages over conventional neural network architectures while remaining equally biologically plausible.

A model of computation in neocortical architecture

We propose that the specific architecture of the neocortex reflects the organization principles of neocortical computation. In this paper, we place the anatomically defined concept of columns into a functional context. It is provided by a large-scale computational hypothesis on visual recognition, which includes both, rapid parallel forward recognition, independent of any feedback prediction, and a feedback controlled refinement system. Short epochs of periodic clocking define a global reference time and introduce a discrete time for cortical processing which enables the combination of parallel categorization and sequential refinement. The presented model differs significantly from conventional neural network architectures and suggests a novel interpretation of the role of gamma oscillations and cognitive binding.

Comparison of neofuzzy and rough neural networks

Conventional neural network architectures generally lack semantics. Both rough and neofuzzy neurons introduce semantic structures in the conventional neural network models. Rough neurons make it possible to process data points with a range of values instead of a single precise value. Neofuzzy neurons make it possible to convert crisp values into fuzzy values. This paper compares rough and neofuzzy neural networks. Rough and neofuzzy neurons are demonstrated to be complementary to each other. It is shown that the introduction of rough and fuzzy semantic structures in neural networks can increase the accuracy of predictions.

An associative memory that can form hypotheses: a phase-coded neural network.

Nonlinear associative memories as realized, e.g., by Hopfield nets are characterized by attractor-type dynamics. When fed with a starting pattern, they converge to exactly one of the stored patterns which is supposed to be most similar. These systems cannot render hypotheses of classification, i.e., render several possible answers to a given classification problem. Inspired by von der Malsburg's correlation theory of brain function, we extend conventional neural network architectures by introducing additional dynamical variables. Assuming an oscillatory time structure of neural firing, i.e., the existence of neural clocks, we assign a so-called phase to each formal neuron. The phases explicitly describe detailed correlations of neural activities neglected in conventional neural network architectures. Implementing this extension into a simple self-organizing network based on a feature map, we present an associative memory that actually is capable of forming hypotheses of classification.