Neural Network Decoder Research Articles

Mainshock–aftershock sequence simulation via latent space encoding of generative adversarial networks

AbstractAftershocks (ASs) following strong mainshocks (MSs) can exacerbate structural damage or lead to collapse. However, the scarcity of recorded data necessitates reliance on artificial sequences, which have difficulty in characterizing the time‐frequency correlation between MSs and ASs. This study innovatively converts the AS time history prediction into an image translation task, exploiting the invertible transformation between accelerograms and time‐frequency representations. An encoder–decoder neural network is developed to encode the MS information into the latent space of a pre‐trained generative adversarial network, enabling accurate AS predictions through the decoder. The integration of seismic parameters further improves the AS prediction performance. Comparative analyses demonstrate that the proposed method outperforms the traditional ones on accuracy and robustness and reproduces the non‐stationarity of ASs.

Unsupervised, piecewise linear decoding enables an accurate prediction of muscle activity in a multi-task brain computer interface.

Creating an intracortical brain-computer interface (iBCI) capable of seamless transitions between tasks and contexts would greatly enhance user experience. However, the nonlinearity in neural activity presents challenges to computing a global iBCI decoder. We aimed to develop a method that differs from a globally optimized decoder to address this issue. We devised an unsupervised approach that relies on the structure of a low-dimensional neural manifold to implement a piecewise linear decoder. We created a distinctive dataset in which monkeys performed a diverse set of tasks, some trained, others innate, while we recorded neural signals from the motor cortex (M1) and electromyographs (EMGs) from upper limb muscles. We used both linear and nonlinear dimensionality reduction techniques to discover neural manifolds and applied unsupervised algorithms to identify clusters within those spaces. Finally, we fit a linear decoder of EMG for each cluster. A specific decoder was activated corresponding to the cluster each new neural data point belonged to. We found clusters in the neural manifolds corresponding with the different tasks or task sub-phases. The performance of piecewise decoding improved as the number of clusters increased and plateaued gradually. With only two clusters it already outperformed a global linear decoder, and unexpectedly, it outperformed even a global recurrent neural network (RNN) decoder with 10-12 clusters. This study introduced a computationally lightweight solution for creating iBCI decoders that can function effectively across a broad range of tasks. EMG decoding is particularly challenging, as muscle activity is used, under varying contexts, to control interaction forces and limb stiffness, as well as motion. The results suggest that a piecewise linear decoder can provide a good approximation to the nonlinearity between neural activity and motor outputs, a result of our increased understanding of the structure of neural manifolds in motor cortex.

Graph neural network coarse-grain force field for the molecular crystal RDX

Condense phase molecular systems organize in wide range of distinct molecular configurations, including amorphous melt and glass as well as crystals often exhibiting polymorphism, that originate from their intricate intra- and intermolecular forces. While accurate coarse-grain (CG) models for these materials are critical to understand phenomena beyond the reach of all-atom simulations, current models cannot capture the diversity of molecular structures. We introduce a generally applicable approach to develop CG force fields for molecular crystals combining graph neural networks (GNN) and data from an all-atom simulations and apply it to the high-energy density material RDX. We address the challenge of expanding the training data with relevant configurations via an iterative procedure that performs CG molecular dynamics of processes of interest and reconstructs the atomistic configurations using a pre-trained neural network decoder. The multi-site CG model uses a GNN architecture constructed to satisfy translational invariance and rotational covariance for forces. The resulting model captures both crystalline and amorphous states for a wide range of temperatures and densities.

Sensing with shallow recurrent decoder networks

Sensing is a universal task in science and engineering. Downstream tasks from sensing include inferring full-state estimates of a system (system identification), control decisions and forecasting. We propose a shallow recurrent decoder (SHRED) neural network structure for sensing, which incorporates (i) a recurrent neural network to learn a latent representation of the temporal dynamics of the sensors and (ii) a shallow decoder that learns a mapping between this latent representation and the high-dimensional state space. SHRED enables accurate reconstructions with far fewer sensors, outperforms existing techniques when more measurements are available and is more robust to random sensor placements. In the example cases explored, complex spatio-temporal dynamics are characterized by exceedingly limited sensors that can be randomly placed with minimal loss of performance.

PolySegNet: improving polyp segmentation through swin transformer and vision transformer fusion.

Colorectal cancer ranks as the second most prevalent cancer worldwide, with a high mortality rate. Colonoscopy stands as the preferred procedure for diagnosing colorectal cancer. Detecting polyps at an early stage is critical for effective prevention and diagnosis. However, challenges in colonoscopic procedures often lead medical practitioners to seek support from alternative techniques for timely polyp identification. Polyp segmentation emerges as a promising approach to identify polyps in colonoscopy images. In this paper, we propose an advanced method, PolySegNet, that leverages both Vision Transformer and Swin Transformer, coupled with a Convolutional Neural Network (CNN) decoder. The fusion of these models facilitates a comprehensive analysis of various modules in our proposed architecture.To assess the performance of PolySegNet, we evaluate it on three colonoscopy datasets, a combined dataset, and their augmented versions. The experimental results demonstrate that PolySegNet achieves competitive results in terms of polyp segmentation accuracy and efficacy, achieving a mean Dice score of 0.92 and a mean Intersection over Union (IoU) of 0.86. These metrics highlight the superior performance of PolySegNet in accurately delineating polyp boundaries compared to existing methods. PolySegNet has shown great promise in accurately and efficiently segmenting polyps in medical images. The proposed method could be the foundation for a new class of transformer-based segmentation models in medical image analysis.

High-performance decoding using deep neural networks

Problem statement. Deep neural networks are successfully used in channel coding to improve the quality of decoding. However, modern neural channel decoders cannot achieve high decoding performance and low complexity at the same time. To overcome this problem, it is necessary to develop a decoder with a double residual neuron. By integrating residual input and residual learning into the neural channel decoder architecture, it will significantly improve decoding performance while maintaining low complexity. Goal. Development of a decoder architecture with a dual residual neural network, which provides high decoding performance in channel coding with low model complexity and computational costs. Results. The simulation results show that the proposed decoder with a double residual neural network provides a significant increase in decoding performance. Compared to a neural decoder based on a trust propagation algorithm, a decoder with a dual residual neural network provides an additional 0.5-1.8 dB encoding gain compared to different channel codes. Compared to the hypergraph neural decoder, which has the best error correction capability among all existing neural channel decoders, the dual residual neural network decoder also provides similar or even better decoding performance for different channel codes. Practical significance. The results obtained can be used to improve decoding efficiency and improve performance in channel coding. The work was carried out with the financial support of the Ministry of Science and Higher Education of the Russian Federation within the framework of the state assignment "youth laboratory" No. FZGM-2024-0003.

Traffic prediction based on spatial-temporal disentangled generative models

Understanding the underlying spatial-temporal patterns in a traffic network is crucial for accurate predictions. However, current deep learning techniques often struggle to effectively identify the generative factors hidden in spatial-temporal traffic data, leading to a lack of interpretability. To address this challenge, we have developed an innovative generative model called Spatial-Temporal Disentangled Neural Relational Inference (STDNRI) for traffic prediction and disentangling interpretable generative factors within observed traffic network data. In this study, we assume that spatial-temporal data is dominated by three generative factors: time-correlated factors, spatial-correlated factors and spatial-temporal joint correlation factors. Based on information bottleneck theory, we establish a novel objective function to maximize the disentanglement of different factors. To model the probability distributions of three factors, we propose three graph neural network encoders to represent time-correlated factors using a graph containing only self-loops, and denote spatial-correlated factors as well as spatial-temporal joint correlation factors adopting a fully-connected graph without self-loops. Then, graph neural network decoder with a spatial-temporal message passing mechanism is used to generate the final predictions. Several comprehensive experiments were conducted based on two real datasets from NYC Yellow Taxi and SHMetro. The results indicate that our STDNRI outperforms ten existing competitive models in terms of high prediction accuracy and provides excellent interpretability for generative factors. Furthermore, the disentangled interpretable factors can afford valuable insights for planners, urban developers, and other stakeholders to gain a deeper understanding of the dynamics of the traffic network, thus enabling more informed decision-making.

A Stochastic Recurrent Encoder Decoder Network for Multistep Probabilistic Wind Power Predictions.

In this article, a stochastic recurrent encoder decoder neural network (SREDNN), which considers latent random variables in its recurrent structures, is developed for the first time for the generative multistep probabilistic wind power predictions (MPWPPs). The SREDNN enables the stochastic recurrent model under the encoder-decoder framework to engage exogenous covariates to produce better MPWPP. The SREDNN consists of five components, the prior network, the inference network, the generative network, the encoder recurrent network, and the decoder recurrent network. The SREDNN is equipped with two critical advantages compared with conventional RNN-based methods. First, the integration over the latent random variable builds an infinite Gaussian mixture model (IGMM) as the observation model, which drastically increases the expressiveness of the wind power distribution. Secondly, hidden states of the SREDNN are updated in a stochastic way, which builds an infinite mixture of the IGMM for describing the ultimate wind power distribution and enables the SREDNN to model complex patterns across wind speed and wind power sequences. Computational experiments are conducted on a dataset of a commercial wind farm having 25 wind turbines (WTs) and two publicly assessable WT datasets to verify the advantages and effectiveness of the SREDNN for MPWPP. Experimental results show that the SREDNN achieves a lower negative form of the continuously ranked probability score (CRPS*) as well as a superior sharpness and comparable reliability of prediction intervals by comparing against considered benchmarking models. Results also show the clear benefit gained from considering latent random variables in SREDNN.

Pixel-wise exposure control for single-shot HDR imaging: A joint optimization approach

Dynamic range is one of the primary limitations that restricts digital image sensors from acquiring more visual information. Current high dynamic range (HDR) imaging techniques entail a trade-off between dynamic range and visual fidelity. In this work, we propose a HDR imaging method, termed PE-HDR, to achieve both a wide dynamic range and high visual fidelity without additional complex post-processing algorithms. Instead of merging a bracketed exposure sequence, the PE-HDR captures HDR images in a single shot using optical coded pixel-wise exposure control, enabling cost-effective and flexible HDR imaging. By incorporating a differentiable optical encoder and a neural network decoder, we jointly optimize the imaging pipeline from light irradiance to digital image signals, thereby refining the pixel-wise exposure control strategy and improving image fidelity. Both simulations and experiments demonstrate that the proposed method achieves a dynamic range of up to 120 dB and an excellent visual fidelity with spatial resolution of up to 2560 × 1600 pixels.

TEDNet: Twin Encoder Decoder Neural Network for 2D Camera and LiDAR Road Detection

Abstract Robust road surface estimation is required for autonomous ground vehicles to navigate safely. Despite it becoming one of the main targets for autonomous mobility researchers in recent years, it is still an open problem in which cameras and LiDAR sensors have demonstrated to be adequate to predict the position, size and shape of the road a vehicle is driving on in different environments. In this work, a novel Convolutional Neural Network model is proposed for the accurate estimation of the roadway surface. Furthermore, an ablation study has been conducted to investigate how different encoding strategies affect model performance, testing 6 slightly different neural network architectures. Our model is based on the use of a Twin Encoder–Decoder Neural Network (TEDNet) for independent camera and LiDAR feature extraction and has been trained and evaluated on the Kitti–Road dataset. Bird’s Eye View projections of the camera and LiDAR data are used in this model to perform semantic segmentation on whether each pixel belongs to the road surface. The proposed method performs among other state-of-the-art methods and operates at the same frame rate as the LiDAR and cameras, so it is adequate for its use in real-time applications.

BRAND: a platform for closed-loop experiments with deep network models.

Objective.Artificial neural networks (ANNs) are state-of-the-art tools for modeling and decoding neural activity, but deploying them in closed-loop experiments with tight timing constraints is challenging due to their limited support in existing real-time frameworks. Researchers need a platform that fully supports high-level languages for running ANNs (e.g. Python and Julia) while maintaining support for languages that are critical for low-latency data acquisition and processing (e.g. C and C++).Approach.To address these needs, we introduce the Backend for Realtime Asynchronous Neural Decoding (BRAND). BRAND comprises Linux processes, termednodes, which communicate with each other in agraphvia streams of data. Its asynchronous design allows for acquisition, control, and analysis to be executed in parallel on streams of data that may operate at different timescales. BRAND uses Redis, an in-memory database, to send data between nodes, which enables fast inter-process communication and supports 54 different programming languages. Thus, developers can easily deploy existing ANN models in BRAND with minimal implementation changes.Main results.In our tests, BRAND achieved <600 microsecond latency between processes when sending large quantities of data (1024 channels of 30 kHz neural data in 1 ms chunks). BRAND runs a brain-computer interface with a recurrent neural network (RNN) decoder with less than 8 ms of latency from neural data input to decoder prediction. In a real-world demonstration of the system, participant T11 in the BrainGate2 clinical trial (ClinicalTrials.gov Identifier: NCT00912041) performed a standard cursor control task, in which 30 kHz signal processing, RNN decoding, task control, and graphics were all executed in BRAND. This system also supports real-time inference with complex latent variable models like Latent Factor Analysis via Dynamical Systems.Significance.By providing a framework that is fast, modular, and language-agnostic, BRAND lowers the barriers to integrating the latest tools in neuroscience and machine learning into closed-loop experiments.

Predicting Time-to-Healing from a Digital Wound Image: A Hybrid Neural Network and Decision Tree Approach Improves Performance

Despite the societal burden of chronic wounds and despite advances in image processing, automated image-based prediction of wound prognosis is not yet in routine clinical practice. While specific tissue types are known to be positive or negative prognostic indicators, image-based wound healing prediction systems that have been demonstrated to date do not (1) use information about the proportions of tissue types within the wound and (2) predict time-to-healing (most predict categorical clinical labels). In this work, we analyzed a unique dataset of time-series images of healing wounds from a controlled study in dogs, as well as human wound images that are annotated for the tissue type composition. In the context of a hybrid-learning approach (neural network segmentation and decision tree regression) for the image-based prediction of time-to-healing, we tested whether explicitly incorporating tissue type-derived features into the model would improve the accuracy for time-to-healing prediction versus not including such features. We tested four deep convolutional encoder–decoder neural network models for wound image segmentation and identified, in the context of both original wound images and an augmented wound image-set, that a SegNet-type network trained on an augmented image set has best segmentation performance. Furthermore, using three different regression algorithms, we evaluated models for predicting wound time-to-healing using features extracted from the four best-performing segmentation models. We found that XGBoost regression using features that are (i) extracted from a SegNet-type network and (ii) reduced using principal components analysis performed the best for time-to-healing prediction. We demonstrated that a neural network model can classify the regions of a wound image as one of four tissue types, and demonstrated that adding features derived from the superpixel classifier improves the performance for healing-time prediction.

Ecurrent neural network decoding of rotated surface codes based on distributed strategy

Abstract Quantum error correction is a crucial technology for realizing quantum computers. These computers achieve fault-tolerant quantum computing by detecting and correcting errors using decoding algorithms. Quantum error correction using neural network-based machine learning methods is a promising approach that is adapted to physical systems without the need to build noise models. In this paper, we use a distributed decoding strategy, which effectively alleviates the problem of exponential growth of the training set required for neural networks as the code distance of quantum error-correcting codes increases. Our decoding algorithm is based on renormalization group decoding and recurrent neural network decoder. The recurrent neural network is trained through the Resnet architecture to improve its decoding accuracy. Then we tested the decoding performance of our distributed strategy decoder, recurrent neural network decoder and the classic minimum weight perfect matching(MWPM) decoder for rotated surface codes with different code distances under the circuit noise model, the thresholds of this three decoders are about 0.0052, 0.0051, and 0.0049 respectively. Our results demonstrate that the distributed strategy decoder outperforms the other two decoders, achieving approximately a 5% improvement in decoding efficiency compared to the MWPM decoder and approximately a 2% improvement compared to the recurrent neural network decoder.

Predicting wind farm wake losses with deep convolutional hierarchical encoder–decoder neural networks

Wind turbine wakes are the most significant factor affecting wind farm performance, decreasing energy production and increasing fatigue loads in downstream turbines. Wind farm turbine layouts are designed to minimize wake interactions using a suite of predictive models, including analytical wake models and computational fluid dynamics simulations (CFD). CFD simulations of wind farms are time-consuming and computationally expensive, which hinder their use in optimization studies that require hundreds of simulations to converge to an optimal turbine layout. In this work, we propose DeepWFLO, a deep convolutional hierarchical encoder–decoder neural network architecture, as an image-to-image surrogate model for predicting the wind velocity field for Wind Farm Layout Optimization (WFLO). We generate a dataset composed of image representations of the turbine layout and undisturbed flow field in the wind farm, as well as images of the corresponding wind velocity field, including wake effects generated with both analytical models and CFD simulations. The proposed DeepWFLO architecture is then trained and optimized through supervised learning with an application-tailored loss function that considers prediction errors in both wind velocity and energy production. Results on a commonly used test case show median velocity errors of 1.0%–8.0% for DeepWFLO networks trained with analytical and CFD data, respectively. We also propose a model-fusion strategy that uses analytical wake models to generate an additional input channel for the network, resulting in median velocity errors below 1.8%. Spearman rank correlations between predictions and data, which evidence the suitability of DeepWFLO for optimization purposes, range between 92.3% and 99.9%.

Brain control of bimanual movement enabled by recurrent neural networks

Brain-computer interfaces have so far focused largely on enabling the control of a single effector, for example a single computer cursor or robotic arm. Restoring multi-effector motion could unlock greater functionality for people with paralysis (e.g., bimanual movement). However, it may prove challenging to decode the simultaneous motion of multiple effectors, as we recently found that a compositional neural code links movements across all limbs and that neural tuning changes nonlinearly during dual-effector motion. Here, we demonstrate the feasibility of high-quality bimanual control of two cursors via neural network (NN) decoders. Through simulations, we show that NNs leverage a neural ‘laterality’ dimension to distinguish between left and right-hand movements as neural tuning to both hands become increasingly correlated. In training recurrent neural networks (RNNs) for two-cursor control, we developed a method that alters the temporal structure of the training data by dilating/compressing it in time and re-ordering it, which we show helps RNNs successfully generalize to the online setting. With this method, we demonstrate that a person with paralysis can control two computer cursors simultaneously. Our results suggest that neural network decoders may be advantageous for multi-effector decoding, provided they are designed to transfer to the online setting.

Seismic vulnerability signal analysis of low tower cable-stayed bridges method based on convolutional attention network

Abstract Due to the particularity and complexity of sedimentary environments, the wave impedance differences between different reflection interfaces in underground media may vary greatly. Therefore, an encoder–decoder neural network is proposed to enhance erroneous seismic weak reflection signals. The convolutional neural network (CNN) has the problem of difficulty in parallel computing, resulting in slow network training and computational efficiency. Considering that attention has an innate global self-attention mechanism, can compensate for long-term dependency deficiencies, and has the ability to perform parallel computing, which greatly compensates for the shortcomings of CNNs and recurrent neural networks, a seismic impedance inversion method based on convolutional attention networks is proposed. To improve the ability to extract noise, residual structure and convolutional attention module (CBAM) were introduced. The residual structure utilizes residual jump to weaken network degradation and reduce the difficulty of feature mapping. The CBAM uses a mixed attention weight of channel and space, which can enhance features with high correlation and suppress features with low correlation. In the decoder, in order to improve the dimension recovery ability of feature fusion, bilinear interpolation is selected for upsampling. The application results of the model and actual data indicate that this method can effectively enhance the weak reflection signals caused by the formation itself and improve the reservoir identification ability of seismic data.

Deep attentive survival analysis in limit order books: estimating fill probabilities with convolutional-transformers

One of the key decisions in execution strategies is the choice between a passive (liquidity providing) or an aggressive (liquidity taking) order to execute a trade in a limit order book (LOB). Essential to this choice is the fill probability of a passive limit order placed in the LOB. This paper proposes a deep learning method to estimate the filltimes of limit orders posted in different levels of the LOB. We develop a novel model for survival analysis that maps time-varying features of the LOB to the distribution of filltimes of limit orders. Our method is based on a convolutional-Transformer encoder and a monotonic neural network decoder. We use proper scoring rules to compare our method with other approaches in survival analysis, and perform an interpretability analysis to understand the informativeness of features used to compute fill probabilities. Our method significantly outperforms those typically used in survival analysis literature. Finally, we carry out a statistical analysis of the fill probability of orders placed in the order book (e.g. within the bid-ask spread) for assets with different queue dynamics and trading activity.

ESGC-MDA: Identifying miRNA-disease associations using enhanced Simple Graph Convolutional Networks.

MiRNAs play an important role in the occurrence and development of human disease. Identifying potential miRNA-disease associations is valuable for disease diagnosis and treatment. Therefore, it is urgent to develop efficient computational methods for predicting potential miRNA-disease associations to reduce the cost and time associated with biological wet experiments. In addition, high-quality feature representation remains a challenge for miRNA-disease association prediction using graph neural network methods. In this paper, we propose a method named ESGC-MDA, which employs an enhanced Simple Graph Convolution Network to identify miRNA-disease associations. We first construct a bipartite attributed graph for miRNAs and diseases by computing multi-source similarity. Then, we enhance the feature representations of miRNA and disease nodes by applying two strategies in the simple convolution network, which include randomly dropping messages during propagation to ensure the model learns more reliable feature representations, and using adaptive weighting to aggregate features from different layers. Finally, we calculate the prediction scores of miRNA-disease pairs by using a fully connected neural network decoder. We conduct 5-fold cross-validation and 10-fold cross-validation on HDMM v2.0 and HMDD v3.2, respectively, and ESGC-MDA achieves better performance than state-of-the-art baseline methods. The case studies for cardiovascular disease, lung cancer and colon cancer also further confirm the effectiveness of ESGC-MDA. The source codes are available at https://github.com/bixuehua/ESGC-MDA.

Image and Video Captioning Using Machine Learning

Abstract: Image based web crawler is the way toward looking through data by utilizing related images. The tremendous assets of images are accessible on the web in that a large number of the images are contain as with named and without named caption. The users are needed to look through the images relying upon their necessities. In that a significant number of the users can't recover the pertinent images as a result of their unpredicted appropriate inscription on their images. Our task is to generate an automatic caption for the images based on the image content. To produce an image caption, firstly, the content of the image should be fully understood; and then the semantic information contained in the image should be described using a phrase or statement that conforms to certain grammatical rules. Thus, it requires techniques from both computer vision and natural language processing to connect the two different media forms together, which is highly challenging. The paper targets producing mechanized inscriptions by learning the contents of the image. At present images are clarified with human intercession and it turns out to be almost unthinkable task for tremendous databases. The picture information base is given as contribution to a deep neural network Convolutional Neural Network encoder for creating caption which extricates the highlights and subtleties out of our image and Recurrent Neural Network decoder is utilized to interpret the highlights and articles given by our image to acquire consecutive, meaningful description of the image.

Convolutional LSTM-Attention Based Encoder–Decoder Neural Network for Prediction of Chaotic Vibrations of Multi-Dimensional Dynamic Systems

The present research proposes a convolutional long-short term memory (ConvLSTM) with an attention mechanism (AM) model, termed as ConvLSTM-AM, to conduct prediction of chaotic vibrations of multi-dimensional dynamic systems. The proposed data-driven model is based on an encoder–decoder architecture where the lengths of inputs and outputs are variable. Different from other conventional benchmarks which consider the temporal correlation solely to deal with the chaotic sequences, this research work takes the spatial information into account. In this sense, the ConvLSTM is adopted as an encoder to acquire useful chaotic spatiotemporal patterns and retain long-term successive dependencies. LSTM and AM in this research are taken as the main structures of the procedure in decoding, in which the LSTM is used as the further temporal processor and AM is stacked on the top to exploit more salient information of the historical data. Among that, a residual connection between the outputs of LSTM and the information of attention is considered in AM to prevent gradient vanishment. Two datasets of chaotic vibrations of multi-dimensional systems are employed to adequately illustrate the effectiveness and feasibility of the proposed model. Besides, five conventional benchmarks are built to demonstrate the advantages of the proposed model in terms of both training and generalization performance. As found in the research, the training time is reduced with lower testing loss in comparing with the other five counterparts, as the spatial information introduced expedites the training convergence. The present research provides a useful guidance for predicting and analysing chaotic vibrations of multi-dimensional dynamic systems.