Forward Network Research Articles

P-Swin: Parallel Swin transformer multi-scale semantic segmentation network for land cover classification

With the recent development of remote sensing technology and deep learning, semantic segmentation methods have been increasingly used in land cover classification. However, this method is faced with the challenge of incomplete recognition caused by big differences in scale of ground objects. Owing to multi-head self-attention, the Swin Transformer Network (Swin) has a large receptive field at its shallow level, which is conducive to the identification of large-scale objects. However, Swin does not fully mine the context information of features, so it is easy to cause incomplete recognition. Based on Swin, we propose a parallel window-based Transformer Network, Parallel Swin Transformer Network (P-Swin). The core of P-Swin is a Parallel Swin Transformer Block (PST Block), which includes Window-based Self Attention Interaction (WSAI) and Feed Forward Network (FFN). WSAI can not only calculate the relationship within windows, but also establish the relationship between windows. Therefore, it improves the ability of network to obtain feature context information. P-Swin outperformed Swin and reached the highest level, with 76.42% mIoU for the test set in the ISPRS Potsdam 2D dataset (Swin: 75.95%), 65.13% mIoU for the test set in the Gaofen Image Dataset (Swin: 63.41%), and 64.61% mIoU for the test set in the WHDLD Dataset (Swin: 63.01%)

Design of an efficient decision support system using evolutionary deep forward network model

With the vast advancements in the medical domain, earlier prediction of disease plays a substantial role in enhancing healthcare quality and making better decisions during tough times. This research concentrates on modelling and automated disease prediction model to offer an earlier prediction model for heart disease and the risk factors. This work considers a standard UCI machine learning-based benchmark dataset for model validation and extracts the risk factors related to the disease. The outliers and imbalanced datasets are pre-processed using data normalization to enhance the classification performance. Here, feature selection is performed using non-linear Particle Swarm Optimization (NL - PSO). Finally, classification is done with the Improved Deep Evolutionary model with Feed Forward Neural Networks (IDEBDFN). The algorithm’s learning nature is used to evaluate the nature of the hidden layers to produce the optimal results. The outcomes demonstrate that the anticipated model provides superior prediction accuracy. The simulation is carried out in a MATLAB environment, and metrics like accuracy, F-measure, precision, recall, and so on are evaluated. The accuracy (without features) of the evolutionary model in the UCI ML dataset is 97.65%, accuracy (with features) is 98.56%, specificity is 95%, specificity is 2% higher than both the datasets, F1-score is 40%, execution time (min) is 0.04 min, and the AUROC is 96.85% which is substantially higher than other datasets. The proposed model works efficiently compared to various prevailing standards and individual approaches.

Deep Learning for the Design of Toroidal Metasurfaces

In recent years, the toroidal dipoles have had a profound impact on several fields including electromagnetism. However, the on-demand design of toroidal metasurfaces is still a very time-consuming process. In this paper, a method of neural network simulating the nonlinear relationship between the structural parameters of metasurfaces and its multipole scattered powers is proposed based on a deep learning algorithm. The forward network can quickly predict the scattered powers from input structural parameters, which can achieve an accuracy comparable to the electromagnetic simulations. In addition, with the required scattering spectrum as input, the appropriate parameters of the structure could be automatically calculated and then output by the inverse network which can achieve a low mean square error of 0.074 in training set and 0.18 in the test set. Compared with the conventional design process, the proposed deep learning model can guide the design of the toroidal dipole metasurface faster and pave the way for the rapid development of toroidal metasurfaces.

Multiview Textured Mesh Recovery by Differentiable Rendering

Although having achieved the promising results on shape and color recovery through self-supervision, the multi-layer perceptrons-based methods usually suffer from heavy computational cost on learning the deep implicit surface representation. Since rendering each pixel requires a forward network inference, it is very computational intensive to synthesize a whole image. To tackle these challenges, we propose an effective coarse-to-fine approach to recover the textured mesh from multi-views in this paper. Specifically, a differentiable Poisson Solver is employed to represent the object's shape, which is able to produce topology-agnostic and watertight surfaces. To account for depth information, we optimize the shape geometry by minimizing the differences between the rendered mesh and the predicted depth from multi-view stereo. In contrast to the implicit neural representation on shape and color, we introduce a physically based inverse rendering scheme to jointly estimate the environment lighting and object's reflectance, which is able to render the high resolution image at real-time. The texture of the reconstructed mesh is interpolated from a learnable dense texture grid. We have conducted the extensive experiments on several multi-view stereo datasets, whose promising results demonstrate the efficacy of our proposed approach. The code is available at https://github.com/l1346792580123/diff.

ReCoMIF: Reading comprehension based multi-source information fusion network for Chinese spoken language understanding

Spoken language understanding (SLU) plays a crucial role in the performance of dialogue systems. It usually includes slot filling and intent detection (SFID) tasks aiming at semantic parsing of utterances. At present, researchers focus mainly on English SLU tasks, while such investigations on Chinese utterances are not sufficient. In this paper, we first propose a reading comprehension based multi-source information fusion network, called ReCoMIF for Chinese SFID tasks by transforming the SLU task into a multi-turn question answering procedure comprising multiple choice for intent detection and span extraction for slot filling. Moreover, we present a TCM_CLS module with a concise architecture composed of TextCNN, MaxPooling, and feed forward network plus the [CLS] representation. Such three TCM_CLS modules are stacked in the proposed ReCoMIF network that can serve as sufficient integration of multi-source information originating from contexts, reading comprehension based queries, and hidden representations concerning intent and slot semantics. Finally, experimental results and ablation studies on three Chinese SLU datasets show that our proposed model can effectively fuse intent and slot information achieved by\\ state-of-the-art performance compared with other baseline methods.

Fast forward approximation and multitask inversion of gravity anomaly based on UNet3+

SUMMARYGravity inversion is a process that obtains the spatial structure and physical properties of underground anomalies using surface collected gravity anomaly data. In recent years, the rapid development of deep learning (DL) has enabled the achievement of good results for gravity inversion methods based on DL. These methods aim to learn the mapping between geological models and gravity anomaly data by training a neural network with geological models as labels. However, using DL inversion requires generating a large amount of training data for each geological target and involves the forward calculation of the generated models, which inevitably consumes a large amount of time and storage space. To address this issue, we propose using a neural network to approximate the expensive forward computation with a fast evaluation alternative. After training, the network can reproduce gravity anomalies at any observation point. To evaluate the effectiveness of the forward model, we use the gravity anomalies predicted by the forward network for inversion network training. Additionally, to mitigate the problem of poor generalization of existing DL inversions, we propose using multitask learning. By learning multiple related tasks simultaneously, the generalization ability of the model improves, thus enhancing the performance of the main task. In this paper, a multitask UNet3+ network is proposed to realize anomaly bodies localization and density contrasts reconstruction simultaneously. Test results on the synthetic data set demonstrate that the gravity anomalies predicted by the forward network can be successfully inverted, and the multitask approach can predict subsurface geology more accurately than the single-task. To further illustrate the effectiveness of the algorithm, we apply this method to the inversion of the San Nicolas deposit in central Mexico.

Inverse design of polarization conversion metasurfaces by deep neural networks.

To address the problem of multiple solutions and improve the calculating speed, we construct a tandem architecture consisting of a forward modeling network and an inverse design network. Using this combined network, we inversely design the circular polarization converter and analyze the effect of different design parameters on the prediction accuracy of the polarization conversion rate. The average mean square error of the circular polarization converter is 0.00121 at an average prediction time of 1.56×10-2 s. If only the forward modeling process is considered, it takes 6.15×10-4 s, which is 2.1×105 times faster than that using the traditional numerical full-wave simulation method. By slightly resizing the network input and output layers, the network is adaptable to the design of both the linear cross-polarization and linear-to-circular polarization converters.

Network motifs and hypermotifs in TGFβ-induced epithelial to mesenchymal transition and metastasis

Epithelial to mesenchymal transition (EMT) is a complex, non-linear, dynamic multistep process that plays an integral role in the development of metastatic cancers. A diverse range of signaling molecules, along with their associated pathways, were observed to be involved in promoting EMT and cancer metastasis. Transforming growth factor–β (TGFβ), through its SMAD-dependent and SMAD-independent signaling, orchestrates numerous regulators that converge on key EMT transcription factors (TFs). These TFs further govern the phenotypic transition of cancer cells from epithelial to mesenchymal states. This study explores the TGFβ signaling pathway and its unique network architecture to understand their information processing roles in EMT. Two coherent type 1 feed forward network motifs regulating the expression of SNAIL and N-cadherin were observed. SNAIL, which is one of the crucial regulators of EMT, links both the coherent type 1 feed forward loops (C1FFLs) leading to hypermotif-like structure (Adler and Medzhitov, 2022). Systems modeling and analysis of these motifs and hypermotifs illustrated several interesting emergent information processing roles of the regulators involved. The known roles of these regulators, as described in the literature, were highly correlated with the emergent properties observed. The motifs illustrated persistence detection and noise filtration in regulating the expression of SNAIL and N-cadherin. Along with these system-level properties, the hypermotif architecture also exhibited temporal expression of GLI, SNAIL, ZEB, and N-cadherin. Furthermore, a hypothetical three-layered C1FFL hypermotif was postulated and analyzed. The analysis revealed various interesting system-level properties. However, possible existence of such real biological networks needs further exploration both theoretically and experimentally. Deciphering these network motifs and hypermotifs has provided an additional understanding of the complex biological phenomenon, such as EMT in cancer metastasis.

A Collaborative Learning Tracking Network for Remote Sensing Videos.

With the increasing accessibility of remote sensing videos, remote sensing tracking is gradually becoming a hot issue. However, accurately detecting and tracking in complex remote sensing scenes is still a challenge. In this article, we propose a collaborative learning tracking network for remote sensing videos, including a consistent receptive field parallel fusion module (CRFPF), dual-branch spatial-channel co-attention (DSCA) module, and geometric constraint retrack strategy (GCRT). Considering the small-size objects of remote sensing scenes are difficult for general forward networks to extract effective features, we propose a CRFPF-module to establish parallel branches with consistent receptive fields to separately extract from shallow to deep features and then fuse hierarchical features adaptively. Since the objects and their background are difficult to distinguish, the proposed DSCA-module uses the spatial-channel co-attention mechanism to collaboratively learn the relevant information, which enhances the saliency of the objects and regresses to precise bounding boxes. Considering the interference of similar objects, we designed a GCRT-strategy to judge whether there is a false detection through the estimated motion trajectory and then recover the correct object by weakening the feature response of interference. The experimental results and theoretical analysis on multiple datasets demonstrate our proposed method's feasibility and effectiveness. Code and net are available at https://github.com/Dawn5786/CoCRF-TrackNet.

Accelerating magnetotelluric forward modeling with deep learning: Conv-BiLSTM and D-LinkNet

Magnetotelluric forward modeling is important for exploring underground electromagnetic anomalies. Although directly solving the electromagnetic wave equation has high accuracy, its computational cost is usually unaffordable for large-scale models. A neural network (NN) can increase the computation of the magnetotelluric forward modeling; however, its numerical accuracy is limited owing to the use of a simple network structure and small-scale training data sets. We have increased the computational efficiency of the magnetotelluric forward modeling by deep learning based on cyclic NN and full convolution NN models. First, we have extracted the basic characteristics of the 2D-magnetotelluric forward modeling, which were important for selecting and optimizing the network. Then, we have developed two forward network models: convolutional bidirectional long short-term memory (Conv-BiLSTM) and LinkNet with pretrained encoder and dilated convolution (D-LinkNet). Next, we have constructed data sets of large-scale multiple anomalies. Finally, we have tested our models using various examples. Existing methods only consider a single anomaly and have low accuracy; in contrast, our methods can handle multianomaly models because they have strong generalization, even though the training is based on models with two or three anomalies. Numerical experiments find that the average accuracy of the Conv-BiLSTM and D-LinkNet forward network models was 87.5% and 95.3%, respectively. Compared with D-LinkNet, the Conv-BiLSTM network model has lower accuracy but higher computational efficiency. Our deep-learning schemes can significantly reduce the computational burdens of the magnetotelluric forward modeling, and thus allow us to perform swift inversions of multianomaly models.

Cycle optimization metric learning for few-shot classification

Metric learning methods are widely used in few-shot learning due to their simplicity and effectiveness. Most existing methods directly predict query labels by comparing the similarity between support and query samples. In this paper, we design a cycle optimization metric network for few-shot classification task that optimizes model performance based on loop-prediction of the labels of query samples and support samples. Specifically, we construct a forward network and reverse network based on a geometric algebra Graph Neural Network (GA-GNN). These two networks form the loop prediction from support samples to query samples and then back to support samples, guided by a cycle-consistency loss. We also introduce an optimization module that is able to correct the predicted results of query samples to further improve the network performance. Our extensive experimental results demonstrate that the proposed cycle optimization metric network outperforms existing state-of-the-art few-shot learning methods on classification tasks.

Application of Feature Pyramid Network and Feature Fusion Single Shot Multibox Detector for Real-Time Prostate Capsule Detection

In the process of feature propagation, the low-level convolution layers of the forward feature propagation network lack semantic information, and information loss occurs when fine-grained information is transferred to higher-level convolution; therefore, multi-stage feature fusion networks are needed to solve the interaction between low-level convolution layers and high-level convolution layers. Based on a two-way feature feedback network and feature fusion mechanism, we created a new object detection network called Feature Pyramid Network (FPN)-based Feature Fusion Single Shot Multibox Detector (FFSSD). A bottom-up and top-down architecture with lateral connections enhances the detector’s ability to extract features, then high-level multi-scale semantic feature maps are utilized to generate a feature pyramid network. The results show that the proposed network the mAP for prostate capsule image detection reaches 83.58%, providing real-time detection ability. The context interaction mechanism can transfer high-level semantic information to low-level convolution, and the resulting convolution after low-level and high-level fusion contains richer location and semantic information.

Predicting origin for bone metastatic cancer using deep learning-based pathology.

Predicting origin for bone metastatic cancer using deep learning-based pathology.

Toward Compact Transformers for End-to-End Object Detection With Decomposed Chain Tensor Structure

DEtection TRansformer (DETR) is a recently proposed method that streamlines the detection pipeline and achieves competitive results against two-stage detectors such as Faster-RCNN. The DETR models get rid of complex anchor generation and post-processing procedures thereby making the detection pipeline more intuitive. However, the numerous redundant parameters in transformers make the computation and storage of the DETR models intensive, which seriously hinder them to be deployed on the resources-constrained devices. In this paper, to obtain a compact end-to-end detection framework, we propose to deeply compress the transformers with low-rank tensor decomposition. The basic idea of our tensor-based compression method is to represent the large-scale weight matrix in one network layer with a chain of low-order matrices. Furthermore, we show that redundant attention heads will hinder the performance of detection transformers. We thus propose a gated multi-head attention (GMHA) module to suppress the redundant attention information by normalizing the attention heads. In GMHA, each attention head has an independent gate to determine the passed attention value, thereby down-weighting the uninformative heads. The accuracy drop of the tensor-compressed DETR models can be mitigated by applying GMHA modules. Lastly, to obtain fully compressed DETR models, a low-bitwidth quantization technique is introduced for further reducing the model storage size. Based on the proposed methods, we can achieve significant parameter and model size reduction while maintaining high detection performance. We conduct extensive experiments on the COCO and PASCAL VOC datasets to validate the effectiveness of our tensor-compressed (tensorized) DETR models. The experimental results on the COCO benchmark show that we can attain <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3.7\times $ </tex-math></inline-formula> full model compression with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$482\times $ </tex-math></inline-formula> feed forward network (FFN) parameter reduction and only 0.6 points accuracy drop.

Imaging and seismic modelling inside volcanoes using machine learning

Despite advances in seismology and computing, the ability to image subsurface volcanic environments is poor, limiting our understanding of the overall workings of volcanic systems. This is related to substantive structural heterogeneities which strongly scatters seismic waves obscuring the ballistic arrivals normally used in seismology for wave velocity determination. Here we address this constraint by, using a deep learning approach, a Fourier neural operator (FNO), to model and invert seismic signals in volcanic settings. The FNO is trained using 40,000+ simulations of elastic wave propagation through complex volcano models, and includes the full scattered wavefield. Once trained, the forward network is used to predict elastic wave propagation and is shown to accurately reproduce the seismic wavefield. The FNO is also trained to predict heterogeneous velocity models given a limited set of input seismograms. It is shown to capture details of the complex velocity structure that lie far outside the ability of current methods available in volcano imagery.

Vibration transmission characteristic prediction and structure inverse design of acoustic metamaterial beams based on deep learning

In this paper, an intelligent prediction and design method of acoustic metamaterial beams based on deep learning is presented. A theoretical model of acoustic metamaterial beams is derived by using the spectral element method to calculate the band structure and transmission characteristic of beam-type acoustic metamaterial. A high-degree-of-freedom design space dataset of band structure and transmission characteristics is constructed. In addition, the vibration transmission characteristics of acoustic metamaterial beams are predicted and the on-demand inverse design of acoustic metamaterial beams is realized by constructing a fully connected deep learning neural network model. Furthermore, the forward prediction and reverse design network model is verified by using the autoencoder neural network model. Results show that the predicted value of the autoencoder neural network structure is in good agreement with the target value, which indicates the feasibility of the intelligent design method of acoustic metamaterials based on deep learning. The presented intelligent design method could be potentially utilized in the field of fast and efficient acoustic metamaterial design for low frequency vibration isolation in engineering structures.

Cloth manipulation planning on basis of mesh representations with incomplete domain knowledge and voxel-to-mesh estimation.

Cloth manipulation is common in both housework and manufacturing. However, robotic cloth manipulation remains challenging, especially for less controlled and open-goal settings. We consider the problem of open-goal planning for robotic cloth manipulation, with focus on the roles of cloth representation and epistemic uncertainty. Core of our system is a neural network trained as a forward model of cloth behaviour under manipulation, with planning performed through backpropagation. We introduce a neural network-based routine for estimating mesh representations from voxel input, and perform planning in mesh format internally. We address the problem of planning with incomplete domain knowledge by introducing an explicit epistemic uncertainty penalty, using prediction divergence between two instances of the forward model network as a proxy of epistemic uncertainty. This allows us to avoid plans with high epistemic uncertainty during planning. Finally, we introduce logic for handling restriction of grasp points to a discrete set of candidates, in order to accommodate graspability constraints imposed by robotic hardware. We evaluate the system's mesh estimation, prediction, and planning ability on simulated cloth for sequences of one to three manipulations. Comparative experiments confirm that planning on basis of estimated meshes improves accuracy compared to voxel-based planning, and that epistemic uncertainty avoidance improves performance under conditions of incomplete domain knowledge. Planning time cost is a few seconds. We additionally present qualitative results on robot hardware. Our results indicate that representation format and epistemic uncertainty are important factors to consider for open-goal cloth manipulation planning.

AFT: Adaptive Fusion Transformer for Visible and Infrared Images.

In this paper, an Adaptive Fusion Transformer (AFT) is proposed for unsupervised pixel-level fusion of visible and infrared images. Different from the existing convolutional networks, transformer is adopted to model the relationship of multi-modality images and explore cross-modal interactions in AFT. The encoder of AFT uses a Multi-Head Self-attention (MSA) module and Feed Forward (FF) network for feature extraction. Then, a Multi-head Self-Fusion (MSF) module is designed for the adaptive perceptual fusion of the features. By sequentially stacking the MSF, MSA, and FF, a fusion decoder is constructed to gradually locate complementary features for recovering informative images. In addition, a structure-preserving loss is defined to enhance the visual quality of fused images. Extensive experiments are conducted on several datasets to compare our proposed AFT method with 21 popular approaches. The results show that AFT has state-of-the-art performance in both quantitative metrics and visual perception.

Adaptive Recurrent Forward Network for Dense Point Cloud Completion

Point cloud completion is an interesting and challenging task in 3D vision, which aims to recover complete shapes from sparse and incomplete point clouds. Existing completion networks often require a vast number of parameters and substantial computational costs to achieve a high performance level, which may limit their practical application. In this work, we propose a novel Adaptive efficient Recurrent Forward Network (ARFNet), which is composed of three parts: Recurrent Feature Extraction (RFE), Forward Dense Completion (FDC) and Raw Shape Protection (RSP). In an RFE, multiple short global features are extracted from incomplete point clouds, while a dense quantity of completed results are generated in a coarse-to-fine pipeline in the FDC. Finally, we propose the Adamerge module to preserve the details from the original models by merging the generated results with the original incomplete point clouds in the RSP. In addition, we introduce the Sampling Chamfer Distance to better capture the shapes of the models and the balanced expansion constraint to restrict the expansion distances from coarse to fine. According to the experiments on ShapeNet and KITTI, our network can achieve state-of-the-art completion performances on dense point clouds with fewer parameters, smaller model sizes, lower memory costs and a faster convergence.

Study of Word Embeddings for Enhanced Cyber Security Named Entity Recognition

A vast majority of cyber security information is in the form of unstructured text. A much-needed task is to have a machine-assisted analysis of such information. Named Entity Recognition (NER) provides a vital step towards this conversion. However, cyber security named entities are not restricted to classical entity types like people, location, organisation, miscellaneous etc but comprise a large set of domain-specific entities. Word embedding has emerged as the dominant choice for the initial transfer of semantics to downstream NLP tasks and impacts performance. Though several word embeddings learned using general purpose large corpus like Google News, Wikipedia etc. are available as pre-trained embeddings and have shown good performance on NER tasks; this trend is not consistent when it comes to domain-specific NER. This work explores the relative performances and suitability of prominent word embeddings for cyber security NER task. Embeddings considered include both general-purpose pre-trained word embeddings (non-contextual and contextual) available in the public domain and task-adapted embedding generated by fine-tuning these pre-trained embeddings on a task-specific supervised dataset. The results indicate that when it comes to using pre-trained embeddings for cyber security NER, fastText performs better than GloVe and BERT. However, when embeddings are further fine-tuned for the cyber-NER task, the performance of all the fine-tuned embeddings improved by +2-7%. Further, BERT embedding fine-tuned using position-wise FFN (Feed Forward Network) produced the state-of-the-art 0.974 F1-Score on the cyber security NER dataset.