Layered Decoding Research Articles

An interpretable health indicator for bearing condition monitoring based on semi-supervised autoencoder latent space variance maximization

Effective health indicator (HI) construction can help equipment managers detect the abnormal state of rotating machinery quickly. However, although the current deep learning-based HI construction methods have good life prediction value, most of them lose the ability to detect device anomalies and little work has been done on model interpretability. Therefore, an interpretable HI construction method based on semi-supervised autoencoder (AE) latent space variance maximization (SSALSVM) was proposed to monitor the health status of bearings. In order to fully excavate degradation features inside the device and make the model focus on the encoding process, a deep convolutional neural network (DCNN) is used as the encoding layer, while only a layer of fully-connected layer is used as the decoding layer. In addition, to enable the latent space to capture the device early degradation point (EDP) successfully, an auxiliary layer is added to the output of the encoder layer. Simultaneously, for improving the sensitivity of the indicator to capture equipment abnormal state and highlight the difference between equipment health state and degradation state, the constraint of variance maximization is added into the latent space. The model optimizing process was presented by observing the projected variance of the test set in latent space of each epoch model. The validity of the proposed HI was verified by comparison experiments on two datasets.

Chinese E-commerce NER Using RoBERTa-wmm under the Machine Reading Comprehension Paradigm

In practical applications within the e-commerce domain, there is often a requirement to identify product entities and their corresponding brand entities, based on their descriptions. However, there has been a relatively limited focus on studies addressing the Named Entities Recognition in the e-commerce domain. we crawled data from e-commerce websites and transformed them into a Named Entity Recognition dataset, which is suitable for Machine Reading Comprehension. Since the questions Machine Reading Comprehension contain a priori semantic information about the types of the entities, we propose a model that uses the MRC modeling paradigm to solve the task of recognizing brand entities as well as commodity entities in the e-commerce domain. The model encodes the contexts and the corresponding questions using the RoBERTa-wwm model, and then further extracts the semantic information of the contexts using an attention network. We utilize SoftMax as the decoding layer to get the head index and tail index of the entity, and finally use the matching module to get the entity index. Through experiments on two e-commerce datasets, the results show that the new method can significantly improve the recognition effect of Chinese NER in e-commerce domain.

Multi-scale spatial-temporal aware transformer for traffic prediction

Traffic prediction is an important part of smart city management. Accurate traffic prediction can be deployed in urban applications such as congestion alerting and route planning, thus providing sustainable services to the public or relevant departments. Although some improvements have been made in existing traffic prediction methods, there are challenges due to the following: (1) Time series has multi-scale nature, that is, from different scale time ranges, traffic flow changes show different trends; (2) Spatial heterogeneity, meaning that traffic conditions in similar functional areas are usually similar. This task remains difficult. To address the above challenges, we propose a new spatial-temporal prediction method, namely Multi-Scale Spatial-Temporal Aware Transformer (MSSTAT), which is a Transformer architecture with multi-scale characteristics. Specifically, compared to the input of encoder, the input of different decoder layers has different scale information, MSSTAT synchronizes model the connection between time steps and scale information by a kind of Parallel Cross Multi-Head Attention, which gives each time step several times the perceived field while also being able to weaken the impact brought by anomaly point. In addition, to add connections between regions with similar functions, we map the traffic data of each node as a probability distribution and then measure the similarity between the nodes by the Wasserstein Distance, which leads to our proposed spatial-temporal aware adjacency matrix. Experimental results on four traffic flow datasets show that MSSTAT outperforms the state-of-the-art baseline.

A Joint Extraction Model for Entity Relationships Based on Span and Cascaded Dual Decoding.

The entity-relationship joint extraction model plays a significant role in entity relationship extraction. The existing entity-relationship joint extraction model cannot effectively identify entity-relationship triples in overlapping relationships. This paper proposes a new joint entity-relationship extraction model based on the span and a cascaded dual decoding. The model includes a Bidirectional Encoder Representations from Transformers (BERT) encoding layer, a relational decoding layer, and an entity decoding layer. The model first converts the text input into the BERT pretrained language model into word vectors. Then, it divides the word vectors based on the span to form a span sequence and decodes the relationship between the span sequence to obtain the relationship type in the span sequence. Finally, the entity decoding layer fuses the span sequences and the relationship type obtained by relation decoding and uses a bi-directional long short-term memory (Bi-LSTM) neural network to obtain the head entity and tail entity in the span sequence. Using the combination of span division and cascaded double decoding, the overlapping relations existing in the text can be effectively identified. Experiments show that compared with other baseline models, the F1 value of the model is effectively improved on the NYT dataset and WebNLG dataset.

DeepCrackAT: An effective crack segmentation framework based on learning multi-scale crack features

The detection of cracks is essential for assessing and maintaining building and road safety. However, the large appearance variations and the complex topological structures of cracks bring challenges to automatic crack detection. To alleviate the above challenges, we propose a deep multi-scale crack feature learning model called DeepCrackAT for crack segmentation, which is based on an encoder–decoder network with feature tokenization mechanism and attention mechanism. Specifically, we use hybrid dilated convolutions in the first three layers of the encoder–decoder to increase the network’s receptive field and capture more crack information. Then, we introduce a tokenized multilayer perceptron (Tok-MLP) in the last two layers of the encoder–decoder to tokenize and project high-dimensional crack features into low-dimensional space. This helps to reduce parameters and enhance the network’s ability of noise resistance. Next, we concatenate the features corresponding to the encoder–decoder layers and introduce the convolutional block attention module (CBAM) to enhance the network’s perception of the critical crack region. Finally, the five-layer features are fused to generate a binary segmentation map of the crack image. We conducted extensive experiments and ablation studies on two real-world crack datasets, and DeepCrackAT achieved 97.41% and 97.25% accuracy on these datasets, respectively. The experimental results show that the proposed method outperforms the current state-of-the-art methods.

A two-step deep learning method for 3DCT-2DUS kidney registration during breathing

This work proposed KidneyRegNet, a novel deep registration pipeline for 3D CT and 2D U/S kidney scans of free breathing, which comprises a feature network, and a 3D–2D CNN-based registration network. The feature network has handcrafted texture feature layers to reduce the semantic gap. The registration network is an encoder-decoder structure with loss of feature-image-motion (FIM), which enables hierarchical regression at decoder layers and avoids multiple network concatenation. It was first pretrained with a retrospective dataset cum training data generation strategy and then adapted to specific patient data under unsupervised one-cycle transfer learning in onsite applications. The experiment was performed on 132 U/S sequences, 39 multiple-phase CT and 210 public single-phase CT images, and 25 pairs of CT and U/S sequences. This resulted in a mean contour distance (MCD) of 0.94 mm between kidneys on CT and U/S images and MCD of 1.15 mm on CT and reference CT images. Datasets with small transformations resulted in MCDs of 0.82 and 1.02 mm, respectively. Large transformations resulted in MCDs of 1.10 and 1.28 mm, respectively. This work addressed difficulties in 3DCT-2DUS kidney registration during free breathing via novel network structures and training strategies.

BiFTransNet: A unified and simultaneous segmentation network for gastrointestinal images of CT & MRI

Gastrointestinal (GI) cancer is a malignancy affecting the digestive organs. During radiation therapy, the radiation oncologist must precisely aim the X-ray beam at the tumor while avoiding unaffected areas of the stomach and intestines. Consequently, accurate, automated GI image segmentation is urgently needed in clinical practice. While the fully convolutional network (FCN) and U-Net framework have shown impressive results in medical image segmentation, their ability to model long-range dependencies is constrained by the convolutional kernel’s restricted receptive field. The transformer has a robust capacity for global modeling owing to its inherent global self-attention mechanism. The TransUnet model leverages the strengths of both the convolutional neural network (CNN) and transformer models through a hybrid CNN-transformer encoder. However, the concatenation of high- and low-level features in the decoder is ineffective in fusing global and local information. To overcome this limitation, we propose an innovative transformer-based medical image segmentation architecture called BiFTransNet, which introduces a BiFusion module into the decoder stage, enabling effective global and local feature fusion by enabling feature integration from various modules. Further, a multilevel loss (ML) strategy is introduced to oversee the learning process of each decoder layer and optimize the use of globally and locally fused contextual features at different scales. Our method achieved a Dice score of 89.51% and an intersection-over-union (IoU) score of 86.54% on the UW-Madison Gastrointestinal Segmentation dataset. Moreover, our method attained a Dice score of 78.77% and a Hausdorff distance (HD) of 27.94% on the Synapse Multi-organ Segmentation dataset. Compared with the state-of-the-art methods, our proposed method achieves superior segmentation performance in gastrointestinal segmentation tasks. More significantly, our method can be easily extended to medical segmentation in different modalities such as CT and MRI. Our method achieves clinical multimodal medical segmentation and provides decision supports for clinical radiotherapy plans.

A new real-time image semantic segmentation framework based on a lightweight deep convolutional encoder-decoder architecture for robotic environment sensing

Background: Image semantic segmentation can be understood as the allocation of a predefined category label to each pixel in the image to achieve the region segmentation of the image. Different categories in the image are identified with different colors. While achieving pixel classification, the position information of pixel points of different categories in the image is retained. Purpose: Due to the influence of background and complex environment, the traditional semantic segmentation methods have low accuracy. To alleviate the above problems, this paper proposes a new real-time image semantic segmentation framework based on a lightweight deep convolutional encoder-decoder architecture for robotic environment sensing. Methodology: This new framework is divided into three stages: encoding stage, decoding stage and dimension reduction stage. In the coding stage, a cross-layer feature map fusion (CLFMF) method is proposed to improve the effect of feature extraction. In the decoding stage, a new lightweight decoder (LD) structure is designed to reduce the number of convolutional layers to speed up model training and prediction. In the dimension reduction stage, the convolution dimension reduction method (CDR) is presented to connect the encoder and decoder layer by layer to enhance the decoder effect. Results: Compared with other state-of-the-art image semantic segmentation methods, we conduct comparison experiments on datasets Cityscapes, SUN RGB-D, CamVid, KITTI. The Category iIoU combined with the proposed method is more than 70%, and the Category IoU is as high as 89.7%. Conclusion: The results reflect that the new method can achieve the better semantic segmentation effect.

Scale-aware dense residual retinal vessel segmentation network with multi-output weighted loss

BackgroundRetinal vessel segmentation provides an important basis for determining the geometric characteristics of retinal vessels and the diagnosis of related diseases. The retinal vessels are mainly composed of coarse vessels and fine vessels, and the vessels have the problem of uneven distribution of coarse and fine vessels. At present, the common retinal blood vessel segmentation network based on deep learning can easily extract coarse vessels, but it ignores the more difficult to extract fine vessels.MethodsScale-aware dense residual model, multi-output weighted loss and attention mechanism are proposed and incorporated into the U-shape network. The model is proposed to extract image features through residual module, and using a multi-scale feature aggregation method to extract the deep information of the network after the last encoder layer, and upsampling output at each decoder layer, compare the output results of each decoder layer with the ground truth separately to obtain multiple output losses, and the last layer of the decoder layers is used as the final prediction output.ResultThe proposed network is tested on DRIVE and STARE. The evaluation indicators used in this paper are dice, accuracy, mIoU and recall rate. On the DRIVE dataset, the four indicators are respectively 80.40%, 96.67%, 82.14% and 88.10%; on the STARE dataset, the four indicators are respectively 83.41%, 97.39%, 84.38% and 88.84%.ConclusionThe experiment result proves that the network in this paper has better performance, can extract more continuous fine vessels, and reduces the problem of missing segmentation and false segmentation to a certain extent.

DenseUNet+: A novel hybrid segmentation approach based on multi-modality images for brain tumor segmentation

Segmentation of brain tumors is of great importance for patients in clinical diagnosis and treatment. For this reason, experts try to identify border regions of special importance using multimodal images from magnetic resonance imaging systems. In some images, border regions may be intertwined. As a result, this situation leads experts to make incomplete or wrong decisions. This paper presents DenseUNet+, a new deep learning-based approach to perform segmentation with high accuracy using multimodal images. In the DenseUNet+ model, data from four different modalities were used together in dense block structures. Afterward, linear operations were applied to these data and then the concatenate operation was performed. The results obtained in this way were transferred to the decoder layer. The proposed method was also compared with state-of-the-art (SOTA) studies using the same dataset by using dice and jaccard metrics in the BraTS2021 and FeTS2021 datasets. As a result of the comparison, dice and jaccard evaluation metrics for the BraTS2021 dataset were 95% and 88%, respectively, and 86% and 87% performance values were obtained for FeTS2021, respectively. It has been determined that the performance results are better than many SOTA brain tumor segmentation methods.

Combination of super-resolution reconstruction and SGA-Net for marsh vegetation mapping using multi-resolution multispectral and hyperspectral images

ABSTRACT Vegetation is crucial for wetland ecosystems. Human activities and climate changes are increasingly threatening wetland ecosystems. Combining satellite images and deep learning for classifying marsh vegetation communities has faced great challenges because of its coarse spatial resolution and limited spectral bands. This study aimed to propose a method to classify marsh vegetation using multi-resolution multispectral and hyperspectral images, combining super-resolution techniques and a novel self-constructing graph attention neural network (SGA-Net) algorithm. The SGA-Net algorithm includes a decoding layer (SCE-Net) to precisely fine marsh vegetation classification in Honghe National Nature Reserve, Northeast China. The results indicated that the hyperspectral reconstruction images based on the super-resolution convolutional neural network (SRCNN) obtained higher accuracy with a peak signal-to-noise ratio (PSNR) of 28.87 and structural similarity (SSIM) of 0.76 in spatial quality and root mean squared error (RMSE) of 0.11 and R2 of 0.63 in spectral quality. The improvement of classification accuracy (MIoU) by enhanced super-resolution generative adversarial network (ESRGAN) (6.19%) was greater than that of SRCNN (4.33%) and super-resolution generative adversarial network (SRGAN) (3.64%). In most classification schemes, the SGA-Net outperformed DeepLabV3 + and SegFormer algorithms for marsh vegetation and achieved the highest F1-score (78.47%). This study demonstrated that collaborative use of super-resolution reconstruction and deep learning is an effective approach for marsh vegetation mapping.

Multiscale feature fusion deep network for single image dehazing with continuous memory mechanism

Haze scenes seriously degrade the performance of computer vision systems. Thus, the single image dehazing based on convolutional neural network (CNN), as an important direction of computer vision, has always attracted extraordinary attentions of researchers and made great progress on dehazing effects. However, there are still some non-trivial issues such as uneven dehazing and color distortion, which are not well addressed. To alleviate these issues, this work proposes a new multiscale feature fusion image dehazing network incorporating a contiguous memory mechanism (MFFDN-CM). Specifically, the pixel attention mechanism, continuous memory strategy and residual dense blocks are jointly integrated into the dehazing model with a prevalent encoder-decoder structure(U-Net). Firstly, our model obtains multiscale feature maps by subsampling operations, and further employs skip connections between the corresponding network layers to connect the feature maps between the encoder and the decoder for good feature fusion. Then, we introduce a continuous memory residual block to strengthen the information flows for feature reuse. Moreover, to leverage detail representation and accomplish adaptive dehazing according to the haze density, MFFDN-CM adopts a pixel attention module on the skip connections to combine the residual dense block module of the corresponding decoding layers. Finally, multiple residual blocks are exploited on the bottleneck in encoder-decoder structure to prevent network performance degradation due to vanishing gradients. Experimental results demonstrate the proposed model can achieve better dehazing performance than the state-of-the-art methods based on other deep neural networks.

MPP-net: Multi-perspective perception network for dense video captioning

Applying deformable transformer for dense video captioning has achieved great success recently. However, deformable transformer only explores local-perspective perception by attending to a small set of key sampling points, which will make the decoder short-sighted and generate semantically incoherent and contradictory dense captions for a long video. In this paper, we propose a novel Multi-Perspective Perception Network to improve this problem. We first introduce a hierarchical temporal-spatial summary method to generate global-perspective summary context for each decoder layer and avoid redundant information. Then our new designed multi-perspective attention encourages the model to selectively incorporate the multi-perspective perception feature. Finally, we propose a novel multi-perspective generator to perform both multi-perspective feature fusion and caption generation. Experiments show that our proposed model outperforms previously published methods and achieves a competitive performance on ActivityNet Captions and YouCook2. The design of our model also shows the universality of other visual tasks that we obtain comparable results by applying our model for Object Detection and Paragraph Video Captioning.

Image Structure-Induced Semantic Pyramid Network for Inpainting

The existing deep-learning-based image inpainting algorithms often suffer from local structure disconnections and blurring when dealing with large irregular defective images. To solve these problems, an image structure-induced semantic pyramid network for inpainting is proposed. The model consists of two parts: the edge inpainting network and the content-filling network. U-Net-based edge inpainting network restores the edge of the image defect with residual blocks. The edge inpainting map is input into the pyramid content-filling network together with the image in the prior condition. In the content-filling network, the attention transfer module (ATM) is designed to reconfigure the encoding features of each scale step by step, and the recovered feature map is linked to the decoding layer and the corresponding potential feature fusion decoding to improve the global consistency of the image and finally obtain the restored image. The quantitative analysis shows that the average L1 loss is reduced by about 1.14%, the peak signal-to-noise ratio (PSNR) is improved by about 3.51, and the structural similarity (SSIM) is improved by about 0.163 on the CelebA-HQ and Places2 datasets compared with the current mainstream algorithms. The qualitative analysis shows that this model not only generates semantically sound content as a whole but also better matches the human visual perception in terms of local structural connectivity and texture synthesis.

Predicting combustion behavior in rotating detonation engines using an interpretable deep learning method

As rotating detonation engine (RDE) is maturing toward engineering implementation, it is a crucial step in developing real-time diagnostics capable of monitoring the combustion state therein to prevent combustion instability, such as detonation quenching, re-initiation, and mode switch. However, previous studies rarely consider monitoring combustion behavior in RDEs, let alone predicting the impending combustion instabilities based on the warning signals. Given active control requirements, a novel Transformer-based neural network, RDE-Transformer, is proposed for monitoring and predicting the combustion states in advance. RDE-Transformer is a multi-horizon forecasting model fed by univariate or multivariate time series data including pressure signals and aft-end photographs. Model hyper-parameters, namely, the number of encoder and decoder layers, the number of attention heads, implementation of positional encoding, and prediction length, are investigated for performance improvements. The results show that the optimal architecture can reliably predict pressures up to 5 detonation periods ahead of the current time, with a mean squared error of 0.0057 and 0.0231 for the training and validation set, respectively. Moreover, the feasibility of predicting combustion instability is validated, and the decision-making process through the attention mechanism is visualized by attention maps, making the model interpretable and superior to other “black-box” deep learning methods. In summary, the high performance and high interpretability of RDE-Transformer make it a promising diagnostics functional component for RDEs toward applied technology.

Heartbeat information prediction based on transformer model using millimetre‐wave radar

AbstractMillimetre‐wave radar offers high ranging accuracy and can capture subtle vibration information of the human heart. This study proposes a heartbeat prediction method based on the transformer model using millimetre‐wave radar. Firstly, the millimetre‐wave radar was used to collect the heartbeat data and conduct normalisation processing. Secondly, a position coding was introduced to assign sine or cosine variables to input data and extract their relative position relationship. Subsequently, the transformer encoder was adopted to allocate attention to input data through the multi‐head attention mechanism, using a mask layer before the decoding layer to prevent the leakage of future information. Finally, we employ the fully connected layer was employed in the linear decoder for regression and output the predicted results. Our experimental results demonstrate that the proposed transformer model achieves nearly 30% higher prediction accuracy than traditional long short‐term memory models while improving both the prediction accuracy and convergence rate. The proposed method has great potential in predicting the heartbeat state of elderly and sick patients.

DAN: A Segmentation-Free Document Attention Network for Handwritten Document Recognition.

Unconstrained handwritten text recognition is a challenging computer vision task. It is traditionally handled by a two-step approach, combining line segmentation followed by text line recognition. For the first time, we propose an end-to-end segmentation-free architecture for the task of handwritten document recognition: the Document Attention Network. In addition to text recognition, the model is trained to label text parts using begin and end tags in an XML-like fashion. This model is made up of an FCN encoder for feature extraction and a stack of transformer decoder layers for a recurrent token-by-token prediction process. It takes whole text documents as input and sequentially outputs characters, as well as logical layout tokens. Contrary to the existing segmentation-based approaches, the model is trained without using any segmentation label. We achieve competitive results on the READ 2016 dataset at page level, as well as double-page level with a CER of 3.43% and 3.70%, respectively. We also provide results for the RIMES 2009 dataset at page level, reaching 4.54% of CER. We provide all source code and pre-trained model weights at https://github.com/FactoDeepLearning/DAN.

TranSEFusionNet: Deep fusion network for colorectal polyp segmentation

Fully convolutional neural (FCN) networks like U-Net have been the state-of-the-art methods in colorectal polyp segmentation. However, U-Net still has some limitations in modelling remote semantic information, especially since the semantic information at different levels can vary greatly, making it difficult to utilize this information fully. To address these issues, we propose a new network architecture called TranSEFusionNet that utilizes the Transformer’s global modelling capability to better focus on global contextual semantic information. In addition, we added two feature fusion modules, a spatial feature fusion module (SFM) and an edge feature fusion module (EFM), to the network. SEF with a skip connection can improve the accuracy of passing deep features to shallow features. EFM in the output part of each decoder layer improves the recognition of edge ambiguous features by refining the semantic information of the network. We validate the model’s performance on five publicly available colorectal polyp datasets, and the experiments show that TranSEFusionNet has higher segmentation accuracy. To measure the generalization ability of TranSEFusionNet, we further applied the model to the cell nuclei dataset, which further verifies the performance of our model.Code: https://github.com/Linaaalin/TranSEFusionNet.

Full convolutional speech enhance network based on with combined amplitude and phase spectra

With the coming of the information age, speech has a pivotal role in modern society, and the study of speech enhancement has become necessary. To implement the often-neglected speech phase spectrum. This research proposes a full convolutional model based on an enhanced U-Net. The standard convolution kernel in the model is replaced with a deformable convolution with adaptive learning bias, a double attention mechanism is added between the encoder and decoder layers, and many residual connections are used to link the different layers, and the phase spectrum is also used as one of the inputs. The simulation experiments show that the added modules all contribute to the speech enhancement effect. Among them, the addition of the deformable convolution has the greatest contribution to the model.

Entity Relationship Extraction Based on a Multi-Neural Network Cooperation Model

Entity relation extraction mainly extracts relations from text, which is one of the important tasks of natural language processing. At present, some special fields have insufficient data; for example, agriculture, the metallurgical industry, etc. There is a lack of an effective model for entity relationship recognition under the condition of insufficient data. Inspired by this, we constructed a suitable small balanced data set and proposed a multi-neural network collaborative model (RBF, Roberta–Bidirectional Gated Recurrent Unit–Fully Connected). In addition, we also optimized the proposed model. This model uses the Roberta model as the coding layer, which is used to extract the word-level features of the text. This model uses BiGRU (Bidirectional Gated Recurrent Unit)–FC (Fully Connected) as the decoding layer, which is used to obtain the optimal relationship of the text. To further improve the effect, the input layer is optimized by feature fusion, and the learning rate is optimized by the cosine annealing algorithm. The experimental results show that, using the small balanced data set, the F1 value of the RBF model proposed in the paper is 25.9% higher than the traditional Word2vec–BiGRU–FC model. It is 18.6% higher than the recent Bert–BiLSTM (Bidirectional Long Short Term Memory)–FC model. The experimental results show that our model is effective.