Output Of Encoder Research Articles

Predicting Residential Energy Consumption by Explainable Deep Learning with Long-Term and Short-Term Latent Variables

Recently, deep learning models proliferate in the prediction of power demand for efficient planning of power consumption. However, the “black-box” characteristics of deep learning hinders from establishing a specific plan because it cannot explain the cause of the prediction. Recently, there are several attempts to explain the result of deep learning through the analysis of the input attributes that influence the prediction, but they lack of appropriate explanation because of ignoring the time-series property of the input data. In this paper, we propose a deep learning model to explain the impact of the input attributes on the prediction by taking account of the long-term and short-term properties of the time-series forecasting. The model consists of (i) two encoders to represent the power information for prediction and explanation, (ii) a decoder to predict the power demand from the concatenated outputs of encoders, and (iii) an explainer to identify the most significant attributes for predicting the energy consumption. Kullback–Leibler divergence in the loss function induces the long-term and short-term dependencies in latent space constructed by the second encoder. Several experiments on the benchmark dataset of household electric energy demand show that the proposed method explains the prediction appropriately with the most influential input attributes in the long-term and short-term dependencies. We can trade off the gain of the time-series explanation of the result against a slight degradation of the prediction performance.

Semi-Supervised Building Footprint Generation With Feature and Output Consistency Training

Accurate and reliable building footprint maps are vital to urban planning and monitoring, and most existing approaches fall back on convolutional neural networks (CNNs) for building footprint generation. However, one limitation of these methods is that they require strong supervisory information from massive annotated samples for network learning. State-of-the-art semi-supervised semantic segmentation networks with consistency training can help to deal with this issue by leveraging a large amount of unlabeled data, which encourages the consistency of model output on data perturbation. Considering that rich information is also encoded in feature maps, we propose to integrate the consistency of both features and outputs in the end-to-end network training of unlabeled samples, enabling to impose additional constraints. Prior semi-supervised semantic segmentation networks have established the cluster assumption, in which the decision boundary should lie in the vicinity of low sample density. In this work, we observe that for building footprint generation, the low-density regions are more apparent at the intermediate feature representations within the encoder than the encoder's input or output. Therefore, we propose an instruction to assign the perturbation to the intermediate feature representations within the encoder, which considers the spatial resolution of input remote sensing imagery and the mean size of individual buildings in the study area. The proposed method is evaluated on three datasets with different resolutions: Planet dataset (3 m/pixel), Massachusetts dataset (1 m/pixel), and Inria dataset (0.3 m/pixel). Experimental results show that the proposed approach can well extract more complete building structures and alleviate omission errors.

Sequence-Level Speaker Change Detection With Difference-Based Continuous Integrate-and-Fire

Speaker change detection is an important task in multi-party interactions such as meetings and conversations. In this paper, we address the speaker change detection task from the perspective of sequence transduction. Specifically, we propose a novel encoder-decoder framework that directly converts the input feature sequence to the speaker identity sequence. The difference-based continuous integrate-and-fire mechanism is designed to support this framework. It detects speaker changes by integrating the speaker difference between the encoder outputs frame-by-frame and transfers encoder outputs to segment-level speaker embeddings according to the detected speaker changes. The whole framework is supervised by the speaker identity sequence, a weaker label than the precise speaker change points. The experiments on the AMI and DIHARD-I corpora show that our sequence-level method consistently outperforms a strong frame-level baseline that uses the precise speaker change labels.

Spectral–Spatial–Temporal Transformers for Hyperspectral Image Change Detection

Convolutional neural networks (CNNs) with excellent spatial feature extraction abilities have become popular in remote sensing (RS) image change detection (CD). However, CNNs often focus on the extraction of spatial information but ignore important spectral and temporal sequences for hyperspectral images (HSIs). In this paper, we propose a joint spectral, spatial, and temporal transformer for hyperspectral image change detection (HSI-CD), named SST-Former. First, the SST-Former position-encodes each pixel on the cube to remember the spectral and spatial sequences. Second, a spectral transformer encoder structure is used to extract spectral sequence information. Then, a class token for storing the class information of a single temporal HSI concatenates the output of the spectral transformer encoder. The spatial transformer encoder is used to extract spatial texture information in the next step. Finally, the features of different temporal HSIs are sent as the input of temporal transformer, which is used to extract useful CD features between the current HSI pairs and obtain the binary CD result through multilayer perception (MLP). We evaluate SST-Former on three HSI-CD datasets by numerous experiments, showing that it performs better than other excellent methods both visually and qualitatively.

Alleviating ASR Long-Tailed Problem by Decoupling the Learning of Representation and Classification

Recently, we have witnessed excellent improvement of end-to-end (E2E) automatic speech recognition (ASR). However, how to tackle the long-tailed data distribution problem while maintaining E2E ASR models' performance for high-frequency tokens is still challenging. To solve this challenge, we propose a novel decoupled ASR learning method for the sequence-to-sequence ASR architecture in this paper. Our method decouples the learning procedure of this model into two stages: representation learning and classification learning. In the representation learning stage, we use the encoder output of a pretrained language model as one of the ASR model’s learning targets, and propose threshold log cosine embedding loss (TLCE-loss) as the objective function. A frequency-mask cross-entropy loss (FMCE-loss) is also designed as an auxiliary loss. In the classification learning stage, we find that introducing a temperature into softmax function helps reduce the influence of negative samples on tail classes, thus mitigating the biased learning process for the classifier. Furthermore, we propose a weighted softmax (w-softmax) to adjust ASR posterior probabilities according to the token appearing frequency during inference. Additionally, we introduce tail word/character error rate (TWER / TCER) and head word/character error rate (HWER / HCER) that respectively evaluate the ASR accuracy for tail and head words/characters. Experimental results on the Switchboard and HKUST corpora show that our proposed method greatly outperforms the baseline, especially in TWER / TCER reduction. To the best of our knowledge, this is the first work to use a decoupled ASR learning method to alleviate the long-tailed problem in sequence-to-sequence ASR.

Double-scale supervised inversion with a data-driven forward model for low-frequency impedance recovery

Low-frequency information is important in reducing the nonuniqueness of absolute impedance inversion and for quantitative seismic interpretation. In traditional model-driven impedance inversion methods, the low-frequency impedance background is from an initial model and is almost unchanged during the inversion process. Moreover, the inversion results are limited by the quality of the modeled seismic data and the extracted wavelet. To alleviate these issues, we have investigated a double-scale supervised impedance inversion method based on the gated recurrent encoder-decoder network (GREDN). We first train the decoder network of GREDN called the forward operator, which can map impedance to seismic data. We then implement the well-trained decoder as a constraint to train the encoder network of GREDN called the inverse operator. Besides matching the output of the encoder with broadband pseudowell impedance labels, data generated by inputting the encoder output into the known decoder match the observed narrowband seismic data. The broadband impedance information and the already-trained decoder largely limit the solution space of the encoder. Finally, after training, only the derived optimal encoder is applied to unseen seismic traces to yield broadband impedance volumes. Our approach is fully data driven and does not involve the initial model, seismic wavelet, and model-driven operator. Tests on the Marmousi model illustrate that our double-scale supervised impedance inversion method can effectively recover low-frequency components of the impedance model, and we determine that low frequencies of the predicted impedance originate from well logs. Furthermore, we apply the strategy of combining the double-scale supervised impedance inversion method with a model-driven impedance inversion method to process field seismic data. Tests on a field data set indicate that the predicted impedance results not only reveal a classic tectonic sedimentation history but also match the corresponding results measured at the locations of two wells.

A de-ambiguous condition monitoring scheme for wind turbines using least squares generative adversarial networks

Wind turbine condition monitoring (WTCM) plays an important role in reducing operation & maintenance (O&M) cost and improving the reliability of wind farms. Supervisory control and data acquisition (SCADA) data have advantages such as easy access and strong timeliness and are used widely for WTCM. However, it is difficult to distinguish and label historical SCADA data as healthy or faulty accurately during the model training process. Therefore, a De-ambiguous Condition Monitoring scheme with Transfer layer (DCMT) based on SCADA data is proposed to overcome this problem. This scheme provides a fault early warning for wind turbines. In this scheme, an improved auto-encoder (AE) network with a transfer layer is designed to eliminate the effect of SCADA data in the ambiguous status (ambiguous data) and enhance the reliability of a training dataset. Meanwhile, a structure of Siamese encoder is designed to calculate the residuals between latent features, i.e., the outputs of the Siamese encoders. These residuals can be utilised to identify wind turbine operational conditions. Further, least squares generative adversarial networks (LSGAN) is introduced to learn the distribution of health data while restricting the discriminator and realising the augmentation of health data for model training. The proposed method is applied to two cases of generator winding and gearbox bearing over-temperature faults of wind turbines from northwest China. Compared with other methods, the proposed method effectively detects potential abnormal conditions in advance without raising false alarms.

Enhance the Language Ability of Humanoid Robot NAO through Deep Learning to Interact with Autistic Children

Autism spectrum disorder (ASD) is a life-long neurological disability, and a cure has not yet been found. ASD begins early in childhood and lasts throughout a person’s life. Through early intervention, many actions can be taken to improve the quality of life of children. Robots are one of the best choices for accompanying children with autism. However, for most robots, the dialogue system uses traditional techniques to produce responses. Robots cannot produce meaningful answers when the conversations have not been recorded in a database. The main contribution of our work is the incorporation of a conversation model into an actual robot system for supporting children with autism. We present the use a neural network model as the generative conversational agent, which aimed at generating meaningful and coherent dialogue responses given the dialogue history. The proposed model shares an embedding layer between the encoding and decoding processes through adoption. The model is different from the canonical Seq2Seq model in which the encoder output is used only to set-up the initial state of the decoder to avoid favoring short and unconditional responses with high prior probability. In order to improve the sensitivity to context, we changed the input method of the model to better adapt to the utterances of children with autism. We adopted transfer learning to make the proposed model learn the characteristics of dialogue with autistic children and to solve the problem of the insufficient corpus of dialogue. Experiments showed that the proposed method was superior to the canonical Seq2sSeq model and the GAN-based dialogue model in both automatic evaluation indicators and human evaluation, including pushing the BLEU precision to 0.23, the greedy matching score to 0.69, the embedding average score to 0.82, the vector extrema score to 0.55, the skip-thought score to 0.65, the KL divergence score to 5.73, and the EMD score to 12.21.

Context-Aware Bidirectional Neural Model for Sindhi Named Entity Recognition

Named entity recognition (NER) is a fundamental task in many natural language processing (NLP) applications, such as text summarization and semantic information retrieval. Recently, deep neural networks (NNs) with the attention mechanism yield excellent performance in NER by taking advantage of character-level and word-level representation learning. In this paper, we propose a deep context-aware bidirectional long short-term memory (CaBiLSTM) model for the Sindhi NER task. The model relies upon contextual representation learning (CRL), bidirectional encoder, self-attention, and sequential conditional random field (CRF). The CaBiLSTM model incorporates task-oriented CRL based on joint character-level and word-level representations. It takes character-level input to learn the character representations. Afterwards, the character representations are transformed into word features, and the bidirectional encoder learns the word representations. The output of the final encoder is fed into the self-attention through a hidden layer before decoding. Finally, we employ the CRF for the prediction of label sequences. The baselines and the proposed CaBiLSTM model are compared by exploiting pretrained Sindhi GloVe (SdGloVe), Sindhi fastText (SdfastText), task-oriented, and CRL-based word representations on the recently proposed SiNER dataset. Our proposed CaBiLSTM model achieved a high F1-score of 91.25% on the SiNER dataset with CRL without relying on additional handmade features, such as hand-crafted rules, gazetteers, or dictionaries.

A dual-head attention model for time series data imputation

Digital agriculture increasingly relies on the availability and accuracy of measurement data collected from various sensors. Of this data, water quality attracts great attention due to its intended use for crop irrigation, livestock, and other farming activities. Accurate and reliable water quality measurements enable farmers to understand the landscape comprehensively, optimising resource utilisation and reducing the negative impacts of agriculture on the environment. In practice, missing and incomplete data can create biased estimations and reduce the efficiency of many of the valuable applications provided by digital agriculture. The purpose of this paper is to propose a dual-head sequence-to-sequence imputation model (Dual-SSIM) designed to impute missing time series data in sensor networks, therefore reducing the negative consequences of missing and incomplete data. Unlike standard sequence-to-sequence architecture, the Dual-SSIM model features two encoders with gated recurrent units (GRUs) which are used to process temporal information before and after the missing gap separately. Furthermore, the attention mechanism is applied to two encoder outputs concurrently, in order to allow the model to focus on the high relative inputs when estimating missing data. The performance efficacy of Dual-SSIM has been investigated through the monitoring of water quality, sourced from an Australian water quality information system. Experimental results of this investigation indicate that Dual-SSIM outperforms associated alternatives based on the mean absolute error (MAE), root mean square error (RMSE), and dynamic time warping (DTW) scores in imputing two different water quality variables. Therefore, it can be concluded that Dual-SSIM provides an effective and promising approach for water quality data imputation.

Deep Convolutional Feature-Based Probabilistic SVDD Method for Monitoring Incipient Faults of Batch Process

Support vector data description (SVDD) has been widely applied to batch process fault detection. However, it often performs poorly, especially when incipient faults occur, because it only considers the shallow data feature and omits the probabilistic information of features. In order to provide better monitoring performance on incipient faults in batch processes, an improved SVDD method, called deep probabilistic SVDD (DPSVDD), is proposed in this work by integrating the convolutional autoencoder and the probability-related monitoring indices. For mining the hidden data features effectively, a deep convolutional features extraction network is designed by a convolutional autoencoder, where the encoder outputs and the reconstruction errors are used as the monitor features. Furthermore, the probability distribution changes of these features are evaluated by the Kullback-Leibler (KL) divergence so that the probability-related monitoring indices are developed for indicating the process status. The applications to the benchmark penicillin fermentation process demonstrate that the proposed method has a better monitoring performance on the incipient faults in comparison to the traditional SVDD methods.

Connectionist temporal classification loss for vector quantized variational autoencoder in zero-shot voice conversion

Vector quantized variational autoencoder (VQ-VAE) has recently become an increasingly popular method in non-parallel zero-shot voice conversion (VC). The reason behind is that VQ-VAE is capable of disentangling the content and the speaker representations from the speech by using a content encoder and a speaker encoder, which is suitable for the VC task that makes the speech of a source speaker sound like the speech of the target speaker without changing the linguistic content. However, the converted speech is not satisfying because it is difficult to disentangle the pure content representations from the acoustic features due to the lack of linguistic supervision for the content encoder. To address this issue, under the framework of VQ-VAE, connectionist temporal classification (CTC) loss is proposed to guide the content encoder to learn pure content representations by using an auxiliary network. Based on the fact that the CTC loss is not affected by the sequence length of the output of the content encoder, adding the linguistic supervision to the content encoder can be much easier. This non-parallel many-to-many voice conversion model is named as CTC-VQ-VAE. VC experiments on the CMU ARCTIC and VCTK corpus are carried out to evaluate the proposed method. Both the objective and the subjective results show that the proposed approach significantly improves the speech quality and speaker similarity of the converted speech, compared with the traditional VQ-VAE method.

Theoretical and experimental study of wind turbine drivetrain fault diagnosis by using torsional vibrations and modal estimation

This paper provides an analytical proof and the theoretical development of the idea of using the torsional vibration measurements for a system-level condition monitoring of the drivetrain system. The method relies on modal parameter estimation of the drivetrain system by using the torsional measurements and subsequent monitoring of the variations in the system eigenfrequencies and normal modes. Angular velocity error function extracted from encoder outputs at both input and output of drivetrain is used to estimate modal parameters including natural frequencies and damping coefficients. In the proposed condition monitoring approach, it is shown that any abnormal deviation from the reference values of the drivetrain system dynamic properties can be translated into the progression of a specific fault in the system. In order to extract the condition monitoring features, local sensitivity analysis is engaged to establish a relationship between different categories of drivetrain faults with the system dynamic properties and the amplitude of torsional response, which helps with both to identify the state of the progressive faults and to localize them. Local sensitive analysis shows that abnormal deviations in stiffness and moment of inertia due to the presence of faults result in considerable changes in natural frequencies and modal responses which can be measured and used as fault detecting features by using the proposed analytical approach. Sensitivity analysis is also employed along with the estimated modal frequency for estimation of modal damping from the amplitude of response at the natural frequencies and their subsequent use for estimation of undamped natural frequencies which are later used in the proposed condition monitoring approach. The proposed approach is computationally inexpensive and can be implemented without additional instrumentation. Two test cases, using 10 MW simulated and 1.75 MW operational drivetrains have been demonstrated.

TaLNet: Voice Reconstruction from Tongue and Lip Articulation with Transfer Learning from Text-to-Speech Synthesis

This paper presents TaLNet, a model for voice reconstruction with ultrasound tongue and optical lip videos as inputs. TaLNet is based on an encoder-decoder architecture. Separate encoders are dedicated to processing the tongue and lip data streams respectively. The decoder predicts acoustic features conditioned on encoder outputs and speaker codes.To mitigate for having only relatively small amounts of dual articulatory-acoustic data available for training, and since our task here shares with text-to-speech (TTS) the common goal of speech generation, we propose a novel transfer learning strategy to exploit the much larger amounts of acoustic-only data available to train TTS models. For this, a Tacotron 2 TTS model is first trained, and then the parameters of its decoder are transferred to the TaLNet decoder. We have evaluated our approach on an unconstrained multi-speaker voice recovery task. Our results show the effectiveness of both the proposed model and the transfer learning strategy. Speech reconstructed using our proposed method significantly outperformed all baselines (DNN, BLSTM and without transfer learning) in terms of both naturalness and intelligibility. When using an ASR model decoding the recovery speech, the WER of our proposed method is relatively reduced over 30% compared to baselines.

Rethinking Boundaries: End-To-End Recognition of Discontinuous Mentions with Pointer Networks

A majority of research interests in irregular (e.g., nested or discontinuous) named entity recognition (NER) have been paid on nested entities, while discontinuous entities received limited attention. Existing work for discontinuous NER, however, either suffers from decoding ambiguity or predicting using token-level local features. In this work, we present an innovative model for discontinuous NER based on pointer networks, where the pointer simultaneously decides whether a token at each decoding frame constitutes an entity mention and where the next constituent token is. Our model has three major merits compared with previous work: (1) The pointer mechanism is memory-augmented, which enhances the mention boundary detection and interactions between the current decision and prior recognized mentions. (2) The encoder-decoder architecture can linearize the complexity of structure prediction, and thus reduce search costs. (3) The model makes every decision using global information, i.e., by consulting all the input, encoder and previous decoder output in a global view. Experimental results on the CADEC and ShARe13 datasets show that our model outperforms flat and hypergraph models as well as a state-of-the-art transition-based model for discontinuous NER. Further in-depth analysis demonstrates that our model performs well in recognizing various entities including flat, overlapping and discontinuous ones. More crucially, our model is effective on boundary detection, which is the kernel source to NER.

ADID-UNET-a segmentation model for COVID-19 infection from lung CT scans.

Currently, the new coronavirus disease (COVID-19) is one of the biggest health crises threatening the world. Automatic detection from computed tomography (CT) scans is a classic method to detect lung infection, but it faces problems such as high variations in intensity, indistinct edges near lung infected region and noise due to data acquisition process. Therefore, this article proposes a new COVID-19 pulmonary infection segmentation depth network referred as the Attention Gate-Dense Network- Improved Dilation Convolution-UNET (ADID-UNET). The dense network replaces convolution and maximum pooling function to enhance feature propagation and solves gradient disappearance problem. An improved dilation convolution is used to increase the receptive field of the encoder output to further obtain more edge features from the small infected regions. The integration of attention gate into the model suppresses the background and improves prediction accuracy. The experimental results show that the ADID-UNET model can accurately segment COVID-19 lung infected areas, with performance measures greater than 80% for metrics like Accuracy, Specificity and Dice Coefficient (DC). Further when compared to other state-of-the-art architectures, the proposed model showed excellent segmentation effects with a high DC and F1 score of 0.8031 and 0.82 respectively.

Spatially coupled serially concatenated codes via parity re‐encoding

The recently introduced spatially coupled turbo-like codes are attractive in trading-off performances between the error-floor region and the water-fall region. This letter is concerned with the construction of spatially coupled serially concatenated codes. Particularly, a new construction of spatially coupled serially concatenated codes is presented, in which a portion of the encoding output of the previous inner encoder is taken as the input of the current inner encoder. The authors show that by proper selection of the component codes and the re-encoding ratio, performance gain about 0.25 dB is obtained with the proposed construction.

EFFICIENT IMPLEMENTATION OF FPGA-BASED FORWARD ERROR CORRECTING COMBINATION AND BIT TO CELL WORD DE-MULTIPLEXER FOR A SECOND GENERATION DIGITAL TERRESTRIAL TELEVISION BROADCASTING SYSTEM

This article focus on the implementation of FEC part. The motivation of the design is that FEC is an effective tool to mitigate problems associated with OFDM which stem from multipath fading channel, high speed data rate. One of the key features of BCH codes is that during code design, there is a precise control over the number of symbol errors that are correctable by the code. BCH coder proccesses parralelly with high-speed operation. 8-bit parallel data input and output helps to maximize the throughput. DVB-T2 used LDPC coder, as inner codes with word length up to 64,800 bits, enabling significant proximity to Shannon limit. This encoder supports all code rates and both normal and short frames. Output of LDPC encoder is interleaved with bit interleaver. This project was fully optimized for speed and memory area, fully synchronized by using a single clock. The design was coded in VHDL, synthesized by using Xilinx ISE Design Suite 14.7. The design has been tested on development Kit NetFPGA-1G-CML of Digilent Corporation and the bit map was downloaded into Xilinx Kintex-7 XC7K325T-1FFG676, which is integrated on experimental transmitter system DVB-T2. This research product belong to program “Research of experimental testing of second generation digital terrestrial television broadcasting system DVB-T2” of Vietnamese Communications Television Development JSC.

A Calibration Scheme With Combination of the Optical Shaft Encoder and Laser Triangulation Sensor for Low-Frequency Angular Acceleration Rotary Table

A low-frequency angular acceleration rotary table (LFAART) is capable of sinusoidal motion around a fixed position and essential to verify dynamic characteristics of a gyroscope device. The table’s performance at different swinging frequencies and amplitudes needs to be calibrated to realize traceability. However, there has been little research on the LFAART calibration. Besides, the existing measurement methods of sinusoidal angular motion parameters are not adequate for the needs of the LFAART calibration in terms of measurement range, precision, and feasibility. This article presents a calibration scheme combining an optical shaft encoder and a laser triangulation sensor. When the rotation amplitude covers sufficient slots of the scale grating, the optical shaft encoder is applied, while the laser triangulation sensor is used for a tiny amplitude motion. An algorithm is proposed to accurately calculate the angular acceleration amplitude and frequency of the rotation utilizing sinusoidal phase-modulated signals of the optical shaft encoder output. Singular zero-crossings of the series are discarded, and intervals of reversing moments are recognized according to the signal derivative’s sign at remaining zero-crossings. Then intervals are divided into subintervals to obtain accurate reversing moments. Finally, angular displacements between two adjacent reversing moments are estimated based on the fringe counting method, and the amplitude and frequency are calculated. The measurement uncertainty analysis and experiment demonstrate that the calibration scheme, including the algorithm, possesses high accuracy and resolution, wide range, and good operability, which perfectly solves the rotation’s angular acceleration amplitude and frequency.

TAP-Net

We introduce the transport-and-pack (TAP) problem, a frequently encountered instance of real-world packing, and develop a neural optimization solution based on reinforcement learning. Given an initial spatial configuration of boxes, we seek an efficient method to iteratively transport and pack the boxes compactly into a target container. Due to obstruction and accessibility constraints, our problem has to add a new search dimension, i.e., finding an optimal transport sequence , to the already immense search space for packing alone. Using a learning-based approach, a trained network can learn and encode solution patterns to guide the solution of new problem instances instead of executing an expensive online search. In our work, we represent the transport constraints using a precedence graph and train a neural network, coined TAP-Net, using reinforcement learning to reward efficient and stable packing. The network is built on an encoder-decoder architecture, where the encoder employs convolution layers to encode the box geometry and precedence graph and the decoder is a recurrent neural network (RNN) which inputs the current encoder output, as well as the current box packing state of the target container, and outputs the next box to pack, as well as its orientation. We train our network on randomly generated initial box configurations, without supervision , via policy gradients to learn optimal TAP policies to maximize packing efficiency and stability. We demonstrate the performance of TAP-Net on a variety of examples, evaluating the network through ablation studies and comparisons to baselines and alternative network designs. We also show that our network generalizes well to larger problem instances, when trained on small-sized inputs.