Non-autoregressive Model Research Articles

Balancing quality and efficiency: An improved non-autoregressive model for pseudocode-to-code conversion

Pseudocode can efficiently represent algorithm logic, but manual conversion to executable code requires more time. Recent works have applied autoregressive (AR) models to automate pseudocode-to-code conversion, achieving good results but slow generation speed. Non-autoregressive (NAR) models offer the advantage of parallel generation. However, they face challenges in effectively capturing contextual information, leading to a potential degradation in the quality of the generated output. This paper presents an improved NAR model for balancing quality and efficiency in pseudocode conversion. Firstly, two strategies are proposed to address out-of-vocabulary and repetition problems. Secondly, an improved NAR model is built using linear smoothing and adaptive techniques in the transition matrix, which can mitigate the “winner takes all” effect. Finally, a new synthesis potential metric is proposed for evaluating pseudocode conversion. Experimental results show that the proposed method matches AR model performance while accelerating generation over 10-fold. Further, the proposed NAR model reduces the gap with the AR model in terms of the BLEU score on the EN-DE and DE-EN tasks of the WMT14 machine translation.

Bridging Implicit and Explicit Geometric Transformation for Single-Image View Synthesis.

Creating novel views from a single image has achieved tremendous strides with advanced autoregressive models, as unseen regions have to be inferred from the visible scene contents. Although recent methods generate high-quality novel views, synthesizing with only one explicit or implicit 3D geometry has a trade-off between two objectives that we call the "seesaw" problem: 1) preserving reprojected contents and 2) completing realistic out-of-view regions. Also, autoregressive models require a considerable computational cost. In this paper, we propose a single-image view synthesis framework for mitigating the seesaw problem while utilizing an efficient non-autoregressive model. Motivated by the characteristics that explicit methods well preserve reprojected pixels and implicit methods complete realistic out-of-view regions, we introduce a loss function to complement two renderers. Our loss function promotes that explicit features improve the reprojected area of implicit features and implicit features improve the out-of-view area of explicit features. With the proposed architecture and loss function, we can alleviate the seesaw problem, outperforming autoregressive-based state-of-the-art methods and generating an image ≈ 100 times faster. We validate the efficiency and effectiveness of our method with experiments on RealEstate10 K and ACID datasets.

Tibetan Speech Synthesis Based on Pre-Traind Mixture Alignment FastSpeech2

Most current research in Tibetan speech synthesis relies primarily on autoregressive models in deep learning. However, these models face challenges such as slow inference, skipped readings, and repetitions. To overcome these issues, we propose an enhanced non-autoregressive acoustic model combined with a vocoder for Tibetan speech synthesis. Specifically, we introduce the mixture alignment FastSpeech2 method to correct errors caused by hard alignment in the original FastSpeech2 method. This new method employs soft alignment at the level of Latin letters and hard alignment at the level of Tibetan characters, thereby improving alignment accuracy between text and speech and enhancing the naturalness and intelligibility of the synthesized speech. Additionally, we integrate pitch and energy information into the model, further enhancing overall synthesis quality. Furthermore, Tibetan has relatively smaller text-to-audio datasets compared to widely studied languages. To address these limited resources, we employ a transfer learning approach to pre-train the model with data from resource-rich languages. Subsequently, this pre-trained mixture alignment FastSpeech2 model is fine-tuned for Tibetan speech synthesis. Experimental results demonstrate that the mixture alignment FastSpeech2 model produces higher-quality speech compared to the original FastSpeech2 model, particularly when pre-trained on an English dataset, resulting in further improvements in clarity and naturalness.

Non-Autoregressive Line-Level Code Completion

Software developers frequently use code completion tools to accelerate software development by suggesting the following code elements. Researchers usually employ AutoRegressive (AR) decoders to complete code sequences in a left-to-right, token-by-token fashion. To improve the accuracy and efficiency of code completion, we argue that tokens within a code statement have the potential to be predicted concurrently. In this article, we first conduct an empirical study to analyze the dependency among the target tokens in line-level code completion. The results suggest that it is potentially practical to generate all statement tokens in parallel. To this end, we introduce SANAR, a simple and effective syntax-aware non-autoregressive model for line-level code completion. To further improve the quality of the generated code, we propose an adaptive and syntax-aware sampling strategy to boost the model’s performance. The experimental results obtained from two widely used datasets indicate that our model outperforms state-of-the-art code completion approaches of similar model size by a considerable margin, and is faster than these models with up to 9× speed-up. Moreover, the extensive results additionally demonstrate that the enhancements achieved by SANAR become even more pronounced with larger model sizes, highlighting their significance.

Dynamic behavior modelling of the silicone-ethanol actuator based on principal dynamic modes of the Laguerre model

The silicone-ethanol actuator is an emerging artificial muscle gaining attention because of its simple construction/stimulation and high-force generation. Most related researches are focused on structural modification methods, while behavior modeling has received less attention. Although the previous models are highly accurate, they suffer from the lack of ethanol content as an input, highly feedback dependency, and fixed output range. In this research, to address the mentioned concerns, the actuator was made according to the conventional procedure and subjected to stimulation, and displacement/blocked-force recording by special custom hardware. The displacement/blocked-force was not limited to a fix amount in order to evaluate ethanol leakage. To predict displacement/blocked-force, a non-autoregressive model was developed with the applied energy, temperature, and ethanol content as inputs. The Laguerre functions was applied to ensure dynamic matching between input-output. Finally, Principal Dynamic Mode analysis was used to optimize the number of filter banks in the model. The results showed that the proposed model provide a reasonable balance between accuracy, interpretability, and simplicity. Also, we found that the prediction of displacement/blocked-force is mainly influenced by the core-temperature/applied-energy, respectively. The proposed model focuses on the effect of stimulation signal and disregards the structural parameters due to their variety in different applications. By considering the both structural parameters and excitation signal, a more comprehensive model can be developed to predict actuator dynamic behavior.

Reinforcement Learning-Based Nonautoregressive Solver for Traveling Salesman Problems.

The traveling salesman problem (TSP) is a well-known combinatorial optimization problem (COP) with broad real-world applications. Recently, neural networks (NNs) have gained popularity in this research area because as shown in the literature, they provide strong heuristic solutions to TSPs. Compared to autoregressive neural approaches, nonautoregressive (NAR) networks exploit the inference parallelism to elevate inference speed but suffer from comparatively low solution quality. In this article, we propose a novel NAR model named, which incorporates a specially designed architecture and an enhanced reinforcement learning (RL) strategy. To the best of our knowledge, is the first TSP solver that successfully combines RL and NAR networks. The key lies in the incorporation of NAR network output decoding into the training process. efficiently represents TSP-encoded information as rewards and seamlessly integrates it into RL strategies, while maintaining consistent TSP sequence constraints during both training and testing phases. Experimental results on both synthetic and real-world TSPs demonstrate that outperforms five state-of-the-art (SOTA) models in terms of solution quality, inference speed, and generalization to unseen scenarios.

MDM: Meta diffusion model for hard-constrained text generation

Hard-constrained text generation (Hard-CTG) task aims to generate text with given keywords, which is helpful for summarization, data augmentation, story writing, etc. Existing Hard-CTG models face two significant challenges. Firstly, hard-CTG models based on the editing method suffer from error propagation, resulting in low generation quality. Secondly, Hard-CTG models utilizing the prompt method cannot guarantee high keyword coverage. To tackle these challenges, we propose Meta Diffusion Model (MDM), a non-autoregressive diffusion model with novel meta-learning module. Specifically, we fix the embedding of keywords in the generation process, while all non-keyword tokens evolve simultaneously and contribute to each other towards the target sentence under given keywords, addressing the above issues. Moreover, existing diffusion models for the text domain have an inconsistency in the training and inference stages. To unify the training and inference processes, we present an adaptively denoising method derived from meta-learning, and further improves generation quality. Experiments on three representative datasets demonstrate that our method achieves state-of-the-art performance evaluated on empirical metrics. Especially, compared with strong baselines, MDM significantly improves the BLEU-4, CIDEr, and ROUGE by 0.48%—11.56%, 17.33%—82.87%, and 23.15%–29.78%, respectively. In terms of keyword coverage, our MDM outperforms ChatGPT by 2.93%–7.88%.

DiffuCom: A novel diffusion model for comment generation

Given that the diffusion model has good advantages in diversity, an increasing number of researchers have recently begun exploring natural language generation. However, despite significant progress in modeling based on text diffusion, there are still several challenges to be overcome in comment tasks. On the one hand, the source sentences in the comment task are long. If each timestep in the diffusion carries and computes the source sentence, it will significantly increase the computational overhead. On the other hand, the diffusion model is essentially an iterative non-autoregressive model, which lacks target decomposition in the decoding. This makes vanilla cross-attention prone to overlook local perceptual content, leading to the problem of poor correlation. To solve the aforementioned problem, we propose a novel diffusion model for comment generation (named DiffuCom). Specifically, DiffuCom employs the transformer-based encoder–decoder structure, encoding only once during the reverse diffusion process, effectively reducing computational overhead. Additionally, we design a context-aware cross-attention module (CACAM) that integrates global and local cross-attention information through a gating mechanism, which more accurately captures relevant fragment information in the source sentence and improves the correlation. Second, self-conditioning technology was introduced in the training process, which can effectively improve the quality of the samples. Furthermore, to enhance the controllability of DiffuCom, we also propose a blended control. It combines class-conditional and classifier guidance by utilizing the difference between training and inference. Extensive experiments on four comment datasets demonstrate that the proposed DiffuCom model can generate comments that are both diverse and relevant.

A Nonautoregressive Dynamic Model Based Welding Parameter Planning Method for Varying Geometry Beads in WAAM

Wire and arc additive manufacturing (WAAM) is a promising method for directly manufacturing parts with complex shapes. However, the accuracy of the existing welding parameter planning methods would dramatically decrease when the bead geometry changes dynamically due to the long-term dependence, strong coupling, and hysteresis properties of the WAAM process. To this end, a nonautoregressive dynamic model is proposed to predict the bead geometry, and an adaptive model predictive control (aMPC) method is proposed to plan welding parameters to achieve high manufacturing accuracy. First, in the proposed dynamic model, the long-term dynamic characteristics of the WAAM process are modeled by a resample long short-term memory network by considering the fluidity of the welding pool, which is the crucial factor of dynamic characteristics of the welding process. Second, in the proposed aMPC method, the strong coupling is addressed by high multiobjective performance, and the hysteresis is considered by a long control horizon. Thus, the aMPC method can reduce the control latency and improve the manufacturing accuracy for varying geometry beads. The proposed methods are validated by experiments, which indicate that the proposed methods are effective.

Capture Salient Historical Information: A Fast and Accurate Non-autoregressive Model for Multi-turn Spoken Language Understanding

Spoken Language Understanding (SLU), a core component of the task-oriented dialogue system, expects a shorter inference facing the impatience of human users. Existing work increases inference speed by designing non-autoregressive models for single-turn SLU tasks but fails to apply to multi-turn SLU in confronting the dialogue history. The intuitive idea is to concatenate all historical utterances and utilize the non-autoregressive models directly. However, this approach seriously misses the salient historical information and suffers from the uncoordinated-slot problems. To overcome those shortcomings, we propose a novel model for multi-turn SLU named Salient History Attention with Layer-Refined Transformer (SHA-LRT), which comprises a SHA module, a Layer-Refined Mechanism (LRM), and a Slot Label Generation (SLG) task. SHA captures salient historical information for the current dialogue from both historical utterances and results via a well-designed history-attention mechanism. LRM predicts preliminary SLU results from Transformer’s middle states and utilizes them to guide the final prediction, and SLG obtains the sequential dependency information for the non-autoregressive encoder. Experiments on public datasets indicate that our model significantly improves multi-turn SLU performance (17.5% on Overall) with accelerating (nearly 15 times) the inference process over the state-of-the-art baseline as well as effective on the single-turn SLU tasks.

Electrolaryngeal speech enhancement based on a two stage framework with bottleneck feature refinement and voice conversion

An electrolarynx (EL) is a medical device that generates speech for people who lost their biological larynx. However, EL speech signals are unnatural and unintelligible due to the monotonous pitch and the mechanical excitation of the EL device. This paper proposes an end-to-end voice conversion method to enhance EL speech. We adopt a speaker-independent automatic speech recognition model to extract bottleneck features as the intermediate phonetic features for enhancement. Our system includes two stages: the bottleneck feature vectors of the EL speech are mapped by a parallel non-autoregressive model to the corresponding feature vectors of the normal speech in stage one. Then another voice conversion model maps normal speech’s bottleneck feature vectors directly to normal speech’s Mel-spectrogram in stage two, followed by a MelGAN-based vocoder to convert the Mel-spectrogram into waveform. In addition, we incorporate data augmentation and transfer learning to improve conversion performance. Experimental results show that the proposed method outperforms our baseline methods and performs well in terms of naturalness and intelligibility. The audio samples are available online.22https://haydencaffrey.github.io/el/index.html.

FastSpeechStyle: Fast, Emotion Controllable, and High-Quality Speech Synthesis

Non-autoregressive text-to-speech models such as Fastspeech2 can fast synthesize high-quality speech. This model also allows explicit control of the speech signal’s pitch, energy, and speed. However, controlling emotion while maintaining natural human-like speech is still a problem. In this work, we propose an expressive speech synthesis model that can synthesize high-quality speech with desired emotion. The proposed model includes two main components (1) Mel Emotion Encoder extracts emotion embedding from the Mel-spectrogram of audio, (2) the FastSpeechStyle, a non-autoregressive model, which is modified from vanilla Fastspeech2. The FastSpeechStyle used an Improved Conformer block, which replaces normal LayerNorm with Style Adaptive LayerNorm to “shift” and “scale” hidden features according to emotion embedding, instead of vanilla FFTBlock [Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao and T.-Y. Liu, Fastspeech: Fast, robust and controllable text to speech, Adv. Neural Inf. Process. Syst. 32 (2019)] to better model the local and global dependency in the acoustic model. We also propose a specific inference strategy to control the desired emotion of speech. The experimental results show that (1) The proposed model with improved Conformer achieved higher scores than the baseline model in all naturalness and emotion similarity scores; (2) The proposed model still maintains fast inference speed like the baseline model.

Hybrid Autoregressive and Non-Autoregressive Transformer Models for Speech Recognition

The autoregressive (AR) models, such as attention-based encoder-decoder models and RNN-Transducer, have achieved great success in speech recognition. They predict the output sequence conditioned on the previous tokens and acoustic encoded states, which is inefficient on GPUs. The non-autoregressive (NAR) models can get rid of the temporal dependency between the output tokens and predict the entire output tokens in at least one step. However, the NAR model still faces two major problems. On the one hand, there is still a great gap in performance between the NAR models and the advanced AR models. On the other hand, it's difficult for most of the NAR models to train and converge. To address these two problems, we propose a new model named the two-step non-autoregressive transformer(TSNAT), which improves the performance and accelerating the convergence of the NAR model by learning prior knowledge from a parameters-sharing AR model. Furthermore, we introduce the two-stage method into the inference process, which improves the model performance greatly. All the experiments are conducted on a public Chinese mandarin dataset ASIEHLL-1. The results show that the TSNAT can achieve a competitive performance with the AR model and outperform many complicated NAR models.

Research on Low Resource Neural Machine Translation Based on Non-autoregressive Model

The autoregressive model can’t make full use of context information because of its single direction of generation, and the autoregressive method can’t perform parallel computation in decoding, which affects the efficiency of translation generation. Therefore, we explore a non-autoregressive translation generation method based on insertion and deletion in low-resource languages, which decomposes translation generation into three steps: deletion-insertion-generation. Therefore, the dynamic editing of the translation can be realized in the iterative updating process. At the same time, each step can be calculated in parallel, which improves the decoding efficiency. In order to reduce the complexity of data sets in non-autoregressive model training, we have trained Uyghur-Chinese training data with sequence-level knowledge distillation. Experiments on Uyghur-Chinese, English-Romanian distilled data sets and standard data sets verify the effectiveness of the non-autoregressive method.

Vehicle Trajectory Prediction Based on Intention-Aware Non-Autoregressive Transformer With Multi-Attention Learning for Internet of Vehicles

As a core function of autonomous driving and the internet of vehicles, accurately predicting the trajectory of vehicles can significantly improve traffic safety and reduce crash injuries. In this paper, we propose an intention-aware non-autoregressive Transformer model with multi-attention learning for multi-modal vehicle trajectory prediction. We first present social attention learning where graph attention is properly integrated with the Transformer encoder so as to model the social interaction between vehicles. Then, the social and temporal dependency across consecutive frames is captured by temporal attention learning. The above social and temporal attention modules can be interleaved and stacked to achieve the coupled modeling and thus extract abundant features from trajectory data. To implement precise prediction as well as efficient inference, we further put forward an intention-aware decoder query generation approach to produce multiple possible trajectories concurrently. Finally, cross-attention learning is devised to make full use of the encoded features, therefore, yielding future predictions. The proposed model is evaluated on two large-scale vehicle trajectory datasets and the experimental results verify that our algorithm achieves better performance compared with some state-of-the-art models. The root-mean-square error (RMSE) of the predicted trajectory over 5s time horizon for the NGSIM and HighD datasets is 3.43m and 1.10m, respectively.

Towards Contextual Spelling Correction for Customization of End-to-End Speech Recognition Systems

Contextual biasing is an important and challenging task for end-to-end automatic speech recognition (ASR) systems, which aims to achieve better recognition performance by biasing the ASR system to particular context phrases such as person names, music list, proper nouns, etc. Existing methods mainly include contextual LM biasing and adding bias encoder into end-to-end ASR models. In this work, we introduce a novel approach to do contextual biasing by adding a contextual spelling correction model on top of the end-to-end ASR system. We incorporate contextual information into a sequence-to-sequence spelling correction model with a shared context encoder. The proposed model includes two different mechanisms: autoregressive (AR) and non-autoregressive (NAR). We also propose filtering algorithms to handle large-size context lists, and performance balancing mechanisms to control the biasing degree of the model. The proposed model is a general biasing solution which is domain-insensitive and can be adopted in different scenarios. Experiments show that the proposed method achieves as much as 51% relative word error rate (WER) reduction over ASR system and outperforms traditional biasing methods. Compared to the AR solution, the NAR model reduces model size by 43.2% and speeds up inference by 2.1 times.

Partially Non-Autoregressive Image Captioning

Current state-of-the-art image captioning systems usually generated descriptions autoregressively, i.e., every forward step conditions on the given image and previously produced words. The sequential attribution causes a unavoidable decoding latency. Non-autoregressive image captioning, on the other hand, predicts the entire sentence simultaneously and accelerates the inference process significantly. However, it removes the dependence in a caption and commonly suffers from repetition or missing issues. To make a better trade-off between speed and quality, we introduce a partially non-autoregressive model, named PNAIC, which considers a caption as a series of concatenated word groups. The groups are generated parallelly in global while each word in group is predicted from left to right, and thus the captioner can create multiple discontinuous words concurrently at each time step. More importantly, by incorporating curriculum learning-based training tasks of group length prediction and invalid group deletion, our model is capable of generating accurate captions as well as preventing common incoherent errors. Extensive experiments on MS COCO benchmark demonstrate that our proposed method achieves more than 3.5× speedup while maintaining competitive performance.

Multitask Non-Autoregressive Model for Human Motion Prediction.

Human motion prediction, which aims at predicting future human skeletons given the past ones, is a typical sequence-to-sequence problem. Therefore, extensive efforts have been devoted to exploring different RNN-based encoder-decoder architectures. However, by generating target poses conditioned on the previously generated ones, these models are prone to bringing issues such as error accumulation problem. In this paper, we argue that such issue is mainly caused by adopting autoregressive manner. Hence, a novel Non-AuToregressive model (NAT) is proposed with a complete non-autoregressive decoding scheme, as well as a context encoder and a positional encoding module. More specifically, the context encoder embeds the given poses from temporal and spatial perspectives. The frame decoder is responsible for predicting each future pose independently. The positional encoding module injects positional signal into the model to indicate the temporal order. Besides, a multitask training paradigm is presented for both low-level human skeleton prediction and high-level human action recognition, resulting in the considerable improvement for the prediction task. Our approach is evaluated on Human3.6M and CMU-Mocap benchmarks and outperforms state-of-the-art autoregressive methods.

Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior

Although neural machine translation models reached high translation quality, the autoregressive nature makes inference difficult to parallelize and leads to high translation latency. Inspired by recent refinement-based approaches, we propose LaNMT, a latent-variable non-autoregressive model with continuous latent variables and deterministic inference procedure. In contrast to existing approaches, we use a deterministic inference algorithm to find the target sequence that maximizes the lowerbound to the log-probability. During inference, the length of translation automatically adapts itself. Our experiments show that the lowerbound can be greatly increased by running the inference algorithm, resulting in significantly improved translation quality. Our proposed model closes the performance gap between non-autoregressive and autoregressive approaches on ASPEC Ja-En dataset with 8.6x faster decoding. On WMT'14 En-De dataset, our model narrows the gap with autoregressive baseline to 2.0 BLEU points with 12.5x speedup. By decoding multiple initial latent variables in parallel and rescore using a teacher model, the proposed model further brings the gap down to 1.0 BLEU point on WMT'14 En-De task with 6.8x speedup.

Adapting Translation Models for Transcript Disfluency Detection

Transcript disfluency detection (TDD) is an important component of the real-time speech translation system, which arouses more and more interests in recent years. This paper presents our study on adapting neural machine translation (NMT) models for TDD. We propose a general training framework for adapting NMT models to TDD task rapidly. In this framework, the main structure of the model is implemented similar to the NMT model. Additionally, several extended modules and training techniques which are independent of the NMT model are proposed to improve the performance, such as the constrained decoding, denoising autoencoder initialization and a TDD-specific training object. With the proposed training framework, we achieve significant improvement. However, it is too slow in decoding to be practical. To build a feasible and production-ready solution for TDD, we propose a fast non-autoregressive TDD model following the non-autoregressive NMT model emerged recently. Even we do not assume the specific architecture of the NMT model, we build our TDD model on the basis of Transformer, which is the state-of-the-art NMT model. We conduct extensive experiments on the publicly available set, Switchboard, and in-house Chinese set. Experimental results show that the proposed model significantly outperforms previous state-ofthe-art models.