Non-parallel Data Research Articles

Chronic Liver Disease and Cirrhosis Mortality Rates Are Disproportionately Increasing in Younger Women in the United States Between 2000-2020

Background and aimsPrevious studies show that mortality from chronic liver disease (CLD) and cirrhosis is increasing in the U.S. However, there are limited data on sex-specific mortality trends by age, race, and geographical location. The aim of this study was to conduct a comprehensive time-trend analysis of liver disease related mortality rates in the National-Center-of-Health-Statistics(NCHS) database. MethodsCLD and cirrhosis mortality rates between 2000-2020(age-adjusted to the 2000-standard-U.S.-population) were collected from the NCHS database and categorized by sex and age into older adults(≥55 years) and younger adults(<55 years), race (Non-Hispanic-White, Non-Hispanic-Black, Hispanic, Non-Hispanic-America- Indian/Alaska-Native, and Non-Hispanic-Asian/Pacific-Islander), U.S. state, and cirrhosis etiology. Time-trends, annual percentage change (APC) and average APC (AAPC), were estimated using Joinpoint Regression utilizing Monte Carlo permutation analysis. We used tests for parallelism and identicalness for sex-specific pairwise comparisons of mortality trends(two-sided P-value cut-off=0.05). ResultsBetween 2000-2020, there were 716,651 deaths attributed to CLD and cirrhosis in the U.S. (35.68% women). In the overall population and in older adults, CLD and cirrhosis-related mortality rates were increasing similarly in men and women. However, in younger adults (246,149 deaths, %32.72 women), the rate of increase was greater in women compared to men(AAPC=3.04 vs 1.08, AAPC-Difference=1.96,P<0.001), with non-identical non-parallel data(P-values<0.001). The disparity was driven by Non-Hispanic-White(AAPC=4.51 vs 1.79, AAPC-difference=2.71,P<0.001) and Hispanic(AAPC=1.89 vs -0.65,AAPC-difference=2.54,P=0.001) individuals. The disparity varied between U.S. states and was seen in 16 states, mostly in West Virginia(AAPC=4.96 vs 0.88, AAPC-difference=4.08,P<0.001) and Pennsylvania(AAPC=2.81 vs -1.02, AAPC-difference=3.84,P<0.001). Etiology-specific analysis did not show significant sex disparity in younger adults. ConclusionMortality rates due to CLD and cirrhosis in the U.S. are increasing disproportionately in younger women. This finding was driven by higher rates in Non-Hispanic White and Hispanic individuals, with variation between U.S. states. Future studies are warranted to identify the reasons for these trends with the ultimate goal of improving outcomes.

The Incidence of Rectal Neuroendocrine Tumors Is Increasing in Younger Adults in the US, 2001-2020.

Prior non-comparative data showed increasing incidence of rectal neuroendocrine tumors (RNET) in the US. We aimed to evaluate age-specific RNET incidence rates and time-trends in demographic- and tumor-specific populations. The RNET age-adjusted incidence rates were calculated from the United States Cancer Statistics (USCS) database between 2001 and 2020. The population was stratified by age into older (≥55 years) and younger adults (<55 years), as well as by sex and race. The tumors were categorized by their stage at diagnosis into early and late. The annual percentage change (APC) and average APC (AAPC) were estimated using joinpoint regression and Monte Carlo permutation analysis. Pairwise comparison assessed for parallelism and coincidence. There were 59,846 patients diagnosed with RNET between 2001 and 2020 (50.3% women). Overall, the RNET incidence rates during this period were increasing in younger but not older adults (AAPC = 3.12 vs. -1.10; AAPC difference = 4.22, p < 0.001), with non-identical non-parallel data (p-values < 0.001). While similar results were seen in men, a greater age-specific difference was noted in women (AAPC = 3.31 vs. -1.10; AAPC difference = 4.41, p = 0.003). The difference between younger and older adults was seen in non-Hispanic White (AAPC-difference = 4.89; p < 0.001) and non-Hispanic Black (AAPC-difference = 3.33; p = 0.03) patients, and, in most years, among Hispanic and Non-Hispanic Asian/Pacific Islander patients, and it was mostly driven by early-stage tumors (AAPC-difference = 3.93; p < 0.001). The nationwide data show a significantly increasing RNET incidence in younger adults, most notably in younger women and in early-stage tumors, seen in various races. Future studies should evaluate RNET risk factors and outcomes in demographic-specific populations.

W2VC: WavLM representation based one-shot voice conversion with gradient reversal distillation and CTC supervision

Non-parallel data voice conversion (VC) has achieved considerable breakthroughs due to self-supervised pre-trained representation (SSPR) being used in recent years. Features extracted by the pre-trained model are expected to contain more content information. However, in common VC with SSPR, there is no special implementation to remove speaker information in the content representation extraction by SSPR, which prevents further purification of the speaker information from SSPR representation. Moreover, in conventional VC, Mel-spectrogram is often selected as the reconstructed acoustic feature, which is not consistent with the input of the content encoder and results in some information lost. Motivated by the above, we proposed W2VC to settle the issues. W2VC consists of three parts: (1) We reconstruct feature from WavLM representation (WLMR) that is more consistent with the input of content encoder; (2) Connectionist temporal classification (CTC) is used to align content representation and text context from phoneme level, content encoder plus gradient reversal layer (GRL) based speaker classifier are used to remove speaker information in the content representation extraction; (3) WLMR-based HiFi-GAN is trained to convert WLMR to waveform speech. VC experimental results show that GRL can purify well the content information of the self-supervised model. The GRL purification and CTC supervision on the content encoder are complementary in improving the VC performance. Moreover, the synthesized speech using the WLMR retrained vocoder achieves better results in both subjective and objective evaluation. The proposed method is evaluated on the VCTK and CMU databases. It is shown the method achieves 8.901 in objective MCD, 4.45 in speech naturalness, and 3.62 in speaker similarity of subjective MOS score, which is superior to the baseline.

Decoupling Speaker-Independent Emotions for Voice Conversion via Source-Filter Networks

Emotional voice conversion (VC) aims to convert a neutral voice to an emotional one while retaining the linguistic information and speaker identity. We note that the decoupling of emotional features from other speech information (such as content, speaker identity, etc.) is the key to achieving promising performance. Some recent attempts of speech representation decoupling on the neutral speech cannot work well on the emotional speech, due to the more complex entanglement of acoustic properties in the latter. To address this problem, here we propose a novel Source-Filter-based Emotional VC model (SFEVC) to achieve proper filtering of speaker-independent emotion cues from both the timbre and pitch features. Our SFEVC model consists of multi-channel encoders, emotion separate encoders, pre-trained speaker-dependent encoders, and the corresponding decoder. Note that all encoder modules adopt a designed information bottleneck auto-encoder. Additionally, to further improve the conversion quality for various emotions, a novel training strategy based on the 2D Valence-Arousal (VA) space is proposed. Experimental results show that the proposed SFEVC along with a VA training strategy outperforms all baselines and achieves the state-of-the-art performance in speaker-independent emotional VC with nonparallel data.

Obtaining Parallel Sentences in Low-Resource Language Pairs with Minimal Supervision.

Machine translation relies on parallel sentences, the number of which is an important factor affecting the performance of machine translation systems, especially in low-resource languages. Recent advances in learning cross-lingual word representations from nonparallel data by machine learning make a new possibility for obtaining bilingual sentences with minimal supervision in low-resource languages. In this paper, we introduce a novel methodology to obtain parallel sentences via only a small-size bilingual seed lexicon about hundreds of entries. We first obtain bilingual semantic by establishing cross-lingual mapping in monolingual languages via a seed lexicon. Then, we construct a deep learning classifier to extract bilingual parallel sentences. We demonstrate the effectiveness of our methodology by harvesting Uyghur-Chinese parallel sentences and constructing a machine translation system. The experiments indicate that our method can obtain large and high-accuracy bilingual parallel sentences in low-resource language pairs.

Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Data

Multimodal pre-training for audio-and-text has recently been proved to be effective and has significantly improved the performance of many downstream speech understanding tasks. However, these state-of-the-art pre-training audio-text models work well only when provided with large amount of parallel audio-and-text data, which brings challenges on many languages that are rich in unimodal corpora but scarce of parallel cross-modal corpus. In this paper, we investigate whether it is possible to pre-train an audio-text multimodal model with extremely low-resource parallel data and extra non-parallel unimodal data. Our pre-training framework consists of the following components: (1) Intra-modal Denoising Auto-Encoding (IDAE), which is able to reconstruct input text (audio) representations from a noisy version of itself. (2) Cross-modal Denoising Auto-Encoding (CDAE), which is pre-trained to reconstruct the input text (audio), given both a noisy version of the input text (audio) and the corresponding translated noisy audio features (text embeddings). (3) Iterative Denoising Process (IDP), which iteratively translates raw audio (text) and the corresponding text embeddings (audio features) translated from previous iteration into the new less-noisy text embeddings (audio features). We adapt a dual cross-modal Transformer as our backbone model which consists of two unimodal encoders for IDAE and two cross-modal encoders for CDAE and IDP. Our method achieves comparable performance on multiple downstream speech understanding tasks compared with the model pre-trained on fully parallel data, demonstrating the great potential of the proposed method.

Deep Learning for Text Style Transfer: A Survey

AbstractText style transfer is an important task in natural language generation, which aims to control certain attributes in the generated text, such as politeness, emotion, humor, and many others. It has a long history in the field of natural language processing, and recently has re-gained significant attention thanks to the promising performance brought by deep neural models. In this article, we present a systematic survey of the research on neural text style transfer, spanning over 100 representative articles since the first neural text style transfer work in 2017. We discuss the task formulation, existing datasets and subtasks, evaluation, as well as the rich methodologies in the presence of parallel and non-parallel data. We also provide discussions on a variety of important topics regarding the future development of this task.1

Automatic syllable stress detection under non-parallel label and data condition

Typically, automatic syllable stress detection is posed as a supervised classification problem, for which, a classifier is trained using manually annotated (existing) syllable data and stress labels. However, in real testing scenarios, syllable data is estimated since manual annotation is not possible. Further, the estimation process could result in a mismatch between the lengths of the estimated and the existing syllable data causing no one-to-one correspondence between the estimated syllable data and the existing labels. Hence, the existing labels and estimated syllable data together cannot be used to train the classifier. This can be avoided by manually labeling the estimated syllable data, which, however, is impractical. In contrast, we, in this work, propose a method to obtain labels for estimated syllable data without using manual annotation. The proposed method considers a weighted version of the well-known Wagner–Fisher algorithm (WFA) to assign the existing labels to the estimated syllable data, where the weights are computed based on a set of three constraints defined in the proposed algorithm. Experiments on ISLE corpus show that the performance obtained on the test set for four different types of estimated syllable data are higher when the assigned labels and estimated syllable data are used for training compared to those when existing labels and existing syllable data are used. Also, the label assignment accuracy using the proposed method is found to be higher than that using a baseline scheme based on WFA.

IELTS translation education corpus construction based on bilingual non-parallel data model

With the rapid development of intelligent technology, IELTS translation education has gradually explored a brand-new education model. In order to solve the problems of small scale, slow speed and incomplete domain of the traditional bilingual parallel corpus machine translation, we construct an IELTS translation education corpus based on bilingual non-parallel data model, which can be used to train Moses, an IELTS translation education machine translation model, for better aids of translation education. In the process of construction, parallel sentence pairs are extracted from non-parallel corpus by using the translation retrieval framework represented by word graph, and a translation retrieval model based on bilingual non-parallel data is constructed. The experimental results of training Moses translation model with elementary IELTS translation corpus show that the bilingual non-parallel data model constructed in this paper has good translation retrieval performance. Compared with existing algorithms, the BLEU value extracted from parallel sentence pairs is increased by 2.58. Therefore, the proposed algorithm and corpus will do favor for the machine translation of IELTS. In addition, the retrieval method based on the structure of translation option word graph proposed in this paper is time-efficient and has better performance and efficiency in assisting IELTS translation education.

Phoneme-guided Dysarthric speech conversion With non-parallel data by joint training

Phoneme-guided Dysarthric speech conversion With non-parallel data by joint training

Parallel data free singing voice conversion with cycle-consistent BEGAN

Singing voice conversion (SVC) is the method to modify the timbre of the source singer with the target singer while retaining the linguistic content. Recent studies are mainly focused with non-parallel training data, since it is difficult to obtain parallel training data in real life applications. In this paper, a parallel data-free SVC technique is proposed using Cycle-consistent Boundary Equilibrium Generative Adversarial Networks (CycleBEGAN) with gated convolutional neural networks (CNNs) and an identity-mapping loss. CycleBEGAN allows the learning of data distribution using both adversarial loss and cycle-consistency loss. Gated CNN and identity mapping loss ensures the sequential and hierarchical structures of information and preservation of linguistic information. This technique produces high quality converted singing voice without any time-alignment procedures and requires only a small amount of training data.

Data augmentation based non-parallel voice conversion with frame-level speaker disentangler

Non-parallel data voice conversion is a popular and challenging research area. The main task is to build acoustic mappings from the source speaker to the target speaker in different units (e.g., frame, phoneme, cluster, sentence). With the help of the recent high-quality speech synthesis techniques, it is possible to directly produce parallel speech using non-parallel data. This paper proposes ParaGen: a data augmentation based technique for non-parallel data voice conversion. The system consists of a speaker disentangler based text-to-speech model and a simple frame-to-frame spectrogram conversion model. The text-to-speech model takes the text and reference audio as input to produce the speech in the target speaker identity with the time-aligned local speaking style from the reference audio. The spectrogram conversion model directly converts the source spectrogram to the target speaker framewisely. The local speaking style is extracted by an acoustic encoder while the speaker identity is eliminated by a conditional convolutional disentangler. The local style encodings are time-aligned with the text encodings by the attention mechanism. The attention contexts are decoded by a conditional recurrent decoder. The experiment shows that the speaker identity of the source speech is converted to the target speaker while the local speaking style (e.g., prosody) is preserved after the augmentation. The method is compared to the augmentation model with typical statistical parameter speech synthesis (SPSS) with pre-aligned phoneme duration. The result shows that the converted speech has better speech naturalness than the SPSS system, while the speaker similarities of the converted speech are close.

High-Quality Many-to-Many Voice Conversion Using Transitive Star Generative Adversarial Networks with Adaptive Instance Normalization

This paper proposes a novel high-quality nonparallel many-to-many voice conversion method based on transitive star generative adversarial networks with adaptive instance normalization (Trans-StarGAN-VC with AdaIN). First, we improve the structure of generator with TransNets to make full use of hierarchical features associated with speech naturalness. In TransNets, many shortcut connections share hierarchical features between encoding and decoding part to capture sufficient linguistic and semantic information, which helps to provide natural sounding converted speech and accelerate the convergence of training process. Second, by incorporating AdaIN for style transfer, we enable the generator to learn sufficient speaker characteristic information directly from speech instead of using attribute labels, which also provides a promising framework for one-shot VC. Objective and subjective experiments with nonparallel training data show that our method significantly outperforms StarGAN-VC in both speech naturalness and speaker similarity. The mean values of mean opinion score (MOS) and ABX are increased by 24.5% and 10.7%, respectively. The comparison of spectrogram also shows that our method can provide more complete harmonic structures and details, and effectively bridge the gap between converted speech and target speech.

Non-parallel text style transfer with domain adaptation and an attention model

Text style transfer, the aim of which is to convert a specific style in a given sentence to another target style while maintaining the style-independent content information of the original sentence, can face challenges when applied to non-parallel text. In this paper, we combine domain adaptation learning and an attention model to propose a new framework to accomplish the task. Domain adaptation can leverage relative information from the source domain to improve the generative model’s capacity for reconstructing data. The attention model can give the importance weights of generated words for the target style in a sentence; therefore, the generative model can concentrate on generating words with higher importance weights to accomplish text style transfer effectively. We evaluate our framework using Yelp, Amazon and Captions corpora. The results of automatic and human evaluation demonstrate the effectiveness of our framework compared with previous works under non-parallel and limited training data. The available codes are in https://github.com/mingxuan007/text-style-transfer-with-adversarial-network-and-domain-adaptation .

Transfer Learning From Speech Synthesis to Voice Conversion With Non-Parallel Training Data

We present a novel voice conversion (VC) framework by learning from a text-to-speech (TTS) synthesis system, that is called TTS-VC transfer learning or TTL-VC for short. We first develop a multi-speaker speech synthesis system with sequence-to-sequence encoder-decoder architecture, where the encoder extracts the linguistic representations of input text, while the decoder, conditioned on target speaker embedding, takes the context vectors and the attention recurrent network cell output to generate target acoustic features. We take advantage of the fact that TTS system maps input text to speaker independent context vectors, thus re-purpose such a mapping to supervise the training of the latent representations of an encoder-decoder voice conversion system. In the voice conversion system, the encoder takes speech instead of text as the input, while the decoder is functionally similar to the TTS decoder. As we condition the decoder on a speaker embedding, the system can be trained on non-parallel data for any-to-any voice conversion. During voice conversion training, we present both text and speech to speech synthesis and voice conversion networks respectively. At run-time, the voice conversion network uses its own encoder-decoder architecture without the need of text input. Experiments show that the proposed TTL-VC system outperforms two competitive voice conversion baselines consistently, namely phonetic posteriorgram and AutoVC methods, in terms of speech quality, naturalness, and speaker similarity.

Extracting Parallel Sentences from Nonparallel Corpora Using Parallel Hierarchical Attention Network.

Collecting parallel sentences from nonparallel data is a long-standing natural language processing research problem. In particular, parallel training sentences are very important for the quality of machine translation systems. While many existing methods have shown encouraging results, they cannot learn various alignment weights in parallel sentences. To address this issue, we propose a novel parallel hierarchical attention neural network which encodes monolingual sentences versus bilingual sentences and construct a classifier to extract parallel sentences. In particular, our attention mechanism structure can learn different alignment weights of words in parallel sentences. Experimental results show that our model can obtain state-of-the-art performance on the English-French, English-German, and English-Chinese dataset of BUCC 2017 shared task about parallel sentences' extraction.

Non-Parallel Sequence-to-Sequence Voice Conversion With Disentangled Linguistic and Speaker Representations

This article presents a method of sequence-to-sequence (seq2seq) voice conversion using non-parallel training data. In this method, disentangled linguistic and speaker representations are extracted from acoustic features, and voice conversion is achieved by preserving the linguistic representations of source utterances while replacing the speaker representations with the target ones. Our model is built under the framework of encoder-decoder neural networks. A recognition encoder is designed to learn the disentangled linguistic representations with two strategies. First, phoneme transcriptions of training data are introduced to provide the references for leaning linguistic representations of audio signals. Second, an adversarial training strategy is employed to further wipe out speaker information from the linguistic representations. Meanwhile, speaker representations are extracted from audio signals by a speaker encoder. The model parameters are estimated by two-stage training, including a pre-training stage using a multi-speaker dataset and a fine-tuning stage using the dataset of a specific conversion pair. Since both the recognition encoder and the decoder for recovering acoustic features are seq2seq neural networks, there are no constrains of frame alignment and frame-by-frame conversion in our proposed method. Experimental results showed that our method obtained higher similarity and naturalness than the best non-parallel voice conversion method in Voice Conversion Challenge 2018. Besides, the performance of our proposed method was closed to the state-of-the-art parallel seq2seq voice conversion method.

Japanese translation teaching corpus based on bilingual non parallel data model

In recent years, with the development of Internet and intelligent technology, Japanese translation teaching has gradually explored a new teaching mode. Under the guidance of natural language processing and intelligent machine translation, machine translation based on statistical model has gradually become one of the primary auxiliary tools in Japanese translation teaching. In order to solve the problems of small scale, slow speed and incomplete field in the traditional parallel corpus machine translation, this paper constructs a Japanese translation teaching corpus based on the bilingual non parallel data model, and uses this corpus to train Japanese translation teaching machine translation model Moses to get better auxiliary effect. In the process of construction, for non parallel corpus, we use the translation retrieval framework based on word graph representation to extract parallel sentence pairs from the corpus, and then build a translation retrieval model based on Bilingual non parallel data. The experimental results of training Moses translation model with Japanese translation corpus show that the bilingual nonparallel data model constructed in this paper has good translation retrieval performance. Compared with the existing algorithm, the Bleu value extracted in the parallel sentence pair is increased by 2.58. In addition, the retrieval method based on the structure of translation option words graph proposed in this paper is time efficient and has better performance and efficiency in assisting Japanese translation teaching.

Using Phonetic Posteriorgram Based Frame Pairing for Segmental Accent Conversion

Accent conversion AC aims to transform non-native utterances to sound as if the speaker had a native accent. This can be achieved by mapping source speech spectra from a native speaker into the acoustic space of the target non-native speaker. In prior work, we proposed an AC approach that matches frames between the two speakers based on their acoustic similarity after compensating for differences in vocal tract length. In this paper, we propose a new approach that matches frames between the two speakers based on their phonetic rather than acoustic similarity. Namely, we map frames from the two speakers into a phonetic posteriorgram using speaker-independent acoustic models trained on native speech. We thoroughly evaluate the approach on a speech corpus containing multiple native and non-native speakers. The proposed algorithm outperforms the prior approach, improving ratings of acoustic quality 22% increase in mean opinion score and native accent 69% preference while retaining the voice quality of the non-native speaker. Furthermore, we show that the approach can be used in the reverse conversion direction, i.e., generating speech with a native speaker's voice quality and a non-native accent. Finally, we show that this approach can be applied to non-parallel training data, achieving the same accent conversion performance.

Style Transfer in Text: Exploration and Evaluation

The ability to transfer styles of texts or images, is an important measurement of the advancement of artificial intelligence (AI). However, the progress in language style transfer is lagged behind other domains, such as computer vision, mainly because of the lack of parallel data and reliable evaluation metrics. In response to the challenge of lacking parallel data, we explore learning style transfer from non-parallel data. We propose two models to achieve this goal. The key idea behind the proposed models is to learn separate content representations and style representations using adversarial networks. Considering the problem of lacking principle evaluation metrics, we propose two novel evaluation metrics that measure two aspects of style transfer: transfer strength and content preservation. We benchmark our models and the evaluation metrics on two style transfer tasks: paper-news title transfer, and positive-negative review transfer. Results show that the proposed content preservation metric is highly correlate to human judgments, and the proposed models are able to generate sentences with similar content preservation score but higher style transfer strength comparing to auto-encoder.