Masked Words Research Articles

IterPrompt: An iterative prompt-tuning method for nested relation extraction with dynamic assignment strategy

Most existing relation extraction methods focus on extracting relations with the flat or overlapping structure, where constituent elements of these relations are only entities. However, in many application scenarios, relations with nested structures are more common, of which the elements can be both entities or relation triples. The almost vacant research on nested relation extraction can be attributed to two difficulties: (1) how to formalize and recover the nested structures of relations from the unstructured texts. (2) how to keep the extraction method effective in low-data settings. To solve these problems, first, we formally define the nested RE task and propose a prompt-tuning-based method, called iterPrompt, through constructing the iterative prompt templates for all possible relation triples to establish the nested structures. Then, we combine the prompts with the pre-trained language model by converting the nested RE task into the masked word prediction task to adapt our method to low-data settings. Furthermore, to alleviate the data sparsity problem, a dynamic relation identifier assignment strategy is proposed by associating the candidate relation triple with the randomly generated relation identifier, for representing the semantics of the current extracted relation. Extensive experiments demonstrate that the proposed iterPrompt method outperforms previous state-of-the-art baselines and has good compatibility for both flat and overlapping relation extraction tasks.

Deployment of attention to facial expressions varies as a function of emotional quality-but not in alexithymic individuals.

Alexithymia is a risk factor for emotional disorders and is characterized by differences in automatic and controlled emotion processing. The multi-stimulus free-viewing task has been used to detect increased negative and reduced positive attentional biases in depression and anxiety. In the present eye-tracking study, we examined whether lexical emotional priming directs attention toward emotion-congruent facial expressions and whether alexithymia is related to impairments in lexical priming and spontaneous attention deployment during multiple face perception. A free-viewing task with happy, fearful, angry, and neutral faces shown simultaneously was administered to 32 alexithymic and 46 non-alexithymic individuals along with measures of negative affect and intelligence. Face presentation was preceded by masked emotion words. Indices of initial orienting and maintenance of attention were analyzed as a function of prime or target category and study group. Time to first fixation was not affected by prime category or study group. Analysis of fixation duration yielded a three-way interaction. Alexithymic individuals exhibited no prime or target category effect, whereas non-alexithymic individuals showed a main effect of target condition, fixating happy faces longer than neutral and angry faces and fearful faces longer than angry faces. Our results show evidence of attentional biases for positive and fearful social information in non-alexithymic individuals, but not in alexithymic individuals. The lack of spontaneous attentional preference for these social stimuli in alexithymia might contribute to a vulnerability for developing emotional disorders. Our data also suggest that briefly presented emotion words may not facilitate gaze orientation toward emotion-congruent stimuli.

Visual features drive attentional bias for threat

ABSTRACT Past studies argued that the attentional capture by threats is hard to inhibit. When using threats as task-irrelevant stimuli, this effect can deteriorate performance on the primary task. Whether attentional capture is driven by affective information (threat) or visual features (shape) is still debated. Here we aimed to investigate the role of threat value and shape in modulating attentional resources by conducting two experiments (total N = 87). Participants engaged in a semantic vigilance task responding to masked words appearing at the centre of the screen while ignoring threat-relevant (threatening or visually similar but nonthreatening) and neutral control distractor images placed at different distances from the target word. We found no performance difference between participants exposed to threat-related stimuli via affective or shape features. Moreover, while performance decreased when a neutral distractor appeared close (compared to further away) to the target word, stimulus eccentricity had no effect when the distractor (irrespective of the conveying feature) was threat relevant. Our findings are in line with previous studies showing an initial capture of attention by threat-relevant information but that this negative effect is compensated by an increase in arousal. We conclude that even the visual features of a stimulus can modulate attention toward threats.

Contributed Session II: Neural correlates of serial processing during divided attention across multipart objects.

Judgments of multiple simultaneously presented stimuli produce a variety of divided attention effects. For example, participants can detect colors in two locations as well as in one, but can recognize only one masked word at a time (White, Palmer, & Boynton, Psych Science 2018). Here, we ask whether judging objects with interchangeable parts, similar to letters in words, also produces performance deficits consistent with serial processing, and which brain areas subserve this process. In a probe recognition task, participants discriminated abstract objects made of Duplo™ parts. On each trial, two objects were presented, with either one or both objects cued as relevant (single- and dual-task conditions, respectively). The distractor probes were made of the same parts in a different order. The difference in performance between the single- and dual-task conditions was 17±1% (n=13), consistent with the prediction of a serial model and rejecting the fixed-capacity parallel model. This result suggests that there is a visual brain area where information can be processed about only one object at a time. Currently, we are using fMRI to examine activity in object-selective regions of the human lateral occipital cortex. To seek evidence of serial processing, we are assessing how stimulus-related modulation is mediated by selective attention; the area of interest should show a modulation for only the attended stimulus location.

Elevated pre-target EEG alpha power enhances the probability of comprehending weakly noise masked words and decreases the probability of comprehending strongly masked words

Ongoing brain activity constitutes a structuring context for stimulus processing and has been shown to predict both neural and behavioural responses. In the present study, we used EEG to investigate the relationship between brain activity during a speech-shaped noise-filled pre-target time window, and the recognition of digit-words embedded in noise at three signal-to-noise ratios calibrated to individual participants' performance. Single-trial mixed-effects logistic regression revealed that alpha power in a left posterior temporal area between at least 300 ms and 50 ms prior to digit onset predicted target recognition. Elevated alpha power increases the likelihood of recognising easier stimuli but decreases the probability of recognising more heavily masked words. The mechanism we uncover might reflect a gain modulation mechanism that can suppress noise in speech-relevant perceptual channels. This is a first step in identifying pre-stimulus activity patterns that may be exploited as targets for neurofeedback or neurostimulation to assist with masked speech comprehension.

Separating the role of perceptual and conceptual fluency on masked word priming using event-related potentials

Masked word repetition increases “old” responses on an episodic recognition test (Jacoby & Whitehouse, 1989). This effect is commonly attributed to perceptual fluency; that is, unconscious perception of the prime speeds reading of the target and this fluency leads to elevated familiarity. Two experiments directly tested the claim that perceptual fluency is responsible for word priming effects. Experiment 1 held prime-target meaning constant and altered the physical characteristics of match primes (e.g., “RIGHT” primes “RIGHT”) by including both lowercase (e.g, “right”) and mixed case primes (e.g., “rIgHt”). If word priming effects are due to perceptual fluency, then lowering the perceptual overlap between the prime and target should decrease or eliminate word priming effects. Instead, all three conditions showed robust priming effects in the behavioral and ERP (i.e., N400) measures. Experiment 2 equated the prime-target perceptual features and lowered the conceptual overlap by using orthographically similar nonwords as primes (e.g., “JIGHT” primes “RIGHT”). Removing prime-target conceptual overlap eliminated behavioral evidence of priming and N400 ERP differences correlated with priming. The evidence suggests that word priming effects on episodic recognition memory are more likely a product of conceptual fluency than perceptual fluency.

Conversational Context-sensitive Ad Generation with a Few Core-Queries

When people are talking together in front of digital signage, advertisements that are aware of the context of the dialogue will work the most effectively. However, it has been challenging for computer systems to retrieve the appropriate advertisement from among the many options presented in large databases. Our proposed system, the Conversational Context-sensitive Advertisement generator (CoCoA), is the first attempt to apply masked word prediction to web information retrieval that takes into account the dialogue context. The novelty of CoCoA is that advertisers simply need to prepare a few abstract phrases, called Core-Queries, and then CoCoA automatically generates a context-sensitive expression as a complete search query by utilizing a masked word prediction technique that adds a word related to the dialogue context to one of the prepared Core-Queries. This automatic generation frees the advertisers from having to come up with context-sensitive phrases to attract users’ attention. Another unique point is that the modified Core-Query offers users speaking in front of the CoCoA system a list of context-sensitive advertisements. CoCoA was evaluated by crowd workers regarding the context-sensitivity of the generated search queries against the dialogue text of multiple domains prepared in advance. The results indicated that CoCoA could present more contextual and practical advertisements than other web-retrieval systems. Moreover, CoCoA acquired a higher evaluation in a particular conversation that included many travel topics to which the Core-Queries were designated, implying that it succeeded in adapting the Core-Queries for the specific ongoing context better than the compared method without any effort on the part of the advertisers. In addition, case studies with users and advertisers revealed that the context-sensitive advertisements generated by CoCoA also had an effect on the content of the ongoing dialogue. Specifically, since pairs unfamiliar with each other more frequently referred to the advertisement CoCoA displayed, the advertisements had an effect on the topics about which the pairs spoke. Moreover, participants of an advertiser role recognized that some of the search queries generated by CoCoA fit the context of a conversation and that CoCoA improved the effect of the advertisement. In particular, they learned how to design of designing a good Core-Query at ease by observing the users’ response to the advertisements retrieved with the generated search queries.

Comparing Selective Masking Methods for Depression Detection in Social Media

Abstract Identifying those at risk for depression is a crucial issue and social media provides an excellent platform for examining the linguistic patterns of depressed individuals. A significant challenge in depression classification problems is ensuring that prediction models are not overly dependent on topic keywords (i.e., depression keywords) such that it fails to predict when such keywords are unavailable. One promising approach is masking—that is, by selectively masking various words and asking the model to predict the masked words, the model is forced to learn the inherent language patterns of depression. This study evaluates seven masking techniques. Moreover, predicting the masked words during the pre-training or fine-tuning phase was also examined. Last, six class imbalanced ratios were compared to determine the robustness of masked words selection methods. Key findings demonstrate that selective masking outperforms random masking in terms of F1-score. The most accurate and robust models are identified. Our research also indicates that reconstructing the masked words during the pre-training phase is more advantageous than during the fine-tuning phase. Further discussion and implications are discussed. This is the first study to comprehensively compare masked words selection methods, which has broad implications for the field of depression classification and general NLP. Our code can be found at: https://github.com/chanapapan/Depression-Detection.

Semantic relatedness and the cocktail party problem in aphasia: A hybrid remote/in-lab study

ABSTRACT Background Previous work has found that individuals with aphasia demonstrate impaired performance, relative to controls, on listening tasks where target speech is masked by other intelligible speech (Villard & Kidd, 2019). While this deficit may be due in part to impaired cognitive skills (e.g., attention, memory), the current study examined whether impaired linguistic-semantic processing could be a contributing factor as well. Aims This study compared the effect of semantic relatedness between target and masker words on performance on a speech-on-speech masking task, in listeners with aphasia vs. age-matched controls. Additionally, the study compared performance on a remote vs. an in-lab version of the experiment, with the goal of establishing the validity of remote listening experiments in these populations. Methods & Procedures Eight persons with aphasia (PWA) and eight age-matched controls (AMC) participated in the remote version of the experiment; similarly, eight PWA and eight AMC participated in the in-lab version of the experiment. In both versions, participants listened to short target sentences while ignoring maskers. Maskers consisted of either intelligible speech from the same semantic category as the target (e.g., target and maskers were all fruits), intelligible speech from a distant semantic category from the target (e.g., target was a fruit; maskers were articles of clothing), or speech-shaped, speech envelope-modulated noise. Procedures and stimuli were kept as similar as possible between the remote and in-lab versions of the experiment. Outcomes & Results In-lab results revealed a significant effect of group, such that PWA performed more poorly than AMC, as well as a significant effect of masking condition, such that participants performed more poorly on speech masking vs. noise masking conditions. No significant differences between same-category and different-category maskers were observed. A comparison of remote vs. in-lab results revealed similar patterns of performance between the two settings but suggested that remote testing resulted in overall noisier data. Conclusions These findings add to the existing body of knowledge about auditory masking in aphasia. Furthermore, they help lead to a clearer understanding of barriers to social engagement and community participation in PWA by characterizing challenges associated with communicating in noisy settings. The study also provided qualified support for the use of remote testing paradigms for listening experiments.

Assessing linguistic generalisation in language models: a dataset for Brazilian Portuguese

Much recent effort has been devoted to creating large-scale language models. Nowadays, the most prominent approaches are based on deep neural networks, such as BERT. However, they lack transparency and interpretability, and are often seen as black boxes. This affects not only their applicability in downstream tasks but also the comparability of different architectures or even of the same model trained using different corpora or hyperparameters. In this paper, we propose a set of intrinsic evaluation tasks that inspect the linguistic information encoded in models developed for Brazilian Portuguese. These tasks are designed to evaluate how different language models generalise information related to grammatical structures and multiword expressions (MWEs), thus allowing for an assessment of whether the model has learned different linguistic phenomena. The dataset that was developed for these tasks is composed of a series of sentences with a single masked word and a cue phrase that helps in narrowing down the context. This dataset is divided into MWEs and grammatical structures, and the latter is subdivided into 6 tasks: impersonal verbs, subject agreement, verb agreement, nominal agreement, passive and connectors. The subset for MWEs was used to test BERTimbau Large, BERTimbau Base and mBERT. For the grammatical structures, we used only BERTimbau Large, because it yielded the best results in the MWE task. In both cases, we evaluated the results considering the best candidates and the top ten candidates. The evaluation was done both automatically (for MWEs) and manually (for grammatical structures). The results obtained for MWEs show that BERTimbau Large surpassed both the other models in predicting the correct masked element. However, the average accuracy of the best model was only 52% when only the best candidates were considered for each sentence, going up to 66% when the top ten candidates were taken into account. As for the grammatical tasks, the results presented better prediction, but also varied depending on the type of morphosyntactic agreement. On the one hand, cases such as connectors and impersonal verbs, which do not require any agreement in the produced candidates, had precision of 100% and 98.78% among the best candidates. On the other hand, tasks that require morphosyntactic agreement had results consistently below 90% overall precision, with the lowest scores being reported for nominal agreement and verb agreement, both having scores below 80% in overall precision among the best candidates. Therefore, we identified that a critical and widely adopted resource for Brazilian Portuguese NLP presents issues concerning MWE vocabulary and morphosyntactic agreement, even if it is prolific in most cases. These models are a core component in many NLP systems, and our findings demonstrate the need of additional improvements in these models and the importance of widely evaluating computational representations of language.

Strength of target source segregation cues affects the outcome of speech-on-speech masking experiments.

In speech-on-speech listening experiments, some means for designating which talker is the "target" must be provided for the listener to perform better than chance. However, the relative strength of the segregation variables designating the target could affect the results of the experiment. Here, we examine the interaction of two source segregation variables-spatial separation and talker gender differences-and demonstrate that the relative strengths of these cues may affect the interpretation of the results. Participants listened to sentence pairs spoken by different-gender target and masker talkers, presented naturally or vocoded (degrading gender cues), either colocated or spatially separated. Target and masker words were temporally interleaved to eliminate energetic masking in either an every-other-word or randomized order of presentation. Results showed that the order of interleaving had no effect on recall performance. For natural speech with strong talker gender cues, spatial separation of sources yielded no improvement in performance. For vocoded speech with degraded talker gender cues, performance improved significantly with spatial separation of sources. These findings reveal that listeners may shift among target source segregation cues contingent on cue viability. Finally, performance was poor when the target was designated after stimulus presentation, indicating strong reliance on the cues.

CWSXLNet: A Sentiment Analysis Model Based on Chinese Word Segmentation Information Enhancement

This paper proposed a method for improving the XLNet model to address the shortcomings of segmentation algorithm for processing Chinese language, such as long sub-word lengths, long word lists and incomplete word list coverage. To address these issues, we proposed the CWSXLNet (Chinese Word Segmentation XLNet) model based on Chinese word segmentation information enhancement. The model first pre-processed Chinese pretrained text by Chinese word segmentation tool, and proposed a Chinese word segmentation attention mask mechanism by combining PLM (Permuted Language Model) and two-stream self-attention mechanism of XLNet. While performing natural language processing at word granularity, it can reduce the degree of masking between masked and non-masked words for two words belonging to the same word. For the Chinese sentiment analysis task, proposed the CWSXLNet-BiGRU-Attention model, which introduces bi-directional GRU as well as self-attention mechanism in the downstream task. Experiments show that CWSXLNet has achieved 89.91% precision, 91.53% recall rate and 90.71% F1-score, and CWSXLNet-BiGRU-Attention has achieved 92.61% precision, 93.19% recall rate and 92.90% F1-score on ChnSentiCorp dataset, which indicates that CWSXLNet has better performance than other models in Chinese sentiment analysis.

Measuring the efficiency of listening effort

Listening effort is a commonly reported difficulty among those who have hearing loss. This project aims to operationalize the concept of effort precision, as the ability to engage or disengage mental resources at strategic times could be an important signature of a person’s capacity to guard against wasted effort. Stimuli were designed to simulate the situation of seeking clarification of a misperceived word or attending to specific information in a sentence. Some stimuli included verbatim repetitions (which would ideally elicit reduced effort) and others featured sentences with a masked word followed by a clear repetition (which would ideally evoke effort aimed only at the newly unmasked word). Pupillometry was used as an index of moment-to-moment changes in listening effort. Data showed pupil dilations linked in time with the unmasked word, enabling precise measurements of the efficiency of effort. Listeners with normal hearing displayed the ability to plan and exert effort at specific times while also reducing effort in situations where it was unnecessary. Conversely, listeners who wear cochlear implants generally did not display these efficient effort characteristics. These results highlight the need to expand the concept of listening effort beyond a “more” or “less” framework, toward a framework of efficiency.

Digital Health Transformers and Opportunities for Artificial Intelligence-Enabled Nephrology.

Digital Health Transformers and Opportunities for Artificial Intelligence-Enabled Nephrology.

Natural language watermarking via paraphraser-based lexical substitution

Although powerful pretrained language models generate high-quality output text, they bring new concerns about the potential misuse of such models for malicious purposes. Natural language watermarking (NLW) is a technique that is desgined to help tracing the provenance of texts for againsting possible attacks, where the watermark signals are embedded into cover texts using synonym substitutions. The up-to-date BERT-based NLW methods have made remarkable progress on performance improvement of watermarking through generating substitutes for a masked target word. Yet, the BERT-based NLWs focus on the context of texts rather than the meaning of target words, which might make the capacity of watermark embeddings being lower.To address the limitations, this study proposes a novel NLW method by incorporating a paraphraser-based lexical substitution method. Under the promise of paraphrase preservation, the proposed NLW method utilizes the knowledge of paraphrase modeling to generate the substitute candidates to replace the words in original sentences capable of carrying the watermark signal in local contexts. We empirically show that our NLW method not only has a better meaning-preserved, but improves the payload more than 2 times compared with the BERT-based NLW method. Besides, compared with previous state-of-the-art method Compared with other state-of-the-art baselines, the experimental results show that the proposed LS method improves the Precision@1 score from 51.7% to 58.3% and from 50.5% to 62.6% on LS07 and CoInCo benchmarks, respectively.

QAScore-An Unsupervised Unreferenced Metric for the Question Generation Evaluation.

Question Generation (QG) aims to automate the task of composing questions for a passage with a set of chosen answers found within the passage. In recent years, the introduction of neural generation models has resulted in substantial improvements of automatically generated questions in terms of quality, especially compared to traditional approaches that employ manually crafted heuristics. However, current QG evaluation metrics solely rely on the comparison between the generated questions and references, ignoring the passages or answers. Meanwhile, these metrics are generally criticized because of their low agreement with human judgement. We therefore propose a new reference-free evaluation metric called QAScore, which is capable of providing a better mechanism for evaluating QG systems. QAScore evaluates a question by computing the cross entropy according to the probability that the language model can correctly generate the masked words in the answer to that question. Compared to existing metrics such as BLEU and BERTScore, QAScore can obtain a stronger correlation with human judgement according to our human evaluation experiment, meaning that applying QAScore in the QG task benefits to a higher level of evaluation accuracy.

Learning to share by masking the non-shared for multi-domain sentiment classification

Multi-domain sentiment classification deals with the scenario where labeled data exists for multiple domains but is insufficient for training effective sentiment classifiers that work across domains. Thus, fully exploiting sentiment knowledge shared across domains is crucial for real-world applications. While many existing works try to extract domain-invariant features in high-dimensional space, such models fail to explicitly distinguish between shared and private features at the text level, which to some extent lacks interpretability. Based on the assumption that removing domain-related tokens from texts would help improve their domain invariance, we instead first transform original sentences to be domain-agnostic. To this end, we propose the BERTMasker model which explicitly masks domain-related words from texts, learns domain-invariant sentiment features from these domain-agnostic texts and uses those masked words to form domain-aware sentence representations. Empirical experiments on the benchmark multiple domain sentiment classification datasets demonstrate the effectiveness of our proposed model, which improves the accuracy on multi-domain and cross-domain settings by 1.91% and 3.31% respectively. Further analysis on masking proves that removing those domain-related and sentiment irrelevant tokens decreases texts’ domain separability, resulting in the performance degradation of a BERT-based domain classifier by over 12%.

The impact of prior topic awareness on listening effort during speech perception

Most standard speech perception tests use unrelated sentences where the topic changes for each stimulus. This unrealistic paradigm is expedient but might overlook an important aspect of speech processing and effort, which is how listeners use expectations and topic awareness to enable easier repair of misperceived words. We hypothesize that the effort of mentally repairing misperceived words will be reduced when individuals have advance topic awareness of upcoming sentences. In this study, individuals with normal hearing and hearing loss heard sentences with a masked target word that required retroactive mental repair using later context. Sentences were either preceded by the presence or absence of a related priming word presented visually to simulate advance topic knowledge. Listening effort for these sentences was measured by changes in pupil size that were linked with the exact timing of the word being repaired. When the target word is masked in a sentence, pupil dilation increases immediately after the missing word, and that dilation is modulated by the helpfulness of the prime before the sentence.

Brains and algorithms partially converge in natural language processing

Deep learning algorithms trained to predict masked words from large amount of text have recently been shown to generate activations similar to those of the human brain. However, what drives this similarity remains currently unknown. Here, we systematically compare a variety of deep language models to identify the computational principles that lead them to generate brain-like representations of sentences. Specifically, we analyze the brain responses to 400 isolated sentences in a large cohort of 102 subjects, each recorded for two hours with functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). We then test where and when each of these algorithms maps onto the brain responses. Finally, we estimate how the architecture, training, and performance of these models independently account for the generation of brain-like representations. Our analyses reveal two main findings. First, the similarity between the algorithms and the brain primarily depends on their ability to predict words from context. Second, this similarity reveals the rise and maintenance of perceptual, lexical, and compositional representations within each cortical region. Overall, this study shows that modern language algorithms partially converge towards brain-like solutions, and thus delineates a promising path to unravel the foundations of natural language processing.

マスクされた単語埋め込みと 2 段階クラスタリングを用いた動詞の意味フレーム推定

Recent studies on semantic frame induction show that relatively high performance has been achieved by using clustering-based methods with contextualized word embeddings. However, there are two potential drawbacks to these methods: one is that they focus too much on the superficial information of the frame-evoking verb and the other is that they tend to divide the instances of the same verb into too many different frame clusters. To overcome these drawbacks, we propose a semantic frame induction method using masked word embeddings and two-step clustering. Through experiments on the English FrameNet data, we demonstrate that using the masked word embeddings is effective for avoiding too much reliance on the surface information of frame-evoking verbs and that two-step clustering can improve the number of resulting frame clusters for the instances of the same verb.