Text Pairs Research Articles

Easy read health information for people with intellectual disabilities: A linguistic discourse analysis. What happens to language when it is simplified?

Easy Read health documents prepared for people with intellectual disabilities are often generated from Standard Texts. Language in Easy Read versions is typically assumed to be simpler. However, simplification of language may have unintended consequences. This study aimed to explore the differences in language used between Easy Read health material and the Standard Text versions of the same material produced for the general population. Five Easy Read/Standard Text pairs were sampled and analysed using Systemic Functional Linguistics. This addressed: how people with intellectual disabilities and others were represented by language, the author stance in relation to the reader and the overall organisation of the text. The Easy Read versions often used language that was less empowering and inclusive. Increased awareness of author power and better knowledge of the impact of language choice could help to redress these issues.

Augmenting Multimodal Content Representation with Transformers for Misinformation Detection

Information sharing on social media has become a common practice for people around the world. Since it is difficult to check user-generated content on social media, huge amounts of rumors and misinformation are being spread with authentic information. On the one hand, most of the social platforms identify rumors through manual fact-checking, which is very inefficient. On the other hand, with an emerging form of misinformation that contains inconsistent image–text pairs, it would be beneficial if we could compare the meaning of multimodal content within the same post for detecting image–text inconsistency. In this paper, we propose a novel approach to misinformation detection by multimodal feature fusion with transformers and credibility assessment with self-attention-based Bi-RNN networks. Firstly, captions are derived from images using an image captioning module to obtain their semantic descriptions. These are compared with surrounding text by fine-tuning transformers for consistency check in semantics. Then, to further aggregate sentiment features into text representation, we fine-tune a separate transformer for text sentiment classification, where the output is concatenated to augment text embeddings. Finally, Multi-Cell Bi-GRUs with self-attention are used to train the credibility assessment model for misinformation detection. From the experimental results on tweets, the best performance with an accuracy of 0.904 and an F1-score of 0.921 can be obtained when applying feature fusion of augmented embeddings with sentiment classification results. This shows the potential of the innovative way of applying transformers in our proposed approach to misinformation detection. Further investigation is needed to validate the performance on various types of multimodal discrepancies.

Leveraging Contrastive Language–Image Pre-Training and Bidirectional Cross-attention for Multimodal Keyword Spotting

In resource-limited keyword spotting scenarios, the scarcity of annotated corpora hinders deep learning’s ability to develop robust models for representing acoustic features. Recent studies focus on contrastive learning, using paired examples for self-supervision, indicating a growing research interest in keyword spotting. Typically, models are trained on segmented spoken words. However, the absence of distinct word boundaries in lengthy audio poses significant challenges. This study presents a novel approach that integrates Contrastive Language–Image Pre-Training with Cross-Attention for Self-Supervised Alignment in the field of Multimodal Keyword Spotting. The proposed method introduces a cross-modal process for matching word pairs, enhancing collaborative effectiveness between audio and text embeddings, achieving word-level congruence across different modalities. Employing a self-supervised learning approach, we use a restricted number of annotated audio–text pairs to discern semantic congruities and divergences, resulting in improved multimodal feature representation. Using this strategy, we input into subsequent keyword detection tasks, adopting a Bidirectional Cross-Attention block for detailed coordination across modalities, bridging the semantic gap between diverse audio and text representations, consequently improving keyword localization precision. Comprehensive experiments conducted on the Aishell-2 and Librispeech datasets demonstrate that the proposed method markedly surpasses existing techniques, especially regarding the comprehensive performance metric F1 for keyword spotting. It attains state-of-the-art outcomes for both pristine and noisy speech, showing respective enhancements of 5.2% and 5.4% on Aishell-2, as well as 5.4% and 5.6% on Librispeech.

IQAGPT: computed tomography image quality assessment with vision-language and ChatGPT models

Large language models (LLMs), such as ChatGPT, have demonstrated impressive capabilities in various tasks and attracted increasing interest as a natural language interface across many domains. Recently, large vision-language models (VLMs) that learn rich vision–language correlation from image–text pairs, like BLIP-2 and GPT-4, have been intensively investigated. However, despite these developments, the application of LLMs and VLMs in image quality assessment (IQA), particularly in medical imaging, remains unexplored. This is valuable for objective performance evaluation and potential supplement or even replacement of radiologists’ opinions. To this end, this study introduces IQAGPT, an innovative computed tomography (CT) IQA system that integrates image-quality captioning VLM with ChatGPT to generate quality scores and textual reports. First, a CT-IQA dataset comprising 1,000 CT slices with diverse quality levels is professionally annotated and compiled for training and evaluation. To better leverage the capabilities of LLMs, the annotated quality scores are converted into semantically rich text descriptions using a prompt template. Second, the image-quality captioning VLM is fine-tuned on the CT-IQA dataset to generate quality descriptions. The captioning model fuses image and text features through cross-modal attention. Third, based on the quality descriptions, users verbally request ChatGPT to rate image-quality scores or produce radiological quality reports. Results demonstrate the feasibility of assessing image quality using LLMs. The proposed IQAGPT outperformed GPT-4 and CLIP-IQA, as well as multitask classification and regression models that solely rely on images.

Mono-lingual text reuse detection for the Urdu language at lexical level

Text reuse is the process of creating new texts from pre-existing ones. In recent years, Urdu Text Reuse Detection (U-TRD) has garnered the attention of researchers due to the ready availability of digital text all over the internet, which can be copied or paraphrased from other sources without proper attribution, making it easier to reuse but challenging to detect. Previous studies have explored the issue of U-TRD at the phrasal, sentence/passage, and document levels, using benchmark corpora and methods. However, the problem of U-TRD has not been investigated at the lexical level in terms of corpora and methods. To address this research gap, our study has developed a large benchmark corpus manually annotated at the lexical level. This corpus consists of 22,184 text pairs categorized into two levels of rewrite: (1) Derived (8,660) and (2) Non-Derived (13,524). Additionally, our research has involved the development, application, evaluation, and comparison of a range of methods, including baseline methods (uni-gram overlap and word embedding-based methods), along with state-of-the-art transformer-based methods and feature-fusion-based methods, using the proposed UTRD-Lex-23 corpus. Our study concludes that one of our proposed feature-fusion methods outperforms all other methods. The model we propose, which combines seven different Sentence Transformers (ST) (each producing 768 dimension vectors) with one uni-gram (at word level) and sixteen different features extracted from four different Word Embedding (WE) based models (yielding 300 dimension vectors), achieves an F1 score of 0.70601 using 10-fold cross validation. To foster and promote research in Urdu (a low-resourced language) proposed corpus will be freely and publicly available for research purposes.

INCEPT: A Framework for Duplicate Posts Classification with Combined Text Representations

Dealing with many of the problems related to the quality of textual content online involves identifying similar content. Algorithmic solutions for duplicate content classification typically rely on text vector representation, which maps textual information into a set of features. Ideally, this representation would capture all aspects of the underlying text, including length, word frequencies, syntax, and semantics. While recent advancements in text representation have led to improved performance, a comprehensive approach that explicitly incorporates all text features has not yet been proposed. In this study, we present the INCEPT framework that utilizes multiple representation methods to detect duplicate text pairs, taking advantage of their individual strengths. The core of our approach involves using a stacking ensemble of pairwise vector distance measurements that are computed from multiple text representation methods. A stacking classifier then utilizes these distance scores as input and learns to identify duplicate posts. We assess the proposed framework’s effectiveness in identifying duplicate posts in an online Question and Answer platform. By combining several text representation methods, INCEPT performs well in the duplicate posts classification task. Our experiments demonstrate that specific framework configurations outperform the accuracy scores obtained from individual text representation methods. Therefore, we also infer that no single text representation method can independently capture a text’s features.

Benchmarking PathCLIP for Pathology Image Analysis.

Accurate image classification and retrieval are of importance for clinical diagnosis and treatment decision-making. The recent contrastive language-image pre-training (CLIP) model has shown remarkable proficiency in understanding natural images. Drawing inspiration from CLIP, pathology-dedicated CLIP (PathCLIP) has been developed, utilizing over 200,000 image and text pairs in training. While the performance the PathCLIP is impressive, its robustness under a wide range of image corruptions remains unknown. Therefore, we conduct an extensive evaluation to analyze the performance of PathCLIP on various corrupted images from the datasets of osteosarcoma and WSSS4LUAD. In our experiments, we introduce eleven corruption types including brightness, contrast, defocus, resolution, saturation, hue, markup, deformation, incompleteness, rotation, and flipping at various settings. Through experiments, we find that PathCLIP surpasses OpenAI-CLIP and the pathology language-image pre-training (PLIP) model in zero-shot classification. It is relatively robust to image corruptions including contrast, saturation, incompleteness, and orientation factors. Among the eleven corruptions, hue, markup, deformation, defocus, and resolution can cause relatively severe performance fluctuation of the PathCLIP. This indicates that ensuring the quality of images is crucial before conducting a clinical test. Additionally, we assess the robustness of PathCLIP in the task of image-to-image retrieval, revealing that PathCLIP performs less effectively than PLIP on osteosarcoma but performs better on WSSS4LUAD under diverse corruptions. Overall, PathCLIP presents impressive zero-shot classification and retrieval performance for pathology images, but appropriate care needs to be taken when using it.

Vision-language joint representation learning for sketch less facial image retrieval

The traditional sketch-based facial image retrieval (SBFIR) framework assumes that a high-quality facial sketch has been prepared prior to the retrieval task. However, drawing such a sketch requires considerable skills and is time consuming, resulting in limited applicability. Sketch less facial image retrieval (SLFIR) framework aims to break these barriers through human–computer interaction during the sketching process. The primary challenges for the SLFIR problem can be noted that initial sketches (at early sketching) contain only local details and exhibit significant differences among users, resulting in poor performance at early stages and weak generalization abilities in practical testing. In this study, we developed a vision–language pretraining model to align the representation of facial images and their associated semantics. Based on this framework, we proposed a method for learning joint representations by fusing sketches with prior semantics, thereby enriching the information of initial sketches. Specifically, (1) we developed a series of well-designed operations to improve the quality of facial image–text pairs in the LAION-Face dataset; we trained a facial vision–language pretraining (FVLP) model to align the facial image and its semantics at the feature level. (2) subsequently, using FVLP as the backbone, we designed a convolutional attention module to fuse the multiscale features extracted from image encoder. This facilitated the learning of a multimodal representation crucial for the final retrieval process. In experiments, our proposed method achieved state-of-the-art performance at early stages on two public datasets; moreover, it exhibited good generalization capabilities during practical testing. Thus, our method significantly outperforms other baselines in terms of early retrieval performance. Codes are available at: https://github.com/ddw2AIGROUP2CQUPT/FVLP.

Unified multimodal fusion transformer for few shot object detection for remote sensing images

Object detection is a fundamental computer vision task with wide applications in remote sensing, but traditional methods strongly rely on large annotated datasets which are difficult to obtain, especially for novel object classes. Few-shot object detection (FSOD) aims to address this by using detectors to learn from very limited labeled data. Recent work fuse multi-modalities like image–text pairs to tackle data scarcity but require external region proposal network (RPN) to align cross-modal pairs which leads to a bias towards base classes and insufficient cross-modal contextual learning. To address these problems, we propose a unified multi-modal fusion transformer (UMFT), which extracts visual features from ViT and textual encodings from BERT to align multi-modal representations in an end-to-end manner. Specifically, affinity-guided fusion (AFM) captures semantically related image–text pairs by modeling their affinity relationships to selectively combine most informative pairs. In addition, cross-modal correlation module (CCM) captures discriminative inter-modal patterns between image and text representations and filters out unrelated features to enhance cross-modal alignment. By leveraging AFM to focus on semantic relationships and CCM to refine inter-modal features, the model better aligns multimodal data without RPN. These representations are passed to detection decoder that predicts bounding boxes, probabilities of class and cross-modal attributes. Evaluation of UMFT on benchmark datasets NWPU VHR-10 and DIOR demonstrated its ability to leverage limited image–text training data via dynamic fusion, achieving new state-of-the-art mean average precision (mAP) for few-shot object detection. Our code will be publicly available at https://github.com/abdullah-azeem/umft.

Generate custom travel magazine layouts

Among the problems of specifying the style and number of elements of a travel magazine, the problem of generating magazine layout by constraining text, and constraining graph layout remains a complex and unsolved problem. In this paper, we generate layouts of text satisfying constraints via GAN. Due to the complexity and variety of graph designs, we enhance the performance of the discriminator and the generator so that the layouts generated by the generator are more constrained. Add non-corresponding constraint text and real layout pairs to the discriminator to enhance the performance of the discriminator; then add a spatial attention mechanism to the layout encoder to extract the features of the layout and generate high-quality layouts. We demonstrate that the proposed method can generate high-quality layouts of text satisfying the constraints, and we validate the effectiveness of this method through user ratings.

Morphosyntactic stereotypes of speakers with different genders and sexual orientations: an experimental investigation

Abstract Previous research has identified differences in language use between speakers representing different genders and sexual orientations, as well as stereotypical beliefs about the existence of such differences. The present paper investigates such stereotypical beliefs among speakers of two varieties of Spanish and explores whether linguistic stereotypes of gender and sexual orientation are stronger in a society with more conservative gender roles. We conducted an experiment where 252 speakers of European and Mexican Spanish were shown pairs of texts and sentences depicting the same event but differing in the use of morphosyntactic resources, creating stereotypically “masculine” and “feminine” variants of the same text or sentence. The informants were asked which of the variants was more likely uttered by a woman (rather than a man), a lesbian woman (rather than a straight woman), or a gay man (rather than a straight man). Some of the morphosyntactic features were strongly associated with speaker gender (e.g., use of first- and second-person singular) and others with lesbian (e.g., use of impersonal structures) or gay speakers (e.g., direct reported speech). Contrary to expectations, the responses did not differ according to informants’ age, gender, or nationality. While the present study did not reveal any differences between the two societies, more cross-linguistic and cultural research is needed to elucidate the effect of society on morphosyntactic variation and stereotypes thereof.

Ta-Adapter: Enhancing few-shot CLIP with task-aware encoders

Contrastive Language-Image Pre-training (CLIP) has shown impressive zero-shot transfer capabilities, but its potential for specific downstream tasks is not fully utilized. To further enhance CLIP’s few-shot capability for specific datasets, some subsequent works have been proposed, such as methods based on lightweight adapters and prompt learning. However, since CLIP is pretrained on a diverse collection of image and text pairs sourced from the internet, it is difficult to sufficiently tune models to specific datasets using only lightweight adaptions. In this paper, we argue that largely modifying the internal representations within CLIP’s encoders can yield better results on downstream datasets. In this work, we introduce Ta-Adapter, a method that equips both the visual and textual encoders of CLIP with task-specific prompts. These prompts are generated using a collaborative prompt learning approach, which allows the encoders to produce representations that are better aligned with specific downstream datasets. Then, we initialize an adapter module using the optimized features generated by the task-aware visual encoder for further feature alignment, and this module can also be further fine-tuned. Our extensive experiments on image classification datasets show that compared to the state-of-the-art few-shot methods Tip-Adapter-F and MaPLe, our model exhibits good performance and obtains an average absolute gain of 2.04% and 1.62% on 11 different image recognition datasets, respectively. In conclusion, this work presents a unique and effective approach to unlocking the full potential of CLIP’s few-shot learning capabilities.

Enhancing Visual Grounding in Vision-Language Pre-Training With Position-Guided Text Prompts.

Vision-Language Pre-Training (VLP) has demonstrated remarkable potential in aligning image and text pairs, paving the way for a wide range of cross-modal learning tasks. Nevertheless, we have observed that VLP models often fall short in terms of visual grounding and localization capabilities, which are crucial for many downstream tasks, such as visual reasoning. In response, we introduce a novel Position-guided Text Prompt (PTP) paradigm to bolster the visual grounding abilities of cross-modal models trained with VLP. In the VLP phase, PTP divides an image into N x N blocks and employs a widely-used object detector to identify objects within each block. PTP then reframes the visual grounding task as a fill-in-the-blank problem, encouraging the model to predict objects in given blocks or regress the blocks of a given object, exemplified by filling "[P]" or "[O]" in a PTP sentence such as "The block [P] has a [O]." This strategy enhances the visual grounding capabilities of VLP models, enabling them to better tackle various downstream tasks. Additionally, we integrate the seconda-order relationships between objects to further enhance the visual grounding capabilities of our proposed PTP paradigm. Incorporating PTP into several state-of-the-art VLP frameworks leads to consistently significant improvements across representative cross-modal learning model architectures and multiple benchmarks, such as zero-shot Flickr30 k Retrieval (+5.6 in average recall@1) for ViLT baseline, and COCO Captioning (+5.5 in CIDEr) for the state-of-the-art BLIP baseline. Furthermore, PTP attains comparable results with object-detector-based methods and a faster inference speed, as it discards its object detector during inference, unlike other approaches.

Automated Assessment of Inferences Using Pre-Trained Language Models

Inference plays a key role in reading comprehension. However, assessing inference in reading is a complex process that relies on the judgment of trained experts. In this study, we explore objective and automated methods for assessing inference in readers’ responses using natural language processing. Specifically, classifiers were trained to detect inference from a pair of input texts and reader responses by fine-tuning three widely used pre-trained language models. The effects of the model size and pre-training strategy on the accuracy of inference classification were investigated. The highest F1 score of 0.92 was achieved via fine-tuning the robustly optimized 12-layer BERT model (RoBERTa-base). Fine-tuning the larger 24-layer model (RoBERTa-large) did not improve the classification accuracy. Error analysis provides insight into the relative difficulty of classifying inference subtypes. The proposed method demonstrates the feasibility of the automated quantification of inference during reading, and offers potential to facilitate individualized reading instructions.

Prompt-Enhanced Generation for Multimodal Open Question Answering

Multimodal open question answering involves retrieving relevant information from both images and their corresponding texts given a question and then generating the answer. The quality of the generated answer heavily depends on the quality of the retrieved image–text pairs. Existing methods encode and retrieve images and texts, inputting the retrieved results into a language model to generate answers. These methods overlook the semantic alignment of image–text pairs within the information source, which affects the encoding and retrieval performance. Furthermore, these methods are highly dependent on retrieval performance, and poor retrieval quality can lead to poor generation performance. To address these issues, we propose a prompt-enhanced generation model, PEG, which includes generating supplementary descriptions for images to provide ample material for image–text alignment while also utilizing vision–language joint encoding to improve encoding effects and thereby enhance retrieval performance. Contrastive learning is used to enhance the model’s ability to discriminate between relevant and irrelevant information sources. Moreover, we further explore the knowledge within pre-trained model parameters through prefix-tuning to generate background knowledge relevant to the questions, offering additional input for answer generation and reducing the model’s dependency on retrieval performance. Experiments conducted on the WebQA and MultimodalQA datasets demonstrate that our model outperforms other baseline models in retrieval and generation performance.

Being critical and critical being in the language classroom

The aim of this conceptual paper is to delve into the notion of criticality and demonstrate how criticality can be nurtured in the language classroom. The premise is that a more holistic concept of criticality, which fuses critical thinking (skills) and critical being (disposition), can nurture language learners to become more critical and responsible ‘prosumers’ of contemporary media. The paper unfurls in three main stages. I begin with an introduction to Critical Discourse Analysis (CDA), a prominent field in applied linguistics, and an explication of its aims and principles by using a pair of texts to draw out the essence and application of these principles. This provides the backdrop against which I argue how the mission of CDA, which focuses on a critically oriented and socially committed praxis, can be distilled into a criticality centered on ‘critical being’. In the third section, I offer some guidelines on how this ‘critical being’ or spirit of criticality can be nurtured in the language classroom by guiding learners to (1) read to question and (2) write to challenge. In so doing, it is hoped that the abstract notion of criticality can be materialized and cultivated to encourage learners to appreciate how language use is intimately and inexorably associated with how they view the world and how they can use language to (re)shape, challenge and (re)constitute existing worldviews.

Visual Clue Guidance and Consistency Matching Framework for Multimodal Named Entity Recognition

The goal of multimodal named entity recognition (MNER) is to detect entity spans in given image–text pairs and classify them into corresponding entity types. Despite the success of existing works that leverage cross-modal attention mechanisms to integrate textual and visual representations, we observe three key issues. Firstly, models are prone to misguidance when fusing unrelated text and images. Secondly, most existing visual features are not enhanced or filtered. Finally, due to the independent encoding strategies employed for text and images, a noticeable semantic gap exists between them. To address these challenges, we propose a framework called visual clue guidance and consistency matching (GMF). To tackle the first issue, we introduce a visual clue guidance (VCG) module designed to hierarchically extract visual information from multiple scales. This information is utilized as an injectable visual clue guidance sequence to steer text representations for error-insensitive prediction decisions. Furthermore, by incorporating a cross-scale attention (CSA) module, we successfully mitigate interference across scales, enhancing the image’s capability to capture details. To address the third issue of semantic disparity between text and images, we employ a consistency matching (CM) module based on the idea of multimodal contrastive learning, facilitating the collaborative learning of multimodal data. To validate the effectiveness of our proposed framework, we conducted comprehensive experimental studies, including extensive comparative experiments, ablation studies, and case studies, on two widely used benchmark datasets, demonstrating the efficacy of the framework.

A Privacy-Preserving Multilingual Comparable Corpus Construction Method in Internet of Things

With the expansion of the Internet of Things (IoT) and artificial intelligence (AI) technologies, multilingual scenarios are gradually increasing, and applications based on multilingual resources are also on the rise. In this process, apart from the need for the construction of multilingual resources, privacy protection issues like data privacy leakage are increasingly highlighted. Comparable corpus is important in multilingual language information processing in IoT. However, the multilingual comparable corpus concerning privacy preserving is rare, so there is an urgent need to construct a multilingual corpus resource. This paper proposes a method for constructing a privacy-preserving multilingual comparable corpus, taking Chinese–Uighur–Tibetan IoT based news as an example, and mapping the different language texts to a unified language vector space to avoid sensitive information, then calculates the similarity between different language texts and serves as a comparability index to construct comparable relations. Through the decision-making mechanism of minimizing the impossibility, it can identify a comparable corpus pair of multilingual texts based on chapter size to realize the construction of a privacy-preserving Chinese–Uighur–Tibetan comparable corpus (CUTCC). Evaluation experiments demonstrate the effectiveness of our proposed provable method, which outperforms in accuracy rate by 77%, recall rate by 34% and F value by 47.17%. The CUTCC provides valuable privacy-preserving data resources support and language service for multilingual situations in IoT.

VL-Few: Vision Language Alignment for Multimodal Few-Shot Meta Learning

Complex tasks in the real world involve different modal models, such as visual question answering (VQA). However, traditional multimodal learning requires a large amount of aligned data, such as image text pairs, and constructing a large amount of training data is a challenge for multimodal learning. Therefore, we propose VL-Few, which is a simple and effective method to solve the multimodal few-shot problem. VL-Few (1) proposes the modal alignment, which aligns visual features into language space through a lightweight model network and improves the multimodal understanding ability of the model; (2) adopts few-shot meta learning in the multimodal problem, which constructs a few-shot meta task pool to improve the generalization ability of the model; (3) proposes semantic alignment to enhance the semantic understanding ability of the model for the task, context, and demonstration; (4) proposes task alignment that constructs training data into the target task form and improves the task understanding ability of the model; (5) proposes generation alignment, which adopts the token-level training and multitask fusion loss to improve the generation ability of the model. Our experimental results show the effectiveness of VL-Few for multimodal few-shot problems.

Authorship style transfer with inverse transfer data augmentation

Authorship style transfer aims to modify the style of neutral text to match the unique speaking or writing style of a particular individual. While Large Language Models (LLMs) present promising solutions, their effectiveness is limited by the small number of in-context learning demonstrations, particularly for authorship styles not frequently seen during pre-training. In response, this paper proposes an inverse transfer data augmentation (ITDA) method, leveraging LLMs to create (neutral text, stylized text) pairs. This method involves removing the existing styles from stylized texts, a process made more feasible due to the prevalence of neutral texts in pre-training. We use this augmented dataset to train a compact model that is efficient for deployment and adept at replicating the targeted style. Our experimental results, conducted across four datasets with distinct authorship styles, establish the effectiveness of ITDA over traditional style transfer methods and forward transfer using GPT-3.5. For further research and application, our dataset and code are openly accessible at https://github.com/AnonymousRole/ITDA.