Language Representation Research Articles

Comparing SMILES and SELFIES tokenization for enhanced chemical language modeling

Life sciences research and experimentation are resource-intensive, requiring extensive trials and considerable time. Often, experiments do not achieve their intended objectives, but progress is made through trial and error, eventually leading to breakthroughs. Machine learning is transforming this traditional approach, providing methods to expedite processes and accelerate discoveries. Deep Learning is becoming increasingly prominent in chemistry, with Convolutional Graph Networks (CGN) being a key focus, though other approaches also show significant potential. This research explores the application of Natural Language Processing (NLP) to evaluate the effectiveness of chemical language representations, specifically SMILES and SELFIES, using tokenization methods such as Byte Pair Encoding (BPE) and a novel approach developed in this study, Atom Pair Encoding (APE), in BERT-based models. The primary objective is to assess how these tokenization techniques influence the performance of chemical language models in biophysics and physiology classification tasks. The findings reveal that APE, particularly when used with SMILES representations, significantly outperforms BPE by preserving the integrity and contextual relationships among chemical elements, thereby enhancing classification accuracy. Performance was evaluated in downstream classification tasks using three distinct datasets for HIV, toxicology, and blood–brain barrier penetration, with ROC-AUC serving as the evaluation metric. This study highlights the critical role of tokenization in processing chemical language and suggests that refining these techniques could lead to significant advancements in drug discovery and material science.

Scale matters: Large language models with billions (rather than millions) of parameters better match neural representations of natural language.

Recent research has used large language models (LLMs) to study the neural basis of naturalistic language processing in the human brain. LLMs have rapidly grown in complexity, leading to improved language processing capabilities. However, neuroscience researchers haven't kept up with the quick progress in LLM development. Here, we utilized several families of transformer-based LLMs to investigate the relationship between model size and their ability to capture linguistic information in the human brain. Crucially, a subset of LLMs were trained on a fixed training set, enabling us to dissociate model size from architecture and training set size. We used electrocorticography (ECoG) to measure neural activity in epilepsy patients while they listened to a 30-minute naturalistic audio story. We fit electrode-wise encoding models using contextual embeddings extracted from each hidden layer of the LLMs to predict word-level neural signals. In line with prior work, we found that larger LLMs better capture the structure of natural language and better predict neural activity. We also found a log-linear relationship where the encoding performance peaks in relatively earlier layers as model size increases. We also observed variations in the best-performing layer across different brain regions, corresponding to an organized language processing hierarchy.

Treatment Recommendation using BERT Personalization

This research work develops a new framework that combines patient feedback with evidence-based best practices across disease states to improve drug recommendations. It employs BERT as its free-text processing engine to deal with sentiment judgment and classification. The functionality of the system, named `PharmaBERT`, includes acceptance of drug review data as a comprehensive input, drug categorization when dealing with a wide range of treatments and fine-tuning the BERT-based model for gaining positive or negative sentiment towards specific medications. PharmaBERT categorizes various drugs and fine-tunes the BERT structure to perceive lots of possible sentiments for specific medications. Consequently, PharmaBERT brings all its training and optimization capabilities together and through this, the system reaches a higher accuracy of up to 91% thus showcasing the potency of the model in capturing patient sentiments. While being a BERT spin-off, PharmaBERT utilizes its own set of experienced techniques to comprehend and sense the health-related text input given by the patient, doctor, or pharmacist. It uses transfer learning, that is, it learns from language representations to adapt quickly to the intricacies of drug reviewing. Through PharmaBERT, healthcare professionals may expand their diagnoses with insights from patient feedback to constitute more neutral decisions.

The relationship between white matter architecture and language lateralisation in the healthy brain.

Interhemispheric anatomical differences have long been thought to be related to language lateralisation. Previous studies have explored whether asymmetries in the diffusion characteristics of white matter language tracts are consistent with language lateralisation. These studies, typically with smaller cohorts, yielded mixed results. This study investigated whether connectomic analysis of quantitative anisotropy (QA) and shape features of white matter tracts across the whole brain are associated with language lateralisation. We analysed 1040 healthy individuals (562 females) from the Human Connectome Project database. Hemispheric language dominance for each participant was quantified using a laterality quotient (LQ) derived from fMRI activation in regions of interest (ROIs) associated with a language comprehension task compared against a math task. A linear regression model was used to examine the relationship between structural asymmetry and functional lateralisation. Connectometry revealed a significant negative correlation between LQs and QA of corpus callosum tracts, indicating that higher QA in these regions is associated with bilateral and right-hemisphere language representation in frontal and temporal regions, respectively. Left language laterality in temporal lobe was significantly associated with longer right inferior fronto-occipital fasciculus (IFOF) and forceps minor tracts. These results suggest that diffusion measures of microstructural architecture as well as geometrical features of reconstructed white matter tracts play a role in language lateralisation. People with increased dependence on right or both frontal hemispheres for language processing may have more developed commissural fibres, which may support more efficient interhemispheric communication.Significance statement The left cerebral hemisphere is dominant for language functions in most people. In some healthy people, language functions are lateralised to the right hemisphere or distributed across both hemispheres. The anatomy underlying patterns of hemispheric language dominance are not well established. Emerging evidence suggests that white matter connectivity and architecture is an important feature of cortical functional organisation. In this work, we report that people who have language functions distributed across both hemispheres have greater inter-hemispheric connectivity compared to lateralised people. Our findings provide further insights into the anatomical basis of language function and may have wider clinical implications.

Visual-guided hierarchical iterative fusion for multi-modal video action recognition

Vision-Language models(VLMs) have shown promising improvements on various visual tasks. Most existing VLMs employ two separate transformer-based encoders, each dedicated to modeling visual and language features independently. Because the visual features and language features are unaligned in the feature space, it is challenging for the multi-modal encoder to learn vision-language interactions. In this paper, we propose a V,isual-guided Hierarchical Iterative Fusion (VgHIF) method for VLMs in video action recognition, which acquires more discriminative vision and language representation. VgHIF leverages visual features from different levels in visual encoder to interact with language representation. The interaction is processed by the attention mechanism to calculate the correlation between visual features and language representation. VgHIF learns grounded video-text representation and supports many different pre-trained VLMs in a flexible and efficient manner with a tiny computational cost. We conducted experiments on the Kinetics-400 Mini Kinetics 200 HMDB51, and UCF101 using VLMs: CLIP, X-CLIP, and ViFi-CLIP. The experiments were conducted under full supervision and few shot settings, and compared with the baseline multi-modal model without VgHIF, the Top-1 accuracy of the proposed method has been improved to varying degrees, and several groups of results have achieved comparable results with state-of-the-art performance, which strongly verified the effectiveness of the proposed method.

Hippocampus supports long-term maintenance of language representations: Evidence of impaired collocation knowledge in amnesia

Traditional systems consolidation theories of memory suggest that the role of the hippocampus in maintaining memory representations diminishes over time, with learned information eventually becoming fully independent of the hippocampus. Knowledge of collocations in one’s native (L1) language are acquired during development and are solidly acquired by adulthood. Remote semantic knowledge of collocations might therefore be expected to be resistant to hippocampal pathology. Patients with hippocampal damage and severe anterograde amnesia completed two tasks testing English collocation knowledge originally designed for use with English language learners. Patients with hippocampal damage demonstrated impairments in recognition of common English collocations, despite a lifetime of language experience (including postsecondary education) prior to sustaining this damage. These results suggest the hippocampus contributes to the long-term maintenance of linguistic representations and provides a challenge to traditional consolidation views of memory and an extension of newer theories to include a role for the hippocampus in maintaining semantic memory.

Early language learning in Croatia: Exploring foreign language representation and participation trends in kindergartens (2008/09 – 2022/23)

Early language learning in Croatia: Exploring foreign language representation and participation trends in kindergartens (2008/09 – 2022/23)

Neural representations of concreteness and concrete concepts are specific to the individual.

Different people listening to the same story may converge upon a largely shared interpretation while still developing idiosyncratic experiences atop that shared foundation. What linguistic properties support this individualized experience of natural language? Here, we investigate how the "concrete-abstract" axis - i.e., the extent to which a word is grounded in sensory experience - relates to within- and across-subject variability in the neural representations of language. Leveraging a dataset of human participants of both sexes who each listened to four auditory stories while undergoing functional MRI, we demonstrate that neural representations of "concreteness" are both reliable across stories and relatively unique to individuals, while neural representations of "abstractness" are variable both within individuals and across the population. Using natural language processing tools, we show that concrete words exhibit similar neural representations despite spanning larger distances within a high-dimensional semantic space, which potentially reflects an underlying representational signature of sensory experience - namely, imageability - shared by concrete words but absent from abstract words. Our findings situate the concrete-abstract axis as a core dimension that supports both shared and individualized representations of natural language.Significance Statement The meaning of spoken language is often ambiguous. As a result, people may form different interpretations despite being presented with the same information. What properties of language does the brain leverage to form this diverse, individual experience? Analyses of functional MRI data demonstrated that "concreteness", the extent to which language is related to sensory experience, evoked reliable neural patterns that were unique to individual subjects and allowed us to identify individuals solely based on their neural data. Application of machine learning methods showed that sets of concrete concepts, but not abstract concepts, show stable neural patterns, potentially due to a sensory signature: imageability. Overall, this study characterizes concreteness as a central property supporting the individualized experience of real-world language.

A Formal Language for Performance Evaluation Based on Reinforcement Learning

Temporal Logics are a rich variety of logical systems designed for specifying properties over time, and about events and changes in the world over time. Traditional temporal logic, however, is limited to binary outcomes true or false and lacks the capacity to specify performance properties of a system such as the maximum, minimum, or average costs between states. Current languages do not accommodate the quantification of such performance properties, especially in scenarios involving infinite execution paths where performance property like cumulative sums may fail to converge. To this end, this paper introduces a novel formal language aimed at assessing system performance, which encapsulates not only temporal dynamics but also various performance-related properties. In this study, this paper utilizes reinforcement learning techniques to compute the values of performance property formulas. Finally, in the experimental part, a formal language representation of system performance properties was implemented, and the values of the performance property formulas were computed using reinforcement learning. The effectiveness and feasibility of the proposed method were validated.

Accurate minority Language Representation in the Public Space: the case of Arabic in Israel

ABSTRACT Although Arabic is the native language of a fifth of the Israeli population, its representation in the Israeli linguistic landscape (LL) is problematic. Adopting a sociolinguistic framework that underscores the political nature of language (Ben-Rafael 2009), this study invited Palestinian-Israeli college students to reflect on the linguistic reality in their area through the participatory documentary photography tool ‘PhotoVoice’. The students captured photos of linguistic landscapes, analysed them, and wrote commentaries voicing their reflections. Students ‘PhotoVoices’ shed light on the abundance of mistakes in Arabic words on the signs. Content analysis of the ‘PhotoVoices’ on this issue highlighted students’ cognitive and emotional perceptions and emerging awareness of the explicit and implicit messages delivered through the signage.

Language Learning, Representation, and Processing in Humans and Machines: Introduction to the Special Issue

Abstract Large Language Models (LLMs) and humans acquire knowledge about language without direct supervision. LLMs do so by means of specific training objectives, while humans rely on sensory experience and social interaction. This parallelism has created a feeling in NLP and cognitive science that a systematic understanding of how LLMs acquire and use the encoded knowledge, could provide useful insights for studying human cognition. Conversely, methods and findings from the field of cognitive science have occasionally inspired language model development. Yet, the differences in the way that language is processed by machines and humans—in terms of learning mechanisms, amounts of data used, grounding and access to different modalities—make a direct translation of insights challenging. The aim of this edited volume has been to create a forum of exchange and debate along this line of research, inviting contributions that further elucidate similarities and differences between humans and LLMs.

Funny words on the screen: Exploring linguistic authority through subtitling practices

AbstractLinguistic authority, though generally understood as the right claimed by some people and institutions to regiment language use and representation, has also been conceptualized as the power of languages to command respect and attention from community members. To explore these two aspects of linguistic authority, this article examines how the producers of a popular Chinese talk show use on‐screen text to construct authority both for themselves and for Putonghua (standard Mandarin) in situated moments of interaction. It focuses on an episode of the show in which the Putonghua‐speaking host interviews a Hong Kong actor/director known for his Gangpu (Hong Kong Mandarin). Through the strategic use of traditional subtitles and impact captions, the show's producers position themselves as anonymous listening subjects who not only provide running commentary on what viewers hear, but also contrast Putonghua with Gangpu and Cantonese, and affirm its legitimacy by presenting it as the unmarked, anonymous language against which these minoritized varieties are compared. To fully understand the (de)legitimation of linguistic authority in media productions, we need to consider both aspects of linguistic authority, examine how they are connected to each other, and attend to the array of contrasting relations that subtitling practices create among linguistic varieties.

Unveiling dynamics of language visibility and vitality in café menus near Labrang Monastery.

This study explores the dynamics of Linguistic Landscapes (LL) in the commercial tourist district near Labrang Monastery by analyzing the use of Tibetan, Mandarin, and English on café menus. Traditionally, LL research has focused on language visibility as an indicator of vitality, overlooking contextual complexities. We challenge this approach by considering geocultural contexts, instigators' language proficiency, and intended audience. This study employed a triangulation approach, combining multiple methods of data collection and analysis. First, we conducted a preliminary observation of shop names in the region. Second, with the consent of shop owners, we comprehensively documented visual and textual artifacts both inside and outside the cafés, including signboards, menus, wall billboards, and displayed books. Third, we conducted extensive semi-structured interviews with shop owners and customers. These interviews, transcribed for analysis, provided insights into language usage preferences and decision-making processes. By advocating for a holistic LL research approach, integrating qualitative insights with quantitative data, this study contributes to a deeper understanding of LL dynamics. Our findings show that geocultural contexts, instigators' language proficiency, and intended audience all play a role in shaping language representation and vitality. The absence of a language on signage does not necessarily signify diminished vitality but can reflect strategic decisions influenced by religious, individual and commercial factors. Beyond mere visibility, languages on café menus serve as symbolic markers of ethnic identity and reflect the functions they assume within the speech community, offering insights into language vitality across different usage contexts. This research enriches scholarly discourse on LL, particularly within the Chinese context, by emphasizing the multifaceted nature of language presence and its vitality.

Raising awareness to the representation of a minority language through engaging pedagogies

ABSTRACT Grounded in critical pedagogies, this study invites Palestinian-Israeli students to reflect on the limited representation of their indigenous language in the Israeli linguistic landscape (LL). Using the participatory-documentary tool “PhotoVoice,” they capture photos of signage in their area, analyze following a set of guiding questions, and share their findings. Content analysis of students’ “PhotoVoices” and follow-up reflections sheds light on their perceptions of the linguistic reality in the area and their emerging awareness and critique. The contribution of the study lies in demonstrating the pedagogical potential of the innovative tool PhotoVoice in engaging local inhabitants in exploration endeavors into linguistic landscapes and developing their awareness to representation of minority languages in the public space.

Large Language Models to Help Appeal Denied Radiotherapy Services

PURPOSE Large language model (LLM) artificial intelligences may help physicians appeal insurer denials of prescribed medical services, a task that delays patient care and contributes to burnout. We evaluated LLM performance at this task for denials of radiotherapy services. METHODS We evaluated generative pretrained transformer 3.5 (GPT-3.5; OpenAI, San Francisco, CA), GPT-4, GPT-4 with internet search functionality (GPT-4web), and GPT-3.5ft. The latter was developed by fine-tuning GPT-3.5 via an OpenAI application programming interface with 53 examples of appeal letters written by radiation oncologists. Twenty test prompts with simulated patient histories were programmatically presented to the LLMs, and output appeal letters were scored by three blinded radiation oncologists for language representation, clinical detail inclusion, clinical reasoning validity, literature citations, and overall readiness for insurer submission. RESULTS Interobserver agreement between radiation oncologists' scores was moderate or better for all domains (Cohen's kappa coefficients: 0.41-0.91). GPT-3.5, GPT-4, and GPT-4web wrote letters that were on average linguistically clear, summarized provided clinical histories without confabulation, reasoned appropriately, and were scored useful to expedite the insurance appeal process. GPT-4 and GPT-4web letters demonstrated superior clinical reasoning and were readier for submission than GPT-3.5 letters ( P < .001). Fine-tuning increased GPT-3.5ft confabulation and compromised performance compared with other LLMs across all domains ( P < .001). All LLMs, including GPT-4web, were poor at supporting clinical assertions with existing, relevant, and appropriately cited primary literature. CONCLUSION When prompted appropriately, three commercially available LLMs drafted letters that physicians deemed would expedite appealing insurer denials of radiotherapy services. LLMs may decrease this task's clerical workload on providers. However, LLM performance worsened when fine-tuned with a task-specific, small training data set.

Revisiting Empathy, Motivational Systems, and a Theory of Cure

ABSTRACT Lichtenberg’s chapter on “Empathy, Motivational Systems, and a Theory of Cure” was a brilliant effort to integrate his new paradigm of multimotivational systems with Kohut’s empathic approach to psychotherapy. Lichtenberg shows how the clinical dialogue with patients can be monitored by using empathic perception to track shifts in motivational systems. At any given moment, a particular motivational system may become dominant. Successfully identifying the dominant motivation through empathic perception creates in patients a feeling of being understood without judgment. This ambience of safety allows for a shift that activates an exploratory motivation with patients and a spirit of inquiry. An observational platform is created where experience from the past with attachment figures and interaction in the here and now in therapy can be explored from new perspectives. Recent research in psychotherapy that uses monitoring the clinical dialogue with patients by tracking shifts in motivational systems supports the view of its positive effects. Lichtenberg’s “theory of cure” is based on the premise that an empathic perception that monitors shifts in motivational systems will mobilize intrinsic self-righting capacities in patients. I note that there is controversy about what mutative factors in psychotherapy are most important: building an ambience of safety and trust or creating new insights into personal and internal conflicts through the use of language and symbolic representations. Both factors are crucial for positive changes in psychotherapy, and their importance will vary from patient to patient and from one patient–therapist relation to another. I note that Wilma Bucci’s multiple code theory that sees experience coded at subsymbolic and verbal symbolic levels supports this view. I close by proposing that we can build on Lichtenberg’s multimotivational system by adding a trans-survival need to give meaning to our lives.

Double-Ended Bit-Stealing for Algebraic Data Types

Algebraic data types are central to the development and evaluation of most functional programs. It is therefore important for compilers to choose compact and efficient representations of such types, in particular to achieve good memory footprints for applications. Algebraic data types are most often represented using blocks of memory where the first word is used as a so-called tag, carrying information about the constructor, and the following words are used for carrying the constructor's arguments. As an optimisation, lists are usually represented more compactly, using a technique called bit-stealing, which, in its simplest form, uses the word-alignment property of pointers to byte-addressed allocated memory to discriminate between the nil constructor (often represented as 0x1) and the cons constructor (aligned pointer to allocated pair). Because the representation supports that all values can be held uniformly in one machine word, possibly pointing to blocks of memory, type erasure is upheld. However, on today's 64-bit architectures, memory addresses (pointers) are represented using only a subset of the 64 bits available in a machine word, which leave many bits unused. In this paper, we explore the use, not only of the least-significant bits of pointers, but also of the most-significant bits, for representing algebraic data types in a full ML compiler. It turns out that, with such a particular utilisation of otherwise unused bits, which we call double-ended bit-stealing, it is possible to choose unboxed representations for a large set of data types, while still not violating the principle of uniform data-representations. Examples include Patricia trees, union-find data structures, stream data types, internal language representations for types and expressions, and mutually recursive ASTs for full language definitions. The double-ended bit-stealing technique is implemented in the MLKit compiler and speedup ranges from 0 to 26 percent on benchmarks that are influenced by the technique. For MLKit, which uses abstract data types extensively, compilation speedups of around 9 percent are achieved for compiling MLton (another Standard ML compiler) and for compiling MLKit itself.

Early language experience and modality affect parietal cortex activation in different hemispheres: Insights from hearing bimodal bilinguals

The goal of this study was to investigate the impact of the age of acquisition (AoA) on functional brain representations of sign language in two exceptional groups of hearing bimodal bilinguals: native signers (simultaneous bilinguals since early childhood) and late signers (proficient sequential bilinguals, who learnt a sign language after puberty). We asked whether effects of AoA would be present across languages – signed and audiovisual spoken – and thus observed only in late signers as they acquired each language at different life stages, and whether effects of AoA would be present during sign language processing across groups. Moreover, we aimed to carefully control participants’ level of sign language proficiency by implementing a battery of language tests developed for the purpose of the project, which confirmed that participants had high competences of sign language.Between-group analyses revealed a hypothesized modulatory effect of AoA in the right inferior parietal lobule (IPL) in native signers, compared to late signers. With respect to within-group differences across languages we observed greater involvement of the left IPL in response to sign language in comparison to spoken language in both native and late signers, indicating language modality effects. Overall, our results suggest that the neural underpinnings of language are molded by the linguistic characteristics of the language as well as by when in life the language is learnt.

Patterns of language

ABSTRACT Neuroimaging studies have increasingly leveraged the information in multivariate patterns of brain activity, transitioning from voxel-by-voxel activation comparisons to multi-voxel pattern analysis (MVPA) and decoding approaches. Representational Similarity Analysis (RSA) has further advanced this field, enabling comparisons of representational structures across neuroimaging modalities and computational models, in particular in fMRI and EEG/MEG research. Recent applications have extended to brain connectivity estimation, enhancing our understanding of neural representations in language and cognition. This special issue, “Patterns Of Language”, showcases recent applications and methodological developments of multivariate analysis approaches in the neuroscience of language.

Enhancing the vision–language foundation model with key semantic knowledge-emphasized report refinement

Recently, vision–language representation learning has made remarkable advancements in building up medical foundation models, holding immense potential for transforming the landscape of clinical research and medical care. The underlying hypothesis is that the rich knowledge embedded in radiology reports can effectively assist and guide the learning process, reducing the need for additional labels. However, these reports tend to be complex and sometimes even consist of redundant descriptions that make the representation learning too challenging to capture the key semantic information. This paper develops a novel iterative vision–language representation learning framework by proposing a key semantic knowledge-emphasized report refinement method. Particularly, raw radiology reports are refined to highlight the key information according to a constructed clinical dictionary and two model-optimized knowledge-enhancement metrics. The iterative framework is designed to progressively learn, starting from gaining a general understanding of the patient’s condition based on raw reports and gradually refines and extracts critical information essential to the fine-grained analysis tasks. The effectiveness of the proposed framework is validated on various downstream medical image analysis tasks, including disease classification, region-of-interest segmentation, and phrase grounding. Our framework surpasses seven state-of-the-art methods in both fine-tuning and zero-shot settings, demonstrating its encouraging potential for different clinical applications.