Domain-specific Tasks Research Articles

Exploring the effectiveness of instruction tuning in biomedical language processing

Large Language Models (LLMs), particularly those similar to ChatGPT, have significantly influenced the field of Natural Language Processing (NLP). While these models excel in general language tasks, their performance in domain-specific downstream tasks such as biomedical and clinical Named Entity Recognition (NER), Relation Extraction (RE), and Medical Natural Language Inference (NLI) is still evolving. In this context, our study investigates the potential of instruction tuning for biomedical language processing, applying this technique to two general LLMs of substantial scale. We present a comprehensive, instruction-based model trained on a dataset that consists of approximately 200,000 instruction-focused samples. This dataset represents a carefully curated compilation of existing data, meticulously adapted and reformatted to align with the specific requirements of our instruction-based tasks. This initiative represents an important step in utilising such models to achieve results on par with specialised encoder-only models like BioBERT and BioClinicalBERT for various classical biomedical NLP tasks. Our work includes an analysis of the dataset’s composition and its impact on model performance, providing insights into the intricacies of instruction tuning. By sharing our codes, models, and the distinctively assembled instruction-based dataset, we seek to encourage ongoing research and development in this area.22Our code repository is available at https://github.com/nlpie-research/BioInstTune-LLM.

Deficits in or Preservation of Basic Number Processing in Parkinson's Disease? A Registered Report.

Neurodegenerative diseases such as Parkinson's disease (PD) have a huge impact on patients, caregivers, and the health care system. Until now, diagnosis of mild cognitive impairments in PD has been established based on domain-general functions such as executive functions, attention, or working memory. However, specific numerical deficits observed in clinical practice have not yet been systematically investigated. PD-immanent deterioration of domain-general functions and domain-specific numerical areas suggests mechanisms of both primary and secondary dyscalculia. The current study systematically investigated basic number processing performance in PD patients for the first time, targeting domain-specific cognitive representations of numerosity and the influence of domain-general factors. The overall sample consisted of patients with a diagnosis of PD, according to consensus guidelines, and healthy controls. PD patients were stratified into patients with normal cognition (PD-NC) or mild cognitive impairment (level I-PD-MCI based on cognitive screening). Basic number processing was assessed using transcoding, number line estimation, and (non-) symbolic number magnitude comparison tasks. Discriminant analysis was employed to assess whether basic number processing tasks can differentiate between a healthy control group and both PD groups. All participants were subjected to a comprehensive numerical and a neuropsychological test battery, as well as sociodemographic and clinical measures. Results indicate a profile of preserved (verbal representation) and impaired (magnitude representation, place × value activation) function in PD-MCI, hinting at basal ganglia dysfunction affecting numerical cognition in PD. Numerical deficits could not be explained by domain-general cognitive impairments, so that future research needs to incorporate domain-specific tasks of sufficient difficulty.

From pre-training to fine-tuning: An in-depth analysis of Large Language Models in the biomedical domain

In this study, we delve into the adaptation and effectiveness of Transformer-based, pre-trained Large Language Models (LLMs) within the biomedical domain, a field that poses unique challenges due to its complexity and the specialized nature of its data. Building on the foundation laid by the transformative architecture of Transformers, we investigate the nuanced dynamics of LLMs through a multifaceted lens, focusing on two domain-specific tasks, i.e., Natural Language Inference (NLI) and Named Entity Recognition (NER). Our objective is to bridge the knowledge gap regarding how these models’ downstream performances correlate with their capacity to encapsulate task-relevant information. To achieve this goal, we probed and analyzed the inner encoding and attention mechanisms in LLMs, both encoder- and decoder-based, tailored for either general or biomedical-specific applications. This examination occurs before and after the models are fine-tuned across various data volumes. Our findings reveal that the models’ downstream effectiveness is intricately linked to specific patterns within their internal mechanisms, shedding light on the nuanced ways in which LLMs process and apply knowledge in the biomedical context. The source code for this paper is available at https://github.com/agnesebonfigli99/LLMs-in-the-Biomedical-Domain.

How Does a Generative Large Language Model Perform on Domain-Specific Information Extraction?─A Comparison between GPT-4 and a Rule-Based Method on Band Gap Extraction.

The advent of generative Large Language Models (LLMs) has greatly impacted the field of Natural Language Processing. However, it is inconclusive how generative LLMs perform on domain-specific information extraction tasks. This study compares the performance of GPT-4 and a rule-based information extraction method based on ChemDataExtractor on band gap information extraction, a task that has important implications for the materials science domain. No training data is required for either method, which is desirable because there is a lack of training data in the materials science domain compared with a variety of material information that is of interest. Manual evaluation on 415 randomly selected articles showed that the GPT-4 model achieved a higher level of accuracy in extracting materials' band gap information than the rule-based method (Correctness 87.95% vs 51.08%, Partial correctness 11.33% vs 36.87%, incorrectness 0.72% vs 12.05%). Further analysis of the errors reveals the strengths and weaknesses of the GPT-4 model compared to the rule-based method. The GPT-4 model shows stronger performance in interdependency resolution and complicated material name recognition, while it also has weaknesses in hallucination, identifying band gap values, and identifying band gap types. Revised prompt based on the error analysis leads to improved accuracy for GPT-4. To the best of our knowledge, this study is the first to compare the GPT-4 model and ChemDataExtractor for the band gap extraction task. This study provides evidence to support using generative LLMs for domain-specific information extraction tasks.

Beyond clean data: Exploring the effects of label noise on object detection performance

In recent years, the growth of large-scale datasets has significantly propelled the progress of deep learning applications. Yet, annotating these datasets remains a labor-intensive endeavor, pushing the reliance on cost-effective but less specialized data collection methods and internet data sources. This often results in noisy and inaccurate labels, compromising data quality. Traditional machine learning models assume clean data, but real-world datasets often exhibit significant label noise. This paper examines the impact of such noise on object detection performance, a pivotal aspect of computer vision. We analyze the influence of noisy labels using three renowned object detection frameworks: YOLOv5, Faster R-CNN, and the recent YOLOv8, alongside established datasets: MS COCO, VOC, and ExDARK. Additionally, experiments with the UVM dataset explore domain-specific tasks in dense object scenarios. Two new metrics — Model Health and Detection Capability — were introduced to evaluate the results. Findings indicate that models maintain over 80% of their health (a 20% decline in mAP from the baseline) with up to 40% label corruption. However, Detection Capability deteriorates more sharply under the same conditions. The research also employs the D-RISE method for model explainability, highlighting crucial image regions affecting detection outcomes. Despite the noise, critical detection areas in models remain similar to those in clean data up to the 40% corruption level, as verified by similarity metrics. This study underscores the resilience of object detection models to label noise and provides insights into maintaining performance amidst data quality challenges.

Self-supervised learning on MeerKAT wide-field continuum images

Context. Self-supervised learning (SSL) applied to natural images has demonstrated a remarkable ability to learn meaningful, low-dimension representations without labels, resulting in models that are adaptable to many different tasks. Until now, applications of SSL to astronomical images have been limited to Galaxy Zoo datasets, which require a significant amount of preprocessing to prepare sparse images centered on a single galaxy. With wide-field survey instruments at the forefront of the Square Kilometer Array (SKA) era, this approach to gathering training data is impractical. Aims. We demonstrate that continuum images from surveys such as the MeerKAT Galactic Cluster Legacy Survey (MGCLS) can be successfully used with SSL, without extracting single-galaxy cutouts. Methods. Using the SSL framework DINO, we experimented with various preprocessing steps, augmentations, and architectures to determine the optimal approach for this data. We trained both ResNet50 and Vision Transformer (ViT) backbones. Results. Our models match state-of-the-art results (trained on Radio Galaxy Zoo) for FRI/FRII morphology classification. Furthermore, they predict the number of compact sources via linear regression with much higher accuracy. Open-source foundation models trained on natural images such as DINOv2 also excel at simple FRI/FRII classification; the advantage of domain-specific backbones is much smaller models trained on far less data. Smaller models are more efficient to fine-tune, and doing so results in a similar performance between our models, the state-of-the-art, and open-source models on multi-class morphology classification. Conclusions. Using source-rich crops from wide-field images to train multi-purpose models is an easily scalable approach that significantly reduces data preparation time. For the tasks evaluated in this work, twenty thousand crops is sufficient training data for models that produce results similar to state-of-the-art. In the future, complex tasks like source detection and characterization, together with domain-specific tasks, ought to demonstrate the true advantages of training models with radio astronomy data over natural-image foundation models.

Comparative Analysis of Single and Multiagent Large Language Model Architectures for Domain-Specific Tasks in Well Construction

Summary This paper explores the application of large language models (LLMs) in the oil and gas (O&G) sector, specifically within well construction and maintenance tasks. The study evaluates the performances of a single agent and a multiagent LLM-based architecture in processing different tasks, offering a comparative perspective on their accuracy and the cost implications of their implementation. The results indicate that multiagent systems offer improved performance in question and answer (Q&A) tasks, with a truthfulness measure 28% higher than single-agent systems but at a higher financial cost. Specifically, the multiagent architecture incurs costs that are, on average, 3.7 times higher than those of the single-agent setup due to the increased number of tokens processed. Conversely, single-agent systems excel in Text-to-SQL (structured query language) tasks, particularly when using the Generative Pre-Trained Transformer 4 (GPT-4), achieving a 15% higher score compared to multiagent configurations, suggesting that simpler architectures can sometimes outpace complexity. The novelty of this work lies in its original examination of the specific challenges presented by the complex, technical, unstructured data inherent in well construction operations, contributing to strategic planning for adopting generative artificial intelligence (AI) (Gen-AI) applications and providing a basis for optimizing solutions against economic and technological parameters.

Language Model Adaptation for Legal Ukrainian Domain

Language models in recent decades make a huge step towards solving the tasks that previously could be done only by humans. Development of NLP area is different scopes gives an opportunity to solve domain specific tasks and transfer knowledge from learnt data towards the useful inferences based on that. This article provides the NLP model approach in specific legal domain. Additionally, this article explores performance of pre-training small models and its utilization and checks the scores on fine-tuned task of checking sentence similarities via SBERT. According to this articles it is proven that domain-specific pre-trained models can perform better results than generally trained language model. This article also provides the language model that is adopted to the Ukrainian legal domain.

PreparedLLM: effective pre-pretraining framework for domain-specific large language models

ABSTRACT The direct application of large language models (LLMs) to specific domain tasks frequently encounters challenges due to the scarcity of domain data, variations in domain semantics, and the complexity of domain knowledge. Further pretraining of advanced foundational models on extensive domain-specific corpora can infuse these models with domain-specific knowledge and enhancing their ability of solving domain-specific tasks. However, the development of most domain-specific models focuses primarily on collecting large-scale domain data, often overlooking the crucial optimization of the pre-pretraining stage, which significantly impacts both model performance and training efficiency. This paper introduces PRE -PretrAining FRamEwork for Domain-specific Large Language Models (PreparedLLM), a framework designed to enhance the pre-pretraining stage for domain specialization of LLMs. PreparedLLM employs advanced techniques in data recipe, data cleaning, vocabulary expansion, and embedding initialization. These techniques are implemented to optimize both the composition and quality of the training data, and to enhance understanding of domain terminologies and concepts, as well as improve token embedding initialization. Utilizing the geoscience domain as a case study, this paper applies PreparedLLM for the domain specialization of the Llama, a widely recognized general-purpose LLM. Experimental results demonstrate that PreparedLLM enhances model convergence speed, training speed, inference speed, the text volume of the context window, and overall performance in domain specialization. The utilization of PreparedLLM in developing domain-specific LLMs has significantly increased performance while reducing both time and resource investment. The case study provides valuable insights into the development of domain-specific LLMs.

Mathematics teachers’ multiple perspectives on adaptive tasks: task evaluation and selection as core practices for teaching quality

The selection of tasks based on the evaluation of task features can be considered a core practice of teaching and a relevant component of teaching quality. This is typically part of teachers’ preparation for their classroom teaching, which prompts the following question: What are the characteristics of the tasks that teachers use when selecting tasks for differentiated teaching? To answer this question, we analyzed systematic differences in the focus of 78 in-service high school and lower secondary school teachers during the evaluation of task features. The teachers had to select eight tasks about the practice of fractions with respect to their differentiation potential—operationalizing their adaptive teaching competence from a mathematics educational perspective. To analyze the differences, we performed a cluster analysis of the task features that the teachers drew upon. Three groups of teachers could be identified with variations in their focus on directly or indirectly relevant, domain-specific or domain-general task features. Taking into account such variations may explain differences in teaching quality and student outcomes and may be relevant when designing teacher professional development programs.

Factor Analysis Study of the Achievement Goal Framework in the Domain-Specific Task of EFL Writing

Factor Analysis Study of the Achievement Goal Framework in the Domain-Specific Task of EFL Writing

AgXQA: A benchmark for advanced Agricultural Extension question answering

Large language models (LLMs) have revolutionized various scientific fields in the past few years, thanks to their generative and extractive abilities. However, their applications in the Agricultural Extension (AE) domain remain sparse and limited due to the unique challenges of unstructured agricultural data. Furthermore, mainstream LLMs excel at general and open-ended tasks but struggle with domain-specific tasks. We proposed a novel QA benchmark dataset, AgXQA, for the AE domain to address these issues. We trained and evaluated our domain-specific LM, AgRoBERTa, which outperformed other mainstream encoder- and decoder- LMs, on the extractive QA downstream task by achieving an EM score of 55.15% and an F1 score of 78.89%. Besides automated metrics, we also introduced a custom human evaluation metric, AgEES, which confirmed AgRoBERTa’s performance, as demonstrated by a 94.37% agreement rate with expert assessments, compared to 92.62% for GPT 3.5. Notably, we conducted a comprehensive qualitative analysis, whose results provide further insights into the weaknesses and strengths of both domain-specific and general LMs when evaluated on in-domain NLP tasks. Thanks to this novel dataset and specialized LM, our research enhanced further development of specialized LMs for the agriculture domain as a whole and AE in particular, thus fostering sustainable agricultural practices through improved extractive question answering.

Puppet Dynasty Recognition System Based on MobileNetV2.

Traditional image classification usually relies on manual feature extraction; however, with the rapid development of artificial intelligence and intelligent vision technology, deep learning models such as CNNs can automatically extract key features from input images to achieve efficient classification. This study focuses on the application of lightweight separable convolutional neural networks in domain-specific image classification tasks. In this paper, we discuss how to use the SSDLite object detection algorithm combined with the MobileNetV2 lightweight convolutional architecture for puppet dynasty recognition from images-a novel and challenging task. By constructing a system that combines object detection and image classification, we aimed to solve the problem of automatic puppet dynasty recognition to reduce manual intervention and improve recognition efficiency and accuracy. We hope that this will have significant implications in the fields of cultural protection and art history research.

Exploring different content creation and display methods for remote collaboration supported by eXtended reality: comparative analysis of distinct task scenarios

AbstractRemote collaboration using eXtended Reality (XR) has been explored to establish a common ground between physically distributed individuals. To achieve usable and impactful solutions, it is paramount to understand how different content creation and display methods contribute to the work effort of each member (remote and on-site). Additionally, explore how various task scenarios influence the collaborative process, specifically examining how activities with unique characteristics and complexities affect remote work. Hence, the question arises, ’How do different display and interaction methods impact the collaborative process within specific task domains?’ In this paper, two user studies with 30 distinct participants each are described, focusing on different content creation support (Laptop Computer; Video Wall & Keyboard; Interactive Projector) and display methods (Hand-Held Device (HHD); HHD & Articulated Support; Head-Mounted Display (HMD)) for remote and on-site collaborators respectively, during scenarios of remote guidance. Plus, different physical tasks were considered for each study (Lego pieces assembly; Tangram puzzle assembly; Maintenance procedures; Resource management in a map; Training activity in a laboratory classroom), to understand which method stands out according to the characteristics of the said tasks. We report the results obtained, suggesting that for the remote part of the collaborative process, using a Video Wall & Keyboard appears to be more suitable for tasks associated with maintenance procedures, learning, and training activities. As for the on-site part, using HMD was considered the better option for maintenance, and learning tasks, closely followed by HHD & Articulated Support.

Unlocking underrepresented use-cases for large language model-driven human-robot task planning

Large language models (LLM) are now the de facto task planners for Embodied AI (EAI) systems. This shift can be attributed to LLMs' powerful, emergent properties which enable their adaptation to downstream tasks with minimal to no fine tuning via prompting. However, we find that LLM-driven task planning is not a solved problem. In this work we measure the extent to which these models can be adapted to complex and domain-specific task planning via few-shot prompting. Additionally, we contribute quantitative and qualitative analysis on prompt robustness. Lastly, to meet the challenges of adapting EAI systems to real-world, industrial domains, we adopt a human-in-the-loop approach to guarantee safe and interpretable task planning and execution. We successfully demonstrate co-located, human-robot teaming where an Augmented Reality (AR) headset mediates information exchanged between an EAI agent and human operator for a variety of inspection tasks. To our knowledge the use of an AR headset for multimodal grounding and the application of EAI to industrial tasks are novel contributions within Embodied AI research.

Coding attitudes of fourth-grade latinx students during distance learning

ABSTRACT Background and Context Despite the growing initiatives in K-12 computer science (CS), there is a continued disparity in the participation of Latinx and multilingual students, a historically underrepresented group in computing. The inequitable participation may be understood by examining students’ early development of CS attitudes. Objective This study aims to explore shifts and elicitation in coding attitudes of fourth-grade, Latinx students (ages 9-10) who underwent a year-long remote coding curriculum, with consideration of gender and language designation. Method Using a mixed method approach, pre-post survey responses on coding attitudes were analyzed to understand shifts and portrayal of Interest, Confidence, Utility, Social Values, and Perception of Coders, with consideration of gender or designation as an English Language Learner. Findings Gender and language designation did not interact with overall attitude shifts. However, there is a significant difference in Social Values and Confidence over time. Student interviews revealed more nuance in social influences with siblings and cousins as key motivators for extended learning, underlying values of perseverance in confidence, mixed perception of what coders do, and the importance of creativity to develop interest. Implications The key role of social influence in driving higher identity among Latinx students points to the important role of extended family. Moreso, developing students’ confidence in domain-specific tasks should be a focus in curricula in order to have a longer-term impact on motivation. Finally, more research on the role that subjective task values pertaining to cultural values should be explored in early coding motivation in order to broaden participation. Keywords: Coding attitudes, elementary students, remote learning, expectancy-value theory, coding curriculum.

Advancing language models through domain knowledge integration: a comprehensive approach to training, evaluation, and optimization of social scientific neural word embeddings

This article proposes a comprehensive strategy for training, evaluating, and optimizing domain-specific word2vec-based word embeddings, using social science literature as an example. Our primary objectives are: (1) to train the embeddings utilizing a corpus of social science text, (2) to test their performance against domain-unspecific embeddings using our developed intrinsic and extrinsic evaluation strategy, and (3) to enhance their performance even further by using domain knowledge. As an integral part of this approach, we present SociRel-461, a domain-knowledge dictionary designed for the intrinsic evaluation and subsequent refinement of social science word embeddings. Using a dataset of 100,000 full-text scientific articles in sociology, we train multiple vector space models, which we then benchmark against a larger, pre-trained general language embedding model as part of our extrinsic evaluation. Furthermore, we developed a transfer learning multi-label classification task for extrinsic evaluation. Our findings reveal that domain-specific embeddings outperform their domain-unspecific counterparts in both intrinsic and extrinsic evaluations. We also investigated the retrofitting post-processing method to enhance domain-unspecific embeddings with the domain knowledge embedded in SociRel-461. While retrofitting does not enhance our domain-specific vector space models, it significantly improves the performance of the domain-unspecific embeddings. This highlights the potential of retrofitting for the transfer of domain knowledge to domain-unspecific embeddings. Our results emphasize the importance of utilizing domain-specific word embeddings for better performance in domain specific transfer learning tasks, as they outperform conventional embeddings trained on everyday language.

Predicting high-level visual areas in the absence of task fMRI.

The ventral visual stream is organized into units, or functional regions of interest (fROIs), specialized for processing high-level visual categories. Task-based fMRI scans ("localizers") are typically used to identify each individual's nuanced set of fROIs. The unique landscape of an individual's functional activation may rely in large part on their specialized connectivity patterns; recent studies corroborate this by showing that connectivity can predict individual differences in neural responses. We focus on the ventral visual stream and ask: how well can an individual's resting state functional connectivity localize their fROIs for face, body, scene, and object perception? And are the neural processors for any particular visual category better predicted by connectivity than others, suggesting a tighter mechanistic relationship between connectivity and function? We found, among 18 fROIs predicted from connectivity for each subject, all but one were selective for their preferred visual category. Defining an individual's fROIs based on their connectivity patterns yielded regions that were more selective than regions identified from previous studies or atlases in nearly all cases. Overall, we found that in the absence of a domain-specific localizer task, a 10-min resting state scan can be reliably used for defining these fROIs.

Enhanced Cognitive Inhibition in Table Tennis Athletes: Insights from Color-Word and Spatial Stroop Tasks.

The ability to inhibit conflicting information is pivotal in the dynamic and high-speed context of fast-ball sports. However, the behavioral and electrophysiological characteristics underlying the cognitive inhibition processes associated with table tennis expertise remain unexplored. This study aims to bridge these research gaps by utilizing the color-word Stroop task and the spatial Stroop task alongside event-related potential (ERP) measurements to investigate domain-general and domain-specific cognitive inhibition among table tennis athletes. The study involved a total of 40 participants, including 20 table tennis athletes (11 males and 9 females; mean age 20.75 years) and 20 nonathletes (9 males and 11 females; mean age 19.80 years). The group differences in the Stroop effect on behavioral outcomes and ERP amplitudes were compared within each task, respectively. In the color-word Stroop tasks, athletes exhibited smaller incongruent-related negative potential amplitudes (Ninc; 300-400 ms; p = 0.036) and a diminished Stroop effect on late sustained potential amplitudes (LSP; 500-650 ms; p = 0.028) than nonathletes, although no significant differences were observed in behavioral outcomes (p > 0.05). Conversely, in the spatial Stroop tasks, athletes not only responded more swiftly but also exhibited reduced Stroop effects on both LSP amplitudes (350-500 ms; p = 0.004) and reaction times (p = 0.002) relative to nonathletes. These findings suggest that table tennis athletes excel in cognitive inhibition in the context of both domain-general and domain-specific tasks, particularly exhibiting enhanced performance in tasks that are closely aligned with the demands of their sport. Our results support the neural efficiency hypothesis and improve our understanding of the interactions between cognitive functions and table tennis expertise.

Enhancing post-hurricane regional loss assessment with crowdsourcing: A probabilistic inference approach

With the emergence of efficient, widespread social networking infrastructure, along with advances in sensing technology, data retrieval, and data processing techniques, crowdsourcing and participatory sensing have garnered attention as promising approaches for leveraging the general public’s decision-making capacity for more effective and scalable disaster management practices. Although these approaches have recently been acknowledged for their potential in technical and domain-specific tasks, such as post-disaster damage assessment, there is still room for exploration and enhancement. This paper represents a significant stride in this direction by introducing a probabilistic approach designed to improve the reliability and quality of data gathered through participatory disaster damage assessments. By rigorously addressing crucial challenges related to task design and implementing sophisticated truth-inference mechanisms, this study aims to minimize uncertainty and subjectivity in the assessment of building damage states from imagery data conducted by non-expert crowd participants. The proposed methodology is demonstrated using a post-disaster damage database from Hurricane Laura, with Amazon Mechanical Turk (M-Turk) serving as the online crowdsourcing platform. With an achieved accuracy of approximately 50% and a ±1-class accuracy of around 95%, the proposed methodology significantly outperforms other inference methods derived from crowd-assigned labels in similar applications. Furthermore, damage assessment is extended to loss estimation, providing a comprehensive view of end-to-end damage and loss assessment. The findings of this study can significantly contribute to the development of efficient and reliable strategies for post-disaster assessments, ultimately facilitating expedited response and recovery efforts in areas affected by disasters.