Foundation Model Research Articles

Foundation models for cardiovascular disease detection via biosignals from digital stethoscopes

Auscultation of the heart and the electrocardiogram (ECG) are two central components of the cardiac exam. Recent innovations of the stethoscope have enabled the simultaneous acquisition of a high-quality digital acoustic signal and ECG. We present foundation models trained on phonocardiogram (PCG) and ECG data collected from digital stethoscopes during routine clinical practice. We show that these foundation models that are pre-trained on large unlabeled datasets in a self-supervised manner can be fine-tuned for a variety of cardiovascular disease detection tasks. This is the first study that builds foundation models specifically for synchronously captured PCG and ECG data. Our approach is based on the recently developed masked autoencoder framework which we extend to handle multiple signals that are synchronously captured. This paradigm makes it possible to use large capacity models leading to superior performance even though the size of datasets with medical label annotations may be limited.

Considerations for governing open foundation models.

Different policy proposals may disproportionately affect the innovation ecosystem.

50+ years of psychological acculturation research: Progress and challenges

Acculturation psychology, now over fifty years old, has evolved significantly since its inception in the mid-1960s. This paper seeks to shape the future of the field by addressing recent criticisms and controversies such as the association between integration and adaptation. The author reflects on acculturation psychology’s journey and highlights how global migration trends have accelerated the growth of acculturation as a critical area in psychology. Despite its popularity, the field has faced methodological challenges that necessitate a review of psychometric issues and foundational models. Given the profound impact of acculturation research on millions of lives, our findings must be clear and uncontested. The paper calls for re-evaluating the field's goals and objectives in light of the evolving ethnic compositions of societies. It advocates for leveraging methodological advancements and sophisticated statistical tools to ensure that acculturation research continues to contribute positively to humanity.

GeneCompass: deciphering universal gene regulatory mechanisms with a knowledge-informed cross-species foundation model.

Deciphering universal gene regulatory mechanisms in diverse organisms holds great potential for advancing our knowledge of fundamental life processes and facilitating clinical applications. However, the traditional research paradigm primarily focuses on individual model organisms and does not integrate various cell types across species. Recent breakthroughs in single-cell sequencing and deep learning techniques present an unprecedented opportunity to address this challenge. In this study, we built an extensive dataset of over 120 million human and mouse single-cell transcriptomes. After data preprocessing, we obtained 101,768,420 single-cell transcriptomes and developed a knowledge-informed cross-species foundation model, named GeneCompass. During pre-training, GeneCompass effectively integrated four types of prior biological knowledge to enhance our understanding of gene regulatory mechanisms in a self-supervised manner. By fine-tuning for multiple downstream tasks, GeneCompass outperformed state-of-the-art models in diverse applications for a single species and unlocked new realms of cross-species biological investigations. We also employed GeneCompass to search for key factors associated with cell fate transition and showed that the predicted candidate genes could successfully induce the differentiation of human embryonic stem cells into the gonadal fate. Overall, GeneCompass demonstrates the advantages of using artificial intelligence technology to decipher universal gene regulatory mechanisms and shows tremendous potential for accelerating the discovery of critical cell fate regulators and candidate drug targets.

Digital Fingerprinting of Complex Liquids Using a Reconfigurable Multi-Sensor System with Foundation Models.

Combining chemical sensor arrays with machine learning enables designing intelligent systems to perform complex sensing tasks and unveil properties that are not directly accessible through conventional analytical chemistry. However, personalized and portable sensor systems are typically unsuitable for the generation of extensive data sets, thereby limiting the ability to train large models in the chemical sensing realm. Foundation models have demonstrated unprecedented zero-shot learning capabilities on various data structures and modalities, in particular for language and vision. Transfer learning from such models is explored by providing a framework to create effective data representations for chemical sensors and ultimately describe a novel, generalizable approach for AI-assisted chemical sensing. The translation of signals produced by remarkably simple and portable multi-sensor systems into visual fingerprints of liquid samples under test is demonstrated, and it is illustrated that how a pipeline incorporating pretrained vision models yields average classification accuracy in four unrelated chemical sensing tasks with limited domain-specific training measurements. This approach matches or outperforms expert-curated sensor signal features, thereby providing a generalization of data processing for ultimate ease-of-use and broad applicability to enable interpretation of multi-signal outputs for generic sensingapplications.

Chemically Transferable Electronic Coarse Graining for Polythiophenes.

Recent advances in machine-learning-based electronic coarse graining (ECG) methods have demonstrated the potential to enable electronic predictions in soft materials at mesoscopic length scales. However, previous ECG models have yet to confront the issue of chemical transferability. In this study, we develop chemically transferable ECG models for polythiophenes using graph neural networks. Our models are trained on a data set that samples over the conformational space of random polythiophene sequences generated with 15 different monomer chemistries and three different degrees of polymerization. We systematically explore the impact of coarse-grained representation on ECG accuracy, highlighting the significance of preserving the C-β coordinates in thiophene. We also find that integrating unique polymer sequences into training enhances the model performance more efficiently than augmenting conformational sampling for sequences already in the training data set. Moreover, our ECG models, developed initially for one property and one level of quantum chemical theory, can be efficiently transferred to related properties and higher levels of theory with minimal additional data. The chemically transferable ECG model introduced in this work will serve as a foundation model for new classes of chemically transferable ECG predictions across chemical space.

Establishing the Foundations of Emotional Intelligence in Care Companion Robots to Mitigate Agitation Among High-Risk Patients With Dementia: Protocol for an Empathetic Patient-Robot Interaction Study.

An estimated 6.7 million persons are living with dementia in the United States, a number expected to double by 2060. Persons experiencing moderate to severe dementia are 4 to 5 times more likely to fall than those without dementia, due to agitation and unsteady gait. Socially assistive robots fail to address the changing emotional states associated with agitation, and it is unclear how emotional states change, how they impact agitation and gait over time, and how social robots can best respond by showing empathy. This study aims to design and validate a foundational model of emotional intelligence for empathetic patient-robot interaction that mitigates agitation among those at the highest risk: persons experiencing moderate to severe dementia. A design science approach will be adopted to (1) collect and store granular, personal, and chronological data using Personicle (an open-source software platform developed to automatically collect data from phones and other devices), incorporating real-time visual, audio, and physiological sensing technologies in a simulation laboratory and at board and care facilities; (2) develop statistical models to understand and forecast the emotional state, agitation level, and gait pattern of persons experiencing moderate to severe dementia in real time using machine learning and artificial intelligence and Personicle; (3) design and test an empathy-focused conversation model, focused on storytelling; and (4) test and evaluate this model for a care companion robot (CCR) in the community. The study was funded in October 2023. For aim 1, architecture development for Personicle data collection began with a search for existing open-source data in January 2024. A community advisory board was formed and met in December 2023 to provide feedback on the use of CCRs and provide personal stories. Full institutional review board approval was received in March 2024 to place cameras and CCRs at the sites. In March 2024, atomic marker development was begun. For aim 2, after a review of open-source data on patients with dementia, the development of an emotional classifier was begun. Data labeling was started in April 2024 and completed in June 2024 with ongoing validation. Moreover, the team established a baseline multimodal model trained and validated on healthy-person data sets, using transformer architecture in a semisupervised manner, and later retrained on the labeled data set of patients experiencing moderate to severe dementia. In April 2024, empathy alignment of large language models was initiated using prompt engineering and reinforcement learning. This innovative caregiving approach is designed to recognize the signs of agitation and, upon recognition, intervene with empathetic verbal communication. This proposal has the potential to have a significant impact on an emerging field of computational dementia science by reducing unnecessary agitation and falls of persons experiencing moderate to severe dementia, while reducing caregiver burden. PRR1-10.2196/55761.

Multi-Organ Foundation Model for Universal Ultrasound Image Segmentation with Task Prompt and Anatomical Prior.

Semantic segmentation of ultrasound (US) images with deep learning has played a crucial role in computer-aided disease screening, diagnosis and prognosis. However, due to the scarcity of US images and small field of view, resulting segmentation models are tailored for a specific single organ and may lack robustness, overlooking correlations among anatomical structures of multiple organs. To address these challenges, we propose the Multi-Organ FOundation (MOFO) model for universal US image segmentation. The MOFO is optimized jointly from multiple organs across various anatomical regions to overcome the data scarcity and explore correlations between multiple organs. The MOFO extracts organ-invariant representations from US images. Simultaneously, the task prompt is employed to refine organ-specific representations for segmentation predictions. Moreover, the anatomical prior is incorporated to enhance the consistency of the anatomical structures. A multi-organ US database, comprising 7039 images from 10 organs across various regions of the human body, has been established to evaluate our model. Results demonstrate that the MOFO outperforms single-organ methods in terms of the Dice coefficient, 95% Hausdorff distance and average symmetric surface distance with statistically sufficient margins. Our experiments in multi-organ universal segmentation for US images serve as a pioneering exploration of improving segmentation performance by leveraging semantic and anatomical relationships within US images of multiple organs.

A Survey of Robot Intelligence with Large Language Models

Since the emergence of ChatGPT, research on large language models (LLMs) has actively progressed across various fields. LLMs, pre-trained on vast text datasets, have exhibited exceptional abilities in understanding natural language and planning tasks. These abilities of LLMs are promising in robotics. In general, traditional supervised learning-based robot intelligence systems have a significant lack of adaptability to dynamically changing environments. However, LLMs help a robot intelligence system to improve its generalization ability in dynamic and complex real-world environments. Indeed, findings from ongoing robotics studies indicate that LLMs can significantly improve robots’ behavior planning and execution capabilities. Additionally, vision-language models (VLMs), trained on extensive visual and linguistic data for the vision question answering (VQA) problem, excel at integrating computer vision with natural language processing. VLMs can comprehend visual contexts and execute actions through natural language. They also provide descriptions of scenes in natural language. Several studies have explored the enhancement of robot intelligence using multimodal data, including object recognition and description by VLMs, along with the execution of language-driven commands integrated with visual information. This review paper thoroughly investigates how foundation models such as LLMs and VLMs have been employed to boost robot intelligence. For clarity, the research areas are categorized into five topics: reward design in reinforcement learning, low-level control, high-level planning, manipulation, and scene understanding. This review also summarizes studies that show how foundation models, such as the Eureka model for automating reward function design in reinforcement learning, RT-2 for integrating visual data, language, and robot actions in vision-language-action models, and AutoRT for generating feasible tasks and executing robot behavior policies via LLMs, have improved robot intelligence.

Crack SAM: enhancing crack detection utilizing foundation models and Detectron2 architecture

Accurate crack detection is crucial for maintaining pavement integrity, yet manual inspections remain labor-intensive and prone to errors, underscoring the need for automated solutions. This study proposes a novel crack segmentation approach utilizing advanced visual models, specifically Detectron2 and the Segment Anything Model (SAM), applied to the CFD and Crack500 datasets, which exhibit intricate and diverse crack patterns. Detectron2 was tested with four configurations—mask_rcnn_R_50_FPN_3x, mask_rcnn_R_101_FPN_3x, faster_rcnn_R_50_FPN_3x, and faster_rcnn_R_101_FPN_3x—while SAM was compared using Focal Loss, DiceCELoss, and DiceFocalLoss. SAM with DiceFocalLoss outperformed Detectron2, achieving mean IoU scores of 0.69 and 0.59 on the CFD and Crack500 datasets, respectively. The integration of Detectron2 with faster_rcnn_R_101_FPN_3x and SAM using DiceFocalLoss involves generating bounding boxes with Detectron2, which serve as prompts for SAM to produce segmentation masks. This approach achieves mIoU scores of 0.83 for CFD dataset and 0.75 for Crack500 dataset. These results highlight the potential of combining foundation models with Detectron2 for advancing crack detection technologies, offering valuable insights for enhancing highway maintenance systems.

Utilizing machine learning algorithm and Carrera unified formulation for vibration analysis of nanocomposite-enhanced bridge structures rested on an innovative elastic foundation: verification of the results via nondestructive testing

This paper presents an advanced vibration analysis of Al2O3 nanocomposite-reinforced concrete bridge structures resting on an innovative elastic foundation using the Carrera Unified Formulation (CUF). The primary objective is to investigate the dynamic response of these bridges under various loading conditions, accounting for the reinforcing effects of Al2O3 nanocomposites within the concrete matrix. The formulation incorporates a novel elastic foundation model designed to more accurately simulate realistic boundary conditions and soil-structure interaction. The accuracy and reliability of the CUF-based vibration analysis are further validated using nondestructive testing (NDT) techniques, which enable the detection of potential damage and anomalies in the bridge structures. Moreover, a machine learning (ML) algorithm is employed to predict the vibrational characteristics, facilitating an efficient comparison with the results obtained from CUF and NDT. The combination of theoretical modeling, experimental verification, and ML predictions highlights the robustness of the proposed method. The results demonstrate the effectiveness of using Al2O3 nanocomposites to enhance the mechanical properties of bridge structures, improving their vibrational performance, stability, and longevity. This study provides a comprehensive framework for future applications in bridge engineering, combining high-fidelity numerical methods with state-of-the-art testing and computational techniques.

TSOTSALearning at LLMs4OL Tasks A and B : Combining Rules to Large Language Model for Ontology Learning

This paper presents our contribution to the Large Language Model For Ontology Learning (LLMs4OL) challenge hosted by ISWC conference. The challenge involves extracting and classifying various ontological components from multiple datasets. The organizers of the challenge provided us with the train set and the test set. Our goal consists of determining in which conditions foundation models such as BERT can be used for ontologies learning. To achieve this goal, we conducted a series of experiments on various datasets. Initially, GPT-4 was tested on the wordnet dataset, achieving an F1-score of 0.9264. Subsequently, we performed additional experiments on the same dataset using BERT. These experiments demonstrated that by combining BERT with rule-based methods, we achieved an F1-score of 0.9938, surpassing GPT-4 and securing the first place for term typing on the Wordnet dataset.

Self-supervised learning on MeerKAT wide-field continuum images

Context. Self-supervised learning (SSL) applied to natural images has demonstrated a remarkable ability to learn meaningful, low-dimension representations without labels, resulting in models that are adaptable to many different tasks. Until now, applications of SSL to astronomical images have been limited to Galaxy Zoo datasets, which require a significant amount of preprocessing to prepare sparse images centered on a single galaxy. With wide-field survey instruments at the forefront of the Square Kilometer Array (SKA) era, this approach to gathering training data is impractical. Aims. We demonstrate that continuum images from surveys such as the MeerKAT Galactic Cluster Legacy Survey (MGCLS) can be successfully used with SSL, without extracting single-galaxy cutouts. Methods. Using the SSL framework DINO, we experimented with various preprocessing steps, augmentations, and architectures to determine the optimal approach for this data. We trained both ResNet50 and Vision Transformer (ViT) backbones. Results. Our models match state-of-the-art results (trained on Radio Galaxy Zoo) for FRI/FRII morphology classification. Furthermore, they predict the number of compact sources via linear regression with much higher accuracy. Open-source foundation models trained on natural images such as DINOv2 also excel at simple FRI/FRII classification; the advantage of domain-specific backbones is much smaller models trained on far less data. Smaller models are more efficient to fine-tune, and doing so results in a similar performance between our models, the state-of-the-art, and open-source models on multi-class morphology classification. Conclusions. Using source-rich crops from wide-field images to train multi-purpose models is an easily scalable approach that significantly reduces data preparation time. For the tasks evaluated in this work, twenty thousand crops is sufficient training data for models that produce results similar to state-of-the-art. In the future, complex tasks like source detection and characterization, together with domain-specific tasks, ought to demonstrate the true advantages of training models with radio astronomy data over natural-image foundation models.

MedLSAM: Localize and segment anything model for 3D CT images

Recent advancements in foundation models have shown significant potential in medical image analysis. However, there is still a gap in models specifically designed for medical image localization. To address this, we introduce MedLAM, a 3D medical foundation localization model that accurately identifies any anatomical part within the body using only a few template scans. MedLAM employs two self-supervision tasks: unified anatomical mapping (UAM) and multi-scale similarity (MSS) across a comprehensive dataset of 14,012 CT scans. Furthermore, we developed MedLSAM by integrating MedLAM with the Segment Anything Model (SAM). This innovative framework requires extreme point annotations across three directions on several templates to enable MedLAM to locate the target anatomical structure in the image, with SAM performing the segmentation. It significantly reduces the amount of manual annotation required by SAM in 3D medical imaging scenarios. We conducted extensive experiments on two 3D datasets covering 38 distinct organs. Our findings are twofold: (1) MedLAM can directly localize anatomical structures using just a few template scans, achieving performance comparable to fully supervised models; (2) MedLSAM closely matches the performance of SAM and its specialized medical adaptations with manual prompts, while minimizing the need for extensive point annotations across the entire dataset. Moreover, MedLAM has the potential to be seamlessly integrated with future 3D SAM models, paving the way for enhanced segmentation performance. Our code is public at https://github.com/openmedlab/MedLSAM.

Review of deep learning-based pathological image classification: From task-specific models to foundation models

Pathological diagnosis is considered the gold standard in cancer diagnosis, playing a crucial role in guiding treatment decisions and prognosis assessment for patients. However, achieving accurate diagnosis of pathology images poses several challenges, including the scarcity of pathologists and the inherent subjective variability in their interpretations. The advancements in whole-slide imaging technology and deep learning methods provide new opportunities for digital pathology, especially in low-resource settings, by enabling effective pathological image classification. In this article, we begin by introducing the datasets, which include both unimodal and multimodal types, as essential resources for advancing pathological image classification. We then provide a comprehensive overview of deep learning-based pathological image classification models, covering task-specific models such as supervised, unsupervised, weakly supervised, and semi-supervised learning methods, as well as unimodal and multimodal foundation models. Next, we review tumor-related indicators that can be predicted from pathological images, focusing on two main categories: indicators that can be recognized by pathologists, such as tumor classification, grading, and region recognition; and those that cannot be recognized by pathologists, including molecular subtype prediction, tumor origin prediction, biomarker prediction, and survival prediction. Finally, we summarize the key challenges in digital pathology and propose potential future directions.

Detecting Novel Malware Classes with a Foundational Multi-Modality Data Analysis Model

Detecting Novel Malware Classes with a Foundational Multi-Modality Data Analysis Model

A novel drying process for knitted fabrics based on heat-wet coupling model

The characteristics of high elasticity and easy deformation of knitted fabrics present challenges for traditional continuous flat drying processes. Shrinkage, uneven drying and dye migration are common problems with these processes. To address those problem, a novel dual-heat-supply drying process for knitted fabrics is proposed, which combines the characteristics of hot air drying and cylinder drying. By establishing a mechanical model of the fabric microelement, the force conditions of the fabric during the drying process are analyzed to prevent the occurrence of shrinkage. Consider the structural attributes of knitted fabrics and the diverse forms of water, a foundational unit model for single-sided weft knitted fabrics is proposed. The effective thermal conductivity of wet knitted fabrics in three directions is calculated through numerical simulation, and the validity of this method is preliminarily verified by comparison with literature experiments. The thermal-humidity coupling model is utilized to develop finite element models of cylinder drying and dual-heat-supply drying, which are then employed to conduct a comparative analysis of the two drying processes. The accuracy of the simulation results is verified through experiments, with the maximum deviation being less than 3.3 %. The results demonstrate that in comparison to cylinder drying, the temperature differential between the upper and lower surfaces of the fabric(ΔT) in dual-heat-supply drying is diminished by approximately 5 K. Additionally, the critical saturation point is reached approximately 10 s earlier, which effectively reduces the proclivity of dye migration and enhances the efficiency of the drying process. Finally, three process parameters (v, Tout and Th) is optimized for the dual-heat-supply drying process, 63.68 % reduction in ΔT, thereby further enhancing the quality of the drying process. The dual-heat-supply drying process represents an effective and significant improvement in the quality of knitted fabrics.

Evaluation of a Task-Specific Self-Supervised Learning Framework in Digital Pathology Relative to Transfer Learning Approaches and Existing Foundation Models

An integral stage in typical digital pathology workflows involves deriving specific features from tiles extracted from a tessellated whole-slide image. Notably, various computer vision neural network architectures, particularly the ImageNet pretrained, have been extensively used in this domain. This study critically analyzes multiple strategies for encoding tiles to understand the extent of transfer learning and identify the most effective approach. The study categorizes neural network performance into 3 weight initialization methods: random, ImageNet-based, and self-supervised learning. Additionally, we propose a framework based on task-specific self-supervised learning, which introduces a shallow feature extraction method, employing a spatial-channel attention block to glean distinctive features optimized for histopathology intricacies. Across 2 different downstream classification tasks (patch classification and weakly supervised whole-slide image classification) with diverse classification data sets, including colorectal cancer histology, Patch Camelyon, prostate cancer detection, The Cancer Genome Atlas, and CIFAR-10, our task-specific self-supervised encoding approach consistently outperforms other convolutional neural network–based encoders. The better performances highlight the potential of task-specific attention-based self-supervised training in tailoring feature extraction for histopathology, indicating a shift from using pretrained models originating outside the histopathology domain. Our study supports the idea that task-specific self-supervised learning allows domain-specific feature extraction, encouraging a more focused analysis.

A Foundation Language-Image Model of the Retina (FLAIR): encoding expert knowledge in text supervision

Foundation vision-language models are currently transforming computer vision, and are on the rise in medical imaging fueled by their very promising generalization capabilities. However, the initial attempts to transfer this new paradigm to medical imaging have shown less impressive performances than those observed in other domains, due to the significant domain shift and the complex, expert domain knowledge inherent to medical-imaging tasks. Motivated by the need for domain-expert foundation models, we present FLAIR, a pre-trained vision-language model for universal retinal fundus image understanding. To this end, we compiled 38 open-access, mostly categorical fundus imaging datasets from various sources, with up to 101 different target conditions and 288,307 images. We integrate the expert’s domain knowledge in the form of descriptive textual prompts, during both pre-training and zero-shot inference, enhancing the less-informative categorical supervision of the data. Such a textual expert’s knowledge, which we compiled from the relevant clinical literature and community standards, describes the fine-grained features of the pathologies as well as the hierarchies and dependencies between them. We report comprehensive evaluations, which illustrate the benefit of integrating expert knowledge and the strong generalization capabilities of FLAIR under difficult scenarios with domain shifts or unseen categories. When adapted with a lightweight linear probe, FLAIR outperforms fully-trained, dataset-focused models, more so in the few-shot regimes. Interestingly, FLAIR outperforms by a wide margin larger-scale generalist image-language models and retina domain-specific self-supervised networks, which emphasizes the potential of embedding experts’ domain knowledge and the limitations of generalist models in medical imaging. The pre-trained model is available at: https://github.com/jusiro/FLAIR.

When segment anything model meets inventorying of roadway assets

Automating the identification, localization, and monitoring of roadway assets distributed widely in the roadway network is critical for the traffic management system. It can efficiently provide up-to-date information in supporting transportation asset management. Collecting videos with vehicle-mounted cameras and processing the data with computer vision-based deep learning methods is garnering increased attention from transportation agencies. While promising, challenges arise due to the lack of high-quality annotations for roadway assets in images, difficulties in identifying these assets, and limited solutions. The Segment Anything Model (SAM), a visual foundation model, demonstrates robust zero-shot capability for general image segmentation under various prompts. This study evaluates SAM’s applicability and efficiency in extracting roadway assets from images. Specifically, it examines the impacts of model size and prompt quality on SAM’s performance in segmenting roadway assets. Five state-of-the-art semantic segmentation models are trained and compared with SAM. Results show that a lightweight SAM with human-rendered prompts outperforms the five semantic segmentation models. Based on the evaluation results, future work will explore incorporating SAM into transportation asset management applications, promoting collaboration between human experts and artificial intelligence.