Foundation Model Research Articles

Effect of scale factor on the dynamic response of frame foundations

Abstract In geotechnical engineering, a scale factor “n” is a numerical ratio that relates the physical properties of a physical model to those of a prototype. The effect of scale factor on the dynamic response of a frame foundation used for high-speed turbo machines is examined through numerical analyses to clarify the influence of scaling dimensions, frequency, static and dynamic loads, and scaled models’ material on the dynamic responses of the frame foundation. The static and dynamic analyses were conducted using Ansys Workbench software. The total number of analyses used to investigate the scale factor effect was 28. Both modal and harmonic analyses were performed to calculate the natural frequencies and displacement responses of the scaled and prototype models under the scaled dynamic loading. Five scale factors were used in this study for concrete models (1, 20, 25, 32, and 40), and two additional analyses on steel models of scale (1 and 20) were used to study the material scaling. The prototype results were first verified, and then the models were investigated after scaling the dynamic loads and dimensions. The results of this study revealed the importance of scaling the frequency when studying the high-speed turbo machine foundation models to reflect the correct responses of the prototype foundation.

Graph-text contrastive learning of inorganic crystal structure toward a foundation model of inorganic materials

Graph-text contrastive learning of inorganic crystal structure toward a foundation model of inorganic materials

Integrating large language models in care, research, and education in multiple sclerosis management.

Use of techniques derived from generative artificial intelligence (AI), specifically large language models (LLMs), offer a transformative potential on the management of multiple sclerosis (MS). Recent LLMs have exhibited remarkable skills in producing and understanding human-like texts. The integration of AI in imaging applications and the deployment of foundation models for the classification and prognosis of disease course, including disability progression and even therapy response, have received considerable attention. However, the use of LLMs within the context of MS remains relatively underexplored. LLMs have the potential to support several activities related to MS management. Clinical decision support systems could help selecting proper disease-modifying therapies; AI-based tools could leverage unstructured real-world data for research or virtual tutors may provide adaptive education materials for neurologists and people with MS in the foreseeable future. In this focused review, we explore practical applications of LLMs across the continuum of MS management as an initial scope for future analyses, reflecting on regulatory hurdles and the indispensable role of human supervision.

Protein Set Transformer: A protein-based genome language model to power high diversity viromics.

Exponential increases in microbial and viral genomic data demand transformational advances in scalable, generalizable frameworks for their interpretation. Standard homology-based functional analyses are hindered by the rapid divergence of microbial and especially viral genomes and proteins that significantly decreases the volume of usable data. Here, we present Protein Set Transformer (PST), a protein-based genome language model that models genomes as sets of proteins without considering sparsely available functional labels. Trained on >100k viruses, PST outperformed other homology- and language model-based approaches for relating viral genomes based on shared protein content. Further, PST demonstrated protein structural and functional awareness by clustering capsid-fold-containing proteins with known capsid proteins and uniquely clustering late gene proteins within related viruses. Our data establish PST as a valuable method for diverse viral genomics, ecology, and evolutionary applications. We posit that the PST framework can be a foundation model for microbial genomics when trained on suitable data.

Fair and optimal prediction via post‐processing

AbstractWith the development of machine learning algorithms and the increasing computational resources available, artificial intelligence has achieved great success in many application domains. However, the success of machine learning has also raised concerns about the fairness of the learned models. For instance, the learned models can perpetuate and even exacerbate the potential bias and discrimination in the training data. This issue has become a major obstacle to the deployment of machine learning systems in high‐stakes domains, for example, criminal judgment, medical testing, online advertising, hiring process, and so forth. To mitigate the potential bias exhibited by machine learning models, fairness criteria can be integrated into the training process to ensure fair treatment across all demographics, but it often comes at the expense of model performance. Understanding such tradeoffs, therefore, is crucial to the design of optimal and fair algorithms. My research focuses on characterizing the inherent tradeoff between fairness and accuracy in machine learning, and developing algorithms that can achieve both fairness and optimality. In this article, I will discuss our recent work on designing post‐processing algorithms for fair classification, which can be applied to a wide range of fairness criteria, including statistical parity, equal opportunity, and equalized odds, under both attribute‐aware and attribute‐blind settings, and is particularly suited to large‐scale foundation models where retraining is expensive or even infeasible. I will also discuss the connections between our work and other related research on trustworthy machine learning, including the connections between algorithmic fairness and differential privacy as well as adversarial robustness.

Nonlinear Analysis of the Mechanical Response of an Existing Tunnel Induced by Shield Tunneling during the Entire Under-Crossing Process

The safety of existing tunnels during the entire under-crossing process of a new shield tunnel is critically important for ensuring the sustainable operation of urban transportation infrastructure. The nonlinear behavior of surrounding soils plays a significant role in the mechanical response of tunnel structures. In order to assess the mechanical response of the existing tunnel more reasonably, this study attempts to propose a novel theoretical solution and calculation method by simultaneously considering the nonlinear characteristics of surrounding soils and the tunneling effects of a new tunnel during its entire under-crossing process. Firstly, the additional stresses acting on the existing tunnel stemming from the tunneling effects of a new shield tunnel during different under-crossing stages are calculated using the typical Mindlin solution, as well as the Loganathan and Poulos solutions. The influences of the additional thrust, friction force, and grouting pressure and the loss of surrounding soils are taken into account. Then, the nonlinear Pasternak foundation model is introduced to characterize the behavior of surrounding soils, and the governing differential equation for the mechanical response of the existing tunnel is derived using the typical Euler–Bernoulli beam model. Subsequently, a novel theoretical solution and calculation approach are established using the finite difference formula and the Newton iteration method for assessing the mechanical response of the existing tunnel. Finally, one case study is performed to illustrate the mechanical behavior of the existing tunnel during the whole under-crossing process of a new shield tunnel, and the validity of the developed solution is verified against both the computed result of finite element simulation and the field measurements. In addition, the influences from the ultimate resistance and reaction coefficient of surrounding soils and those from the vertical distance and intersection angle between existing and newly constructed tunnels are analyzed and discussed in detail.

Leveraging foundation and large language models in medical artificial intelligence.

Recent advancements in the field of medical artificial intelligence (AI) have led to the widespread adoption of foundational and large language models. This review paper explores their applications within medical AI, introducing a novel classification framework that categorizes them as disease-specific, general-domain, and multi-modal models. The paper also addresses key challenges such as data acquisition and augmentation, including issues related to data volume, annotation, multi-modal fusion, and privacy concerns. Additionally, it discusses the evaluation, validation, limitations, and regulation of medical AI models, emphasizing their transformative potential in healthcare. The importance of continuous improvement, data security, standardized evaluations, and collaborative approaches is highlighted to ensure the responsible and effective integration of AI into clinical applications.

Road network traffic flow prediction: A personalized federated learning method based on client reputation

Accurate traffic flow prediction can provide effective decision-making support for traffic management, alleviate traffic congestion, and improve road traffic efficiency. Traffic flow data contains personal privacy information, such as vehicle trajectories, driving speed, etc. However, most existing research focuses on using all local data to jointly construct prediction models, facing data security and privacy issues. In response to these challenges, this paper presents a Personalized Federated Learning method based on Client Reputation for traffic flow prediction. This method avoids direct sharing of original data by performing model training and information aggregation locally, thereby better protecting user privacy. We initially constructed the foundational model by integrating Graph Neural Networks (GCNs) and Gated Recurrent Units (GRUs) for federated learning. GCNs are capable of capturing the spatial relationships between nodes in a road network, thereby extracting the spatial features of traffic flow. On the other hand, GRUs has advantages at handling sequential data, effectively capturing the dynamic temporal characteristics of traffic flow. Consequently, by combining GCNs and GRUs, we are able to build a model that captures both the spatial and temporal features of traffic flow. Subsequently, we employed a federated learning framework to train the model. In the training process, we propose two evaluation algorithms to evaluate the reputation of all clients participating in federated learning. Then, different weights are customized for each client through a personalized algorithm based on the client's reputation to improve its personalization. Finally, we validated the method on the License Plate Recognition (LPR) datasets collected in Changsha city, China. The experimental results indicate that the method can achieve good prediction performance and stability while protecting the privacy of all participants.

Zero shot health trajectory prediction using transformer

Integrating modern machine learning and clinical decision-making has great promise for mitigating healthcare’s increasing cost and complexity. We introduce the Enhanced Transformer for Health Outcome Simulation (ETHOS), a novel application of the transformer deep-learning architecture for analyzing high-dimensional, heterogeneous, and episodic health data. ETHOS is trained using Patient Health Timelines (PHTs)—detailed, tokenized records of health events—to predict future health trajectories, leveraging a zero-shot learning approach. ETHOS represents a significant advancement in foundation model development for healthcare analytics, eliminating the need for labeled data and model fine-tuning. Its ability to simulate various treatment pathways and consider patient-specific factors positions ETHOS as a tool for care optimization and addressing biases in healthcare delivery. Future developments will expand ETHOS’ capabilities to incorporate a wider range of data types and data sources. Our work demonstrates a pathway toward accelerated AI development and deployment in healthcare.

Analytical Solution to Estimate the Effect of Underlying Tunnel Excavations on the Existing Tunnel

ABSTRACTThe tunneling underlying inevitably leads to the displacement of adjacent soil, greatly influencing the deformation of the tunnel above. Most theoretical studies primarily concentrate on analyzing the mechanical equilibrium of individual tunnel sections, ignoring the energy generated by the system during tunnel deformation. Based on this, in view of the energy relationship, the overlying tunnel's deformation is simulated by using the Rayleigh–Ritz method. Further, its potential energy equation can be established based on the Vlasov foundation. The corresponding energy variational solution is solved according to the minimum potential energy principle, leading to an analytical solution for the shield tunneling‐induced overlying tunnel's response. The efficacy of the suggested method is verified by contrasting it with centrifuge experiments and field case studies derived from prior studies. Relative to the Winkler foundation model, which deviated from the suggested approach, the results derived from the suggested method show a closer correlation with the collected measurement data. Further parameter studies show that the vertical clearance and skew angle between two tunnels, the volume loss rate, and elastic modulus are significant factors affecting the tunnel behaviors due to tunneling underneath. The suggested theoretical model can be applied to forecast potential risks that an existing tunnel may encounter during the excavation of a new tunnel underlying in similar engineering projects.

Revisiting the self‐supply model: A foundational, not a complimentary model for water supply in rural households

AbstractIn this opinion article, we propose that self‐supply, as a model for household‐managed water supply in rural areas, is the foundational model for water supply in rural households. Any other model is only complementary while it lasts. Therefore, self‐supply needs more emphasis in research and intentional exposure and support in practice. This is underscored by the fact that despite the use of taxes, transfers, and tariffs that fund other service models, major water service gaps still exist in rural households forcing them to invest directly in self‐supply. Over 2.2 billion people globally still lack access to safe water. Our argument is not that other models such as communal or government models are irrelevant; rather, it is that self‐supply is the foundational and often preferred model for rural households due to reliability, predictability, control, and flexibility. Yet it is overlooked often. Therefore, we call for investing in research informing self‐supply policy, practice, and funding especially where it links with other models to be prioritized. This will likely accelerate the achievement of SDG 6.

A hybrid non‐parametric ground motion model of power spectral density based on machine learning

AbstractIn the fields of engineering seismology and earthquake engineering, researchers have predominantly focused on ground motion models (GMMs) for intensity measures. However, there has been limited research on power spectral density GMMs (PSD‐GMMs) that characterize spectral characteristics. PSD, being structure‐independent, offers unique advantages. This study aims to construct PSD‐GMMs using non‐parametric machine learning (ML) techniques. By considering 241 different frequencies from 0.1 to 25.12 Hz and evaluating eight performance indicators, seven highly accurate and stable ML techniques are selected from 12 different ML techniques as foundational models for the PSD‐GMM. Through mixed effects regression analysis, inter‐event, intra‐event, and inter‐site standard deviations are derived. To address inherent modeling uncertainty, this study uses the ratio of the reciprocal of the standard deviation of the total residuals of the foundational models to the sum of the reciprocals of the total residuals of the seven ML GMMs as weight coefficients for constructing a hybrid non‐parametric PSD‐GMM. Utilizing this model, ground motion records can be simulated, and seismic hazard curves and uniform hazard PSD can be obtained. In summary, the hybrid non‐parametric PSD‐GMM demonstrates remarkable efficacy in simulating and predicting ground motion records and holds significant potential for guiding seismic hazard and risk analysis.

Rapid assessment of seismic performance of large monopile-supported offshore wind turbines under scour

This study focuses on investigating the seismic performance of a reference 15 MW offshore wind turbine (OWT) supported by a large-diameter monopile, under various scour conditions. Fully coupled nonlinear finite element models considering 3D soil continuum for seismic response analysis of monopile-supported OWTs are very complicated and computationally expensive. Therefore, it is important to have good insight on the dynamic behaviour of monopile-supported OWTs subjected to earthquake excitations before performing high-fidelity fully nonlinear 3D simulations, which will not generally be practical in cases considering various seismic loading and scouring scenarios. The purpose of this study is to provide a fast and efficient tool to understand the interaction between the seismic behaviour of emerging large-capacity OWTs when scour erosion affects the dynamic properties of the structure. The dynamic stiffness method (DSM) is applied for the free vibration analysis of the structure. Modal superposition is used to calculate the seismic responses of the OWT by using the exact mode shapes obtained from the dynamic stiffness formulations. The rotor-nacelle-assembly (RNA) is idealized as a lumped mass at the tower top. The tower and monopile are modelled as Timoshenko beam-columns under axial compressive loading. The soil-structure interaction (SSI) is modelled using the Winkler framework, whereby the soil contribution is assumed as elastic to take advantage of the modal superposition method. Two different approaches are considered for the soil stiffness; a constant profile, and a linearly-varying subgrade modulus along the embedded depth of the pile, and both are appraised in this paper. Firstly, natural frequencies obtained from dynamic stiffness formulations are validated using experimental data from literature for the case of controlled dynamic SSI experiments, and for operating OWTs. Additionally, the natural frequency provided by a technical document for the reference 15 MW OWT is compared to the result from the DSM, where a very good agreement is observed. The seismic response time-histories of the OWT model are obtained under different input earthquake conditions, and subsequently verified by comparison with the results of finite element simulations. The effects of different scour scenarios and foundation modelling approaches on the free vibration and seismic responses are subsequently investigated. A parametric analysis is also performed using different soil properties to observe the interaction between the soil stiffness, scour condition, and seismic response. Finally, the influence of the frequency content of seismic records on the dynamic response of the OWT, specifically the tower top acceleration, the mudline rotation, and the mudline stress, is investigated considering various scour scenarios. The specific numerical case study undertaken for a reference 15 MW OWT, which is supported by a large monopile in very dense sand, shows that the seismic responses are very limited and that the elastic soil behaviour assumption is reasonable. Additionally, considering the computational efficiency, the proposed combination of DSM and modal superposition can be used for rapid seismic performance assessment of large diameter monopile-supported OWTs at the preliminary design stage to gain insight to the global dynamic response of the whole vibrating system.

A meshless model for three-dimensional direct numerical simulation of local scour around cylinder bridge foundation

A meshless local scour model for cylinder bridge foundation based on the smoothed particle hydrodynamics (SPH) method is proposed in this paper. Differing from the traditional Eulerian mesh model used for monopile local scour, it overcomes the high requirements of mesh quality and density for capturing large deformations at the fluid interface. Unlike the transient sediment erosion SPH model used for dam-break, sediment flushing, and water jet, the meshless approach is innovatively applied to the continuous simulation of bridge local scour under steady flow conditions. The present SPH model is established based on the SPH multi-phase fluid model, integrating a classical fluid model combined with numerical stabilization algorithms to govern water particle motion and a sediment model from Zhang et al. (2024) to govern sediment evolution, and tests a new scheme for boundary treatment in the model based on DualSPHysics. Furthermore, a novel scour visualization method that aims at extracting both visible and actual bed surfaces from discrete particles is proposed. The model is validated through comparison with rigid bed and live bed experiments, demonstrating favorable agreement in terms of flow and scour simulation. Importantly, the model results reveal a discrepancy between the elevation of the post-scour visible bed surface derived from the conventional Eulerian mesh model and experimental data, compared to the actual bed surface unaffected by water flow erosion. This indicates the necessity for incorporating a safety margin in foundation design when utilizing experimental results to determine the maximum scour depth.

Toward a personalized autonomous transportation system: Vision, challenges, and solutions

The fragmented design of Intelligent Transportation Systems creates isolated intelligent systems. Resource competition and information gaps are fierce and widespread, worsening traffic issues and degrading overall service levels. Therefore, empowered by advanced technologies, an evolution towards Autonomous Transportation System (ATS) is observed. This evolution aims to develop a collaborative and sustainable ecosystem, prompting interoperability within the cloud-edge-device continuum. It can, accordingly, dismantle internal resource barriers and achieve a systematic balance between demand and supply with less human intervention. Despite the promising vision of ATS, it encounters three key challenges: disparate data, deficient models, and conflicting interests in supporting autonomous and personalized mobility. Hence, as an innovative solution, a trustworthy, private, and equal-serving framework called TPE is designed. It seamlessly integrates Blockchain, Federated Learning, and Large-scale Model to deploy a trustworthy operating environment, process private data for globally shareable knowledge, and develop a foundation model for personalized adaptation, respectively. Consequently, ATS empowered by TPE can serve diverse user groups both privately and equally.

Evaluating segment anything model (SAM) on MRI scans of brain tumors

Addressing the challenge of automatically segmenting anatomical structures from brain images has been a long-standing problem, attributed to subject- and image-based variations and constraints in available data annotations. The Segment Anything Model (SAM), developed by Meta, is a foundational model trained to provide zero-shot segmentation outputs with or without interactive user inputs, demonstrating notable performance on various objects and image domains without explicit prior training. This study evaluated SAM’s performance in brain tumor segmentation using two publicly available Magnetic Resonance Imaging (MRI) datasets. The study analyzed SAM’s standalone segmentation as well as its performance when provided user interaction through point prompts and bounding box inputs. SAM exhibited versatility across configurations and datasets, with the bounding box consistently outperforming others in achieving superior localized precision, with average Dice scores of 0.68 for TCGA and 0.56 for BRATS, along with average IoU values of 0.89 and 0.65, respectively, especially for tumors with low-to-medium curvature. Inconsistencies were observed, particularly in relation to variations in tumor size, shape, and textural features. The conclusion drawn from the study is that while SAM can automate medical image segmentation, further training and careful implementation are necessary for diagnostic purposes, especially with challenging cases such as MRI scans of brain tumors.

Real-world cooking robot system from recipes based on food state recognition using foundation models and PDDL

Although there is a growing demand for cooking behaviors as one of the expected tasks for robots, a series of cooking behaviors based on new recipe descriptions by robots in the real world has not yet been realized. In this study, we propose a robot system that integrates real-world executable robot cooking behavior planning using the Large Language Model (LLM) and classical planning of PDDL descriptions, and food ingredient state recognition learning from a small number of data using the Vision Language model (VLM). We succeeded in experiments in which PR2, a dual-armed wheeled robot, performed cooking from arranged new recipes in a real-world environment, and confirmed the effectiveness of the proposed system.

Special issue on real-world robot applications of the foundation models

Special issue on real-world robot applications of the foundation models

Foundational model aided automatic high-throughput drug screening using self-controlled cohort study.

Developing medicine from scratch to governmental authorization and detecting adverse drug reactions (ADR) have barely been economical, expeditious, and risk-averse investments. The availability of large-scale observational healthcare databases and the popularity of large language models offer an unparalleled opportunity to enable automatic high-throughput drug screening for both repurposing and pharmacovigilance. To demonstrate a general workflow for automatic high-throughput drug screening with the following advantages: (i) the association of various exposure on diseases can be estimated; (ii) both repurposing and pharmacovigilance are integrated; (iii) accurate exposure length for each prescription is parsed from clinical texts; (iv) intrinsic relationship between drugs and diseases are removed jointly by bioinformatic mapping and large language model - ChatGPT; (v) causal-wise interpretations for incidence rate contrasts are provided. Using a self-controlled cohort study design where subjects serve as their own control group, we tested the intention-to-treat association between medications on the incidence of diseases. Exposure length for each prescription is determined by parsing common dosages in English free text into a structured format. Exposure period starts from initial prescription to treatment discontinuation. A same exposure length preceding initial treatment is the control period. Clinical outcomes and categories are identified using existing phenotyping algorithms. Incident rate ratios (IRR) are tested using uniformly most powerful (UMP) unbiased tests. We assessed 3,444 medications on 276 diseases on 6,613,198 patients from the Clinical Practice Research Datalink (CPRD), an UK primary care electronic health records (EHR) spanning from 1987 to 2018. Due to the built-in selection bias of self-controlled cohort studies, ingredients-disease pairs confounded by deterministic medical relationships are removed by existing map from RxNorm and nonexistent maps by calling ChatGPT. A total of 16,901 drug-disease pairs reveals significant risk reduction, which can be considered as candidates for repurposing, while a total of 11,089 pairs showed significant risk increase, where drug safety might be of a concern instead. This work developed a data-driven, nonparametric, hypothesis generating, and automatic high-throughput workflow, which reveals the potential of natural language processing in pharmacoepidemiology. We demonstrate the paradigm to a large observational health dataset to help discover potential novel therapies and adverse drug effects. The framework of this study can be extended to other observational medical databases.

Abstract A029: Evaluating and interpreting scGPT: A foundation model for single-cell biology in real-world cancer clinical trial data

Abstract The success of large language models (LLMs) with attention mechanisms in natural language processing has inspired a series of LLMs for single-cell analysis, such as scGPT. Such LLMs claim to have excellent performance as foundation models across various downstream tasks. This study aims to validate scGPT and explore its potential in cancer research using real-world clinical trial data. We assessed scGPT's performance in both zero-shot and fine-tuned scenarios on various tasks using single-nucleus sequencing data from pancreatic ductal adenocarcinoma (PDAC) patients undergoing different treatments, including ones treated at the Dana-Farber Cancer Institute. Our methodology encompassed a comprehensive evaluation of scGPT's capabilities, including zero-shot clustering, gene expression prediction, and cell reconstruction. We fine-tuned the model for downstream tasks like cell type annotation, treatment perturbation prediction, and gene regulatory network inference. Additionally, we extracted and analyzed multi-head, multi-layer embeddings and attention matrices, visualizing the flow of information within the model and how they respond to different perturbations or fine-tuning objectives to investigate the model's learning process and its correlation with biological information. The study utilized two main datasets for fine-tuning and evaluating scGPT: an unpublished clinical trial dataset featuring metastatic PDAC patients under three comparative treatment arms with pre- and on-treatment single-nucleus sequencing data, and a published clinical trial dataset containing sequencing data from PDAC patients who underwent various treatment regimens. Our results validate scGPT's potential to effectively extract biological information as a foundation model for single-cell biology. For zero-shot, scGPT demonstrated strong performance in tasks like clustering, while showing areas with suboptimal performance like gene expression prediction and reconstruction. In addition, fine-tuning significantly boosts the model's capabilities across various tasks. The model responds well to multiple fine-tuning objectives, capturing information that distinguishes between individuals and treatments, and accurately predicting patient and treatment groups at the single-cell level. The analysis of model structure and information flow revealed that multi-head attention and representations could, to some extent, capture biological information. For example, the model’s attention to certain genes varies in different heads and layers, with being altered to certain patterns responding to fine-tuning objectives. Our analysis of the model's architecture and attention mechanisms offers preliminary insights into the relationship between model behavior and biological processes. It also provides a framework for interpreting complex biological information through the lens of attention mechanisms in single cell LLMs, paving the way for future studies exploring the intersection of foundation models and cancer biology, potentially leading to transformative progress in clinical cancer care. Citation Format: Runzi Tan, Haotian Cui, Bo Wang, Kimberly Perez, Andressa Dias Costa, Alexander Jordan, Thomas Karacic, Dalia Elganainy, Dan Y Gui, Suryun Kim, Chen Yuan, Morgan Truitt, Michael Downes, Ronald Evans, Tae Gyu Oh, Peter O’Dwyer, Andrew Aguirre, Jonathan A Nowak, Brian Wolpin, Simona Cristea. Evaluating and interpreting scGPT: A foundation model for single-cell biology in real-world cancer clinical trial data [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Advances in Pancreatic Cancer Research; 2024 Sep 15-18; Boston, MA. Philadelphia (PA): AACR; Cancer Res 2024;84(17 Suppl_2):Abstract nr A029.