Cross-modal Analysis Research Articles

Cross-Modality Medical Image Segmentation via Enhanced Feature Alignment and Cross Pseudo Supervision Learning.

Given the diversity of medical images, traditional image segmentation models face the issue of domain shift. Unsupervised domain adaptation (UDA) methods have emerged as a pivotal strategy for cross modality analysis. These methods typically utilize generative adversarial networks (GANs) for both image-level and feature-level domain adaptation through the transformation and reconstruction of images, assuming the features between domains are well-aligned. However, this assumption falters with significant gaps between different medical image modalities, such as MRI and CT. These gaps hinder the effective training of segmentation networks with cross-modality images and can lead to misleading training guidance and instability. To address these challenges, this paper introduces a novel approach comprising a cross-modality feature alignment sub-network and a cross pseudo supervised dual-stream segmentation sub-network. These components work together to bridge domain discrepancies more effectively and ensure a stable training environment. The feature alignment sub-network is designed for the bidirectional alignment of features between the source and target domains, incorporating a self-attention module to aid in learning structurally consistent and relevant information. The segmentation sub-network leverages an enhanced cross-pseudo-supervised loss to harmonize the output of the two segmentation networks, assessing pseudo-distances between domains to improve the pseudo-label quality and thus enhancing the overall learning efficiency of the framework. This method's success is demonstrated by notable advancements in segmentation precision across target domains for abdomen and brain tasks.

Photoacoustic Tomography with Temporal Encoding Reconstruction (PATTERN) for cross-modal individual analysis of the whole brain

Cross-modal analysis of the same whole brain is an ideal strategy to uncover brain function and dysfunction. However, it remains challenging due to the slow speed and destructiveness of traditional whole-brain optical imaging techniques. Here we develop a new platform, termed Photoacoustic Tomography with Temporal Encoding Reconstruction (PATTERN), for non-destructive, high-speed, 3D imaging of ex vivo rodent, ferret, and non-human primate brains. Using an optimally designed image acquisition scheme and an accompanying machine-learning algorithm, PATTERN extracts signals of genetically-encoded probes from photobleaching-based temporal modulation and enables reliable visualization of neural projection in the whole central nervous system with 3D isotropic resolution. Without structural and biological perturbation to the sample, PATTERN can be combined with other whole-brain imaging modalities to acquire the whole-brain image with both high resolution and morphological fidelity. Furthermore, cross-modal transcriptome analysis of an individual brain is achieved by PATTERN imaging. Together, PATTERN provides a compatible and versatile strategy for brain-wide cross-modal analysis at the individual level.

Cross-modal analysis of facial EMG in micro-expressions and data annotation algorithm

Cross-modal analysis of facial EMG in micro-expressions and data annotation algorithm

A cross-modal analysis of lexical sophistication: EFL and ESL learners in written and spoken production

Abstract The present study investigated the effects of two usage-related factors – modality (written vs. spoken) and language learning contexts (EFL vs. ESL) – on lexical sophistication in second language (L2) production. We measured 14 features of lexical sophistication in written and spoken texts produced by EFL and ESL learners with matched proficiency. The results showed significant interactions of modality and L2 learning contexts in several indices. In three indices, the EFL learners used more sophisticated words in writing than in speaking. In six indices, the gaps of lexical sophistication scores between writing and speaking were greater for the EFL than the ESL group. The conjoined effects of these factors are argued to stem from disparity in the amount and type of L2 input provided in EFL and ESL contexts.

Frequency dependent emotional responses to sound: A cross-modal analysis of electroencephalogram band power and emotion perception

Enabling machines to comprehend human emotions serves as the motivating factor for our research. This effort significantly contributes to advancing emotion recognition and human-computer interaction. Our study explores the fundamental associations between emotional responses and auditory characteristics, with a primary focus on frequency. To achieve this, we conducted a two-stage perception test, aiming to identify the interplay between EEG band power and dominant emotion class as a function of frequency. Musical notes, ranging between 110 Hz and 973 Hz, served as our stimuli. During the first stage, we collected participants' EEG data, categorized it into distinct energy bands (Alpha (α), Beta (β), Delta (δ), Theta (θ), and Gamma (γ)), and analyzed band power and ratios. In the subsequent stage, we had participants evaluate their emotional responses to each stimulus. Our results from this two-stage perception test suggest that an increased tangent of the α / β band power ratio corresponds to Low-Arousal emotions, while a diminishing tangent correlates with High-Arousal emotions. Furthermore, we observed a crossover point for four primary emotions within the 417–440 Hz frequency range. This finding supports the hypothesis that the 432–440 Hz range is emotionally neutral.

Explainable multi-task learning for multi-modality biological data analysis

Current biotechnologies can simultaneously measure multiple high-dimensional modalities (e.g., RNA, DNA accessibility, and protein) from the same cells. A combination of different analytical tasks (e.g., multi-modal integration and cross-modal analysis) is required to comprehensively understand such data, inferring how gene regulation drives biological diversity and functions. However, current analytical methods are designed to perform a single task, only providing a partial picture of the multi-modal data. Here, we present UnitedNet, an explainable multi-task deep neural network capable of integrating different tasks to analyze single-cell multi-modality data. Applied to various multi-modality datasets (e.g., Patch-seq, multiome ATAC + gene expression, and spatial transcriptomics), UnitedNet demonstrates similar or better accuracy in multi-modal integration and cross-modal prediction compared with state-of-the-art methods. Moreover, by dissecting the trained UnitedNet with the explainable machine learning algorithm, we can directly quantify the relationship between gene expression and other modalities with cell-type specificity. UnitedNet is a comprehensive end-to-end framework that could be broadly applicable to single-cell multi-modality biology. This framework has the potential to facilitate the discovery of cell-type-specific regulation kinetics across transcriptomics and other modalities.

Multimodal charting of molecular and functional cell states via in situ electro-sequencing

Paired mapping of single-cell gene expression and electrophysiology is essential to understand gene-to-function relationships in electrogenic tissues. Here, we developed in situ electro-sequencing (electro-seq) that combines flexible bioelectronics with in situ RNA sequencing to stably map millisecond-timescale electrical activity and profile single-cell gene expression from the same cells across intact biological networks, including cardiac and neural patches. When applied to human-induced pluripotent stem-cell-derived cardiomyocyte patches, in situ electro-seq enabled multimodal in situ analysis of cardiomyocyte electrophysiology and gene expression at the cellular level, jointly defining cell states and developmental trajectories. Using machine-learning-based cross-modal analysis, in situ electro-seq identified gene-to-electrophysiology relationships throughout cardiomyocyte development and accurately reconstructed the evolution of gene expression profiles based on long-term stable electrical measurements. In situ electro-seq could be applicable to create spatiotemporal multimodal maps in electrogenic tissues, potentiating the discovery of cell types and gene programs responsible for electrophysiological function and dysfunction.

Destin2: Integrative and cross-modality analysis of single-cell chromatin accessibility data.

We propose Destin2, a novel statistical and computational method for cross-modality dimension reduction, clustering, and trajectory reconstruction for single-cell ATAC-seq data. The framework integrates cellular-level epigenomic profiles from peak accessibility, motif deviation score, and pseudo-gene activity and learns a shared manifold using the multimodal input, followed by clustering and/or trajectory inference. We apply Destin2 to real scATAC-seq datasets with both discretized cell types and transient cell states and carry out benchmarking studies against existing methods based on unimodal analyses. Using cell-type labels transferred with high confidence from unmatched single-cell RNA sequencing data, we adopt four performance assessment metrics and demonstrate how Destin2 corroborates and improves upon existing methods. Using single-cell RNA and ATAC multiomic data, we further exemplify how Destin2's cross-modality integrative analyses preserve true cell-cell similarities using the matched cell pairs as ground truths. Destin2 is compiled as a freely available R package available at https://github.com/yuchaojiang/Destin2.

Cross Analysis of Genomic-Pathologic Features on Multiple Primary Hepatocellular Carcinoma.

Hepatocellular carcinoma (HCC) is a prevalent malignancy cancer worldwide with a poor prognosis. Hepatic resection is indicated as a potentially curative option for HCC patients in the early stage. However, due to multiple nodules, it leads to clinical challenges for surgical management. Approximately 41%–75% of HCC cases are multifocal at initial diagnosis, which may arise from multicentric occurrence (MO-HCC) or intrahepatic metastasis (IM-HCC) pattern with significantly different clinical outcomes. Effectively differentiating the two mechanisms is crucial to prioritize the allocation of surgery for multifocal HCC. In this study, we collected a multifocal hepatocellular carcinoma cohort of 17 patients with a total of 34 samples. We performed whole-exome sequencing and staining of pathological HE sections for each lesion. Reconstruction of the clonal evolutionary pattern using genome mutations showed that the intrahepatic metastogenesis pattern had a poorer survival performance than independent origins, with variants in the TP53, ARID1A, and higher CNV variants occurring more significantly in the metastatic pattern. Cross-modality analysis with pathology showed that molecular classification results were consistent with pathology results in 70.6% of patients, and we found that pathology results could further complement the classification for undefined patterns of occurrence. Based on these results, we propose a model to differentiate the pattern of multifocal hepatocellular carcinoma based on the pathological results and genome mutations information, which can provide guidelines for diagnosing and treating multifocal hepatocellular carcinoma.

Integrative analysis of multimodal traffic data: addressing open challenges using big data analytics in the city of Lisbon

Worldwide cities are establishing efforts to collect urban traffic data from various modes and sources. Integrating traffic data, together with their situational context, offers more comprehensive views on the ongoing mobility changes and supports enhanced management decisions accordingly. Hence, cities are becoming sensorized and heterogeneous sources of urban data are being consolidated with the aim of monitoring multimodal traffic patterns, encompassing all major transport modes—road, railway, inland waterway—, and active transport modes such as walking and cycling. The research reported in this paper aims at bridging the existing literature gap on the integrative analysis of multimodal traffic data and its situational urban context. The reported work is anchored on the major findings and contributions from the research and innovation project Integrative Learning from Urban Data and Situational Context for City Mobility Optimization (ILU), a multi-disciplinary project on the field of artificial intelligence applied to urban mobility, joining the Lisbon city Council, public carriers, and national research institutes. The manuscript is focused on the context-aware analysis of multimodal traffic data with a focus on public transportation, offering four major contributions. First, it provides a structured view on the scientific and technical challenges and opportunities for data-centric multimodal mobility decisions. Second, rooted on existing literature and empirical evidence, we outline principles for the context-aware discovery of multimodal patterns from heterogeneous sources of urban data. Third, Lisbon is introduced as a case study to show how these principles can be enacted in practice, together with some essential findings. Finally, we instantiate some principles by conducting a spatiotemporal analysis of multimodality indices in the city against available context. Concluding, this work offers a structured view on the opportunities offered by cross-modal and context-enriched analysis of traffic data, motivating the role of Big Data to support more transparent and inclusive mobility planning decisions, promote coordination among public transport operators, and dynamically align transport supply with the emerging urban traffic dynamics.

Distributed Storage Strategy and Visual Analysis for Economic Big Data

With the increasing popularity of a large number of Internet-based services and a large number of services hosted on cloud platforms, a more powerful back-end storage system is needed to support these services. At present, it is very difficult or impossible to implement a distributed storage to meet all the above assumptions. Therefore, the focus of research is to limit different characteristics to design different distributed storage solutions to meet different usage scenarios. Economic big data should have the basic requirements of high storage efficiency and fast retrieval speed. The large number of small files and the diversity of file types make the storage and retrieval of economic big data face severe challenges. This paper is oriented to the application requirements of cross-modal analysis of economic big data. According to the source and characteristics of economic big data, the data types are analyzed and the database storage architecture and data storage structure of economic big data are designed. Taking into account the spatial, temporal, and semantic characteristics of economic big data, this paper proposes a unified coding method based on the spatiotemporal data multilevel division strategy combined with Geohash and Hilbert and spatiotemporal semantic constraints. A prototype system was constructed based on Mongo DB, and the performance of the multilevel partition algorithm proposed in this paper was verified by the prototype system based on the realization of data storage management functions. The Wiener distributed memory based on the principle of Wiener filter is used to store the workload of each workload distributed storage window in a distributed manner. For distributed storage workloads, this article adopts specific types of workloads. According to its periodicity, the workload is divided into distributed storage windows of specific duration. At the beginning of each distributed storage window, distributed storage is distributed to the next distributed storage window. Experiments and tests have verified the distributed storage strategy proposed in this article, which proves that the Wiener distributed storage solution can save platform resources and configuration costs while ensuring Service Level Agreement (SLA).

A multimodal cell census and atlas of the mammalian primary motor cortex

Here we report the generation of a multimodal cell census and atlas of the mammalian primary motor cortex as the initial product of the BRAIN Initiative Cell Census Network (BICCN). This was achieved by coordinated large-scale analyses of single-cell transcriptomes, chromatin accessibility, DNA methylomes, spatially resolved single-cell transcriptomes, morphological and electrophysiological properties and cellular resolution input–output mapping, integrated through cross-modal computational analysis. Our results advance the collective knowledge and understanding of brain cell-type organization1–5. First, our study reveals a unified molecular genetic landscape of cortical cell types that integrates their transcriptome, open chromatin and DNA methylation maps. Second, cross-species analysis achieves a consensus taxonomy of transcriptomic types and their hierarchical organization that is conserved from mouse to marmoset and human. Third, in situ single-cell transcriptomics provides a spatially resolved cell-type atlas of the motor cortex. Fourth, cross-modal analysis provides compelling evidence for the transcriptomic, epigenomic and gene regulatory basis of neuronal phenotypes such as their physiological and anatomical properties, demonstrating the biological validity and genomic underpinning of neuron types. We further present an extensive genetic toolset for targeting glutamatergic neuron types towards linking their molecular and developmental identity to their circuit function. Together, our results establish a unifying and mechanistic framework of neuronal cell-type organization that integrates multi-layered molecular genetic and spatial information with multi-faceted phenotypic properties.

Cross-modality analysis of left ventricular function and volume in patients with st-segment elevation myocardial infarction: impact of infarct patterns

Abstract Funding Acknowledgements Type of funding sources: Public hospital(s). Main funding source(s): Sonderforschungsbereich 1116, University Düsseldorf Background Left ventricular (LV) function is considered of major importance to estimate prognosis of patients with ST-segment elevation myocardial infarction (STEMI) [1]. Although transthoracic echocardiography (TTE) after STEMI is recommended to assess for LV function and identify patients at high risk for rehospitalisation due to heart failure, Cardiovascular Magnetic Resonance (CMR) is still considered the gold standard for an adequate quantification of volumes, function and infarct size (IS) [2]. In addition, unenhanced TTE has been shown to underestimate LV function [3]. It is complicating that parameters such as IS and area-at risk (AAR) are thought to further deteriorate prognosis [4,5]. Purpose To test the hypothesis that cross-modality 2D/3D analyses using TTE, CMR and cineventriculography are comparable for the determination of LV-function in STEMI and that infarct patterns influence the determination of LV-function and volumes across modalities. Methods In a group of 70 patients after STEMI, estimation of LV function and volumes was performed by biplane cineventriculography during acute catheterization, biplane (2D) CMR 28 ± 16 hours (1 day) as well as 2D/3D-CMR and TTE 114 ± 31 hours (5 days) post STEMI. 3D-TTE after STEMI was only performed in a subset of 36 patients respectively. IS and AAR were assessed by T2-weighted and contrast enhanced CMR (1.5T) as percentage of the LV mass. Results Both, 2D- and 3D-, LV ejection fraction (LVEF) were comparable between the single modalities. Stroke volume (SV) and LV end-diastolic volume (LVEDV) were underestimated when comparing 2D- to 3D-CMR and when comparing TTE/cineventriculography to CMR. Correlating IS to the difference of 2D- and 3D-CMR, higher IS/AAR did not alter previous results, except from a fairly increased difference in SV (R = 0.3). For 2D-TTE versus 2D-CMR, higher IS/AAR increased differences in LVEDV (IS & AAR: p <0.001 both, R = -0.58, R=-0.54). For 3D-TTE and 3D-CMR, IS and AAR increased the difference in LVEF (p = 0.009, R = 0.50; p < 0.001, R = 0.60) and LVEDV (p = 0.021, R=-0.5; p = 0.027, R=-0.46). A general linear model (GLM) for repeated measures including IS as covariate supported this finding by showing a significant interaction effect between the imaging modalities (TTE/CMR) for LVEF estimation and IS. AAR increased the difference in LVEF comparing cineventriculography and 2D-CMR as well (p = 0.001, R = 0.47). This was again supported by a significant interaction effect as detected by the GLM. Conclusions There is good agreement between LVEF analysis across dimensions (2D and 3D) and different modalities (TTE, cineventriculography and CMR) whereas the comparison of SV and LVEDV is hampered. However, with increasing IS or AAR, LVEF differences between modalities increase and 3D-CMR, if available, or alternatively 3D-TTE should be used for the most accurate estimation of LVEF.

Neuroimaging Phenotyping and Assessment of Structural-Metabolic-Electrophysiological Alterations in the Temporal Neocortex of Focal Cortical Dysplasia IIIa.

Focal cortical dysplasia IIIa (FCD IIIa) is a common histopathological finding in temporal lobe epilepsy. However, subtle alterations in the temporal neocortex of FCD IIIa renders presurgical diagnosis and definition of the resective range challenging. To explore neuroimaging phenotyping and structural-metabolic-electrophysiological alterations in FCD IIIa. Retrospective. One hundred and sixty-seven subjects aged 4-39 years, including 64 FCD IIIa patients, 89 healthy controls and 14 FCD I patients as disease controls. 3 T, fast-spin-echo T2 -weighted fluid-attenuated inversion recovery (FLAIR), synthetic T1 -weighted magnetization prepared rapid acquisition gradient echo (MPRAGE). Surface-based linear model was applied to reveal neuroimaging phenotyping in FCD IIIa and assess its relationship with clinical variables. Logistic regression was implemented to identify FCD IIIa patients. Epileptogenicity mapping (EM) was conducted to explore the structural-metabolic-electrophysiological alterations in temporal neocortex of FCD IIIa. Student's t-test was applied to determine the significance of paired differences. Calibration curves were plotted to assess the goodness-of-fit (GOF) of the models, combined with the Hosmer-Lemeshow test. FCD IIIa exhibited widespread hyperintensities in temporal neocortex, and these alterations correlated with disease duration (Puncorrected < 0.01). Machine learning model accurately identified 84.4% of FCD IIIa patients, 92.1% of healthy controls and 92.9% of FCD I patients. Cross-modality analysis showed a significant negative correlation between FLAIR hyperintensity and positron emission tomography hypometabolism P < 0.01). Furthermore, epileptogenic cortices were located predominantly in brain regions with FLAIR hyperintensity and hypometabolism. FCD IIIa exhibited widespread temporal neocortex FLAIR hyperintensity. Automated machine learning of neuroimaging patterns is conducive for accurate identification of FCD IIIa. The degree and distribution of morphological alterations related to the extent of metabolic and epileptogenic abnormalities, lending support to its potential value for reduction of the radiative and invasive approaches during presurgical workup. 3 TECHNICAL EFFICACY STAGE: 2.

Cross-modal learning with prior visual relation knowledge

Visual relational reasoning is a central component in recent cross-modal analysis tasks, which aims at reasoning about the visual relationships between objects and their properties. These relationships provide rich semantics and help to enhance the visual representation for improving cross-modal learning. Previous works have succeeded in modeling latent visual relationships or rigid-categorized visual relationships. However, these kinds of methods leave out the problem of ambiguity inherent in the visual relationships because of the diverse relational semantics of different visual appearances. In this work, we explore to model the visual relationships by context-aware representations based on human prior knowledge. Based on such representations, we novelly propose a plug-and-play visual relational reasoning module to enhance image encoding. Specifically, we design an Anisotropic Graph Convolution to utilize the information of relation embeddings and relation directionality between objects for generating relation-aware image representations. We demonstrate the effectiveness of the relational reasoning module by applying it to both Visual Question Answering (VQA) and Cross-Modal Information Retrieval (CMIR) tasks. Extensive experiments are conducted on VQA 2.0 and CMPlaces datasets and superior performance is reported when comparing with state-of-the-art works.

DeepDance: Music-to-Dance Motion Choreography With Adversarial Learning

The creation of improvised dancing choreographies is an important research field of cross-modal analysis. A key point of this task is how to effectively create and correlate music and dance with a probabilistic one-to-many mapping, which is essential to create realistic dances of various genres. To address this issue, we propose a GAN-based cross-modal association framework, DeepDance, which correlates two different modalities (dance motion and music) together, aiming at creating the desired dance sequence in terms of the input music. Its generator is to predictively produce the dance movements best-fit to current music piece by learning from examples. In another hand, its discriminator acts as an external evaluation from the audience and judges the whole performance. The generated dance movements and the corresponding input music are considered to be well-matched if the discriminator cannot distinguish the generated movements from the training samples according to the estimated probability. By adding motion consistency constraints in our loss function, the proposed framework is able to create long realistic dance sequences. To alleviate the problem of expensive and inefficient data collection, we propose an effective approach to create a large-scale dataset, YouTube-Dance3D, from open data source. Extensive experiments on currently available music-dance datasets and our YouTube-Dance3D dataset demonstrate that our approach effectively captures the correlation between music and dance and can be used to choreograph appropriate dance sequences.

Reasoning on the Relation: Enhancing Visual Representation for Visual Question Answering and Cross-Modal Retrieval

Cross-modal analysis has become a promising direction for artificial intelligence. Visual representation is crucial for various cross-modal analysis tasks that require visual content understanding. Visual features which contain semantical information can disentangle the underlying correlation between different modalities, thus benefiting the downstream tasks. In this paper, we propose a Visual Reasoning and Attention Network (VRANet) as a plug-and-play module to capture rich visual semantics and help to enhance the visual representation for improving cross-modal analysis. Our proposed VRANet is built based on the bilinear visual attention module which identifies the critical objects. We propose a novel Visual Relational Reasoning (VRR) module to reason about pair-wise and inner-group visual relationships among objects guided by the textual information. The two modules enhance the visual features at both relation level and object level. We demonstrate the effectiveness of the proposed VRANet by applying it to both Visual Question Answering (VQA) and Cross-Modal Information Retrieval (CMIR) tasks. Extensive experiments conducted on VQA 2.0, CLEVR, CMPlaces, and MS-COCO datasets indicate superior performance comparing with state-of-the-art work.

A Multimodal Saliency Model for Videos with High Audio-Visual Correspondence.

Audio information has been bypassed by most of current visual attention prediction studies. However, sound could have influence on visual attention and such influence has been widely investigated and proofed by many psychological studies. In this paper, we propose a novel multi-modal saliency (MMS) model for videos containing scenes with high audio-visual correspondence. In such scenes, humans tend to be attracted by the sound sources and it is also possible to localize the sound sources via cross-modal analysis. Specifically, we first detect the spatial and temporal saliency maps from the visual modality by using a novel free energy principle. Then we propose to detect the audio saliency map from both audio and visual modalities by localizing the moving-sounding objects using cross-modal kernel canonical correlation analysis, which is first of its kind in the literature. Finally we propose a new two-stage adaptive audiovisual saliency fusion method to integrate the spatial, temporal and audio saliency maps to our audio-visual saliency map. The proposed MMS model has captured the influence of audio, which is not considered in the latest deep learning based saliency models. To take advantages of both deep saliency modeling and audio-visual saliency modeling, we propose to combine deep saliency models and the MMS model via a later fusion, and we find that an average of 5% performance gain is obtained. Experimental results on audio-visual attention databases show that the introduced models incorporating audio cues have significant superiority over state-of-the-art image and video saliency models which utilize a single visual modality.

Transfer Correlation Between Textual Content to Images for Sentiment Analysis

In social media, images and texts are used to convey individuals’ attitudes and feelings; thus, social media has become an indispensable part of people’s lives. To understand social behavior and provide better recommendations, sentiment analysis on social media is helpful. One sentiment analysis task is polarity prediction. Although current research on visual or textual sentiment analysis has achieved quite good progress, multimodal and cross-modal analysis combining visual and textual correlation is still in the exploration stage. To capture a semantic connection between images and captions, this paper proposes a cross-modal approach that considers both images and captions in classifying image sentiment polarity. This method transfers the correlation between textual content to images. First, the image and its corresponding caption are sent into an inner-class mapping model, where they are transformed into vectors in Hilbert space to obtain their labels by calculating the inner-class maximum mean discrepancy (MMD). Then, a class-aware sentence representation (CASR) model assigns the distributed representation to the labels with a class-aware attention-based gated recurrent unit (GRU). Finally, an inner-class dependency LSTM (IDLSTM) classifies the sentiment polarity. Experiments carried out on the Getty Images dataset and Twitter 1269 dataset demonstrate the effectiveness of our approach. Moreover, extensive experimental results show that our model outperforms baseline solutions.

Abstract 3776: Spatially resolved immunogenomic analyses reveal diverse sub tumoral microenvironments in the context of melanoma immunotherapy

Abstract Sustained periods of apparent clinical benefit despite lack of objective response are well known in a subpopulation of advanced melanoma patients. Inter-individual heterogeneity in response of separate tumors is common, characterizing complex overall response patterns. The molecular and cellular dynamics facilitating such long-term survival and heterogeneous response is poorly understood, particularly in the era of exposure to multiple potentially active therapies. We studied an exceptional case of long-term survival in a patient with non-responding metastatic melanoma in order to characterize the clonal and microenvironmental factors active across 3 time points. We performed immunogenomic analyses of 3 metachronous tumors, including a systemic therapy-naïve mass, 67 intratumor sub-regions of a non-responding mass during PD-1 inhibitor therapy, and a post-PD-1 inhibitor mass. We profiled samples using whole exome sequencing, RNA-sequencing (RNA-seq), immunohistochemistry (IHC), and T cell receptor sequencing. Longitudinal, spatial, and cross-modal analyses were performed. Longitudinal analyses identified mutations in several genes known to be associated with targeted or immune therapy resistance affecting distinct metastases. Genomic intratumoral heterogeneity (ITH) was primarily driven by subclonal copy number alterations that showed evidence of spatially-distinct evolution which may be in response to selective pressures at the tumor margin. RNA-seq revealed an unexpectedly high degree of ITH characterized by limited group-level gene or pathway associations with physical or immune characteristics of each site. Spatially-distinct pockets of immune activation and suppression were observed throughout the PD-1 inhibitor resistant metastasis despite a largely immune-excluded phenotype seen on IHC. A specific T cell Vβ CDR3 rearrangement was identified as dominant and recurrent not only across multiple spatial points within a single tumor mass, but also across metachronous tumors spanning the patient’s disease course. Immunophenotyping of the T cell population with single-cell RNA-seq suggested repeated T-cell priming events leading to the persistence of both activated and exhausted T cells bearing the same TCR-β at any given time. Our findings highlight an unexpected level of genomic and immune heterogeneity in metastatic melanoma tumors of a long-term surviving patient. The observed degree of ITH across local tumor microenvironments reiterates the inherent limitations to identifying robust and reproducible predictive biomarkers of therapy response based on limited physical sampling of tumors. Further spatiotemporal analysis of metastatic lesions in the context of immune checkpoint blockade will be required to determine how the mechanisms driving convergent microenvironmental phenotypes may be harnessed for therapeutic gain. Citation Format: Akash Mitra, Miles C. Andrews, Whijae Roh, Mariana P. de Macedo, Alexandre Reuben, Fernando Carapeto, Feng Wang, Sangeetha M. Reddy, Khalida Wani, Christine Spencer, John Miller, Aislyn Schalck, Latasha D. Little, Donald A. Sakellariou-Thompson, Curtis Gumbs, Wen-Jen Hwu, Chantale Bernatchez, Jianhua Zhang, Patrick Hwu, Nicholas Navin, Padmanee Sharma, James P. Allison, Jennifer Wargo, Alexander J. Lazar, Philip A. Futreal. Spatially resolved immunogenomic analyses reveal diverse sub tumoral microenvironments in the context of melanoma immunotherapy [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2019; 2019 Mar 29-Apr 3; Atlanta, GA. Philadelphia (PA): AACR; Cancer Res 2019;79(13 Suppl):Abstract nr 3776.