A multimodal neural network with gradient blending improves predictions of survival and metastasis in sarcoma

The 5-year survival of patients with soft tissue sarcoma (STS) is only 65% and has not improved over the past 3 decades. New innovations in matching patients to optimal treatments are needed to improve outcomes. While not indicated for all STS patients, the decision to use chemotherapy is influenced by the predicted aggressivity of a patient's sarcoma, and thus the patient's risk of developing metastases. Furthermore, the ideal surveillance regimen may be one tailored to a patient's perspectives and individualized risk of metastases. Whereas currently available nomograms use only clinical variables and cannot guide individualized management, the inclusion of rich data like MRIs in prediction models may result in predictions accurate enough to enable individualized treatment. The objective of this study is to develop a multimodal neural network model (MMNN) that analyzes clinical variables as well as MRI images of a sarcoma, to predict an STS patient's overall survival and risk of distant metastases. We compare the performance of this MMNN to other models based on clinical variables alone, radiomics models, and a unimodal neural network. All patients aged 18 or older with biopsy-confirmed non-retroperitoneal STS who underwent primary resection at MSKCC between January 1st, 2005, and December 31st, 2020 were reviewed. We included all patients with complete outcome data and a pre-treatment MRI with both a T1 post-contrast sequence and a T2 fat-sat sequence available. Preprocessing of the MRI data included N4 bias correction and z-score normalization. A total of 9380 MRI slices containing sarcomas are available for analysis. Our MMNN [Figure 1] accepts the T1 and T2 MRIs which are fed through an image subnetwork consisting of a 2-channel DenseNet-121. The T1 and T2 sequences are masked and cropped to contain only the tumor volume. Clinical variables are analyzed in a parallel deep neural network and this information is concatenated with the image features. A fully connected layer analyzes the combined multimodal features before outputting the predicted risk of each of our outcomes. Gradient blending is used to moderate the loss contributions of the different modalities during the training of multimodal neural networks. Visualization of the image features using heat maps was obtained using the Grad-CAM methodology. Our MMNN outperformed all other models in predicting overall survival and the risk of distant metastases [Table 1]. The C-Index of our MMNN for overall survival is 0.769, an absolute increase of 11.4% in AUC compared to the next best performing model. The heat maps demonstrate the areas of the sarcomas deemed most salient for the predictions [Figure 2]. This is the first multimodal neural network in sarcoma. Given the rarity of STS, the use of multimodal data in prediction algorithms is essential to overcoming limits of small sample sizes. Future work will seek to externally validate this model using federated learning. For any figures or tables, please contact the authors directly.

Development and Validation of an MRI-Based Signature to Predict Distant Metastasis in Nasopharyngeal Carcinoma Before Initial Treatment: A Retrospective Cohort Study

Computed Tomography-Based Radiomics Prediction of Biochemical Failure and Distant Metastasis in Patients With High- and Very High-Risk Localized Prostate Cancer

Introduction. Treatment of patients with brain metastases is an important problem that should be considered in the framework of combination approach. Introduction of new techniques of drug therapy as well as radiotherapy and neurosurgical treatment allows to significantly increase patient survival. Effective drug therapy and local control of brain metastases are of utmost importance in prediction of overall survival and patient quality of life.Aim. To investigate the prognostic factors for overall survival and intracranial progression (local recurrences, distant metastases) in patients with brain metastases of renal cancer after neurosurgical resection.Materials and methods. Retrospective analysis of the treatment results of 114 patients with metastatic brain lesions due to renal cancer who underwent neurosurgical resection (NSR) at the N. N. Blokhin National medical Research Center of Oncology was performed. Clinical data of 102 (89.5 %) of 114 patients for whom data on survival was available were evaluated. Among them, 80 (78.4 %) of patients died, 22 (21.5 %) are under observation. Extracranial disease status at the time of NSR was known in 82 (71.9 %) patients: 45 (54.8 %) patients had extracranial metastases, and 37 (45.1 %) did not. Total resection of brain metastases with perifocal and perivascular zones was performed in 92 (90.1 %) patients; in other cases, fragmental lesion resection was performed.Results. median overall survival after NSR was 13.8 months (95 % confidence interval 10.3–18.6). per study data, factors affecting overall survival of patients with brain metastases of renal cancer after neurosurgical resection were presence / absence of extracranial metastases and patient’s functional status. Local recurrences in the postoperative cavity after NSR were observed in 24 (21 %) of 114 patients. median time of local recurrence was not achieved. Statistically significant factor of high risk of recurrence in the postoperative cavity was presence of lesions with maximal diameter ≥2 cm. Development of new (distant) metastases was observed in 31 (27.2 %) of 114 patients. median survival without distant metastases in patients with brain metastases after NSR was not achieved. frequencies of distant metastases at 6, 12 and 24 months were 15.5; 24.1 and 35.8 % respectively. per multifactor analysis, factors affecting development of distant metastases in the brain after NSR are multiple metastatic brain lesions and presence of extracranial metastases.Conclusion. Neurosurgical resection in patients with cerebral metastases of renal cancer in the total group leads to median overall survival of 13.8 months. predictors of better overall survival are absence of extracranial metastases and high functional status.

Hematoma expansion occasionally occurs in patients with intracerebral hemorrhage (ICH), associating with poor outcome. Multimodal neural networks incorporating convolutional neural network (CNN) analysis of images and neural network analysis of tabular data are known to show promising results in prediction and classification tasks. We aimed to develop a reliable multimodal neural network model that comprehensively analyzes CT images and clinical variables to predict hematoma expansion. We retrospectively enrolled ICH patients at four hospitals between 2017 and 2021, assigning patients from three hospitals to the training and validation dataset and patients from one hospital to the test dataset. Admission CT images and clinical variables were collected. CT findings were evaluated by experts. Three types of models were developed and trained: (1) a CNN model analyzing CT images, (2) a multimodal CNN model analyzing CT images and clinical variables, and (3) a non-CNN model analyzing CT findings and clinical variables with machine learning. The models were evaluated on the test dataset, focusing first on sensitivity and second on area under the receiver operating curve (AUC). Two hundred seventy-three patients (median age, 71 years [59–79]; 159 men) in the training and validation dataset and 106 patients (median age, 70 years [62–82]; 63 men) in the test dataset were included. Sensitivity and AUC of a CNN model were 1.000 (95% confidence interval [CI] 0.768–1.000) and 0.755 (95% CI 0.704–0.807); those of a multimodal CNN model were 1.000 (95% CI 0.768–1.000) and 0.799 (95% CI 0.749–0.849); and those of a non-CNN model were 0.857 (95% CI 0.572–0.982) and 0.733 (95% CI 0.625–0.840). We developed a multimodal neural network model incorporating CNN analysis of CT images and neural network analysis of clinical variables to predict hematoma expansion in ICH. The model was externally validated and showed the best performance of all the models.

A multimodal machine learning framework to predict treatment duration in multiple myeloma patients

Biologically inspired control system is necessary to be increased. This paper proposed the new design of multimodal neural network inspired from human learning system which takes different action in different condition. The multimodal neural network consists of some recurrent neural networks (RNNs) those are separated into different condition. There is selector system that decides certain RNN system depending the current condition of the robot. In this paper, we implemented this system in pendulum mobile robot as the basic object of study. Several certain number of RNNs are implemented into certain different condition of tilt robot. RNN works alternately depending on the condition of robot. In order to prove the effectiveness of the proposed model, we simulated in the computer simulation Open Dynamic Engine (ODE) and compared with ordinary RNN. The proposed neural model successfully stabilize the applied robot (2-wheeled robot). This model is developed for implemented into humanoid balancing learning system as the final object of study.

Utilization of Image-Based Deep Learning in Multimodal Glaucoma Detection Neural Network from a Primary Patient Cohort.

<div>Abstract<p>Dysregulation of miRNA expression may influence breast cancer progression, and experimental evidence suggests that miRNA silencing might suppress breast cancer metastasis. However, the relationship between miRNA and metastasis must be confirmed before this approach can be applied in the clinic. To this end, we conducted a two-stage study in a cohort of 3,760 patients with breast cancer to first identify and then validate the association between miRNA expression and risk of distant metastasis. The first stage (discovery) entailed miRNA sequencing of 126 case–control pairs; qPCR was used to validate the findings in a separate set of 80 case–control pairs. The 13 miRNAs most differentially expressed between cases and controls were combined into an miRNA score that was significantly associated with risk of distant metastasis in a logistic regression model that also included clinical variables (tumor size and number of positive lymph nodes) (OR<sub>per unit increase in score</sub> = 1.30; 95% confidence interval, 1.03–1.66). The results of this study suggest that in women with invasive breast cancer, a miRNA score that incorporates both clinical variables and miRNA expression levels in breast tumor tissue is moderately predictive of risk of subsequent distant metastasis.</p>Significance:<p>A novel predictive scoring system for patients with breast cancer includes clinical variables and the expression levels of 13 miRNAs and may help to identify those at increased risk of distant metastasis.</p></div>

<div>Abstract<p>Dysregulation of miRNA expression may influence breast cancer progression, and experimental evidence suggests that miRNA silencing might suppress breast cancer metastasis. However, the relationship between miRNA and metastasis must be confirmed before this approach can be applied in the clinic. To this end, we conducted a two-stage study in a cohort of 3,760 patients with breast cancer to first identify and then validate the association between miRNA expression and risk of distant metastasis. The first stage (discovery) entailed miRNA sequencing of 126 case–control pairs; qPCR was used to validate the findings in a separate set of 80 case–control pairs. The 13 miRNAs most differentially expressed between cases and controls were combined into an miRNA score that was significantly associated with risk of distant metastasis in a logistic regression model that also included clinical variables (tumor size and number of positive lymph nodes) (OR<sub>per unit increase in score</sub> = 1.30; 95% confidence interval, 1.03–1.66). The results of this study suggest that in women with invasive breast cancer, a miRNA score that incorporates both clinical variables and miRNA expression levels in breast tumor tissue is moderately predictive of risk of subsequent distant metastasis.</p>Significance:<p>A novel predictive scoring system for patients with breast cancer includes clinical variables and the expression levels of 13 miRNAs and may help to identify those at increased risk of distant metastasis.</p></div>

Dysregulation of miRNA expression may influence breast cancer progression, and experimental evidence suggests that miRNA silencing might suppress breast cancer metastasis. However, the relationship between miRNA and metastasis must be confirmed before this approach can be applied in the clinic. To this end, we conducted a two-stage study in a cohort of 3,760 patients with breast cancer to first identify and then validate the association between miRNA expression and risk of distant metastasis. The first stage (discovery) entailed miRNA sequencing of 126 case-control pairs; qPCR was used to validate the findings in a separate set of 80 case-control pairs. The 13 miRNAs most differentially expressed between cases and controls were combined into an miRNA score that was significantly associated with risk of distant metastasis in a logistic regression model that also included clinical variables (tumor size and number of positive lymph nodes) (ORper unit increase in score = 1.30; 95% confidence interval, 1.03-1.66). The results of this study suggest that in women with invasive breast cancer, a miRNA score that incorporates both clinical variables and miRNA expression levels in breast tumor tissue is moderately predictive of risk of subsequent distant metastasis. SIGNIFICANCE: A novel predictive scoring system for patients with breast cancer includes clinical variables and the expression levels of 13 miRNAs and may help to identify those at increased risk of distant metastasis.

With an expectation of excellent locoregional control, ongoing efforts to de-intensify therapy for patients with human papillomavirus-associated squamous cell oropharyngeal cancer necessitate a better understanding of the metastatic risk for patients with this disease. The objective of this study was to determine what factors affect the risk of metastases in patients with squamous cell cancers of the oropharynx. Under a shared use agreement, 547 patients from Radiation Therapy Oncology Group 0129 and 0522 with nonmetastatic oropharyngeal squamous cell cancers who had a known p16 status and smoking status were analyzed to assess the association of clinical features with the development of distant metastases. The analyzed factors included the p16 status, sex, T stage, N stage, age, and smoking history. A multivariate analysis of 547 patients with a median follow-up of 4.8 years revealed that an age ≥ 50 years (hazard ratio [HR], 3.28; P = .003), smoking for more than 0 pack-years (HR, 3.09; P < .001), N3 disease (HR, 2.64; P < .001), T4 disease (HR, 1.63; P = .030), and a negative p16 status (HR, 1.60; P = .044) were all factors associated with an increased risk of distant disease. Age, smoking, N3 disease, T4 disease, and a negative p16 status were associated with the development of distant metastases in patients with squamous cell cancers of the oropharynx treated definitively with concurrent chemoradiation.

To develop a model based on the clinical variables for evaluating the risk of distant metastasis in patients with advanced nasopharyngeal carcinoma (NPC). From September,2007 to June,2015,a total of 238 consecutive patients with biopsy-proven NPC in stage Ⅲ-Ⅳ(M0) based on the AJCC TNM staging manual were enrolled in this study,including 106 male and 34 female patients with a median age of 45 years (range 18-68 years).In this cohort,126 patients received concurrent chemoradiotherapy,and 24 received chemotherapy and radiotherapy,and 40 had induction chemotherapy.We used the least absolute shrinkage and selection operator (LASSO) method to select the most significant features for establishing the model for assessing the risks of distant metastasis. Among the 18 clinical variables tested,5 were significantly associated with distant metastasis in advanced NPC,including plasma Epstein-Barr virus (EBV) DNA,neutrophil/lymphocytes (NLR),VCA-IgA,concurrent chemoradiotherapy,and induction chemotherapy.Based on these 5 clinical variables,we established the following model:risk score=1.73×EBV DNA+0.54×NLR+0.38×VCA-IgA-0.95×concurrent chemoradiotherapy-2.37×induction chemotherapy+0.51.The cutoff point of this model was-0.62,which classified the patients into high-risk and low-risk groups for distant metastasis.This model showed a good performance in predicting distant metastasis in patients with advanced NPC (P&lt;0.01). The model we established herein can be used for evaluating the risks of distant metastasis in patients with advanced NPC and provides assistance in the clinical decision-making on individualized treatment strategy.

Background Patients with triple-negative breast cancers (TNBC) are at an increased risk of distant metastasis compared to patients with other subtypes of breast cancer. While TNBCs are aggressive as a group, many are potentially curable, reflecting an underlying heterogeneity. As such, there is an interest in identifying factors that may allow further stratification of patients in relation to the risk of distant metastasis and, ultimately, better tailor treatment plans to individual patients. With the increasing use of neoadjuvant chemotherapy (NAC), past studies have shown that patients who achieve a pathological complete response (pCR) following NAC have a decreased risk of distant metastasis. However, beyond the presence or absence of pCR, other risk factors for distant metastasis have not been well characterized. Methods This is a single institution, retrospective study of women with TNBC treated with NAC, surgery, and radiation therapy (RT) between 2000 and 2013. The rate of distant metastasis was estimated and compared between patients who achieved pCR versus those who did not achieve pCR using Kaplan-Meier method. In patients who failed to achieve pCR, patient-specific and treatment-specific factors including age, race, menopause status, family history, smoking history, clinical stage, histology, NAC regimen, whether breast conserving surgery was performed, response to NAC, treatment with adjuvant chemotherapy, and use of RT boost were analyzed using multivariable cox proportional hazards method to evaluate factors associated with distant metastasis. Results A total of 153 patients with a median follow up of 48.6 months were included. Of the 153 patients, 108 (70.9%) were identified as not having pCR following NAC. Among those 45 patients that did achieve a pCR, only 1 patient (2.2%) went on to have distant metastasis. In contrast, of the 108 patients that failed to achieve a pCR, 47 (43.5%) went on to have distant metastasis. On univariable analysis, factors associated with distant metastasis in patients that did not achieve a pCR included increasing clinical and pathological T and N stage, positive pathologic lymph node status, multifocality, lymphovascular space invasion (LVSI), extranodal extension, and failure of downstaging after NAC. After controlling for potential confounders in multivariable analysis, higher pathological N stage (HR 2.18, 95% CI 1.12 - 4.22), positive pathologic lymph nodes (HR 2.21, 95% CI 1.02 - 4.80), LVSI (HR 1.87, 95% CI 1.04 - 3.37), and multifocality (HR 2.05, 95% CI 1.05 - 4.03) were found to be independent predictors of distant metastasis. Conclusions Approximately 43.5% of patients with TNBC that did not achieve a pCR went on to develop distant metastasis, perhaps reflecting an underlying chemo-resistance of these non-pCR tumors. Here we identify multiple risk factors associated with distant metastasis among patients not achieving a pCR, including positive lymph nodes, LVSI, high pathologic N stage, and multifocality. This data can be used to inform prognoses and treatment decisions in this high-risk cohort of patients and future clinical trials are warranted to lower the risk of distant metastasis in this population. Citation Format: Tricarico CJ, Gabani P, Weiner AA, Ochoa LL, Thomas MA, Zoberi I. Predictors of distant metastasis in patients with triple negative breast cancer who failed to achieve a pathological complete response after neoadjuvant chemotherapy [abstract]. In: Proceedings of the 2018 San Antonio Breast Cancer Symposium; 2018 Dec 4-8; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2019;79(4 Suppl):Abstract nr P2-08-39.

Achieving acoustically optimized structural design for the Guqin is crucial, combining its acoustic quality evaluation with craftsmanship. This paper proposes a parametric design method for non-uniform thickness distribution in the Guqin soundboard, enabling segmented thickness control and establishing 26 experimental models. Acoustic-structural coupled modal analysis models were constructed, and the acoustic quality of soundboards with varying thicknesses was evaluated. By comparing coupled modal characteristics and key acoustic parameters, the impact of panel structure on acoustic quality was quantitatively analyzed. The study reveals that adjusting the panel thickness distribution (±10 mm) alters the dominant vibration modes and their distribution, thereby influencing Guqin’s acoustic characteristics. To characterize the effect of all thickness distributions on acoustic quality and validate the correlations among thickness distribution, modal frequencies, and acoustic parameters, a multi-modal deep neural network (Mm-DNN) prediction model was developed. It integrates three subnetworks for predicting modal frequencies, acoustic parameters, and modal-acoustic parameters. The root mean square error of the model was < 0.13, and the loss value was < 0.52 × 10−2. Validation confirms Mm-DNN’s effectiveness in predicting Guqin’s acoustic performance from structural parameters. This research supports the industrialized manufacturing of Guqin and provides a digital design approach for predicting acoustic quality in traditional instruments.