MSInet: A Self-Supervised CNN Framework Integrating Global and Local Context for Robust Mass Spectrometry Imaging Segmentation.

Historically, studies of drug biodistribution are traditionally carried out in the later stages of pre-clinical pharmaceutical research and development (R&D) using radio-labelled techniques. Such studies are often slow, expensive and unselective, meaning resulting data can be complicated to deconvolute and too late in the development pipeline to change the medicine under investigation. Mass spectrometry imaging (MSI) has the potential to provide an unlabelled, multiplex method of mapping and quantifying molecular distributions within tissues at a much earlier stage in the R&D timeline, informing researchers of exposure in target tissue or providing evidence of localised and accumulated drug concentration in tissues exhibiting symptoms of toxicity. The research presented in this thesis begins by exploring the use of MALDI, DESI and LESA-MSI in early pharmacokinetic cassette dosing studies. Furthermore, MSI techniques were applied to blood brain barrier penetration studies to assess compound penetration profiles. Quantitative MSI (qMSI) methods were studied using tissue mimetics to generate accurate calibration lines and produce in situ concentration data. Finally, region specific qMSI was used to quantify endogenous metabolite concentrations and evaluate tumour heterogeneity in several different tumour models, identifying a model that would be used in pre-clinical efficacy studies. The results indicate that MSI drug distribution studies can be performed much earlier in the lead optimisation stage of the drug discovery process. This was done using a range of MSI platforms with different sensitivity, spatial resolution and chemical scope. The use of LESA-MSI to assess drug blood brain barrier penetration revealed benefits 2 over non-spatially resolved analytical methods. The multiplex nature of MSI analysis was shown to mitigate residual blood contamination in brain tissue sections giving greater differentiation of poorly BBB permeable drugs. Development of quantitative LESA and DESI-MSI methods were used in conjunction with tissue mimetics to show that qMSI is a reliable way of generating in-tissue concentration data. qMSI results compared favourably with ‘gold standard’ LC-MS approaches. Finally, MALDI-qMSI was shown to be capable of generating region-specific concentration data of endogenous metabolites in heterogeneous tumour tissues. This culminated in drug project selection of a tumour model with a less heterogeneous lactate distribution, less intra-tumour lactate variability and a better platform to discriminate lactate modulation in drug dosed animals versus control in efficacy studies. The research presented in this thesis has shown that the MSI methodology developed can be successfully applied to pharmaceutical R&D. The validated protocols can be employed earlier in the development timeline allowing researchers time to evaluate and react to any data produced. Furthermore, MSI has been shown to be applicable in pharmacokinetic, pharmacodynamic and toxicity studies, offering spatially enhanced results that complement the data generated using existing analytical techniques and hence can make a contribution to safer, more efficacious medicines being brought to patients.

Inkjet ink classification and source prediction based on direct analysis in real-time mass spectrometry (DART-MS) via mass imaging and convolutional neural network (CNN)

Changes of Mass Spectra Patterns on a Brain Tissue Section Revealed by Deep Learning with Imaging Mass Spectrometry Data.

Mass spectrometry imaging is a powerful tool for investigating the spatial distribution of chemical compounds in a biological sample such as tissue. Two common goals of these experiments are unsupervised segmentation of images into newly discovered homogeneous segments and supervised classification of images into predefined classes. In both cases, the important secondary goals are to characterize the uncertainty associated with the segmentation and with the classification and to characterize the spectral features that define each segment or class. Recent analysis methods have focused on the spatial structure of the data to improve results. However, they either do not address these secondary goals or do this with separate post hoc procedures.We introduce spatial shrunken centroids, a statistical model-based framework for both supervised classification and unsupervised segmentation. It takes as input sets of previously detected, aligned, quantified, and normalized spectral features and expresses both spatial and multivariate nature of the data using probabilistic modeling. It selects informative subsets of spectral features that define each unsupervised segment or supervised class and quantifies and visualizes the uncertainty in spatial segmentations and in tissue classification. In the unsupervised setting, it also guides the choice of an appropriate number of segments. We demonstrate the usefulness of this framework in a supervised human renal cell carcinoma experimental dataset and several unsupervised experimental datasets, including a pig fetus cross-section, three rodent brains, and a controlled image with known ground truth. This framework is available for use within the open-source R package Cardinal as part of a full pipeline for the processing, visualization, and statistical analysis of mass spectrometry imaging experiments.

Mass spectrometry imaging (MSI)-based spatially resolved metabolomics addresses the limitations inherent in traditional liquid chromatography-tandem mass spectrometry (LC-MS)-based metabolomics, particularly the loss of spatial context within heterogeneous tissues. MSI not only enhances our understanding of disease aetiology but also aids in the identification of biomarkers and the assessment of drug toxicity and therapeutic efficacy by converting invisible metabolites and biological networks into visually rendered image data. In this comprehensive review, we illuminate the key advancements in MSI-driven spatially resolved metabolomics over the past few years. We first outline recent innovations in preprocessing methodologies and MSI instrumentation that improve the sensitivity and comprehensiveness of metabolite detection. We then delve into the progress made in functional visualization techniques, which enhance the precision of metabolite identification and annotation. Ultimately, we discuss the significant potential applications of spatially resolved metabolomics technology in translational medicine and drug development, offering new perspectives for future research and clinical translation. HIGHLIGHTS: MSI-driven spatial metabolomics preserves metabolite spatial information, enhancing disease analysis and biomarker discovery. Advances in MSI technology improve detection sensitivity and accuracy, expanding bioanalytical applications. Enhanced visualization techniques refine metabolite identification and spatial distribution analysis. Integration of MSI with AI promises to advance precision medicine and accelerate drug development.

MotivationMass spectrometry imaging (MSI) characterizes the molecular composition of tissues at spatial resolution, and has a strong potential for distinguishing tissue types, or disease states. This can be achieved by supervised classification, which takes as input MSI spectra, and assigns class labels to subtissue locations. Unfortunately, developing such classifiers is hindered by the limited availability of training sets with subtissue labels as the ground truth. Subtissue labeling is prohibitively expensive, and only rough annotations of the entire tissues are typically available. Classifiers trained on data with approximate labels have sub-optimal performance.ResultsTo alleviate this challenge, we contribute a semi-supervised approach mi-CNN. mi-CNN implements multiple instance learning with a convolutional neural network (CNN). The multiple instance aspect enables weak supervision from tissue-level annotations when classifying subtissue locations. The convolutional architecture of the CNN captures contextual dependencies between the spectral features. Evaluations on simulated and experimental datasets demonstrated that mi-CNN improved the subtissue classification as compared to traditional classifiers. We propose mi-CNN as an important step toward accurate subtissue classification in MSI, enabling rapid distinction between tissue types and disease states.Availability and implementationThe data and code are available at https://github.com/Vitek-Lab/mi-CNN_MSI.

To design a convolutional recurrent neural network (CRNN) that calculates three-dimensional (3D) positions of lung tumors from continuously acquired cone beam computed tomography (CBCT) projections, and facilitates the sorting and reconstruction of 4D-CBCT images. Under an IRB-approved clinical lung protocol, kilovoltage (kV) projections of the setup CBCT were collected in free-breathing. Concurrently, an electromagnetic signal-guided system recorded motion traces of three transponders implanted in or near the tumor. Convolutional recurrent neural network was designed to utilize a convolutional neural network (CNN) for extracting relevant features of the kV projections around the tumor, followed by a recurrent neural network for analyzing the temporal patterns of the moving features. Convolutional recurrent neural network was trained on the simultaneously collected kV projections and motion traces, subsequently utilized to calculate motion traces solely based on the continuous feed of kV projections. To enhance performance, CRNN was also facilitated by frequent calibrations (e.g., at 10° gantry rotation intervals) derived from cross-correlation-based registrations between kV projections and templates created from the planning 4DCT. Convolutional recurrent neural network was validated on a leave-one-out strategy using data from 11 lung patients, including 5500kV images. The root-mean-square error between the CRNN and motion traces was calculated to evaluate the localization accuracy. Three-dimensional displacement around the simulation position shown in the Calypso traces was 3.4±1.7mm. Using motion traces as ground truth, the 3D localization error of CRNN with calibrations was 1.3±1.4mm. CRNN had a success rate of 86±8% in determining whether the motion was within a 3D displacement window of 2mm. The latency was 20ms when CRNN ran on a high-performance computer cluster. CRNN is able to provide accurate localization of lung tumors with aid from frequent recalibrations using the conventional cross-correlation-based registration approach, and has the potential to remove reliance on the implanted fiducials.

Mass spectrometry imaging (MSI), which quantifies the underlying chemistry with molecular spatial information in tissue, represents an emerging tool for the functional exploration of pathological progression. Unsupervised machine learning of MSI datasets usually gives an overall interpretation of the metabolic features derived from the abundant ions. However, the features related to the latent lesions are always concealed by the abundant ion features, which hinders precise delineation of the lesions. Herein, we report a data-driven MSI data segmentation approach for recognizing the hidden lesions in the heterogeneous tissue without prior knowledge, which utilizes one-step prediction for feature selection to generate function-specific segmentation maps of the tissue. The performance and robustness of this approach are demonstrated on the MSI datasets of the ischemic rat brain tissues and the human glioma tissue, both possessing different structural complexity and metabolic heterogeneity. Application of the approach to the MSI datasets of the ischemic rat brain tissues reveals the location of the ischemic penumbra, a hidden zone between the ischemic core and the healthy tissue, and instantly discovers the metabolic signatures related to the penumbra. In view of the precise demarcation of latent lesions and the screening of lesion-specific metabolic signatures in tissues, this approach has great potential for in-depth exploration of the metabolic organization of complex tissue.

Mass spectrometry imaging (MSI) provides an untargeted characterization of the chemical composition of samples at spatial resolution. With the ability to quantify analytes from small metabolites to proteins in high throughput, and the applicability to various samples, such as tissues, plants, and microbiomes, MSI has become an important tool in spatial metabolomics and proteomics. In the spatial domain, the chemical composition varies across locations in the sample, e.g., due to different morphological structures, different pathological types, and different conditions. In the spectral domain, molecules can have similar spatial distributions, due to technical reasons, e.g., fragment ions, isotopic ions, and sodium adducts of a same analyte, or due to biological reasons, e.g., molecules being specific to a tissue compartment or cell type. The main objectives of MSI data analyses are 1) identify the spatial structure of the tissues defined by chemical compositions; 2) understand the associations of molecules in terms of spatial distribution. The first objective is achieved in part by segmenting MSI images into regions of homogeneous chemical compositions. To overcome the limitations of multivariate segmentation, I developed a method called spatial-DGMM for single ion image segmentation. It performs ion-specific tissue segmentation that accounts for spatial dependence between pixels and generates spatial-structure preserved summaries that are useful for downstream analyses. I evaluated this method on a simulated dataset and two experimental datasets from different ionization sources. In addition, I applied this method to an MSI investigation of metabolic perturbations in rat fibrotic liver tissues, induced by the dose of the combination of nevirapine and galactosamine. The first objective can also be achieved by distinguishing various sample types, such as tumor and non-tumor tissues, by supervised classification of mass spectra. Supervised classification methods require ground truth labels. In practice, while ground-truth labels are available at the whole tissue level, accessing such labels at the sub-tissue level is challenging. To address this issue, I developed mi-CNN, a multiple instance learning-based algorithm that classifies sub-tissue locations under weak supervision from tissue-level annotations. This method uses a convolutional neural network to capture the dependencies between spectral features. I extensively evaluated the proposed methods on datasets of diverse MSI workflows and biological samples and provided in-depth discussions on potential applications in MSI data analyses. The second objective of MSI data analysis can be achieved in part by unsupervised clustering of ion images. I contributed a deep clustering approach for ion images that accounts for both spatial contextual features and noise. This approach improved the interpretability of MSI data in the spectral domain by grouping ions from the same source into a same cluster more frequently than existing methods. In addition, it also improved the downstream interpretation in the spatial domain by using ions in representative cluster profiles as input for segmentation methods. These methods are implemented as a module CardinalNN, which is a part of Cardinal open-source software. The implementations are tested on multiple simulated and experimental datasets. The source codes, the documentation, and the vignettes with case studies are available on Bioconductor. Altogether, my work presents methods and workflows for MSI data analysis in both spatial and spectral domains. It complements the manual explorations and overcomes the limitations of current computational tools. --Author's abstract

Imaging the Clear Cell Renal Cell Carcinoma Proteome

Simple SummaryHigh-grade serous ovarian cancer (HGSOC) accounts for 70% of ovarian carcinomas with sobering survival rates. The mechanisms mediating treatment efficacy are still poorly understood with no adequate biomarkers of response to treatment and risk assessment. This variability of treatment response might be due to its molecular heterogeneity. Therefore, identification of biomarkers or molecular signatures to stratify patients and offer personalized treatment is of utmost priority. Currently, comprehensive gene expression profiling is time- and cost-extensive and limited by tissue heterogeneity. Thus, it has not been implemented into clinical practice. This study demonstrates for the first time a spatially resolved, time- and cost-effective approach to stratifying HGSOC patients by combining novel matrix-assisted laser desorption/ionization imaging mass spectrometry (MALDI-IMS) technology with machine-learning algorithms. Eventually, MALDI-derived predictive signatures for treatment efficacy, recurrent risk, or, as demonstrated here, molecular subtypes might be utilized for emerging clinical challenges to ultimately improve patient outcomes.Despite the correlation of clinical outcome and molecular subtypes of high-grade serous ovarian cancer (HGSOC), contemporary gene expression signatures have not been implemented in clinical practice to stratify patients for targeted therapy. Hence, we aimed to examine the potential of unsupervised matrix-assisted laser desorption/ionization imaging mass spectrometry (MALDI-IMS) to stratify patients who might benefit from targeted therapeutic strategies. Molecular subtyping of paraffin-embedded tissue samples from 279 HGSOC patients was performed by NanoString analysis (ground truth labeling). Next, we applied MALDI-IMS paired with machine-learning algorithms to identify distinct mass profiles on the same paraffin-embedded tissue sections and distinguish HGSOC subtypes by proteomic signature. Finally, we devised a novel approach to annotate spectra of stromal origin. We elucidated a MALDI-derived proteomic signature (135 peptides) able to classify HGSOC subtypes. Random forest classifiers achieved an area under the curve (AUC) of 0.983. Furthermore, we demonstrated that the exclusion of stroma-associated spectra provides tangible improvements to classification quality (AUC = 0.988). Moreover, novel MALDI-based stroma annotation achieved near-perfect classifications (AUC = 0.999). Here, we present a concept integrating MALDI-IMS with machine-learning algorithms to classify patients according to distinct molecular subtypes of HGSOC. This has great potential to assign patients for personalized treatment.

Purpose: Brain tumors are diagnosed and classified manually and noninvasively by radiologists using Magnetic Resonance Imaging (MRI) data. The risk of misdiagnosis may exist due to human factors such as lack of time, fatigue, and relatively low experience. Deep learning methods have become increasingly important in MRI classification. To improve diagnostic accuracy, researchers emphasize the need to develop Computer-Aided Diagnosis (CAD) computational diagnostics based on artificial intelligence (AI) systems by using deep learning methods such as convolutional neural networks (CNN) and improving the performance of CNN by combining it with other data analysis tools such as wavelet transform. In this study, a novel diagnostic framework based on CNN and DWT data analysis is developed for the diagnosis of glioma tumors in the brain, among other tumors and other diseases, with T2-SWI MRI scans. It is a binary CNN classifier that treats the disease “glioma tumor” as positive and the other pathologies as negative, resulting in a very unbalanced binary problem. The study includes a comparative analysis of a CNN trained with wavelet transform data of MRIs instead of their pixel intensity values in order to demonstrate the increased performance of the CNN and DWT analysis in diagnosing brain gliomas. The results of the proposed CNN architecture are also compared with a deep CNN pre-trained on VGG16 transfer learning network and with the SVM machine learning method using DWT knowledge. Methods: To improve the accuracy of the CNN classifier, the proposed CNN model uses as knowledge the spatial and temporal features extracted by converting the original MRI images to the frequency domain by performing Discrete Wavelet Transformation (DWT), instead of the traditionally used original scans in the form of pixel intensities. Moreover, no pre-processing was applied to the original images. The images used are MRIs of type T2-SWI sequences parallel to the axial plane. Firstly, a compression step is applied for each MRI scan applying DWT up to three levels of decomposition. These data are used to train a 2D CNN in order to classify the scans as showing glioma or not. The proposed CNN model is trained on MRI slices originated from 382 various male and female adult patients, showing healthy and pathological images from a selection of diseases (showing glioma, meningioma, pituitary, necrosis, edema, non-enchasing tumor, hemorrhagic foci, edema, ischemic changes, cystic areas, etc.). The images are provided by the database of the Medical Image Computing and Computer-Assisted Intervention (MICCAI) and the Ischemic Stroke Lesion Segmentation (ISLES) challenges on Brain Tumor Segmentation (BraTS) challenges 2016 and 2017, as well as by the numerous records kept in the public general hospital of Chania, Crete, “Saint George”. Results: The proposed frameworks are experimentally evaluated by examining MRI slices originating from 190 different patients (not included in the training set), of which 56% are showing gliomas by the longest two axes less than 2 cm and 44% are showing other pathological effects or healthy cases. Results show convincing performance when using as information the spatial and temporal features extracted by the original scans. With the proposed CNN model and with data in DWT format, we achieved the following statistic percentages: accuracy 0.97, sensitivity (recall) 1, specificity 0.93, precision 0.95, FNR 0, and FPR 0.07. These numbers are higher for this data format (respectively: accuracy by 6% higher, recall by 11%, specificity by 7%, precision by 5%, FNR by 0.1%, and FPR is the same) than it would be, had we used as input data the intensity values of the MRIs (instead of the DWT analysis of the MRIs). Additionally, our study showed that when our CNN takes into account the TL of the existing network VGG, the performance values are lower, as follows: accuracy 0.87, sensitivity (recall) 0.91, specificity 0.84, precision 0.86, FNR of 0.08, and FPR 0.14. Conclusions: The experimental results show the outperformance of the CNN, which is not based on transfer learning, but is using as information the MRI brain scans decomposed into DWT information instead of the pixel intensity of the original scans. The results are promising for the proposed CNN based on DWT knowledge to serve for binary diagnosis of glioma tumors among other tumors and diseases. Moreover, the SVM learning model using DWT data analysis performs with higher accuracy and sensitivity than using pixel values.

Convolutional Neural Networks (CNNs) have demonstrated impressive performance in complex machine learning tasks such as classification and regression problems. A reliable neural network structure plays a decisive role in CNN studies. Through comparing and analyzing the structure of neural networks, a model structure for better visualization performance has been discovered, and such a method supports the development of deep learning research. These studies are of particular importance in end-to-end systems for autonomous driving to imitate human driving, where the interpretability of the system is limited. Because of the uncertainty of the ground truth, for the determination of human steering in an image, it is difficult to accurately compare the visualization performance of different CNN models or different visualization methods. For practical applications, however, an objective and quantitative measure for assessing visualization performance is necessary. Therefore, a method to evaluate the visualization performance of CNN models using a driver model instead of human drivers is proposed, to generate a data set which can be used to determine the decisional point (ground truth) in the input image. Then, an exclusive method is also put forth, to quantitatively calculate the relationship between the decisional point (ground truth) and the visualization results produced by CNN models. In this paper, five CNN models as an autonomous steering controller are designed based on PilotNet, and the visualization abilities of each CNN models is compared by three evaluation indicators. By comparing the visualization performance of five different CNN models, it is shown that the proposed method can successfully assess the visualization level of the CNN model.

We present our results from training and evaluating a convolutional neural network (CNN) to predict galaxy shapes from wide-field survey images of the first data release of the Dark Energy Survey (DES DR1). We use conventional shape measurements as ‘ground truth’ from an overlapping, deeper survey with less sky coverage, the Canada–France–Hawaii Telescope Lensing Survey (CFHTLenS). We demonstrate that CNN predictions from single band DES images reproduce the results of CFHTLenS at bright magnitudes and show higher correlation with CFHTLenS at fainter magnitudes than maximum likelihood model fitting estimates in the DES Y1 im3shape catalogue. Prediction of shape parameters with a CNN is also extremely fast, it takes only 0.2 ms per galaxy, improving more than 4 orders of magnitudes over forward model fitting. The CNN can also accurately predict shapes when using multiple images of the same galaxy, even in different colour bands, with no additional computational overhead. The CNN is again more precise for faint objects, and the advantage of the CNN is more pronounced for blue galaxies than red ones when compared to the DES Y1 metacalibration catalogue, which fits a single Gaussian profile using riz band images. We demonstrate that CNN shape predictions within the metacalibration self-calibrating framework yield shear estimates with negligible multiplicative bias, m < 10−3, and no significant point spread function (PSF) leakage. Our proposed set-up is applicable to current and next-generation weak lensing surveys where higher quality ‘ground truth’ shapes can be measured in dedicated deep fields.

Imaging mass spectrometry (IMS) is a novel molecular imaging technique to investigate how molecules are distributed between tumors and within tumor region in order to shed light into tumor biology or find potential biomarkers. Convolutional neural networks (CNNs) have proven to be very potent classifiers often outperforming other machine learning algorithms, especially in computational pathology. To overcome the challenge of complexity and high-dimensionality of the IMS data, the proposed CNNs are either very deep or use large kernels, which results in large amount of parameters and therefore a high computational complexity. An alternative is down-sampling the data, which inherently leads to a loss of information. In this paper, we propose using dilated CNNs as a possible solution to this challenge, since it allows for an increase of the receptive field size, neither by increasing the network parameters nor by decreasing the input signal resolution. Since the mass signature of cancer biomarkers are distributed over the whole mass spectrum, both locally- and globally-distributed patterns need to be captured to correctly classify the spectrum. By experiment, we show that employing dilated convolutions in the architecture of a CNN leads to a higher performance in tumor classification. Our proposed model outperforms the state-of-the-art for tumor classification in both clinical lung and bladder datasets by 1-3%.