Interpretable Representation Research Articles

TorchSISSO: A PyTorch-based implementation of the sure independence screening and sparsifying operator for efficient and interpretable model discovery

Symbolic regression (SR) is a powerful machine learning approach that searches for both the structure and parameters of algebraic models, offering interpretable and compact representations of complex data. Unlike traditional regression methods, SR explores progressively complex feature spaces, which can uncover simple models that generalize well, even from small datasets. Among SR algorithms, the Sure Independence Screening and Sparsifying Operator (SISSO) has proven particularly effective in the natural sciences, helping to rediscover fundamental physical laws as well as discover new interpretable equations for materials property modeling. However, its widespread adoption has been limited by performance inefficiencies and the challenges posed by its FORTRAN-based implementation, especially in modern computing environments. In this work, we introduce TorchSISSO, a native Python implementation built in the PyTorch framework. TorchSISSO leverages GPU acceleration, easy integration, and extensibility, offering a significant speed-up and improved accuracy over the original. We demonstrate that TorchSISSO matches or exceeds the performance of the original SISSO across a range of tasks, while dramatically reducing computational time and improving accessibility for broader scientific applications.

A cell atlas foundation model for scalable search of similar human cells.

Single-cell RNA-seq (scRNA-seq) has profiled hundreds of millions of human cells across organs, diseases, development, and perturbations to date. Mining these growing atlases could reveal cell-disease associations, discover cell states in unexpected tissue contexts, and relate in vivo biology to in vitro models. These require a common measure of cell similarity across the body and an efficient way to search. Here, we develop SCimilarity, a metric learning framework to learn a unified and interpretable representation that enables rapid queries of tens of millions of cell profiles from diverse studies for cells that are transcriptionally similar to an input cell profile or state. We use SCimilarity to query a 23.4 million cell atlas of 412 scRNA-seq studies for macrophage and fibroblast profiles from interstitial lung disease1 and reveal similar cell profiles across other fibrotic diseases and tissues. The top scoring in vitro hit for the macrophage query was a 3D hydrogel system2, which we experimentally demonstrated reproduces this cell state. SCimilarity serves as a foundation model for single-cell profiles that enables researchers to query for similar cellular states across the human body, providing a powerful tool for generating biological insights from the Human Cell Atlas.

Mathematical foundation of High-Dimensional Data Analysis: Leveraging Topology and Geometry for Enhanced Model Interpretability in AI

One of the most important challenges for modern AI and machine learning is the analysis of high-dimensional data. Traditional methods face serious complications in such cases due to high complexity of datasets: the curse of dimensionality, overfitting, and lack of transparency of model behavior. In this paper, we adopt a novel approach to analyze high-dimensional data; topological and geometric techniques will be exploited, taking advantage of better model interpretability and deeper insights into the structure. Precisely, we discuss Topological Data Analysis, mainly Persistent Homology (Edelsbrunner et al., 2002), which allows the extraction of topological features-like loops and connected components that enable the extracting knowledge about the global structure of data. We also see how some concepts of differential geometry and Riemannian geometry (Do Carmo, 1976) can be used to cast light on manifold data structure lying at the heart of any attempt at modeling intrinsic patterns in high-dimensional spaces. We will review how these mathematical pillars, combined with state-of-the-art techniques for dimensionality reduction like t-SNE, UMAP, Principal Component Analysis, are able to provide interpretable and low-dimensional representations of high-dimensional data that can be used to understand models and make decisions. Case studies are also included, which explain the practical working of these methods in AI systems and show how much complex models can be made transparent using these, especially in domains that are very critical, such as healthcare (Caruana et al., 2015), finance (Chen et al., 2018), and autonomous systems ( Wang et al., 2019). We also discuss some of the difficulties in using these methods for practical applications: computational complexity; the need for large-scale data processing (Bengio et al., 2007); and integration of topological and geometric intuition with the rest of the machine learning pipeline (Zhu et al., 2020). We conclude with possible future directions of research toward fine-tuning these methods and exploring their broader applicability to AI in its quest for more robust, interpretable, and reliable AI models. Given this work, we focus on how linking topology, geometry, and AI bears great promise for solving one of today's critical challenges: model interpretability in high-dimensional data analysis.

Synchronization-Inspired Interpretable Neural Networks.

Synchronization is a ubiquitous phenomenon in nature that enables the orderly presentation of information. In the human brain, for instance, functional modules such as the visual, motor, and language cortices form through neuronal synchronization. Inspired by biological brains and previous neuroscience studies, we propose an interpretable neural network incorporating a synchronization mechanism. The basic idea is to constrain each neuron, such as a convolution filter, to capture a single semantic pattern while synchronizing similar neurons to facilitate the formation of interpretable functional modules. Specifically, we regularize the activation map of a neuron to surround its focus position of the activated pattern in a sample. Moreover, neurons locally interact with each other, and similar ones are synchronized together during the training phase adaptively. Such local aggregation preserves the globally distributed representation nature of the neural network model, enabling a reasonably interpretable representation. To analyze the neuron interpretability comprehensively, we introduce a series of novel evaluation metrics from multiple aspects. Qualitative and quantitative experiments demonstrate that the proposed method outperforms many state-of-the-art algorithms in terms of interpretability. The resulting synchronized functional modules show module consistency across data and semantic specificity within modules.

Chimeric U-Net – Modifying the standard U-Net towards explainability

Healthcare guided by semantic segmentation has the potential to improve our quality of life through early and accurate disease detection. Convolutional Neural Networks, especially the U-Net-based architectures, are currently the state-of-the-art learning-based segmentation methods and have given unprecedented performances. However, their decision-making processes are still an active field of research. In order to reliably utilize such methods in healthcare, explainability of how the segmentation was performed is mandated. To date, explainability is studied and applied heavily in classification tasks. In this work, we propose the Chimeric U-Net, a U-Net architecture with an invertible decoder unit, that inherently brings explainability into semantic segmentation tasks. We find that having the restriction of an invertible decoder does not hinder the performance of the segmentation task. However, the invertible decoder helps to disentangle the class information in the latent space embedding and to construct meaningful saliency maps. Furthermore, we found that with a simple k-Nearest-Neighbours classifier, we could predict the Intersection over Union scores of unseen data, demonstrating that the latent space, constructed by the Chimeric U-Net, encodes an interpretable representation of the segmentation quality. Explainability is an emerging field, and in this work, we propose an alternative approach, that is, rather than building tools for explaining a generic architecture, we propose constraints on the architecture which induce explainability. With this approach, we could peer into the architecture to reveal its class correlations and local contextual dependencies, taking an insightful step towards trustworthy and reliable AI. Code to build and utilize the Chimeric U-Net is made available under:https://github.com/kenrickschulze/Chimeric-UNet---Half-invertible-UNet-in-Pytorch

A novel classification of dyslipidaemia through the analysis of five million lipid profiles: an unsupervised machine learning approach

Abstract Background Dyslipidemia encompasses a wide range of lipoprotein disorders categorised through two classifications (Fredrickson-Levy [FL] or Sniderman).(1,2) However, both classifications are criticised for relying on incomplete knowledge of lipoprotein metabolism, especially with the emergence of novel treatment options and variations in individual treatment responses.(3) Clustering, an unsupervised machine learning (ML) algorithm that can process a wide range of variables, has the potential to unmask patient groups with distinct molecular profiles and unique therapeutic targets that can inform more effective prevention strategies for cardiovascular disease (CVD).(4) Aim We aimed to use unsupervised ML algorithms to discover intrinsic dyslipidaemia categories from lipoprotein measurements, recognise the necessary components of lipid panels for classification, and analyse the similarities between the newly formed clusters, FL and Sniderman classifications. Methods Lipid profiles of 5,080,248 patients were obtained from the ‘Very Large Database of Lipids’ database. This yielded up to 78 blood components per patient, including at least 31 lipoprotein variables. The analysis involved unsupervised K-means clustering with optimised values for K and the subset of variables, determined in an unsupervised manner using a suitable measure of complexity. We then interpreted our clusters using probabilistic decision trees to provide compact and interpretable representations. Finally, we compared the clusters with Sniderman and FL categories. Results In a completely unsupervised fashion, we identified 14 clusters that could be matched to Sniderman categories. The confusion matrix showed total agreement of 76% (see Figure 1, left panel), relative Cohen’s kappa of 0.78 (the relative version captures accuracy on categories containing smaller numbers of patient profiles) and an accuracy of 96% on the small Type III class. Similar results were observed when matching to FL types. We accurately represented our clusters using probabilistic decision trees of small depth (see Figure 2). We discovered that the data had low intrinsic dimension and a manifold-like structure in which the different clusters could be illustrated (see Figure 1, right panel). Specifically, only 3 variables were needed to obtain our classification: apolipoprotein b, total cholesterol and triglycerides. Conclusion We showed that completely unsupervised ML techniques can uncover dyslipidaemia categories in lipoprotein profiles from a large patient population. The categories largely align with existing classifications based on prior knowledge of lipoprotein metabolism. Furthermore, few lipoprotein variables were required for categorisation (low-dimension data), which could aid in determining which lipoproteins should be measured in a clinical setting. Further analysis of the differences between ML clusters and traditional classifications is needed, which may enhance CVD risk management.

Multi-adversarial autoencoders: Stable, faster and self-adaptive representation learning

The variational autoencoder (VAE) and generative adversarial networks (GAN) are two prominent approaches to achieving a probabilistic generative model by way of an autoencoder and a two-player minimax game. While VAEs often suffer from over-simplified posterior approximations, the adversarial autoencoder (AAE) has shown promise by adopting GAN to match the variational posterior to an arbitrary prior through adversarial training. Both VAEs and GANs face significant challenges such as training stability, mode collapse, and difficulty in extracting meaningful latent representations. In this paper, we propose the Multi-adversarial Autoencoder (MAAE), which extends the AAE framework by incorporating multiple discriminators and enabling soft-ensemble feedback. By adaptively regulating the collective feedback from multiple discriminators, MAAE captures a balance between fitting the data distribution and performing accurate inference and accelerates training stability while extracting meaningful and interpretable latent representations. Experimental evaluations on MNIST, CIFAR10, and CelebA datasets demonstrate significant improvements in latent representation, quality of generated samples, log-likelihood, and a pairwise comparison metric, with comparisons to recent methods.

Interpretable Representation and Customizable Retrieval of Traffic Congestion Patterns Using Causal Graph-Based Feature Associations

The substantial increase in traffic data offers new opportunities to inspect traffic congestion dynamics from different perspectives. This paper presents a novel framework for the interpretable representation and customizable retrieval of traffic congestion patterns using causal relation graphs, which harnesses many of these opportunities. By integrating domain knowledge with innovative data management techniques, we address the challenges of effectively handling and retrieving the growing volume of traffic data for diverse analytical purposes. The framework leverages causal graphs to encode traffic congestion patterns, capturing fundamental phenomena and their spatiotemporal relationships, thus facilitating an interpretable high-level view of traffic dynamics. Moreover, a customizable similarity measurement function is introduced based on inexact graph matching, allowing users to tailor the retrieval process to specific requirements. This framework’s capability to retrieve customizable and interpretable congestion patterns is demonstrated through extensive experiments with real-world highway traffic data in the Netherlands, highlighting its value in supporting diverse data-driven studies and applications.

Graph Fourier transform for spatial omics representation and analyses of complex organs

Spatial omics technologies decipher functional components of complex organs at cellular and subcellular resolutions. We introduce Spatial Graph Fourier Transform (SpaGFT) and apply graph signal processing to a wide range of spatial omics profiling platforms to generate their interpretable representations. This representation supports spatially variable gene identification and improves gene expression imputation, outperforming existing tools in analyzing human and mouse spatial transcriptomics data. SpaGFT can identify immunological regions for B cell maturation in human lymph nodes Visium data and characterize variations in secondary follicles using in-house human tonsil CODEX data. Furthermore, it can be integrated seamlessly into other machine learning frameworks, enhancing accuracy in spatial domain identification, cell type annotation, and subcellular feature inference by up to 40%. Notably, SpaGFT detects rare subcellular organelles, such as Cajal bodies and Set1/COMPASS complexes, in high-resolution spatial proteomics data. This approach provides an explainable graph representation method for exploring tissue biology and function.

Deciphering 3'UTR Mediated Gene Regulation Using Interpretable Deep Representation Learning.

The 3' untranslated regions (3'UTRs) of messenger RNAs contain many important cis-regulatory elements that are under functional and evolutionary constraints. It is hypothesized that these constraints are similar to grammars and syntaxes in human languages and can be modeled by advanced natural language techniques such as Transformers, which has been very effective in modeling complex protein sequence and structures. Here 3UTRBERT is described, which implements an attention-based language model, i.e., Bidirectional Encoder Representations from Transformers (BERT). 3UTRBERT is pre-trained on aggregated 3'UTR sequences of human mRNAs in a task-agnostic manner; the pre-trained model is then fine-tuned for specific downstream tasks such as identifying RBP binding sites, m6A RNA modification sites, and predicting RNA sub-cellular localizations. Benchmark results show that 3UTRBERT generally outperformed other contemporary methods in each of these tasks. More importantly, the self-attention mechanism within 3UTRBERT allows direct visualization of the semantic relationship between sequence elements and effectively identifies regions with important regulatory potential. It is expected that 3UTRBERT model can serve as the foundational tool to analyze various sequence labeling tasks within the 3'UTR fields, thus enhancing the decipherability of post-transcriptional regulatorymechanisms.

Interpretable representation learning for 3D multi-piece intracellular structures using point clouds.

A key challenge in understanding subcellular organization is quantifying interpretable measurements of intracellular structures with complex multi-piece morphologies in an objective, robust and generalizable manner. Here we introduce a morphology-appropriate representation learning framework that uses 3D rotation invariant autoencoders and point clouds. This framework is used to learn representations of complex multi-piece morphologies that are independent of orientation, compact, and easy to interpret. We apply our framework to intracellular structures with punctate morphologies (e.g. DNA replication foci) and polymorphic morphologies (e.g. nucleoli). We systematically compare our framework to image-based autoencoders across several intracellular structure datasets, including a synthetic dataset with pre-defined rules of organization. We explore the trade-offs in the performance of different models by performing multi-metric benchmarking across efficiency, generative capability, and representation expressivity metrics. We find that our framework, which embraces the underlying morphology of multi-piece structures, facilitates the unsupervised discovery of sub-clusters for each structure. We show how our approach can also be applied to phenotypic profiling using a dataset of nucleolar images following drug perturbations. We implement and provide all representation learning models using CytoDL, a python package for flexible and configurable deep learning experiments.

Unsupervised Anomaly Detection via Nonlinear Manifold Learning

Abstract Anomalies are samples that significantly deviate from the rest of the data and their detection plays a major role in building machine learning models that can be reliably used in applications such as data-driven design and novelty detection. The majority of existing anomaly detection methods either are exclusively developed for (semi) supervised settings, or provide poor performance in unsupervised applications where there are no training data with labeled anomalous samples. To bridge this research gap, we introduce a robust, efficient, and interpretable methodology based on nonlinear manifold learning to detect anomalies in unsupervised settings. The essence of our approach is to learn a low-dimensional and interpretable latent representation (aka manifold) for all the data points such that normal samples are automatically clustered together and hence can be easily and robustly identified. We learn this low-dimensional manifold by designing a learning algorithm that leverages either a latent map Gaussian process (LMGP) or a deep autoencoder (AE). Our LMGP-based approach, in particular, provides a probabilistic perspective on the learning task and is ideal for high-dimensional applications with scarce data. We demonstrate the superior performance of our approach over existing technologies via multiple analytic examples and real-world datasets.

Leveraging Brain Modularity Prior for Interpretable Representation Learning of fMRI.

Resting-state functional magnetic resonance imaging (rs-fMRI) can reflect spontaneous neural activities in the brain and is widely used for brain disorder analysis. Previous studies focus on extracting fMRI representations using machine/deep learning methods, but these features typically lack biological interpretability. The human brain exhibits a remarkable modular structure in spontaneous brain functional networks, with each module comprised of functionally interconnected brain regions-of-interest (ROIs). However, existing learning-based methods cannot adequately utilize such brain modularity prior. In this paper, we propose a brain modularity-constrained dynamic representation learning framework for interpretable fMRI analysis, consisting of dynamic graph construction, dynamic graph learning via a novel modularity-constrained graph neural network (MGNN), and prediction and biomarker detection. The designed MGNN is constrained by three core neurocognitive modules (i.e., salience network, central executive network, and default mode network), encouraging ROIs within the same module to share similar representations. To further enhance discriminative ability of learned features, we encourage the MGNN to preserve network topology of input graphs via a graph topology reconstruction constraint. Experimental results on 534 subjects with rs-fMRI scans from two datasets validate the effectiveness of the proposed method. The identified discriminative brain ROIs and functional connectivities can be regarded as potential fMRI biomarkers to aid in clinical diagnosis.

Developing a fair and interpretable representation of the clock drawing test for mitigating low education and racial bias

The clock drawing test (CDT) is a neuropsychological assessment tool to screen an individual’s cognitive ability. In this study, we developed a Fair and Interpretable Representation of Clock drawing test (FaIRClocks) to evaluate and mitigate classification bias against people with less than 8 years of education, while screening their cognitive function using an array of neuropsychological measures. In this study, we represented clock drawings by a priorly published 10-dimensional deep learning feature set trained on publicly available data from the National Health and Aging Trends Study (NHATS). These embeddings were further fine-tuned with clocks from a preoperative cognitive screening program at the University of Florida to predict three cognitive scores: the Mini-Mental State Examination (MMSE) total score, an attention composite z-score (ATT-C), and a memory composite z-score (MEM-C). ATT-C and MEM-C scores were developed by averaging z-scores based on normative references. The cognitive screening classifiers were initially tested to see their relative performance in patients with low years of education (< = 8 years) versus patients with higher education (> 8 years) and race. Results indicated that the initial unweighted classifiers confounded lower education with cognitive compromise resulting in a 100% type I error rate for this group. Thereby, the samples were re-weighted using multiple fairness metrics to achieve sensitivity/specificity and positive/negative predictive value (PPV/NPV) balance across groups. In summary, we report the FaIRClocks model, with promise to help identify and mitigate bias against people with less than 8 years of education during preoperative cognitive screening.

Improved diabetic retinopathy severity classification using squeeze-and-excitation and sparse light weight multi-level attention u-net with transfer learning from xception.

Diabetic Retinopathy (DR) is a significant cause of vision loss in diabetic patients, making early detection and accurate severity classification essential for effective management and prevention. This study aims to develop an enhanced DR severity classification approach using advanced model architectures and transfer learning to improve diagnostic accuracy and support better patient care. We propose a novel model, Xception Squeeze-and-Excitation Sparse Lightweight Multi-Level Attention U-Net (XceSE_SparseLwMLA-UNet), designed to classify DR severity using fundus images from the Messidor 1 and Messidor 2 datasets. The XceSE_SparseLwMLA-UNet integrates several advanced mechanisms: the Squeeze-and-Excitation (SE) mechanism for adaptive feature recalibration, the Sparse Lightweight Multi-Level Attention (SparseLwMLA) mechanism for effective contextual information integration, and transfer learning from the Xception architecture to enhance feature extraction capabilities. The SE mechanism refines channel-wise feature responses, while SparseLwMLA enhances the model's ability to identify complex DR patterns. Transfer learning utilizes pre-trained weights from Xception to improve generalization across DR severity levels. The proposed XceSE_SparseLwMLA-UNet model demonstrates superior performance in DR severity classification, achieving higher accuracy and improved multi-class F1 scores compared to existing models. The model's color-coded segmentation outputs offer interpretable visual representations, aiding medical professionals in assessing DR severity levels. The XceSE_SparseLwMLA-UNet model shows promise for advancing early DR diagnosis and management by enhancing classification accuracy and providing valuable visual insights. Its integration of advanced architectural features and transfer learning contributes to better patient care and improved visual health outcomes.

Multiview representation learning for identification of novel cancer genes and their causative biological mechanisms.

Tumorigenesis arises from the dysfunction of cancer genes, leading to uncontrolled cell proliferation through various mechanisms. Establishing a complete cancer gene catalogue will make precision oncology possible. Although existing methods based on graph neural networks (GNN) are effective in identifying cancer genes, they fall short in effectively integrating data from multiple views and interpreting predictive outcomes. To address these shortcomings, an interpretable representation learning framework IMVRL-GCN is proposed to capture both shared and specific representations from multiview data, offering significant insights into the identification of cancer genes. Experimental results demonstrate that IMVRL-GCN outperforms state-of-the-art cancer gene identification methods and several baselines. Furthermore, IMVRL-GCN is employed to identify a total of 74 high-confidence novel cancer genes, and multiview data analysis highlights the pivotal roles of shared, mutation-specific, and structure-specific representations in discriminating distinctive cancer genes. Exploration of the mechanisms behind their discriminative capabilities suggests that shared representations are strongly associated with gene functions, while mutation-specific and structure-specific representations are linked to mutagenic propensity and functional synergy, respectively. Finally, our in-depth analyses of these candidates suggest potential insights for individualized treatments: afatinib could counteract many mutation-driven risks, and targeting interactions with cancer gene SRC is a reasonable strategy to mitigate interaction-induced risks for NR3C1, RXRA, HNF4A, and SP1.

Interpretable wind power forecasting combining seasonal-trend representations learning with temporal fusion transformers architecture

Wind power forecasting plays a critical role in the efficient integration of wind energy into power systems. Accurate short-term WPF helps maintain grid stability, optimize power dispatch, and reduce operational costs. However, current deep learning-based WPF architectures rely on mapping spatial–temporal features into a single high-dimensional representation, compromising performance and interpretability. In this work, the interpretable seasonal-trend representation algorithm (ISTR) is proposed to learn a couple of disentangled latent representations that describe seasonal-trend of wind power time series. The ISTR is developed in a self-supervised manner with no need for prior knowledge, providing sufficient interpretable information while avoiding the feature entanglement problem. The ISTR is then integrated into the temporal fusion transformer (TFT) model through seasonal time-point embedding to perform accurate multi-horizon WPF with interpretable meanings. In the case study, ISTR-TFT outperforms the current state-of-the-art methods by achieving an 8.9% improvement in mean absolute percentage error (MAPE) compared to DeepAR and a 15.7% improvement in pinball score. The proposed ISTR-TFT also shows enhanced robustness to noise and missing data scenarios. The interpretability results generated by ISTR-TFT reveal the importance of various input features and the learned temporal patterns, offering valuable insights for decision-making in power system operations.

Unsupervised Learning of Disentangled and Interpretable Representations of Material Appearance

As humans, we have learned through experience how to interpret the visual appearance of materials in our environment, enabling us to predict the properties of an object just by looking at it for a few seconds. Although this seems like a straightforward task, the final appearance showed by a material is the result of a non-trivial interaction between confounding factors like geometry, illumination, or viewing angle, which we do not completely understand. Creating an algorithm able to disentangle the perceptual factors of material appearance present in an image, just like humans do, would be a breakthrough in several fields (e.g., in architecture, it could help to create prototypes that accurately reproduce how they will be perceived in the real life; following the inverse path of going from perceptual factors to images, it could be used in product design to intuitively define the perceptual features of a desired material, instead of delving into its mathematical definition). Here, we propose a learning-based algorithm capable of effectively disentangling certain perceptual features of images in an unsupervised way.

Santiago Jiménez - Unsupervised Learning of Disentangled and Interpretable Representations of Material Appearance

Unsupervised Learning of Disentangled and Interpretable Representations of Material Appearance

P-149 A visual interpretability method to unbox ‘black-box’ deep learning image-based classification of embryo properties

Abstract Study question Can we decipher the underlying visual properties that drive image-based AI embryo classification models to assist clinical decisions and biological discovery? Summary answer Our framework interpreted which annotated and non-explicitly-annotated phenotypes impact model predictions and rank their importance. These discoveries were aligned with known blastocyst quality criteria. What is known already Deep learning models have shown great promise for complex pattern recognition when applied to embryo images. The success of these models relies on their ability to perform non-linear optimization of feature extraction during model construction. However, this involves their entanglement of multiple classification-driving image properties, thereby producing ‘black-box’ systems that lack user confidence, trust and interpretability. Therefore, there is an urgent need for an interpretability method that can uncover the semantic image properties that contribute to ‘black box’ embryo image-based AI classification model predictions to assist in blastocyst selection. Study design, size, duration 11,211 time-lapse videos were retrospectively collected from three IVF centers. A deep convolutional neural network is first trained to discriminate high-versus-low quality blastocysts. We then developed DISCOVER, a general-purpose interpretability method designed to discover underlying visual properties driving the classifier. DISCOVER encodes an image to an interpretable lower dimensional representation which is correlated to the classifier and encapsulates a different distinct phenotype in each one of the dimensions. Participants/materials, setting, methods The encoding of embryo images to low dimensional representations enables interpretability globally and locally. Globally, the embryo images are synthetically altered by amplifying subtle properties that affect the classification decision. With our method this can be done one property at a time, therefore separating confounding properties. By evaluating the altered images, embryologists can decipher their meaning. Locally, each one of the discovered properties can be ranked by its importance for a specific embryo instance. Main results and the role of chance Using DISCOVER, we interpreted the classification model driving features. We quantitatively linked the top two classification features as blastocyst size (as proxy to degree of expansion and development) and trophectoderm quality, by embryologists evaluation and annotations. We then asked whether DISCOVER can identify non-explicitly annotated latent features that encode morphologic properties not defined by ASEBIR/Gardner criteria. Expert embryologist interpreted the third top classification feature to be the blastocoel. DISCOVER interpreted high quality embryos as having denser and more granular blastocoelic regions, suggesting that this change in the blastocoel appearance is one of the encoded classification-driving morphologic properties. This visualization indicates that there are additional parameters of the blastocoel beyond its volume expansion associated with its quality. We showed how embryo properties can be weighted differently by the classifier on a per embryo basis, giving clinical insight to which properties influence the classification of a specific instance. These results indicate that DISCOVER enables expert-in-the-loop interpretation of the classification model both globally, discovering the overall main properties driving the classifier, and locally, showing a per instance explanation. Limitations, reasons for caution DISCOVER failed to interpret the inner cell mass (ICM) as a classification-driving feature in its latent representation, though it was explicitly used to label the data for training the classification model. It is possible that other properties collectively contained the discriminative information encoded in the ICM. Wider implications of the findings This deep analysis demonstrates the feasibility of providing interpretability for biomedical image-based classification models for clinical use in the IVF clinic. Trial registration number not applicable