A deep learning based method for large-scale classification, registration, and clustering of in-situ hybridization experiments in the mouse olfactory bulb

Abstract

BackgroundThe Allen Mouse Brain Atlas allows study of the brain’s molecular anatomy at cellular scale, for thousands genes. To fully leverage this resource, one must register histological images of brain tissue – a task made challenging by the brain’s structural complexity and heterogeneity, as well as inter-experiment variability. New methodWe have developed a deep-learning based methodology for classification and registration of thousands of sections of brain tissue, using the mouse olfactory bulb (OB) as a case study. ResultsWe trained a convolutional neural network (CNN) to derive an image similarity measure for in-situ hybridization experiments, and embedded these in a low-dimensional feature space to guide the design of registration templates. We then compiled a high quality, registered atlas of gene expression for the OB (the first such atlas for the OB, to our knowledge). As proof-of-principle, the atlas was clustered using non-negative matrix factorization to reveal canonical expression motifs, and to identify novel, lamina-specific marker genes. Comparison with existing methodsOur method leverages virtues of CNNs for a set of important problems in molecular neuroanatomy, with performance comparable to existing methods. ConclusionThe atlas we have complied allows for intra- and inter-laminar comparisons of gene expression patterns in the OB across thousands of genes, as well identification of canonical expression profiles through clustering. We anticipate that this will be a useful resource for investigators studying the bulb’s development and functional topography. Our methods are publicly available for those interested in extending them to other brain areas.