Background: Segmentation of white matter lesions and deep grey matter structures is an important task in the quantification of magnetic resonance imaging in multiple sclerosis. In this paper we explore segmentation solutions based on convolutional neural networks (CNNs) for providing fast, reliable segmentations of multimodal magnetic resonance images into lesion classes and normal-appearing grey and white-matter structures. We also explore the ability of such methods to transfer between institutions. Methods: We trained two state-of-the-art CNN architectures in cross-validation on the 2016 MSSEG training dataset: a reference implementation of a 3D Unet, and a more recently proposed architecture (DeepSCAN), to set a baseline for MS lesion segmentation performance. We then retrained those methods on a larger dataset from a single centre, with and without labels for other brain structures, and compared performance with the reference methods, and with other reference methods from recent publications. Results: Both CNN methods substantially outperform other approaches in the literature when trained on the MSSEG dataset, showing agreement with human raters in the range of human inter-rater variability. Both showed small drops in performance when trained on single-centre data and tested on the MSSEG dataset instead. When trained with the addition of anatomical labels, the performance of the 3D Unet degraded, while the performance of the DeepSCAN net improved. Overall, the DeepSCAN network predicting both lesion and anatomical labels was the best-performing network. A retrospective study of this classifier found that the classifier successfully identified several lesions missed by human raters. Interpretation: We present a method for simultaneously segmenting MS lesions and neuroanatomical structures, whose performance on lesion segmentation is close to that of human raters. The drop in performance when evaluated on external data is less drastic than reported by other researchers. Performance of 3D Unets, while excellent on datasets with small numbers of structures labelled, may suffer when the number of labels increases. Funding Statement: This research was supported by the Swiss Multiple Sclerosis Society and a grant from the Novartis Forschungsstiftung. Declaration of Interests: Richard McKInley, Rik Wepfer, Fabian Aschwanden, Lorenz Grunder, Raphaela Muri, CHristian Rummel, Rajeev Verma, Christian Weisstanner, Mauricio Reyes, Franca Wagner and Roland Wiest have no interests to declare. Anke Salmen has recieved speaker honoraria and/or travel compensation for activities with Almirall Hermal GmbH, Biogen, Merck, Novartis, Roche, and Sanofi Genzyme, none related to this work. Andrew Chan received honoraria for board and speaker honoraria from Actelion, Bayer, Biogen, Celgene, Merck, Novartis, Sanofi-Genzyme, Roche, Teva, all for hospital/university research funds. He has recieved reserach funds from Research funds: UCB, Biogen, Sanofi-Genzyme, and is an editor for European Journal of Neurology, and Clin Transl Neurosci. Ethics Approval Statement: Ethical approval for the study was granted by the local ethical commission (Cantonal Ethical Commission Bern, ’MS segmentation disease monitoring’, approval number 2016-02035).