Abstract Purpose: The clustering of pediatric patients based on their unlabeled brain magnetic resonance (MR) images reveal the latent features and group structures. It is an unbiased approach to finding image-based markers and risk-stratification for neurocognitive functions. Analyzing large sets of MR images is time-intensive and challenging due to the MR images' high dimensionality and complexity. The extraction of low-dimension informative feature space instead of high-dimension MR image is an effective and fast solution to cluster images accurately. We aimed at developing a feature extraction method using deep learning. Our long-term research goal is to develop a feature extraction model using deep learning to automatically and accurately identify brain regions in association with neurocognitive outcomes. Methods: We used 124 registered fractional anisotropy (FA)-mapped of diffusion tensor (DT) images of pediatric medulloblastoma patients obtained after tumor surgery and before radiation from the SJMB03 clinical trial. First, we developed a 3D convolutional auto-encoder model based on a deep neural network for the voxel-wise analysis of 3D images to produce low-dimensional features, which can reconstruct input images accurately. Then, we applied the density-based clustering model, named OPTICS (Ordering Points To Identify the Clustering Structure), to the features. To train two models, we used a soft and dynamic algorithm to simultaneously combine two loss functions. Distance measure and CAHL (Cluster Assignment Loss Hardening) are used to measure feature learning and clustering performance. Results: Our experiment demonstrates that the proposed models yield results comparable in quality to manual clustering. The model trained regarding using a sufficient dataset and reducing dynamic loss function. Our model can cluster patients precisely and extract representative hidden features of the images, which is much more time- and labor-efficient compared to manual clustering by experts. Conclusions: In this study, we focused on a new feature extraction method and clustering of brain images. Because of the deep neural networks' unique advantages in image processing, we considered a 3D convolutional auto-encoder for the feature extraction part and OPTICS algorithm for the clustering part of the model. This method is highly useful in neuroimaging and neurocognitive research. Citation Format: Hadi Hosseini, Cheng Cheng, John Glass, Gene Reddick, Amar Gajjar, Zhaohua Lu. Image clustering of brain tumor patients using a deep neural network [abstract]. In: Proceedings of the AACR Virtual Special Conference on Artificial Intelligence, Diagnosis, and Imaging; 2021 Jan 13-14. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(5_Suppl):Abstract nr PO-077.