Computational human phantoms are a well-established tool used to simulate the performance of medical imaging devices. The maximum resolution of voxelized phantoms is bounded in practice by the available computer memory, which is especially limited for graphics processing unit (GPU)-based simulations. For phantoms that have been segmented into a small number of tissue types, the use of a uniform grid is inefficient because large numbers of adjacent voxels are composed of the same tissue. The computer files used to store these phantoms can be compressed into a small fraction of their original size using standard lossless compression algorithms. Compression is used for storage and distribution, but not during the simulations because compression algorithms compress the data incrementally from the first byte to the last. This approach intrinsically prevents access to random locations along the data stream, which is an essential requirement of the majority of simulation algorithms. In this paper, we propose to store voxelized phantoms in memory using a binary tree structure, as a way to accomplish both data compression and fast random access. The composition of a particular voxel can be efficiently retrieved by traversing the binary tree down from the root node to the corresponding leaf node. As a drawback, the memory savings come at the cost of more frequent memory access, which might slow down simulations in which memory access constitutes a substantial fraction of the execution time. Methods to generate and efficiently store in memory a binary tree geometry, and an example implementation in a GPU-accelerated Monte Carlo code for X-ray imaging virtual clinical trials, are presented. Results of a simulation of mammography and tomosynthesis imaging with a computational breast phantom voxelized at 50 μm show that the implemented binary tree geometry can achieve a 50-fold reduction of memory use, with a 4.7% increase in simulation time.