Detailed simulation of optical photon transport and detection in radiation detectors is often used for crystal-based gamma detector optimization. However, the time and memory burden associated with the track-wise approach to particle transport and detection in commonly used Monte Carlo codes makes optical simulation prohibitive at a system level, where hundreds to thousands of scintillators must be modeled. Consequently, current large system simulations do not include detailed detector models to analyze the potential performance gain with new radiation detector technologies. Generative adversarial networks (GANs) are explored as a tool to speed up the optical simulation of crystal-based detectors. These networks learn training datasets made of high-dimensional data distributions. Once trained, the resulting model can produce distributions belonging to the training data probability distribution. In this work, we present the proof of concept of using a GAN to enable high-fidelity optical simulations of nuclear medicine systems, mitigating their computational complexity. The architecture of the first network version and high-fidelity training dataset is discussed. The latter is generated through accurate optical simulation with GATE/Geant4, and contains the position, direction, and energy distributions of the optical photons emitted by 511 keV gamma rays in bismuth germanate and detected on the photodetector face. We compare the GAN and simulation-generated distributions in terms of similarity using the Jensen–Shannon distance. Excellent agreement was found with similarity values higher than 93.5% for all distributions. Moreover, the GAN speeded the optical photon distribution generation by up to two orders of magnitude. These very promising results have the potential to drastically change the use of nuclear imaging system optical simulations by enabling high-fidelity system-level simulations in reasonable computation times. The ultimate is to integrate the GAN within GATE/Geant4 since numerous applications (large detectors, bright scintillators, Cerenkov-based timing positron emission tomography) can benefit from these improvements.