As a leading cause of death, worldwide, cardiovascular disease is of great clinical importance. Among cardiovascular diseases, coronary artery disease (CAD) is a key contributor, and it is the attributed cause of death for 10% of all deaths annually. The prevalence of CAD is commensurate with the rise in new medical imaging technologies intended to aid in its diagnosis and treatment. The necessary clinical trials required to validate and optimize these technologies require a large cohort of carefully controlled patients, considerable time to complete, and can be prohibitively expensive. A safer, faster, less expensive alternative is using virtual imaging trials (VITs), utilizing virtual patients or phantoms combined with accurate computer models of imaging devices. In this work, we develop realistic, physiologically-informed models for coronary plaques for application in cardiac imaging VITs. Histology images of plaques at micron-level resolution were used to train a deep convolutional generative adversarial network (DC-GAN) to create a library of anatomically variable plaque models with clinical anatomical realism. The stability of each plaque was evaluated by finite element analysis (FEA) in which plaque components and vessels were meshed as volumes, modeled as specialized tissues, and subjected to the range of normal coronary blood pressures. To demonstrate the utility of the plaque models, we combined them with the whole-body XCAT computational phantom to perform initial simulations comparing standard energy-integrating detector (EID) CT with photon-counting detector (PCD) CT. Our results show the network is capable of generating realistic, anatomically variable plaques. Our simulation results provide an initial demonstration of the utility of the generated plaque models as targets to compare different imaging devices. Vast, realistic, and variable CAD pathologies can be generated to incorporate into computational phantoms for VITs. There they can serve as a known truth from which to optimize and evaluate cardiac imaging technologies quantitatively.