Accurate and fast simulation of X-ray projection data from mesh models has many applications in academia and industry, ranging from 3D X-ray computed tomography (XCT) reconstruction algorithms to radiograph-based object inspection and quality control. While software tools for the simulation of X-ray projection data from mesh models are available, they lack either performance, public availability, flexibility to implement non-standard scanning geometries, or easy integration with existing 3D XCT software. In this paper, we propose CAD-ASTRA, a highly versatile toolbox for fast simulation of X-ray projection data from mesh models. While fully functional as standalone software, it is also compatible with the ASTRA toolbox, an open-source toolbox for flexible tomographic reconstruction. CAD-ASTRA provides three specialized GPU projectors based on state-of-the-art algorithms for 3D rendering, implemented using the NVIDIA CUDA Toolkit and the OptiX engine. First, it enables X-ray phase contrast simulations by modeling refraction through ray tracing. Second, it allows the back-propagation of projective errors to mesh vertices, enabling immediate application in mesh reconstruction, deep learning, and other optimization routines. Finally, CAD-ASTRA allows simulation of polychromatic X-ray projections from heterogeneous objects with a source of finite focal spot size. Use cases on a CAD-based inspection task, a phase contrast experiment, a combined mesh-volumetric data projection, and a mesh reconstruction demonstrate the wide applicability of CAD-ASTRA.