Computational models of tumor growth are valuable for simulating the dynamics of cancer progression and treatment responses. In particular, agent-based models (ABMs) tracking individual agents and their interactions are useful for their flexibility and ability to model complex behaviors. However, ABMs have often been confined to small domains or, when scaled up, have neglected crucial aspects like vasculature. Additionally, the integration into tumor ABMs of precise radiation dose calculations using gold-standard Monte Carlo (MC) methods, crucial in contemporary radiotherapy, has been lacking. Here, we introduce AMBER, an Agent-based fraMework for radioBiological Effects in Radiotherapy that computationally models tumor growth and radiation responses. AMBER is based on a voxelized geometry, enabling realistic simulations at relevant pre-clinical scales by tracking temporally discrete states stepwise. Its hybrid approach, combining traditional ABM techniques with continuous spatiotemporal fields of key microenvironmental factors such as oxygen and vascular endothelial growth factor, facilitates the generation of realistic tortuous vascular trees. Moreover, AMBER is integrated with TOPAS, an MC-based particle transport algorithm that simulates heterogeneous radiation doses. The impact of radiation on tumor dynamics considers the microenvironmental factors that alter radiosensitivity, such as oxygen availability, providing a full coupling between the biological and physical aspects. Our results show that simulations with AMBER yield accurate tumor evolution and radiation treatment outcomes, consistent with established volumetric growth laws and radiobiological understanding. Thus, AMBER emerges as a promising tool for replicating essential features of tumor growth and radiation response, offering a modular design for future expansions to incorporate specific biological traits.