Fibroblasts are a key component of the tumor microenvironment (TME) that can serve as a scaffold for tumor cell migration and augment the tumor's ability to withstand harsh conditions. When activated by external or endogenous stimuli, normal fibroblasts become cancer associated fibroblasts (CAFs), a heterogeneous group of stromal cells in the tumor that are phenotypically and epigenetically different from normal fibroblasts. Dynamic crosstalk between cancer cells, immune cells, and CAFs through chemokines and surface signaling makes the TME conducive to tumor growth. When activated, CAFs promote tumorigenesis and metastasis through several phenomena including regulation of tumor immunity, metabolic reprogramming of the TME, extracellular matrixremodeling and contraction, and induction of therapeutic resistance. Ionizing radiation (radiation theraphy [RT]) is a potent immunological stimulant that has been shown to increase cytotoxic Teff infiltration and IFN-I stimulated genes. RT, however, is unable to overcome the infiltration and activation of immunosuppressive cells which can contribute to tumor progression. Another paradox of RT is that, while very effective at killing cancer cells, it can contribute to the formation of CAFs. This review examines how the interplay between CAFs and immune cells during RT contributes to organ fibrosis, immunosuppression, and tumor growth. We focus on targeting mechanistic pathways of CAF formation as a potentially effective strategy not only for preventing organ fibrosis, but also in hampering tumor progression in response to RT.