Circadian rhythm disruption impacts the efficiency of both chemotherapy and immunotherapy, yet identifying the key factors involved remains challenging. Circadian rhythm disruption can trigger aberrant fibroblasts activation, suggesting potential roles of cancer-associated fibroblasts (CAFs) in addressing this issue. In this paper, TCGA-BLCA patients were classified into two subgroups based on the expression of core circadian rhythm genes (CCRGs). The CCRG-based subgroups showed distinct fibroblast-related signals, from which a risk model composed of five fibroblast-related genes was finally established with excellent survival prognostic value in both TCGA and GEO datasets. The risk model was positively associated with the infiltration of CAFs and can efficiently predict the immunotherapy response in BLCA. Besides, high-risk score was associated with reduced sensitivity to a majority of traditional chemotherapeutic drugs such as oxaliplatin and gemcitabine. Further, the correlation between CCRGs and the risk genes was analyzed. Among the five risk genes, FAM20C displayed the most extensive correlation with the CCRGs and exhibited the strongest connection with CAFs infiltration. Moreover, FAM20C independently served as a predictor for the response to immunotherapy in BLCA. In conclusion, this study has identified a circadian-based signature for evaluating CAFs infiltration and predicting the efficacy of chemotherapy and immunotherapy. The central gene FAM20C has emerged as a promising candidate which merits further investigations.