Agents that target PD-1 and PD-L1 have been developed in the treatment of bladder cancer (BC). However, the diversity of immune cell infiltration in non-muscle-invasive BC (NMIBC) and the dynamics of the microenvironment as it progresses to muscle-invasive/metastatic disease remains unknown. To assess tumor immune activity, hierarchical clustering was applied to 159 BC samples based on cellular positivity for the defined immune cellular markers (CD3/CD4/CD8/FOXP3/CD20/PD-1/PD-L1/LAG3/TIGIT), divided into two clusters. There was a "hot cluster" (25%) consisting of patients with a high expression of these markers and a "cold cluster" (75%) comprising those without. The expression of CD39, CD44, CD68, CD163, IDO1, and Ki67 was significantly higher in tumors in the hot cluster. Immunologically, high-grade T1 tumors were significantly hotter, whereas tumors that had progressed to muscle invasion turned cold. However, a certain number of high-grade NMIBC patients were in the cold cluster, and these patients had a significantly higher risk of disease progression. Using an externally available TCGA dataset, RB1 and TP53 alterations were more frequently observed in TCGA hot cluster; rather FGFR3, KDM6A, and KMT2A alterations were common in TCGA cold/intermediate cluster. Analyses of recurrent tumors after BCG therapy revealed that tumor immune activity was widely maintained before and after treatment, and high FGFR3 expression was detected after recurrence in tumors initially classified into the cold cluster. Collectively, we revealed the dynamics of the tumor microenvironment in BC as a whole and identified candidate molecules as therapeutic targets for recurrent NMIBC, e.g., after BCG therapy.