Immunotherapies have changed the way how we treat cancer at all stages. The understanding of the immune system in individual tumor specimens guides the selection of immune-modulating agents such as immune checkpoint inhibitors alone or in combination with other therapeutic agents that target, modulate or unleash the patient's immune system. Despite the similar histopathological diagnosis, each tumor is unique at its primary site and site of metastasis, also depending on previous treatment regimens or genetic alterations, such as chromosomal instability or acquired mutations. The clinically well-established use of PD-1/PD-L1 inhibitors already requires the assessment of its target molecules in different cells (viable tumor cells alone or in combination with immune cells or immune cells alone) with different thresholds in various indications. Anyhow, checkpoint inhibitors show the best overall response rate when immune effector cells like tumor-infiltrating lymphocytes are in close spatial proximity without being suppressed by other humoral or cellular regulatory mechanisms. Therefore, immune cell-rich tumors (“hot tumors”) are usually quite reactive to immune-modulating agents, whereas other immune-depleted or immune-excluded tumor areas are less responsive and require alternative treatment regimens such as modified immune effectors cells or immune-stimulating agents, for example, oncolytic viruses. Here, we summarize the relevance to understand the entire tumor heterogeneity and its environment, the contextual relationship and spatial quantification of all immune and tumor cells along with the genetic background of the individual cancer through the application of multiplex in-situ technologies and the application of machine learning tools.