Contrastive learning (CL) has emerged as a potent tool for building meaningful latent representations of galaxy properties across a broad spectrum of wavelengths, ranging from optical and infrared to radio frequencies. These latent representations facilitate a variety of downstream tasks, including galaxy classification, similarity searches in extensive datasets, and parameter estimation, which is why they are often referred to as foundation models for galaxies. In this study, we employ CL on the latest extended data release from the Calar Alto Legacy Integral Field Area (CALIFA) survey, which encompasses a total of 895 galaxies with enhanced spatial resolution that reaches the limits imposed by natural seeing (FWHMPSF ∼ 1.5). We demonstrate that CL can be effectively applied to Integral Field Unit (IFU) surveys, even with relatively small training sets, to construct meaningful embedding where galaxies are well separated based on their physical properties. We discover that the strongest correlations in the embedding space are observed with the equivalent width of Hα, galaxy morphology, stellar metallicity, luminosity-weighted age, stellar surface mass density, the [NII]/Hα ratio, and stellar mass, in descending order of correlation strength. Additionally, we illustrate the feasibility of unsupervised separation of galaxy populations along the star formation main sequence, successfully identifying the blue cloud and the red sequence in a two-cluster scenario, and the green valley population in a three-cluster scenario. Our findings indicate that galaxy luminosity profiles have minimal impact on the construction of the embedding space, suggesting that morphology and spectral features play a more significant role in distinguishing between galaxy populations. Moreover, we explore the use of CL for detecting variations in galaxy population distributions across different large-scale structures, including voids, clusters, and filaments and walls. Nonetheless, we acknowledge the limitations of the CL framework and our specific training set in detecting subtle differences in galaxy properties, such as the presence of an AGN or other minor scale variations that exceed the scope of primary parameters such as the stellar mass or morphology. Conclusively, we propose that CL can serve as an embedding function for the development of larger models capable of integrating data from multiple datasets, thereby advancing the construction of more comprehensive foundation models for galaxies.