Spontaneous hybrids between the native elms (genus Ulmus L.) have been observed in the forests of Europe. Gene conservation raises questions regarding the genetic background for the complex morphology and taxonomy of elms. Our objective was to dissect morphological and genetic variation in the natural swamps of Ulmus species groups in Lithuanian forests with the aid of leaf morphology and microsatellite (SSR) markers. We sampled leaves from 189 elms at 26 locations to grasp the phenotypic diversity in variable natural habitats in Lithuanian forests. We assigned the elms into six taxonomic and genetics groups based on 31 leaf morphology parameters and tested the genetic differentiation between these six groups at six nuclear SSR loci by using Bayesian and genetic distance-based clustering. The genetic and leaf morphometric analyses of putative elm hybrid swamps indicated a low genetic exchange between U. laevis Pall. and the other Ulmus groups. The genetic and morphometric data supported the differentiation of U. glabra Huds. and U. glabra (female) × U. minor Mill. (male) spontaneous hybrids. In addition, the results of the genetic analysis also confirmed the high level of genome sharing among U. minor and U. minor subsp. minor Richens., where leaf morphology failed to differentiate genetically discrete groups. For gene conservation, we would suggest considering separate gene conservation units selected based on leaf and stem morphology for U. laevis, U. glabra, U. glabra × minor, and the U. minor species complex.