We report the zygotic encystment of geographically dispersed isolates in the dinoflagellate species complex Alexandrium tamarense, in particular, successful mating of toxic Group I and nontoxic Group III isolates. However, hypnozygotes produced in Group I/III co-cultures complete no more than three divisions after germinating. Previous reports have suggested a mate recognition mechanism whereby hypnozygotes produced in co-cultures could arise from either homotypic (inbred) or heterotypic (outbred) gamete pairs. To determine the extent to which each occurs, a nested PCR assay was developed to determine parentage of individual hypnozygotes. The vast majority of hypnozygotes from pairwise Group I/III co-cultures were outbred, so that inviability was a result of hybridization, not inbreeding. These findings support the assertion that complete speciation underlies the phylogenetic structure of the Alexandrium tamarense species complex. Additionally, the ribosomal DNA (rDNA) copy numbers of both hybrid and single ribotype hypnozygotes were reduced substantially from those of haploid motile cells. The destruction of rDNA loci may be crucial for the successful mating of genetically distant conjugants and appears integral to the process of encystment. The inviability of Group I/III hybrids is important for public health because the presence of hybrid cysts may indicate ongoing displacement of a nontoxic population by a toxic one (or vice versa). Hybrid inviability also suggests a bloom control strategy whereby persistent, toxic Group I blooms could be mitigated by introduction of nontoxic Group III cells. The potential for hybridization in nature was investigated by applying the nested PCR assay to hypnozygotes from Belfast Lough, Northern Ireland, a region where Group I and III populations co-occur. Two hybrid cysts were identified in 14 successful assays, demonstrating that Group I and III populations do interbreed in that region. However, an analysis of mating data collected over an 18-year period indicated a leaky pre-mating barrier between ribosomal species (including Groups I and III). Whether the observed selectivity inhibits hybridization in nature is dependent on its mechanism. If the point of selectivity is the induction of gametogenesis, dissimilar ribotypes could interbreed freely, promoting displacement in cases where hybridization is lethal. If instead, selectivity occurs during the adhesion of gamete pairs, it could enable stable coexistence of A. tamarense species. In either case, hybrid inviability may impose a significant obstacle to range expansion. The nested PCR assay developed here is a valuable tool for investigation of interspecies hybridization and its consequences for the global biogeography of these important organisms.