Hybrid sterility is a reproductive isolation barrier between diverging taxa securing the early steps of speciation. Hybrid sterility is ubiquitous in the animal and plant kingdoms, but its genetic control is poorly understood. In our previous studies, we have uncovered the sterility of hybrids between musculus and domesticus subspecies of the house mouse, which is controlled by the Prdm9 gene, the X-linked Hstx2 locus, and subspecific heterozygosity for genetic background. To further investigate this form of genic-driven chromosomal sterility, we constructed a simplified hybrid sterility model within the genome of the domesticus subspecies by swapping domesticus autosomes with their homologous partners from the musculus subspecies. We show that the "sterility" allelic combination of Prdm9 and Hstx2 can be activated by a musculus/domesticus heterozygosity of as few as two autosomes, Chromosome 17 (Chr 17) and Chr 18 and is further enhanced when another heterosubspecific autosomal pair is present, whereas it has no effect on meiotic progression in the pure domesticus genome. In addition, we identify a new X-linked hybrid sterility locus, Hstx3, at the centromeric end of Chr X, which modulates the incompatibility between Prdm9 and Hstx2. These results further support our concept of chromosomal hybrid sterility based on evolutionarily accumulated divergence between homologous sequences. Based on these and previous results, we believe that future studies should include more information on the mutual recognition of homologous chromosomes at or before the first meiotic prophase in interspecific hybrids, as this may serve as a general reproductive isolation checkpoint in mice and other species.