Genomic scans of multiple populations often reveal marker loci with greatly increased differentiation between populations. Often this differentiation coincides in space with contrasts in ecological factors, forming a genetic-environment association (GEA). GEAs imply a role for local adaptation, and so it is tempting to conclude that the strongly differentiated markers are themselves under ecologically based divergent selection, or are closely linked to loci under such selection. Here, we highlight an alternative and neglected explanation: intrinsic (i.e. environment-independent) pre- or post-zygotic genetic incompatibilities rather than local adaptation can be responsible for increased differentiation. Intrinsic genetic incompatibilities create endogenous barriers to gene flow, also known as tension zones, whose location can shift over time. However, tension zones have a tendency to become trapped by, and therefore to coincide with, exogenous barriers due to ecological selection. This coupling of endogenous and exogenous barriers can occur easily in spatially subdivided populations, even if the loci involved are unlinked. The result is that local adaptation explains where genetic breaks are positioned, but not necessarily their existence, which can be best explained by endogenous incompatibilities. More precisely, we show that (i) the coupling of endogenous and exogenous barriers can easily occur even when ecological selection is weak; (ii) when environmental heterogeneity is fine-grained, GEAs can emerge at incompatibility loci, but only locally, in places where habitats and gene pools are sufficiently intermingled to maintain linkage disequilibria between genetic incompatibilities, local-adaptation genes and neutral loci. Furthermore, the association between the locally adapted and intrinsically incompatible alleles (i.e. the sign of linkage disequilibrium between endogenous and exogenous loci) is arbitrary and can form in either direction. Reviewing results from the literature, we find that many predictions of our model are supported, including endogenous genetic barriers that coincide with environmental boundaries, local GEA in mosaic hybrid zones, and inverted or modified GEAs at distant locations. We argue that endogenous genetic barriers are often more likely than local adaptation to explain the majority of Fst-outlying loci observed in genome scan approaches - even when these are correlated to environmental variables.