Heat stress negatively affects wheat production in several ways, mainly by reducing growth rate, photosynthetic capacity and reducing spike fertility. Modeling stress response means analyzing simultaneous relationships among traits affecting the whole plant response and determinants of grain yield. The aim of this study was to dissect the diverse impacts of heat stress on key yield traits and to identify the most promising sources of alleles for heat tolerance. We evaluated a diverse durum wheat panel of 183 cultivars and breeding lines from worldwide, for their response to long-term heat stress under field conditions (HS) with respect to non stress conditions (NS), considering phenological traits, grain yield (GY) and its components as a function of the timing of heat stress and climatic covariates. We investigated the relationships among plant and environmental variables by means of a structural equation model (SEM) and Genetic SEM (GSEM). Over two years of experiments at CENEB, CIMMYT, the effects of HS were particularly pronounced for the normalized difference vegetation index, NDVI (-51.3%), kernel weight per spike, KWS (-40.5%), grain filling period, GFP (-38.7%), and GY (-56.6%). Average temperatures around anthesis were negatively correlated with GY, thousand kernel weight TKW and test weight TWT, but also with spike density, a trait determined before heading/anthesis. Under HS, the correlation between the three major determinants of GY, i.e., fertile spike density, spike fertility and kernel size, were of noticeable magnitude. NDVI measured at medium milk-soft dough stage under HS was correlated with both spike fertility and grain weight while under NS it was less predictive of grain weight but still highly correlated with spike fertility. GSEM modeling suggested that the causal model of performance under HS directly involves genetic effects on GY, NDVI, KWS and HD. We identified consistently suitable sources of genetic resistance to heat stress to be used in different durum wheat pre-breeding programs. Among those, Desert Durums and CIMMYT'80 germplasm showed the highest degree of adaptation and capacity to yield under high temperatures and can be considered as a valuable source of alleles for adaptation to breed new HS resilient cultivars.