Existing climate-economy models assume climate change has equal impacts on the productivity of firms that produce consumption and investment goods and services. We develop a model of structural change to show that the split between damage to consumption and investment productivity matters for the aggregate consequences of climate change. When investment is more vulnerable to climate, we find smaller short-run consumption losses than leading models suggest, but larger long-run consumption losses. We provide a quantitative illustration of these effects for one type of climate damage in the US economy: labor productivity losses from heat stress. We find that accounting for heterogeneous damages increases the welfare cost of the climate damage from heat stress by approximately 4 to 23%, depending on the discount factor.