Climate change is increasingly impacting temperate forest ecosystems and many forest herbs might be unable to track the changing climate due to dispersal limitation. Forest herbs with a low adaptive capacity may therefore benefit from conservation strategies that mitigate dispersal limitation and evolutionary constraints, such as assisted migration. However, assisted migration strategies rarely consider evolutionary constraints of potential source populations that may jeopardize their success. In cases where climate adaptation is overshadowed by competing evolutionary processes, assisted migration is unlikely to support adaptation to future climates. Using a combination of population and landscape genomic analyses, we disentangled local adaptation drivers and quantified the adaptability and vulnerability to climate change of the self-incompatible deciduous forest herb Primula elatior. Southern populations displayed a sharp genetic turnover and a considerable amount of local adaptation under diversifying selection was discovered. However, most of the outlier loci could not be linked to climate variables (71%) and were likely related to other local adaptation drivers, such as photoperiodism. Furthermore, specific adaptations to climate extremes, such as drought stress, could not be detected. This is in line with the typical occurrence of forest herbs in buffered climatic conditions, which can be expected to reduce selection pressures imposed by climate. Finally, populations in the south of the distribution area had increased sensitivity to climate change due to a reduced adaptive capacity and a moderate genetic offset, while central European populations were sensitive due to a high genetic offset. We conclude that assisted migration from southern source populations could bear significant risk due to nonclimatic maladaptation and a low adaptive capacity. Regional admixture and restoration of ecological connectivity to increase the adaptive capacity, and assisted range expansion to suitable habitat in the north might be more appropriate mitigation strategies.