In the development of supervised semantic segmentation models for building outline detection, a significant challenge is obtaining sufficient accurately labelled training data. Old building outline datasets captured against historic imagery often exist covering the same locations as new imagery. By training Deep Convolutional Neural Networks to recognise buildings in these areas we can detect new and changed buildings. Ideally, historic building outlines as training labels paired with new orthophotography could be used to train models to detect buildings in new imagery. However, historic imagery may be orthorectified to a different standard, or have different off-nadir angles resulting in an apparent displacement of buildings between historic and current imagery sets and in labelled buildings that do not align accurately with the building image. This displacement problem presents difficulties in training accurate models. This paper proposes a novel loss function that searches the prediction segmentation masks for the best matching crop to a central crop of the training segmentation mask, thus reducing error due to linear displacements between an image and mask. According to the Intersection-over-Union metric, a ground truth test set for the building outlines detection problem improved from 0.751 to 0.802 on the Christchurch Buildings dataset when using the mask shift loss versus binary cross-entropy.