Template matching has been widely used in image processing for visual inspection of complicated, patterned surfaces. The currently existing methods such as golden template matching and normalized cross correlation are very sensitive to the displacement even the object under test is carefully aligned with respect to the template. This paper proposes a dissimilarity measure based on the optical-flow technique for surface defect detection, and aims at light-emitting diode (LED) wafer die inspection. The dissimilarity measure of each pixel derived from the optical flow field does not represent the true translation distance, but is reliable enough to indicate the degree of difference between an image pair. It is well tolerated to misalignment and random product variation. The integral image technique is applied to replace the sum operations in optical flow computation, and speeds up the intensive computation. We also point out the pitfall of the Lucas-Kanade optical flow when it is applied for defect detection, and propose a swapping process to tackle the problem. The experiment on LED wafer dies has shown that the proposed method can achieve a 100% recognition rate based on a test set of 357 die images.