Rail breakage can cause train derailment that leads to catastrophic consequences, prevention of which is worthy of a significant amount of investment in finance and time. Non-destructive evaluation can provide qualitative and quantitative inspection of rail tracks in-situ to detect dangerous faults, even though these defects are primarily invisible or hardly noticed from the surface of the rail. To make it efficient and convenient, infrared thermography technique offers a remote operation without interfering with rail transportation. The image processing for edge detection can further quantify the defect deterioration by depicting the contour of the defect. Moreover, the optimal parameters for various edge operators provide reliable detection for natural fracture defects.