Localized residual stresses were correlated to ductile fracture toughness quantitatively in steel and aluminium alloys using in situ neutron diffraction method coupled with elastic-plastic finite element modelling. Local out-of-plane compression (LOPC) method generated compressive and tensile residual stresses in the vicinity of the fatigue pre-crack front in two compact tension (CT) specimens, respectively, and the evolution of the stress fields was simultaneously measured using in situ neutron diffraction technique under mode-I fracture loading. The results clearly showed that the localized tensile residual stress apparently accelerated stress transfer at the mid-thickness of the CT specimen compared to the specimen having compressive residual stress. The coupled quasi-static ductile fracture simulations and neutron diffraction results revealed a clear correspondence of fracture initiation toughness with localized residual stresses in both steel and aluminium alloys. In the aluminium case, tensile residual stress of 208 MPa was obviously detrimental to the fracture toughness resulting in a 43% reduction while compression of −220 MPa increases by up to 14%. On the other hand, localized residual stress in steel hardly affected fracture initiation toughness due to high plastic dissipation energy. This experiment-simulation coupled study quantitatively elucidates the distinctive role of plastic deformation and stress triaxiality in ductile fracture initiation between steel and aluminium alloy.