The effect of residual stress on fracture of materials or structures has been widely studied. However, its influence on ductile-to-brittle transition (DBT), a crucial phenomenon of structural materials, has rarely been investigated so far. In the present study, employing the eigenstrain method residual stresses are introduced into a bi-material specimen, where two configurations of crack and interface, e.g., one with interface perpendicular and one parallel to the crack extension, are designed to study the influence of residual stress. The DBT of the bi-material specimen in the presence of residual stresses is numerically studied by using the CAFE method where temperature dependent surface energy is implemented to calculate absorbed energy of Charpy impact testing specimen. It is found that residual stress generated in the two configurations affect the DBT in a similar manner. The DBT curves generally shift to higher temperature due to the decrease of absorbed energy with the increase of residual stress. It is found that the decrease of absorbed energy in both configurations is caused by the additional constraint on the notch root, which is induced by the residual stress and can facilitate the fracture.