Fatigue of welded joint is a scientific issue of great significance, because welding process always causes the loss of fatigue strength up to 40–60% for welded joint of the key components, resulting in a relatively poor reliability in service. In the present work, the fatigue behavior and damage mechanism of a medium-carbon steel welded joint were systematically investigated by microstructure observation and fatigue tests. It was found that, for the medium-carbon steel welded joint, fatigue crack mainly initiated at the welding defects, e.g., welding porosity, slag inclusion and heterogeneous microstructure with a poor mechanical property (i.e., the network-block proeutectoid ferrite). The competitive impact of various defects on the fatigue damage behaviors of welded joint was discussed, i.e., the competition between the volume defects (e.g., porosity and inclusions) and the heterogeneous structure. It is revealed that the smaller surface defects are more susceptible to fatigue damage due to the low constraint of plane stress state and higher stress intensity factor at high stress amplitude, while the larger internal defects easily cause fatigue damage at low stress amplitude because the driving force for crack propagation is larger than that of the damage case from the sample surface. Generally, the fatigue lives of samples with fatigue crack initiating from the weak phase are relatively longer, about 10 times as long as those of the samples with fatigue crack starting from the sample surface. In addition, the residual fatigue crack growth life, Nf, and the initial stress intensity factor (SIF), ΔK1 at the volume defect tip exhibit a linear relation in the double logarithmic coordinate system. The present findings can provide a theoretical basis for the anti-fatigue design and fatigue life extension technology for the metallic component with welded structures.