Residual stress of electron beam welding (EBW) is associated with the heat transfer, fluid flow, and keyhole dynamics occurred in the process. In engineering, it was commonly simulated by pure heat conduction-based thermomechanical modeling with an experimentally calibrated heat source model. This method, however, is difficult to be used to consider the effect of weld imperfections such as the spiking and porosity defect. In this study, we proposed a novel combined computational fluid dynamics and finite element method (CFD-FEM) model to predicate the residual stresses of EBW. This model allows us to include the integral effect of heat transfer, fluid flow, and keyhole dynamics as well as process defects on residual stresses for the first time. A phenomenological heat source model dependent on the dynamic keyhole was considered in the CFD model part. A mapping algorithm was developed to build a new heat source model of FEM modeling part from the CFD simulations. Parametrical simulations and experiments of EBW of a 10-mm-thickness Ti-6-Al-4-V alloy plate were carried out and good agreements were obtained by the comparisons. It was shown that for deep penetration EBW the keyhole oscillates in radial and depth directions and the spiking defect occurs, particularly under large heat input parameters. Interestingly, the equivalent and principle residual stresses exhibits strong oscillations in the bottom part of welds along the welding direction, in which the amplitudes are around 9 pct (73.69 MPa) of the equivalent stress and 24 pct (121.84 MPa) of the principle stress in penetration direction, respectively, when the power of electron beam is 1.5 kW. As the power of electron beam increases, the effect of spiking on the residual stress enhances. When the welding power increases to 2.25 kW, the amplitudes of equivalent residual stress increase to 12 pct (119.2 MPa). Our results provide a quantitative method connecting the weld pool dynamics and process defects with the residual stress of EBW, which previously could be understood only in a qualitative way, and demonstrate the possibility of residual stress predications only from the information of materials and process parameters before welding, without the knowledge of post-weld profiles.