To optimize the mechanical properties of low-Ag lead-free solder, different amount of Bi element was added into the Sn–1.0Ag–0.5Cu (SAC105) to form SAC105-xBi (0 ≤ x ≤ 4) solder alloy. The microstructural evolution and shear fracture of SAC105-xBi solder joints were systematically investigated to clarify the failure mechanism after reflow and after 175 °C aging compared to SAC305. It was found that Bi element could provide solution strengthening and second-phase strengthening in the low-Ag SAC solder joints depending on the amount of added Bi element. The shear strength after reflow increased from 34.79 to 70.16 MPa with the increase of Bi addition from x = 0.0 to x = 4.0, while that after aging at 175 °C for 168 h increased from 28.97 MPa (x = 0.0) to 55.02 MPa (x = 4.0). The thickness of IMC decreased with the addition of Bi, and excessive Bi would precipitate in the matrix when the Bi content reached 2 wt% or even form brittle Bi barrier layer after 4 wt% addition. During shear tests, three kinds of fracture mode including ductile fracture, quasi-cleavage fracture, and cleavage brittle fracture were observed, and three kinds of failure position including solder matrix, Cu6Sn5/solder interface, and Cu3Sn/Cu interface were revealed. The failure mechanism of different solder joints was closely related to the coupling effects among the Bi amount in the matrix, the thicknesses of IMC, and the status of Kirkendall voids, which significantly affected the reliability of SAC solder joints. Finally, SAC105-1.0Bi solder alloy was considered having the most suitable Bi addition with anticipated comprehensive properties compared with SAC305.