The assistance of ultrasonic vibration during the friction stir welding (FSW) process has been verified as an effective approach for the improvement of joint strength. In the present study, experimentation on Al/Mg dissimilar alloys in butt joint configuration is implemented by employing FSW with and without the assistance of ultrasonic vibration. An optimized tool shoulder diameter of 12 mm is utilized, and the ultrasonic vibration is applied perpendicularly onto the tool along the welding direction, which is named UVaFSW. The results of joint appearance and macrostructure, characteristics of the intermetallic compounds (IMCs), as well as joint strength and fracture appearance are compared between Al/Mg FSW joints with and without ultrasonic vibration. It is demonstrated that the material intermixing between Al and Mg alloys is substantially strengthened in the UVaFSW joint compared with that in the FSW joint. Additionally, the ultrasonic vibration can be beneficial for the reduction of IMC thickness, as well as the formation of intermittently distributed IMC phases at the Al–Mg bonding interface. Consequently, the mechanical properties of Al/Mg FSW joints are significantly improved with the assistance of ultrasonic vibration. The maximum ultimate tensile strength is 206 MPa at tool rotation speed of 800 rpm and welding speed of 50 mm/min for the Al/Mg UVaFSW joint.