Friction stir welding (FSW) and ultrasonic vibration enhanced FSW (UVeFSW) experiments were conducted by using 6061-T6 Al alloy and AZ31B-H24 Mg alloy sheets of thickness 2 mm. The suitable process parameters windows were obtained for the butt joining of Al/Mg sheets. The effect of ultrasonic vibration on the macrostructure and mechanical properties of the dissimilar joints was studied. The results showed that the width of the weld nugget zone (WNZ) was enlarged to some extent and the hardness distribution in WNZ was more uniform in UVeFSW. In addition, the application of ultrasonic vibration effectively promoted the interpenetration degree of dissimilar materials in the WNZ so that the mechanical interlocking on the bonding interface of dissimilar Al/Mg materials was enhanced. The facture positions were changed from the bonding interface in FSW to the boundary between WNZ and the thermo-mechanical affected zone, and the ductile fracture zone was expanded. The highest ultimate tensile strength was 205 MPa at the process parameters set of 1200 rpm-50 mm/min in UVeFSW in this experiment. The average ultimate tensile strength of FSW/UVeFSW joints was 172.3 MPa and 184.4 MPa, respectively, and the average ultimate tensile strength was increased by 7.02% with the introduction of ultrasonic vibration.