One of the mechanisms for the dissimilar materials joining is the formation of interfacial reaction layers. However, this reaction is disrupted owing to the presence of oxide layers and voids, leading to lower strength, and the elimination of these factors is essential. Thus far, studies on linear fraction welding (LFW) with regard to the relationship between the joint strength and the microstructure of the base alloy and interface are lacking. Therefore, herein, the influence of microstructural changes in base alloys and interfacial structures on the joint tensile strength was investigated for the dissimilar joint between the stainless steel and an A5083 alloy after high-frequency linear friction welding at 245 Hz. After the initial joining process up to 1 s, the joint strength decreased as the friction increased. The crystal orientation analyses using electron backscatter diffraction demonstrated a minimal change in the microstructure of the A5083 alloy, particularly in the grain size in the geometric dynamic recrystallization region. This variation in the microstructure was minimal, regardless of the friction time, which was consistent with the result of the hardness distributions. The observation of microstructure using transmission electron microscopy revealed the formation of metastable Al-Fe intermetallic compounds and Al-Mg-Cr compounds derived from the reaction between the oxide layer and alloying elements in the A5083 after experiencing friction for a short time. Conversely, we confirmed the formation of a fragile boundary within the interfacial reaction layer after the exposure to friction for an extended period due to differences in the formation process of the layer: mechanical mixing or interdiffusion. Hence, it was proposed that mechanical removal as well as the interfacial reaction between the oxide layer and the Mg contained in the aluminum alloy function in removing the surface oxide layer, which plays a significant role in designing the interfacial structure and joint properties.