High-strength pipeline steel was subjected to friction stir welding (FSW) at rotation rates of 400–700 rpm, and the grain refinement mechanism of the nugget zone (NZ) was determined. The thermo-mechanical process during FSW in the NZ was simulated by multi-pass thermal compression, thereby achieving the austenitic non-recrystallization temperature (Tnr). The austenitic non-recrystallization in the NZ at the lowest rotation rate of 400 rpm caused a significant grain refinement. Furthermore, the reduced rotation rate also resulted in the formation of a high ratio of island-like martensite-austenite (M-A) constituent. The toughness of the NZs was enhanced as the rotation rate decreased, which is attributed to the fine effective grains and homogeneously distributed fine M-A constituents dramatically inhibiting crack initiation and propagation.