The lateral displacement of a column-supported modular steel structure with semi-rigid connections (CSMSS-SRC) owing to lateral loading was investigated to determine the corresponding lateral displacement mode and establish a foundation for a direct displacement-based seismic design method. Theoretical equations for the lateral displacement caused by the bending deformation of beams and columns, axial deformation of columns, and shear deformation of the vertical inter-module connections were theoretically derived using slope–deflection equations, unit-load method, and shear deformation definition, respectively. Two simplified single frame finite element models were then established to verify the theoretical equations and study the effects of the rotational and horizontal shear stiffnesses of the vertical inter-module connections on the lateral displacement of the CSMSS-SRC. The results showed that the proposed lateral displacement equations are highly accurate. Furthermore, although the rotational stiffness of the vertical inter-module connection did not exert an appreciable effect on the lateral displacement of the CSMSS-SRC, its horizontal shear stiffness did. These results are expected to provide a reference for the engineering design of CSMSS-SRC.