Inter-module connections serve as crucial components, horizontally and vertically connecting individual modules to form modular buildings. This paper presents a component-based model that divides inter-module connections into vertical and horizontal connection components, facilitating the simplification of shear and axial behaviors for input into the global numerical model. Initially, experimental studies and numerical simulations were conducted to investigate the shear and axial behavior of these connection components, following which simplified connection component models were derived. Subsequently, a component-based model of inter-module connections was established, in which the shear and axial properties of the vertical and horizontal components were represented by nonlinear connector elements. Finally, the proposed component-based model was incorporated directly into the global frame models of high-rise steel modular buildings, and the seismic behavior was evaluated. The accuracy of the proposed component-based model was investigated by comparing the natural vibration characteristics and global building responses of high-rise steel modular buildings with traditional empirical models established based on assumptions of rigid or pinned fixities. The results reveal that traditional empirical models fail to accurately represent the discontinuous properties of inter-module connections, resulting in an overestimation of structural lateral stiffness. In contrast, the component-based model accounts for the actual load-transfer mechanisms of vertical and horizontal components, providing a more accurate understanding of the natural vibration characteristics and seismic response of high-rise steel modular buildings. The proposed component-based model establishes a critical link between isolated connection behavior and structural response assessment for high-rise steel modular buildings, making it well-suited for accurately predicting global building responses.