Accurate measurement of the railway track geometry is a task of fundamental importance to ensure the track quality in both the construction phase and the regular maintenance stage. Conventional track geometry measuring trolleys (TGMTs) in combination with classical geodetic surveying apparatus such as total stations alone cannot meet the requirements of measurement accuracy and surveying efficiency at the same time. Accurate and fast track geometry surveying applications call for an innovative surveying method that can measure all or most of the track geometric parameters in short time without interrupting the railway traffic. We provide a novel solution to this problem by integrating an inertial navigation system (INS) with a geodetic surveying apparatus, and design a modular TGMT system based on aided INS, which can be configured according to different surveying tasks including precise adjustment of slab track, providing tamping measurements, measuring track deformation and irregularities, and determination of the track axis. TGMT based on aided INS can operate in mobile surveying mode to significantly improve the surveying efficiency. Key points in the design of the TGMT’s architecture and the data processing concept and workflow are introduced in details, which should benefit subsequent research and provide a reference for the implementation of this kind of TGMT. The surveying performance of proposed TGMT with different configurations is assessed in the track geometry surveying experiments and actual projects.