Magnetorheological (MR) dampers are widely used in industrial applications due to its simple structure, fast response and adjustable damping performance. However, the volume of the MR damper will be limited by the installation space, which hinders the application to some extent. In this paper, a hybrid fluid flow MR damper which can be used in antiseismic building is designed to achieve better damping performance under constrained volume size. The multiphysics coupling simulation model and circuit simulation model are also established. Numerical results show the output damping force is 6.367 kN, the dynamic adjustable range is 50.768, and the current input time is 31 ms under the applied current of 2.0 A. In order to improve the dynamic performance of the designed MR damper under constrained volume size, a multi-objective structural optimization method based on theoretical calculation model is proposed. Experiments are also conducted to investigate the dynamic performance of the initial and optimal MR damper. Compared with the initial MR damper, the output damping force and the dynamic adjustable range of the optimal damper are improved by 6.6 % and 46.3 %, while the energy consumption is reduced by 10.7 %, respectively.