Understanding the structure–property relationship of polyurethane (PU) binders is of critical importance in the construction engineering applications. A series of rapid room temperature-curable PU binders were prepared using polyoxypropylene glycol as the soft segments, along with polymethylene polyphenylene isocyanate and 4,4′-methylene-bis(2-chlorobenzenamine) as the hard segments. This work investigated the influence of the hard segment content (40 wt%, 45 wt%, and 50 wt%) and the soft segment molecular weight (1000 g/mol, 2000 g/mol, 3000 g/mol, and 4000 g/mol) on the microphase structure of the PU binders. Their influence on the compressive behaviors, flexural behaviors, impact strengths, and slant shear strengths of the PU-based polymer concretes was also investigated. Structural characterization revealed that increasing either the hard segment content or the larger soft segment molecular weight effectively enhanced hydrogen bonding as well as microphase separation. Our results also demonstrated that a higher hard segment content resulted in better mechanical performance and shorter pot life. In addition, the PU-based polymer concrete using 2000 g/mol polyoxypropylene glycol reached optimal mechanical performance when the hard segment was held constant. Hence, varying the hard segment content and soft segment length can be an effective method to adjust the engineering performance of PU-based polymer concrete.