Plastic-bonded granular materials (PBM) are widely used in industrial sectors, including building construction, abrasive applications, and defense applications such as plastic-bonded explosives. The mechanical behavior of PBM is highly nonlinear, irreversible, rate dependent, and temperature sensitive governed by various micromechanical attributions such as grain crushing and binder damage. This paper presents a thermodynamically consistent, microstructure-informed constitutive model to capture these characteristic behaviors of PBM. Key features of the model include a breakage internal variable to upscale the grain-scale information to the continuum level and to predict grain size evolution under mechanical loading. In addition, a damage internal state variable is introduced to account for the damage, deterioration, and debonding of the binder matrix upon loading. Temperature is taken as a fundamental external state variable to handle non-isothermal loading paths. The proposed model is able to capture with good accuracy several important aspects of the mechanical properties of PBM, such as pressure-dependent elasticity, pressure-dependent yield strength, brittle-to-ductile transition, temperature dependency, and rate dependency in the post-yielding regime. The model is validated against multiple published datasets obtained from confined and unconfined compression tests, covering various PBM compositions, confining pressures, temperatures, and strain rates.